Flowfile Core API Reference

This section provides a detailed API reference for the core Python objects, data models, and API routes in flowfile-core. The documentation is generated directly from the source code docstrings.

---

Core Components

This section covers the fundamental classes that manage the state and execution of data pipelines. These are the main "verbs" of the library.

FlowGraph

The FlowGraph is the central object that orchestrates the execution of data transformations. It is built incrementally as you chain operations. This DAG (Directed Acyclic Graph) represents the entire pipeline.

flowfile_core.flowfile.flow_graph.FlowGraph

A class representing a Directed Acyclic Graph (DAG) for data processing pipelines.

It manages nodes, connections, and the execution of the entire flow.

Methods:

Name	Description
__init__	Initializes a new FlowGraph instance.
__repr__	Provides the official string representation of the FlowGraph instance.
add_api_response	Adds an API-response sink node.
add_apply_model	Adds an Apply Model node.
add_catalog_reader	Adds a node that reads a table from the catalog.
add_catalog_writer	Adds a node that writes its input to the catalog as a Delta table or virtual table.
add_cloud_storage_reader	Adds a cloud storage read node to the flow graph.
add_cloud_storage_writer	Adds a node to write data to a cloud storage provider.
add_cross_join	Adds a cross join node to the graph.
add_database_reader	Adds a node to read data from a database.
add_database_writer	Adds a node to write data to a database.
add_datasource	Adds a data source node to the graph.
add_dependency_on_polars_lazy_frame	Adds a special node that directly injects a Polars LazyFrame into the graph.
add_dynamic_rename	Adds a node that renames many columns at once via a single rule.
add_evaluate_model	Adds an Evaluate Model node.
add_explore_data	Adds a specialized node for data exploration and visualization.
add_external_source	Adds a node for a custom external data source.
add_filter	Adds a filter node to the graph.
add_flow_input	Adds a named subflow-input placeholder source.
add_flow_output	Adds a named subflow-output sink (passthrough, always materialized).
add_formula	Adds a node that applies a formula to create or modify a column.
add_fuzzy_match	Adds a fuzzy matching node to join data on approximate string matches.
add_google_analytics_reader	Adds a node that reads from a Google Analytics 4 property.
add_graph_solver	Adds a node that solves graph-like problems within the data.
add_group_by	Adds a group-by aggregation node to the graph.
add_include_cols	Adds columns to both the input and output column lists.
add_initial_node_analysis	Adds a data exploration/analysis node based on a node promise.
add_join	Adds a join node to combine two data streams based on key columns.
add_kafka_source	Adds a node to read data from a Kafka or Redpanda topic.
add_manual_input	Adds a node for manual data entry.
add_missing_user_defined_node	Adds a placeholder for a custom node that cannot be loaded on this machine.
add_node_promise	Adds a placeholder node to the graph that is not yet fully configured.
add_node_step	The core method for adding or updating a node in the graph.
add_node_to_starting_list	Adds a node to the list of starting nodes for the flow if not already present.
add_nodes_to_group	Add nodes to an existing group and refit its bounds.
add_output	Adds an output node to write the final data to a destination.
add_pivot	Adds a pivot node to the graph.
add_polars_code	Adds a node that executes custom Polars code.
add_python_script	Adds a node that executes Python code on a kernel container.
add_random_split	Adds a node that randomly partitions rows into N labeled outputs.
add_read	Adds a node to read data from a local file (e.g., CSV, Parquet, Excel).
add_record_count	Adds a filter node to the graph.
add_record_id	Adds a node to create a new column with a unique ID for each record.
add_rest_api_reader	Adds a node that reads from a REST API.
add_run_flow	Adds a node that executes a catalog-registered flow as a subflow.
add_sample	Adds a node to take a random or top-N sample of the data.
add_select	Adds a node to select, rename, reorder, or drop columns.
add_sort	Adds a node to sort the data based on one or more columns.
add_sql_query	Adds a node that executes a SQL query against connected data sources.
add_sql_source	Adds a node that reads data from a SQL source.
add_text_to_rows	Adds a node that splits cell values into multiple rows.
add_train_model	Adds a Train Model node.
add_union	Adds a union node to combine multiple data streams.
add_unique	Adds a node to find and remove duplicate rows.
add_unpivot	Adds an unpivot node to the graph.
add_user_defined_node	Adds a user-defined custom node to the graph.
add_wait_for	Adds a Wait For node — passes the left input through and waits on the right.
add_window_functions	Adds a window-functions node (rolling, cumulative, rank, tile).
apply_layout	Calculates and applies a layered layout to all nodes in the graph.
assign_node_to_named_group	Assign a node to a group identified by name, creating it if absent (find-or-create).
cancel	Cancels an ongoing graph execution.
capture_history_if_changed	Capture history only if the flow state actually changed.
capture_history_snapshot	Capture the current state before a change for undo support.
check_flow_laziness	Check whether the flow supports lazy execution for virtual tables.
close_flow	Performs cleanup operations, such as clearing node caches.
copy_node	Creates a copy of an existing node.
create_group	Create a visual group. Organizational only.
delete_group	Remove a group box (ungroup). Members and sub-groups lift up one level.
delete_node	Deletes a node from the graph and updates all its connections.
generate_code	Generates code for the flow graph.
get_frontend_data	Formats the graph structure into a JSON-like dictionary for a specific legacy frontend.
get_history_state	Get the current state of the history system.
get_implicit_starter_nodes	Finds nodes that can act as starting points but are not explicitly defined as such.
get_node	Retrieves a node from the graph by its ID.
get_node_data	Retrieves all data needed to render a node in the UI.
get_node_storage	Serializes the entire graph's state into a storable format.
get_nodes_overview	Gets a list of dictionary representations for all nodes in the graph.
get_run_info	Gets a summary of the most recent graph execution.
get_vue_flow_input	Formats the graph's nodes and edges into a schema suitable for the VueFlow frontend.
has_unsaved_changes	Return True if the flow has changed since the last save point.
mark_as_saved	Mark the current flow state as the saved baseline (for dirty tracking).
print_tree	Print flow_graph as a visual tree structure, showing the DAG relationships with ASCII art.
redo	Redo the last undone action.
release_run	Release the single-run slot claimed by try_claim_run (idempotent).
remove_from_output_cols	Removes specified columns from the list of expected output columns.
remove_nodes_from_group	Remove nodes from whatever group they belong to; prune groups left empty.
reset	Forces a deep reset on all nodes in the graph.
restore_from_snapshot	Clear current state and rebuild from a snapshot.
restore_groups	Replace the runtime group registry (used by open_flow and restore_from_snapshot).
run_graph	Executes the entire data flow graph from start to finish.
save_flow	Saves the current state of the flow graph to a file.
set_group_bounds	Persist group box bounds (used together with set_node_positions on drag/resize).
set_node_positions	Persist dragged node positions (absolute canvas coordinates) onto setting_input.
trigger_fetch_node	Executes a specific node in the graph by its ID.
try_claim_run	Atomically claim the flow's single-run slot; False when a run is already in flight.
undo	Undo the last action by restoring to the previous state.
update_group	Rename / recolor / move / resize / collapse a group box.

Attributes:

Name	Type	Description
execution_location	ExecutionLocationsLiteral	Gets the current execution location.
execution_mode	ExecutionModeLiteral	Gets the current execution mode ('Development' or 'Performance').
flow_id	int	Gets the unique identifier of the flow.
graph_has_functions	bool	Checks if the graph has any nodes.
graph_has_input_data	bool	Checks if the graph has an initial input data source.
node_connections	list[tuple[int, int]]	Computes and returns a list of all connections in the graph.
nodes	list[FlowNode]	Gets a list of all FlowNode objects in the graph.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

class FlowGraph:
    """A class representing a Directed Acyclic Graph (DAG) for data processing pipelines.

    It manages nodes, connections, and the execution of the entire flow.
    """

    uuid: str
    depends_on: dict[
        int,
        Union[
            ParquetFile,
            FlowDataEngine,
            "FlowGraph",
            pl.DataFrame,
        ],
    ]
    _flow_id: int
    _input_data: Union[ParquetFile, FlowDataEngine, "FlowGraph"]
    _input_cols: list[str]
    _output_cols: list[str]
    _node_db: dict[str | int, FlowNode]
    _node_ids: list[str | int]
    _results: FlowDataEngine | None = None
    cache_results: bool = False
    schema: list[FlowfileColumn] | None = None
    has_over_row_function: bool = False
    _flow_starts: list[int | str] = None
    latest_run_info: RunInformation | None = None
    start_datetime: datetime = None
    end_datetime: datetime = None
    _flow_settings: schemas.FlowSettings = None
    flow_logger: FlowLogger

    def __init__(
        self,
        flow_settings: schemas.FlowSettings | schemas.FlowGraphConfig,
        name: str = None,
        input_cols: list[str] = None,
        output_cols: list[str] = None,
        path_ref: str = None,
        input_flow: Union[ParquetFile, FlowDataEngine, "FlowGraph"] = None,
        cache_results: bool = False,
    ):
        """Initializes a new FlowGraph instance.

        Args:
            flow_settings: The configuration settings for the flow.
            name: The name of the flow.
            input_cols: A list of input column names.
            output_cols: A list of output column names.
            path_ref: An optional path to an initial data source.
            input_flow: An optional existing data object to start the flow with.
            cache_results: A global flag to enable or disable result caching.
        """
        if isinstance(flow_settings, schemas.FlowGraphConfig):
            flow_settings = schemas.FlowSettings.from_flow_settings_input(flow_settings)

        self._flow_settings = flow_settings
        self.uuid = str(uuid1())
        self.start_datetime = None
        self.end_datetime = None
        self.latest_run_info = None
        self._flow_id = flow_settings.flow_id
        self.flow_logger = FlowLogger(flow_settings.flow_id)
        self._flow_starts: list[FlowNode] = []
        self._results = None
        self.schema = None
        self.has_over_row_function = False
        self._input_cols = [] if input_cols is None else input_cols
        self._output_cols = [] if output_cols is None else output_cols
        self._node_ids = []
        self._node_db = {}
        # Visual node groups: organizational only, never read by the executor.
        # Membership lives on each node's setting_input.group_id; this is the box registry.
        self._groups: dict[int, schemas.GroupInformation] = {}
        self._group_id_seq: int = 0  # monotonic group-id allocator; never reuses a freed id
        self._active_group_id: int | None = None
        # Serializes claiming flow_settings.is_running: the bare check-then-set in the
        # run entry points raced when callers arrive from non-asyncio threads.
        self._run_claim_lock = threading.Lock()
        self.cache_results = cache_results
        self.__name__ = name if name else "flow_" + str(id(self))
        self.depends_on = {}
        self.artifact_context = ArtifactContext()
        # Subflow recursion guards: resolved paths of every ancestor flow file and
        # this graph's nesting depth. Attributes (not contextvars) because stages
        # execute on ThreadPoolExecutor threads.
        self._subflow_ancestry: frozenset[str] = frozenset()
        self._subflow_depth: int = 0
        # Last user_id seen on any node settings (stamped by the editor routes /
        # open_flow). Lets restore_from_snapshot re-stamp the owner even when the
        # live graph is empty at undo time (snapshots intentionally omit user_id).
        self._owner_user_id: int | None = None

        from flowfile_core.flowfile.history_manager import HistoryManager
        from flowfile_core.schemas.history_schema import HistoryConfig

        history_config = HistoryConfig(enabled=flow_settings.track_history)
        self._history_manager = HistoryManager(config=history_config)

        if path_ref is not None:
            self.add_datasource(input_schema.NodeDatasource(file_path=path_ref))
        elif input_flow is not None:
            self.add_datasource(input_file=input_flow)

        # Mark the empty initial state as the saved baseline so an unmodified
        # flow is not considered dirty.
        self._history_manager.mark_saved(self)

    @property
    def flow_settings(self) -> schemas.FlowSettings:
        return self._flow_settings

    @flow_settings.setter
    def flow_settings(self, flow_settings: schemas.FlowSettings):
        if (self._flow_settings.execution_location != flow_settings.execution_location) or (
            self._flow_settings.execution_mode != flow_settings.execution_mode
        ):
            self.reset()
        else:

            def _param_state(params: list[schemas.FlowParameter]) -> dict:
                return {p.name: (p.default_value, p.type, tuple(p.enum_values or [])) for p in params}

            old_params = _param_state(self._flow_settings.parameters)
            new_params = _param_state(flow_settings.parameters)
            if old_params != new_params:
                for node in self.nodes:
                    if node.setting_input is not None and find_unresolved_in_model(node.setting_input):
                        node.reset(deep=True)
        self._flow_settings = flow_settings

    # ==================== History Management Methods ====================

    def capture_history_snapshot(
        self,
        action_type: HistoryActionType,
        description: str,
        node_id: int = None,
    ) -> bool:
        """Capture the current state before a change for undo support.

        Args:
            action_type: The type of action being performed.
            description: Human-readable description of the action.
            node_id: Optional ID of the affected node.

        Returns:
            True if snapshot was captured, False if skipped.
        """
        return self._history_manager.capture_snapshot(self, action_type, description, node_id)

    def capture_history_if_changed(
        self,
        pre_snapshot: schemas.FlowfileData,
        action_type: HistoryActionType,
        description: str,
        node_id: int = None,
    ) -> bool:
        """Capture history only if the flow state actually changed.

        Use this for settings updates where the change might be a no-op.
        Call this AFTER the change is applied.

        Args:
            pre_snapshot: The FlowfileData captured BEFORE the change.
            action_type: The type of action that was performed.
            description: Human-readable description of the action.
            node_id: Optional ID of the affected node.

        Returns:
            True if a change was detected and snapshot was captured.
        """
        return self._history_manager.capture_if_changed(self, pre_snapshot, action_type, description, node_id)

    def undo(self) -> UndoRedoResult:
        """Undo the last action by restoring to the previous state.

        Returns:
            UndoRedoResult indicating success or failure.
        """
        return self._history_manager.undo(self)

    def redo(self) -> UndoRedoResult:
        """Redo the last undone action.

        Returns:
            UndoRedoResult indicating success or failure.
        """
        return self._history_manager.redo(self)

    def get_history_state(self) -> HistoryState:
        """Get the current state of the history system.

        Returns:
            HistoryState with information about available undo/redo operations.
        """
        return self._history_manager.get_state()

    def mark_as_saved(self) -> None:
        """Mark the current flow state as the saved baseline (for dirty tracking)."""
        self._history_manager.mark_saved(self)

    def has_unsaved_changes(self) -> bool:
        """Return True if the flow has changed since the last save point."""
        return self._history_manager.has_unsaved_changes(self)

    def _execute_with_history(
        self,
        operation: Callable[[], Any],
        action_type: HistoryActionType,
        description: str,
        node_id: int = None,
    ) -> Any:
        """Execute an operation with automatic history capture.

        This helper captures the state before the operation, executes it,
        and records history only if the state actually changed.

        Args:
            operation: A callable that performs the actual operation.
            action_type: The type of action being performed.
            description: Human-readable description of the action.
            node_id: Optional ID of the affected node.

        Returns:
            The result of the operation (if any).
        """
        # Skip history capture if tracking is disabled for this flow
        if not self.flow_settings.track_history:
            return operation()

        pre_snapshot = self.get_flowfile_data()
        result = operation()
        self._history_manager.capture_if_changed(self, pre_snapshot, action_type, description, node_id)
        return result

    def restore_from_snapshot(self, snapshot: schemas.FlowfileData) -> None:
        """Clear current state and rebuild from a snapshot.

        This method is used internally by undo/redo to restore a previous state.

        Args:
            snapshot: The FlowfileData snapshot to restore from.
        """
        from flowfile_core.flowfile.manage.io_flowfile import (
            _flowfile_data_to_flow_information,
            determine_insertion_order,
        )

        identity = _FlowIdentity.capture(self)
        node_owners = _NodeOwners.capture(self)

        flow_info = _flowfile_data_to_flow_information(snapshot)

        self._node_db.clear()
        self._node_ids.clear()
        self._flow_starts.clear()
        self._groups.clear()
        self._results = None

        self._flow_settings = flow_info.flow_settings
        identity.restore_onto(self)

        ingestion_order = determine_insertion_order(flow_info)

        for node_id in ingestion_order:
            node_info = flow_info.data[node_id]
            if getattr(node_info.setting_input, "is_user_defined", False) and node_info.type not in CUSTOM_NODE_STORE:
                register_missing_node_template(node_info.type)
            node_promise = input_schema.NodePromise(
                flow_id=identity.flow_id,
                node_id=node_info.id,
                pos_x=node_info.x_position or 0,
                pos_y=node_info.y_position or 0,
                node_type=node_info.type,
            )
            if hasattr(node_info.setting_input, "cache_results"):
                node_promise.cache_results = node_info.setting_input.cache_results
            self.add_node_promise(node_promise)

        for node_id in ingestion_order:
            node_info = flow_info.data[node_id]
            if node_info.is_setup and node_info.setting_input is not None:
                if hasattr(node_info.setting_input, "flow_id"):
                    node_info.setting_input.flow_id = identity.flow_id

                if hasattr(node_info.setting_input, "user_id"):
                    node_info.setting_input.user_id = node_owners.owner_of(node_id)

                if hasattr(node_info.setting_input, "is_user_defined") and node_info.setting_input.is_user_defined:
                    # .get() execs the node module lazily; on any failure the node
                    # lands in the missing/error path so the flow still opens.
                    self._place_user_defined_node(node_info.type, node_info.setting_input)
                else:
                    add_method = getattr(self, "add_" + node_info.type, None)
                    if add_method:
                        add_method(node_info.setting_input)

        for node_id in ingestion_order:
            node_info = flow_info.data[node_id]
            from_node = self.get_node(node_id)
            if from_node is None:
                continue

            for output_node_id in node_info.outputs or []:
                to_node = self.get_node(output_node_id)
                if to_node is None:
                    continue
                if to_node.accepts_dynamic_inputs:
                    continue  # keyed edges are restored from input_connections below

                output_node_info = flow_info.data.get(output_node_id)
                if output_node_info is None:
                    continue

                is_left_input = (output_node_info.left_input_id == node_id) and (
                    to_node.left_input is None or to_node.left_input.node_id != node_id
                )
                is_right_input = (output_node_info.right_input_id == node_id) and (
                    to_node.right_input is None or to_node.right_input.node_id != node_id
                )
                is_main_input = node_id in (output_node_info.input_ids or [])

                if is_left_input:
                    insert_type = "left"
                elif is_right_input:
                    insert_type = "right"
                elif is_main_input:
                    insert_type = "main"
                else:
                    continue

                to_node.add_node_connection(from_node, insert_type)

        restore_dynamic_input_connections(self, flow_info)

        # Member group_ids were re-applied above via add_<type>(setting_input);
        # repopulate the box registry (name/color/bounds) from the snapshot.
        self.restore_groups(flow_info.groups)

        logger.info(f"Restored flow from snapshot with {len(self._node_db)} nodes")

    # ==================== End History Management Methods ====================

    # ==================== Group Management Methods ====================
    # Groups are purely visual containers. They never affect execution; the only
    # link to a node is that node's setting_input.group_id. The group box props
    # (name/color/bounds) live in self._groups and ride along in FlowfileData.

    def _next_group_id(self) -> int:
        """Allocate a monotonically increasing group id (never reuses a freed id this session)."""
        self._group_id_seq = max([self._group_id_seq, *self._groups]) + 1
        return self._group_id_seq

    def _member_node_ids(self, group_id: int) -> list[int]:
        """Derive a group's members by scanning node group_id (single source of truth)."""
        return [node.node_id for node in self.nodes if getattr(node.setting_input, "group_id", None) == group_id]

    def _set_node_group(self, node_id: int, group_id: int | None) -> None:
        node = self.get_node(node_id)
        if node is not None and node.setting_input is not None and hasattr(node.setting_input, "group_id"):
            node.setting_input.group_id = group_id

    def _child_group_ids(self, group_id: int) -> list[int]:
        """Sub-groups whose immediate parent is this group."""
        return [gid for gid, g in self._groups.items() if g.parent_group_id == group_id]

    def _group_depth(self, group_id: int) -> int:
        """Nesting depth (0 = top-level); cycle-safe."""
        depth, seen, current = 0, set(), self._groups.get(group_id)
        while current is not None and current.parent_group_id is not None and current.id not in seen:
            seen.add(current.id)
            depth += 1
            current = self._groups.get(current.parent_group_id)
        return depth

    def _is_ancestor_group(self, ancestor_id: int, group_id: int) -> bool:
        """True if ancestor_id equals group_id or one of its ancestors (cycle-safe)."""
        seen, current = set(), self._groups.get(group_id)
        while current is not None and current.id not in seen:
            if current.id == ancestor_id:
                return True
            seen.add(current.id)
            current = self._groups.get(current.parent_group_id) if current.parent_group_id is not None else None
        return False

    def _recompute_group_bounds(self, group_id: int | None = None) -> None:
        """Refit one or all group boxes around their member nodes and child groups.

        Uses nominal node dimensions since the backend doesn't know rendered sizes;
        the frontend refines bounds on first user interaction. Groups with no members
        keep their current bounds. When refitting all groups, deepest first so a parent
        unions already-fitted child-group boxes.
        """
        node_width, node_height, padding, header = 180.0, 80.0, 40.0, 36.0
        if group_id is not None:
            target_ids = [group_id]
        else:
            target_ids = sorted(self._groups, key=self._group_depth, reverse=True)
        for gid in target_ids:
            group = self._groups.get(gid)
            if group is None:
                continue
            boxes: list[tuple[float, float, float, float]] = []  # (x, y, w, h)
            for nid in self._member_node_ids(gid):
                node = self.get_node(nid)
                if node is not None and node.setting_input is not None:
                    nx = float(node.setting_input.pos_x or 0)
                    ny = float(node.setting_input.pos_y or 0)
                    boxes.append((nx, ny, node_width, node_height))
            for cid in self._child_group_ids(gid):
                child = self._groups.get(cid)
                if child is not None:
                    boxes.append((child.x_position, child.y_position, child.width, child.height))
            if not boxes:
                continue
            min_x = min(b[0] for b in boxes) - padding
            min_y = min(b[1] for b in boxes) - padding - header
            max_x = max(b[0] + b[2] for b in boxes) + padding
            max_y = max(b[1] + b[3] for b in boxes) + padding
            group.x_position = min_x
            group.y_position = min_y
            group.width = max_x - min_x
            group.height = max_y - min_y

    def create_group(
        self,
        name: str,
        node_ids: list[int],
        *,
        color: schemas.GroupColor | None = None,
        bounds: schemas.GroupBounds | None = None,
        parent_group_id: int | None = None,
        child_group_ids: list[int] | None = None,
    ) -> schemas.GroupInformation:
        """Create a visual group. Organizational only.

        Members are the given nodes (group_id) and child groups (their parent_group_id).
        The new group itself nests under parent_group_id. Bounds are computed when not supplied.
        """

        def _do() -> schemas.GroupInformation:
            group_id = self._next_group_id()
            group = schemas.GroupInformation(id=group_id, name=name, color=color, parent_group_id=parent_group_id)
            if bounds is not None:
                group.x_position, group.y_position, group.width, group.height = bounds
            self._groups[group_id] = group
            for node_id in node_ids:
                self._set_node_group(node_id, group_id)
            for cid in child_group_ids or []:
                child = self._groups.get(cid)
                if child is not None and not self._is_ancestor_group(cid, group_id):
                    child.parent_group_id = group_id
            if bounds is None:
                self._recompute_group_bounds(group_id)
            return group

        return self._execute_with_history(_do, HistoryActionType.CREATE_GROUP, f"Create group '{name}'")

    def update_group(
        self,
        group_id: int,
        *,
        name: str | None = None,
        color: schemas.GroupColor | None = None,
        bounds: schemas.GroupBounds | None = None,
        collapsed: bool | None = None,
    ) -> schemas.GroupInformation:
        """Rename / recolor / move / resize / collapse a group box."""
        group = self._groups.get(group_id)
        if group is None:
            raise ValueError(f"Group {group_id} does not exist")

        def _do() -> schemas.GroupInformation:
            if name is not None:
                group.name = name
            if color is not None:
                group.color = color
            if bounds is not None:
                group.x_position, group.y_position, group.width, group.height = bounds
            if collapsed is not None:
                group.collapsed = collapsed
            return group

        return self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP, f"Update group '{group.name}'")

    def delete_group(self, group_id: int) -> None:
        """Remove a group box (ungroup). Members and sub-groups lift up one level."""
        group = self._groups.get(group_id)
        if group is None:
            return
        new_parent = group.parent_group_id

        def _do() -> None:
            for node_id in self._member_node_ids(group_id):
                self._set_node_group(node_id, new_parent)
            for cid in self._child_group_ids(group_id):
                child = self._groups.get(cid)
                if child is not None:
                    child.parent_group_id = new_parent
            self._groups.pop(group_id, None)

        self._execute_with_history(_do, HistoryActionType.DELETE_GROUP, f"Delete group '{group.name}'")

    def add_nodes_to_group(self, group_id: int, node_ids: list[int]) -> schemas.GroupInformation:
        """Add nodes to an existing group and refit its bounds."""
        group = self._groups.get(group_id)
        if group is None:
            raise ValueError(f"Group {group_id} does not exist")

        def _do() -> schemas.GroupInformation:
            for node_id in node_ids:
                self._set_node_group(node_id, group_id)
            self._recompute_group_bounds(group_id)
            return group

        return self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP_MEMBERSHIP, "Add nodes to group")

    def remove_nodes_from_group(self, node_ids: list[int]) -> None:
        """Remove nodes from whatever group they belong to; prune groups left empty."""

        def _do() -> None:
            affected: set[int] = set()
            for node_id in node_ids:
                node = self.get_node(node_id)
                current = getattr(node.setting_input, "group_id", None) if node is not None else None
                if current is not None:
                    affected.add(current)
                    self._set_node_group(node_id, None)
            for gid in affected:
                if not self._member_node_ids(gid) and not self._child_group_ids(gid):
                    self._groups.pop(gid, None)
                else:
                    self._recompute_group_bounds(gid)

        self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP_MEMBERSHIP, "Remove nodes from group")

    def assign_node_to_named_group(
        self, node_id: int, name: str, *, color: schemas.GroupColor | None = None
    ) -> schemas.GroupInformation:
        """Assign a node to a group identified by name, creating it if absent (find-or-create)."""
        existing = next((group for group in self._groups.values() if group.name == name), None)
        if existing is not None:
            return self.add_nodes_to_group(existing.id, [node_id])
        return self.create_group(name, [node_id], color=color)

    def set_node_positions(self, updates: list[schemas.NodePositionUpdate]) -> None:
        """Persist dragged node positions (absolute canvas coordinates) onto setting_input.

        Plain mutator: the caller (update_layout route) captures history once for the
        whole drag-end batch so node moves and group-bounds changes share one snapshot.
        """
        for update in updates:
            node = self.get_node(update.node_id)
            if node is not None and node.setting_input is not None and hasattr(node.setting_input, "pos_x"):
                node.setting_input.pos_x = update.pos_x
                node.setting_input.pos_y = update.pos_y

    def set_group_bounds(self, updates: list[schemas.GroupBoundsUpdate]) -> None:
        """Persist group box bounds (used together with set_node_positions on drag/resize)."""
        for update in updates:
            group = self._groups.get(update.group_id)
            if group is not None:
                group.x_position = update.x_position
                group.y_position = update.y_position
                group.width = update.width
                group.height = update.height

    def restore_groups(self, groups: list[schemas.GroupInformation]) -> None:
        """Replace the runtime group registry (used by open_flow and restore_from_snapshot)."""
        self._groups = {group.id: group for group in groups}
        self._group_id_seq = max(self._groups, default=0)  # next id resumes above the highest restored
        for group in self._groups.values():
            if group.width <= 0 or group.height <= 0:
                self._recompute_group_bounds(group.id)

    # ==================== End Group Management Methods ====================

    def add_node_to_starting_list(self, node: FlowNode) -> None:
        """Adds a node to the list of starting nodes for the flow if not already present.

        Args:
            node: The FlowNode to add as a starting node.
        """
        if node.node_id not in {self_node.node_id for self_node in self._flow_starts}:
            self._flow_starts.append(node)

    def add_node_promise(self, node_promise: input_schema.NodePromise, track_history: bool = True):
        """Adds a placeholder node to the graph that is not yet fully configured.

        Useful for building the graph structure before all settings are available.
        Automatically captures history for undo/redo support.

        Args:
            node_promise: A promise object containing basic node information.
            track_history: Whether to track this change in history (default True).
        """

        def _do_add():
            def placeholder(n: FlowNode = None):
                if n is None:
                    return FlowDataEngine()
                return n

            self.add_node_step(
                node_id=node_promise.node_id,
                node_type=node_promise.node_type,
                function=placeholder,
                setting_input=node_promise,
            )
            if node_promise.is_user_defined:
                node_needs_settings: bool
                custom_node = CUSTOM_NODE_STORE.get(node_promise.node_type)
                if custom_node is None:
                    raise ValueError(missing_custom_node_error(node_promise.node_type))
                settings_schema = custom_node.model_fields["settings_schema"].default
                node_needs_settings = settings_schema is not None and not settings_schema.is_empty()
                if not node_needs_settings:
                    user_defined_node_settings = input_schema.UserDefinedNode(settings={}, **node_promise.model_dump())
                    initialized_model = custom_node()
                    self.add_user_defined_node(
                        custom_node=initialized_model, user_defined_node_settings=user_defined_node_settings
                    )

        if track_history:
            self._execute_with_history(
                _do_add,
                HistoryActionType.ADD_NODE,
                f"Add {node_promise.node_type} node",
                node_id=node_promise.node_id,
            )
        else:
            _do_add()

    def apply_layout(self, y_spacing: int = 150, x_spacing: int = 200, initial_y: int = 100):
        """Calculates and applies a layered layout to all nodes in the graph.

        This updates their x and y positions for UI rendering.

        Args:
            y_spacing: The minimum vertical spacing between two nodes in a layer.
            x_spacing: The horizontal spacing between layers.
            initial_y: The y-position of the topmost node.
        """
        self.flow_logger.info("Applying layered layout...")
        start_time = time()
        try:
            new_positions = calculate_layered_layout(
                self, y_spacing=y_spacing, x_spacing=x_spacing, initial_y=initial_y
            )

            if not new_positions:
                self.flow_logger.warning("Layout calculation returned no positions.")
                return

            updated_count = 0
            for node_id, (pos_x, pos_y) in new_positions.items():
                node = self.get_node(node_id)
                if node and hasattr(node, "setting_input"):
                    setting = node.setting_input
                    if hasattr(setting, "pos_x") and hasattr(setting, "pos_y"):
                        setting.pos_x = pos_x
                        setting.pos_y = pos_y
                        updated_count += 1
                    else:
                        self.flow_logger.warning(
                            f"Node {node_id} setting_input ({type(setting)}) lacks pos_x/pos_y attributes."
                        )
                elif node:
                    self.flow_logger.warning(f"Node {node_id} lacks setting_input attribute.")
                # else: node removed between calculation and apply; skip it

            # Reflowed node positions invalidate group boxes — refit them.
            self._recompute_group_bounds()

            end_time = time()
            self.flow_logger.info(
                f"Layout applied to {updated_count}/{len(self.nodes)} nodes in {end_time - start_time:.2f} seconds."
            )

        except Exception as e:
            self.flow_logger.error(f"Layout failed, keeping current positions: {e}")

    @property
    def flow_id(self) -> int:
        """Gets the unique identifier of the flow."""
        return self._flow_id

    @flow_id.setter
    def flow_id(self, new_id: int):
        """Sets the unique identifier for the flow and updates all child nodes.

        Args:
            new_id: The new flow ID.
        """
        self._flow_id = new_id
        for node in self.nodes:
            if hasattr(node.setting_input, "flow_id"):
                node.setting_input.flow_id = new_id
        self.flow_settings.flow_id = new_id

    def __repr__(self):
        """Provides the official string representation of the FlowGraph instance."""
        settings_str = "  -" + "\n  -".join(f"{k}: {v}" for k, v in self.flow_settings)
        return f"FlowGraph(\nNodes: {self._node_db}\n\nSettings:\n{settings_str}"

    def print_tree(self):
        """Print flow_graph as a visual tree structure, showing the DAG relationships with ASCII art."""
        if not self._node_db:
            self.flow_logger.info("Empty flow graph")
            return

        node_info = build_node_info(self.nodes)

        for node_id in node_info:
            calculate_depth(node_id, node_info)

        depth_groups, max_depth = group_nodes_by_depth(node_info)

        for depth in depth_groups:
            depth_groups[depth].sort()

        lines = ["=" * 80, "Flow Graph Visualization", "=" * 80, ""]

        merge_points = define_node_connections(node_info)

        max_label_length = {}
        for depth in range(max_depth + 1):
            if depth in depth_groups:
                max_len = max(len(node_info[nid].label) for nid in depth_groups[depth])
                max_label_length[depth] = max_len

        drawn_nodes = set()
        merge_drawn = set()

        paths_by_merge = {}
        standalone_paths = []

        paths = build_flow_paths(node_info, self._flow_starts, merge_points)

        for path in paths:
            if len(path) > 1 and path[-1] in merge_points and len(merge_points[path[-1]]) > 1:
                merge_id = path[-1]
                if merge_id not in paths_by_merge:
                    paths_by_merge[merge_id] = []
                paths_by_merge[merge_id].append(path)
            else:
                standalone_paths.append(path)

        draw_merged_paths(node_info, merge_points, paths_by_merge, merge_drawn, drawn_nodes, lines)

        draw_standalone_paths(drawn_nodes, standalone_paths, lines, node_info)

        add_un_drawn_nodes(drawn_nodes, node_info, lines)

        try:
            execution_plan = compute_execution_plan(
                nodes=self.nodes, flow_starts=self._flow_starts + self.get_implicit_starter_nodes()
            )
            ordered_nodes = execution_plan.all_nodes
            if ordered_nodes:
                for i, node in enumerate(ordered_nodes, 1):
                    lines.append(f"  {i:3d}. {node_info[node.node_id].label}")
        except Exception as e:
            lines.append(f"  Could not determine execution order: {e}")

        output = "\n".join(lines)

        print(output)

    def get_nodes_overview(self):
        """Gets a list of dictionary representations for all nodes in the graph."""
        output = []
        for v in self._node_db.values():
            output.append(v.get_repr())
        return output

    def remove_from_output_cols(self, columns: list[str]):
        """Removes specified columns from the list of expected output columns.

        Args:
            columns: A list of column names to remove.
        """
        cols = set(columns)
        self._output_cols = [c for c in self._output_cols if c not in cols]

    def get_node(self, node_id: int | str = None) -> FlowNode | None:
        """Retrieves a node from the graph by its ID.

        Args:
            node_id: The ID of the node to retrieve. If None, retrieves the last added node.

        Returns:
            The FlowNode object, or None if not found.
        """
        if node_id is None:
            node_id = self._node_ids[-1]
        node = self._node_db.get(node_id)
        if node is not None:
            return node

    def add_user_defined_node(
        self, *, custom_node: CustomNodeBase, user_defined_node_settings: input_schema.UserDefinedNode
    ):
        """Adds a user-defined custom node to the graph.

        When the custom node has a ``kernel_id`` set, the process code is sent
        to the kernel for execution instead of running locally.  This enables
        custom nodes to use external packages installed on the kernel.

        Args:
            custom_node: The custom node instance to add.
            user_defined_node_settings: The settings for the user-defined node.
        """
        kernel_id = user_defined_node_settings.kernel_id or custom_node.kernel_id
        if (custom_node.environment == "kernel" or custom_node.requires_kernel) and not kernel_id:
            raise KernelRequiredError(custom_node.item)

        registry_entry = user_defined_registry.get(custom_node.item)
        if registry_entry is not None and registry_entry.source_hash:
            user_defined_node_settings.node_source_hash = registry_entry.source_hash

        # Output handles are structural — the node class declares them; the settings
        # copy is a persistence snapshot kept in sync for save/codegen.
        output_names = list(custom_node.output_names or user_defined_node_settings.output_names)
        user_defined_node_settings.output_names = output_names

        if kernel_id:
            _func = self._make_kernel_user_defined_func(
                custom_node=custom_node,
                user_defined_node_settings=user_defined_node_settings,
                kernel_id=kernel_id,
                output_names=output_names,
                registry_entry=registry_entry,
            )
        else:
            _func = self._make_local_user_defined_func(
                custom_node=custom_node,
                user_defined_node_settings=user_defined_node_settings,
                output_names=output_names,
                registry_entry=registry_entry,
            )

        # Wire the hook through add_node_step so user_provided_schema_callback is set
        # BEFORE setting_input triggers reset(): otherwise a 0-input node's eager
        # schema prefetch would run the real function (kernel/worker) in the background.
        schema_callback = None
        if type(custom_node).predict_output_schema is not CustomNodeBase.predict_output_schema:
            schema_callback = self._make_user_defined_schema_callback(
                custom_node=custom_node,
                node_id=user_defined_node_settings.node_id,
                output_names=output_names,
            )
        elif not kernel_id and bool(getattr(custom_node, "requires_data_for_prediction", False)):
            # Hookless data-dependent node: never predict by executing; block until run.
            schema_callback = self._make_blocked_prediction_callback(node_id=user_defined_node_settings.node_id)
        else:
            # Traceability: a stale registry class (or a genuinely hook-less node)
            # lands here and schema prediction degrades to the execution tier.
            logger.info(
                f"custom node {custom_node.item}: no predict_output_schema override on "
                f"{type(custom_node).__module__}.{type(custom_node).__name__}; "
                f"schema prediction uses the execution tier"
            )

        self.add_node_step(
            node_id=user_defined_node_settings.node_id,
            function=_func,
            setting_input=user_defined_node_settings,
            input_node_ids=user_defined_node_settings.depending_on_ids,
            node_type=custom_node.item,
            schema_callback=schema_callback,
        )
        node = self.get_node(user_defined_node_settings.node_id)
        node._executes_on_kernel = bool(kernel_id)
        node._prediction_requires_data = bool(getattr(custom_node, "requires_data_for_prediction", False))
        if custom_node.number_of_inputs == 0:
            self.add_node_to_starting_list(node)
        if custom_node.settings_schema is not None and user_defined_node_settings.settings:
            report = custom_node.settings_schema.populate_values_report(user_defined_node_settings.settings)
            if report.has_drift:
                unknown = report.unknown_sections + report.unknown_components
                node.results.warnings = (
                    f"Stored settings no longer match the node's schema; ignored keys: {', '.join(sorted(unknown))}"
                )

    def add_missing_user_defined_node(
        self, *, user_defined_node_settings: input_schema.UserDefinedNode, node_type: str, error: str
    ):
        """Adds a placeholder for a custom node that cannot be loaded on this machine.

        The stored settings are preserved verbatim (lossless re-save), the node
        renders with its connections, and running the flow fails this node with
        ``error`` instead of silently dropping it.
        """
        register_missing_node_template(node_type)

        def _missing_custom_node(*_flow_data_engine: FlowDataEngine) -> FlowDataEngine:
            raise ValueError(error)

        self.add_node_step(
            node_id=user_defined_node_settings.node_id,
            function=_missing_custom_node,
            setting_input=user_defined_node_settings,
            input_node_ids=user_defined_node_settings.depending_on_ids,
            node_type=node_type,
        )
        node = self.get_node(user_defined_node_settings.node_id)
        node.results.errors = error

    def _place_user_defined_node(
        self, node_type: str, user_defined_node_settings: input_schema.UserDefinedNode
    ) -> None:
        """Place a custom node from the store, degrading to a missing-node placeholder when
        its type isn't installed. Shared by copy and both flow-restore paths."""
        user_defined_node_class = CUSTOM_NODE_STORE.get(node_type)
        if user_defined_node_class is not None:
            self.add_user_defined_node(
                custom_node=user_defined_node_class.from_settings(user_defined_node_settings.settings),
                user_defined_node_settings=user_defined_node_settings,
            )
        else:
            self.add_missing_user_defined_node(
                user_defined_node_settings=user_defined_node_settings,
                node_type=node_type,
                error=missing_custom_node_error(node_type),
            )

    @staticmethod
    def _predicted_value_to_columns(value) -> list[FlowfileColumn] | None:
        """Normalize a predict_output_schema return value into FlowfileColumns.

        LazyFrames/DataFrames contribute their (lazily resolved) schema; a plain
        ``pl.Schema`` is tolerated for hand-declared shapes. None means unusable.
        """
        if isinstance(value, pl.LazyFrame):
            return pl_schema_to_flowfile_columns(value.collect_schema())
        if isinstance(value, pl.DataFrame):
            return pl_schema_to_flowfile_columns(value.schema)
        if isinstance(value, pl.Schema):
            return pl_schema_to_flowfile_columns(value)
        return None

    def _make_blocked_prediction_callback(self, *, node_id: int) -> Callable:
        """Hookless ``requires_data_for_prediction=True``: prediction must never
        execute ``process()``. Wired through ``add_node_step`` so a 0-input
        node's eager prefetch hits this cheap callback instead of the real
        function."""

        def schema_callback() -> list[FlowfileColumn]:
            node = self.get_node(node_id)
            if node is None:
                return []
            if node.node_stats.has_completed_last_run and node.node_schema.result_schema:
                node._schema_prediction_blocked = None
                return node.node_schema.result_schema
            reason = data_needed_block_reason(node)
            node._schema_prediction_blocked = reason
            node.results.warnings = reason
            return []

        return schema_callback

    def _make_user_defined_schema_callback(
        self, *, custom_node: CustomNodeBase, node_id: int, output_names: list[str]
    ) -> Callable:
        """Build a schema callback from the node's ``predict_output_schema`` hook.

        The hook always runs in core, even for kernel nodes, and returns a frame
        (or dict of frames) whose schema is read lazily. Data-needing hooks
        (``requires_data_for_prediction=True``) get real upstream data —
        materialized in-core when the un-run chain is kernel-free, or a kernel
        warning instead (never an implicit kernel run). Returning ``[]`` makes
        the prediction ladder fall back to the execution-based path. The node is
        resolved lazily so the callback can be passed into ``add_node_step``
        before the node exists.
        """
        resolved_output_names = output_names or ["main"]
        requires_data = bool(getattr(custom_node, "requires_data_for_prediction", False))

        def _hook_input_frame(input_node: FlowNode, src_handle: str) -> pl.LazyFrame:
            # Real lazy data when the upstream has run (worker results are
            # scan_ipc plans, cheap to sample). For data-needing hooks on a
            # kernel-free chain, materialize the un-run upstream in-core,
            # pivot-style — the hook's own collect bounds what is computed.
            if input_node.node_stats.has_completed_last_run:
                engine = (input_node._named_outputs or {}).get(src_handle) or input_node.results.resulting_data
                if engine is not None:
                    frame = engine.data_frame
                    return frame if isinstance(frame, pl.LazyFrame) else frame.lazy()
            if requires_data:
                engine = input_node.get_output(src_handle) or input_node.get_resulting_data()
            else:
                engine = input_node.get_predicted_resulting_data(src_handle)
            frame = engine.data_frame
            return frame if isinstance(frame, pl.LazyFrame) else frame.lazy()

        def schema_callback() -> list[FlowfileColumn]:
            node = self.get_node(node_id)
            if node is None:
                return []
            node._schema_prediction_blocked = None
            if requires_data:
                reason = kernel_block_reason(node, include_self=False)
                if reason:
                    # Never execute a kernel implicitly for prediction: surface
                    # the warning and let the exec-tier gate suppress fallback.
                    node._schema_prediction_blocked = reason
                    node.results.warnings = reason
                    return []
            try:
                input_frames = []
                for input_node, src_handle in node._slot_input_pairs():
                    if input_node is None:
                        input_frames.append(pl.LazyFrame())
                        continue
                    input_frames.append(_hook_input_frame(input_node, src_handle))
                predicted = custom_node.predict_output_schema(*input_frames)
            except Exception as e:
                logger.warning(f"predict_output_schema failed for node {node_id}: {e}")
                return []
            if predicted is None:
                return []
            if isinstance(predicted, dict) and not isinstance(predicted, pl.Schema):
                named: dict[str, list[FlowfileColumn]] = {}
                for i, name in enumerate(resolved_output_names):
                    columns = self._predicted_value_to_columns(predicted.get(name))
                    if columns is None:
                        logger.warning(
                            f"predict_output_schema for node {node_id} missing or unsupported "
                            f"declared output '{name}'"
                        )
                        return []
                    named[output_handle(i)] = columns
                node._named_schemas = named
                return named.get(DEFAULT_OUTPUT_HANDLE, [])
            columns = self._predicted_value_to_columns(predicted)
            if columns is None:
                logger.warning(f"predict_output_schema for node {node_id} returned an unsupported value")
                return []
            if len(resolved_output_names) > 1:
                logger.warning(
                    f"predict_output_schema for node {node_id} returned a single frame but the node "
                    f"declares outputs {resolved_output_names}; falling back to execution-based prediction"
                )
                return []
            return columns

        return schema_callback

    def _make_local_user_defined_func(
        self,
        *,
        custom_node: CustomNodeBase,
        user_defined_node_settings: input_schema.UserDefinedNode,
        output_names: list[str] | None = None,
        registry_entry=None,
    ) -> Callable:
        """Create the execution function for a non-kernel custom node.

        Offloads process() to the worker (which owns dataset memory in a
        killable subprocess) whenever the flow doesn't run in local mode and
        the node came from the registry; otherwise runs in-process (the
        --run-flow / offload-disabled fallback, and inline test classes that
        have no source file on disk).
        """
        resolved_output_names = output_names or custom_node.output_names or ["main"]

        def _run_in_core(*flow_data_engine: FlowDataEngine) -> FlowDataEngine | None:
            user_id = user_defined_node_settings.user_id
            if user_id is not None:
                custom_node.set_execution_context(user_id)

            output = custom_node.process(*(fde.data_frame.lazy() for fde in flow_data_engine))

            accessed_secrets = custom_node.get_accessed_secrets()
            if accessed_secrets:
                logger.info(f"Node '{user_defined_node_settings.node_id}' accessed secrets: {accessed_secrets}")
            if isinstance(output, dict):
                node = self.get_node(user_defined_node_settings.node_id)
                primary = None
                for i, name in enumerate(resolved_output_names):
                    if name not in output:
                        raise ValueError(f"process() did not return declared output '{name}'")
                    fde = FlowDataEngine(output[name])
                    node._named_outputs[f"output-{i}"] = fde
                    if i == 0:
                        primary = fde
                return primary
            if isinstance(output, pl.LazyFrame | pl.DataFrame):
                return FlowDataEngine(output)
            return None

        def _func(*flow_data_engine: FlowDataEngine) -> FlowDataEngine | None:
            if self.execution_location == "local" or registry_entry is None or registry_entry.source_text is None:
                return _run_in_core(*flow_data_engine)

            node = self.get_node(user_defined_node_settings.node_id)
            request = CustomNodeExecuteInput(
                task_id=custom_node_task_id(node.hash),
                node_source=registry_entry.source_text,
                class_name=registry_entry.class_name,
                settings_values=user_defined_node_settings.settings or {},
                secrets=resolve_secret_payload(custom_node, user_defined_node_settings.user_id),
                inputs=[fde.data_frame.lazy().serialize() for fde in flow_data_engine],
                output_names=resolved_output_names,
                user_id=user_defined_node_settings.user_id,
                flowfile_flow_id=self.flow_id,
                flowfile_node_id=user_defined_node_settings.node_id,
            )
            fetcher = ExternalCustomNodeFetcher(request)
            node._fetch_cached_df = fetcher
            payload = fetcher.get_payload()

            primary: FlowDataEngine | None = None
            for i, name in enumerate(resolved_output_names):
                info = payload["outputs"].get(name)
                if info is None:
                    raise ValueError(f"Worker did not return declared output '{name}'")
                fde = FlowDataEngine(pl.scan_ipc(info["path"]), number_of_records=info["row_count"])
                node._named_outputs[f"output-{i}"] = fde
                if i == 0:
                    primary = fde
            return primary

        return _func

    def _execute_on_kernel(
        self,
        *,
        node_id: int,
        kernel_id: str,
        code: str,
        output_names: list[str],
        flow_data_engine: tuple[FlowDataEngine, ...],
        declared_publishes: list[str] | None = None,
        required_dependencies: list[str] | None = None,
        node_type: str | None = None,
    ) -> FlowDataEngine | None:
        """Execute code on a kernel container and return the primary output.

        Shared logic for both custom-node kernel execution and python_script nodes.
        Handles artifact context, directory setup, input writing, kernel execution,
        log forwarding, artifact recording, and output reading.
        """
        manager = get_kernel_manager()
        if required_dependencies:
            # Fail fast with a clear message instead of a ModuleNotFoundError
            # from inside process(). Only provable mismatches block; an unknown
            # kernel id falls through to execute_sync's own error path.
            kernel_info = manager.get_kernel_sync(kernel_id)
            if kernel_info is not None:
                missing = verify_kernel_for_node(kernel_info, required_dependencies)
                if missing:
                    raise KernelDependencyError(node_type or "", kernel_id, kernel_info.name, missing)
        flow_id = self.flow_id
        node_logger = self.flow_logger.get_node_logger(node_id)

        self.artifact_context.clear_nodes({node_id})

        available = self.artifact_context.compute_available(
            node_id=node_id,
            kernel_id=kernel_id,
            upstream_node_ids=self._get_upstream_node_ids(node_id),
        )

        shared_base = manager.shared_volume_path
        input_dir = os.path.join(shared_base, str(flow_id), str(node_id), "inputs")
        output_dir = os.path.join(shared_base, str(flow_id), str(node_id), "outputs")
        os.makedirs(input_dir, exist_ok=True)
        os.makedirs(output_dir, exist_ok=True)
        clear_stale_parquets(input_dir)
        clear_stale_parquets(output_dir)

        node = self.get_node(node_id)
        input_names = self._resolve_input_names(node, len(flow_data_engine))
        input_paths = write_inputs_to_parquet(
            flow_data_engine, manager, input_dir, flow_id, node_id, input_names=input_names
        )

        request = build_execute_request(
            node_id=node_id,
            code=code,
            input_paths=input_paths,
            output_dir=output_dir,
            flow_id=flow_id,
            manager=manager,
            source_registration_id=self._flow_settings.source_registration_id,
            available_artifacts={name: ref.source_node_id for name, ref in available.items()},
        )

        cancel_event = threading.Event()
        if node is not None:
            node._kernel_cancel_context = (kernel_id, manager, request.exec_token)
            node._kernel_cancel_event = cancel_event
        try:
            result = manager.execute_sync(kernel_id, request, self.flow_logger, cancel_event=cancel_event)
        finally:
            if node is not None:
                node._kernel_cancel_context = None
                node._kernel_cancel_event = None

        forward_kernel_logs(result, node_logger)
        if not result.success:
            raise RuntimeError(f"Kernel execution failed: {result.error}")

        if result.artifacts_published:
            self.artifact_context.record_published(
                node_id=node_id,
                kernel_id=kernel_id,
                artifacts=[a.model_dump() for a in result.artifacts_published],
            )
        if result.artifacts_deleted:
            self.artifact_context.record_deleted(
                node_id=node_id,
                kernel_id=kernel_id,
                artifact_names=result.artifacts_deleted,
            )

        if declared_publishes:
            observed_names = {a.name for a in result.artifacts_published}
            for name in declared_publishes:
                if name not in observed_names:
                    node_logger.warning(f"Declared artifact '{name}' (in publishes) was not published in this run.")

        primary_result = read_kernel_outputs(output_dir=output_dir, output_names=output_names, result=result, node=node)

        if primary_result is not None:
            return primary_result
        if not flow_data_engine:
            node_logger.warning(
                "Script published no outputs — call flowfile_ctx.publish_output(df, name=...) "
                "to return data; resulting in an empty table."
            )
        return flow_data_engine[0] if flow_data_engine else FlowDataEngine(pl.LazyFrame())

    def _make_kernel_user_defined_func(
        self,
        *,
        custom_node: CustomNodeBase,
        user_defined_node_settings: input_schema.UserDefinedNode,
        kernel_id: str,
        output_names: list[str],
        registry_entry=None,
    ) -> Callable:
        """Create the execution function for a kernel-executed custom node.

        Registry-backed nodes get an AST-generated script (JSON-baked settings,
        no return-rewriting), generated eagerly so KernelCodegenError surfaces
        before the flow runs. Inline test classes with no source file fall back
        to the deprecated ``generate_kernel_code`` so existing behavior survives.
        """
        if registry_entry is not None and registry_entry.source_text and registry_entry.class_name:
            code = generate_kernel_script(
                node_source=registry_entry.source_text,
                class_name=registry_entry.class_name,
                settings_values=custom_node._extract_settings_values(),
                output_names=output_names,
                number_of_inputs=custom_node.number_of_inputs,
            )
        else:
            code = custom_node.generate_kernel_code()

        declared_publishes: list[str] | None = None
        if registry_entry is not None and registry_entry.manifest is not None:
            declared_publishes = [d.name for d in registry_entry.manifest.publishes]

        required_dependencies = list(custom_node.dependencies or [])
        node_type = custom_node.item

        def _func(*flow_data_engine: FlowDataEngine) -> FlowDataEngine | None:
            return self._execute_on_kernel(
                node_id=user_defined_node_settings.node_id,
                kernel_id=kernel_id,
                code=code,
                output_names=output_names,
                flow_data_engine=flow_data_engine,
                declared_publishes=declared_publishes,
                required_dependencies=required_dependencies,
                node_type=node_type,
            )

        return _func

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_pivot(self, pivot_settings: input_schema.NodePivot):
        """Adds a pivot node to the graph.

        Args:
            pivot_settings: The settings for the pivot operation.
        """

        def _func(fl: FlowDataEngine):
            return fl.do_pivot(pivot_settings.pivot_input, self.flow_logger.get_node_logger(pivot_settings.node_id))

        self.add_node_step(
            node_id=pivot_settings.node_id,
            function=_func,
            node_type="pivot",
            setting_input=pivot_settings,
            input_node_ids=[pivot_settings.depending_on_id],
        )

        node = self.get_node(pivot_settings.node_id)
        node._prediction_requires_data = True

        def schema_callback():
            node._schema_prediction_blocked = None
            reason = kernel_block_reason(node, include_self=False)
            if reason:
                # Pivot columns need real data; never run a kernel implicitly for it.
                node._schema_prediction_blocked = reason
                node.results.warnings = reason
                return []
            input_data = node.singular_main_input.get_resulting_data()
            # Runs on a background thread: never mutate the shared memoized
            # engine (input_data.lazy = ...); build a local lazy frame instead.
            input_lf = input_data.data_frame.lazy()
            return pre_calculate_pivot_schema(input_data.schema, pivot_settings.pivot_input, input_lf=input_lf)

        node.schema_callback = schema_callback

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_unpivot(self, unpivot_settings: input_schema.NodeUnpivot):
        """Adds an unpivot node to the graph.

        Args:
            unpivot_settings: The settings for the unpivot operation.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.unpivot(unpivot_settings.unpivot_input)

        self.add_node_step(
            node_id=unpivot_settings.node_id,
            function=_func,
            node_type="unpivot",
            setting_input=unpivot_settings,
            input_node_ids=[unpivot_settings.depending_on_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_union(self, union_settings: input_schema.NodeUnion):
        """Adds a union node to combine multiple data streams.

        Args:
            union_settings: The settings for the union operation.
        """

        def _func(*flowfile_tables: FlowDataEngine):
            dfs: list[pl.LazyFrame] | list[pl.DataFrame] = [flt.data_frame for flt in flowfile_tables]
            return FlowDataEngine(pl.concat(dfs, how="diagonal_relaxed"))

        self.add_node_step(
            node_id=union_settings.node_id,
            function=_func,
            node_type="union",
            setting_input=union_settings,
            input_node_ids=union_settings.depending_on_ids,
        )

    def add_initial_node_analysis(self, node_promise: input_schema.NodePromise, track_history: bool = True):
        """Adds a data exploration/analysis node based on a node promise.

        Automatically captures history for undo/redo support.

        Args:
            node_promise: The promise representing the node to be analyzed.
            track_history: Whether to track this change in history (default True).
        """

        def _do_add():
            node_analysis = create_graphic_walker_node_from_node_promise(node_promise)
            self.add_explore_data(node_analysis)

        if track_history:
            self._execute_with_history(
                _do_add,
                HistoryActionType.ADD_NODE,
                f"Add {node_promise.node_type} node",
                node_id=node_promise.node_id,
            )
        else:
            _do_add()

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_explore_data(self, node_analysis: input_schema.NodeExploreData):
        """Adds a specialized node for data exploration and visualization.

        Args:
            node_analysis: The settings for the data exploration node.
        """

        def analysis_preparation(flowfile_table: FlowDataEngine) -> FlowDataEngine:
            """Pass-through: Graphic Walker aggregates on the worker, not in the browser.

            Charts read the node's result plan through ``/analysis_data/compute``,
            so the run itself owes the explorer nothing.
            """
            return flowfile_table

        def schema_callback():
            node = self.get_node(node_analysis.node_id)
            if len(node.all_inputs) == 1:
                input_node = node.all_inputs[0]
                return input_node.schema
            else:
                return [FlowfileColumn.from_input("col_1", "na")]

        self.add_node_step(
            node_id=node_analysis.node_id,
            node_type="explore_data",
            function=analysis_preparation,
            setting_input=node_analysis,
            schema_callback=schema_callback,
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_group_by(self, group_by_settings: input_schema.NodeGroupBy):
        """Adds a group-by aggregation node to the graph.

        Args:
            group_by_settings: The settings for the group-by operation.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.do_group_by(group_by_settings.groupby_input, False)

        self.add_node_step(
            node_id=group_by_settings.node_id,
            function=_func,
            node_type="group_by",
            setting_input=group_by_settings,
            input_node_ids=[group_by_settings.depending_on_id],
        )

        node = self.get_node(group_by_settings.node_id)

        def schema_callback():
            output_columns = [(c.old_name, c.new_name, c.output_type) for c in group_by_settings.groupby_input.agg_cols]
            depends_on = node.node_inputs.main_inputs[0]
            input_schema_dict: dict[str, str] = {s.name: s.data_type for s in depends_on.schema}
            output_schema = []
            for old_name, new_name, data_type in output_columns:
                data_type = input_schema_dict[old_name] if data_type is None else data_type
                output_schema.append(FlowfileColumn.from_input(data_type=data_type, column_name=new_name))
            return output_schema

        node.schema_callback = schema_callback

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_filter(self, filter_settings: input_schema.NodeFilter):
        """Adds a filter node to the graph.

        Args:
            filter_settings: The settings for the filter operation.
        """

        def _func(fl: FlowDataEngine):
            is_advanced = filter_settings.filter_input.is_advanced()

            if is_advanced:
                expression = filter_settings.filter_input.advanced_filter
            else:
                basic_filter = filter_settings.filter_input.basic_filter
                if basic_filter is None:
                    logger.warning("Basic filter is None, returning unfiltered data")
                    return fl

                try:
                    field_data_type = fl.get_schema_column(basic_filter.field).generic_datatype()
                except Exception:
                    field_data_type = None

                expression = build_filter_expression(basic_filter, field_data_type)
                filter_settings.filter_input.advanced_filter = expression

            if filter_settings.split_mode:
                return fl.filter_split(expression)
            return fl.do_filter(expression)

        self.add_node_step(
            filter_settings.node_id,
            _func,
            node_type="filter",
            renew_schema=False,
            setting_input=filter_settings,
            input_node_ids=[filter_settings.depending_on_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_record_count(self, node_number_of_records: input_schema.NodeRecordCount):
        """Adds a filter node to the graph.

        Args:
            node_number_of_records: The settings for the record count operation.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.get_record_count()

        self.add_node_step(
            node_id=node_number_of_records.node_id,
            function=_func,
            node_type="record_count",
            setting_input=node_number_of_records,
            input_node_ids=[node_number_of_records.depending_on_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_polars_code(self, node_polars_code: input_schema.NodePolarsCode):
        """Adds a node that executes custom Polars code.

        Args:
            node_polars_code: The settings for the Polars code node.
        """

        def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
            return execute_polars_code(*flowfile_tables, code=node_polars_code.polars_code_input.polars_code)

        self.add_node_step(
            node_id=node_polars_code.node_id,
            function=_func,
            node_type="polars_code",
            setting_input=node_polars_code,
            input_node_ids=node_polars_code.depending_on_ids,
        )

        try:
            polars_code_parser.validate_code(node_polars_code.polars_code_input.polars_code)
        except Exception as e:
            node = self.get_node(node_id=node_polars_code.node_id)
            node.results.errors = str(e)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_sql_query(self, node_sql_query: input_schema.NodeSqlQuery):
        """Adds a node that executes a SQL query against connected data sources.

        Args:
            node_sql_query: The settings for the SQL query node.
        """

        def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
            return execute_sql_query(*flowfile_tables, sql_code=node_sql_query.sql_query_input.sql_code)

        self.add_node_step(
            node_id=node_sql_query.node_id,
            function=_func,
            node_type="sql_query",
            setting_input=node_sql_query,
            input_node_ids=node_sql_query.depending_on_ids,
        )

        node = self.get_node(node_id=node_sql_query.node_id)

        def schema_callback() -> list[FlowfileColumn]:
            # Resolve the output schema by running the query plan lazily over
            # 0-row upstream frames (input_1..N); no data is collected.
            inputs = [
                v.get_predicted_resulting_data(src_handle) if v is not None else FlowDataEngine()
                for v, src_handle in node._slot_input_pairs()
            ]
            return execute_sql_query(*inputs, sql_code=node_sql_query.sql_query_input.sql_code).schema

        node.schema_callback = schema_callback

        try:
            validate_sql_query(node_sql_query.sql_query_input.sql_code)
        except Exception as e:
            node.results.errors = str(e)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_python_script(self, node_python_script: input_schema.NodePythonScript):
        """Adds a node that executes Python code on a kernel container."""

        def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
            kernel_id = node_python_script.python_script_input.kernel_id
            if not kernel_id:
                raise ValueError("No kernel selected for python_script node")
            result = self._execute_on_kernel(
                node_id=node_python_script.node_id,
                kernel_id=kernel_id,
                code=node_python_script.python_script_input.code,
                output_names=node_python_script.output_names,
                flow_data_engine=flowfile_tables,
            )
            return result or (flowfile_tables[0] if flowfile_tables else FlowDataEngine(pl.LazyFrame()))

        def schema_callback():
            """Best-effort schema prediction for python_script nodes.

            Returns the input node(s) schema as a reasonable default
            (most python_script nodes transform and pass through).
            If nothing is available, returns [] — never raises.
            """
            try:
                node = self.get_node(node_python_script.node_id)
                if node is None:
                    return []

                main_inputs = node.node_inputs.main_inputs
                if main_inputs:
                    first_input = main_inputs[0]
                    input_node_schema = first_input.schema
                    if input_node_schema:
                        return input_node_schema
                return []
            except Exception:
                return []

        self.add_node_step(
            node_id=node_python_script.node_id,
            function=_func,
            node_type="python_script",
            setting_input=node_python_script,
            input_node_ids=node_python_script.depending_on_ids,
            schema_callback=schema_callback,
        )

        node = self.get_node(node_python_script.node_id)
        if node is not None:
            node._executes_on_kernel = bool(node_python_script.python_script_input.kernel_id)
        output_names = node_python_script.output_names
        if len(output_names) > 1:
            if node is not None:
                node.node_template = node.node_template.model_copy(update={"output": len(output_names)})

    def add_dependency_on_polars_lazy_frame(self, lazy_frame: pl.LazyFrame, node_id: int):
        """Adds a special node that directly injects a Polars LazyFrame into the graph.

        Note: This is intended for backend use and will not work in the UI editor.

        Args:
            lazy_frame: The Polars LazyFrame to inject.
            node_id: The ID for the new node.
        """

        def _func():
            return FlowDataEngine(lazy_frame)

        node_promise = input_schema.NodePromise(
            flow_id=self.flow_id, node_id=node_id, node_type="polars_lazy_frame", is_setup=True
        )
        self.add_node_step(
            node_id=node_promise.node_id, node_type=node_promise.node_type, function=_func, setting_input=node_promise
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_unique(self, unique_settings: input_schema.NodeUnique):
        """Adds a node to find and remove duplicate rows.

        Args:
            unique_settings: The settings for the unique operation.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.make_unique(unique_settings.unique_input)

        self.add_node_step(
            node_id=unique_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="unique",
            setting_input=unique_settings,
            input_node_ids=[unique_settings.depending_on_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_graph_solver(self, graph_solver_settings: input_schema.NodeGraphSolver):
        """Adds a node that solves graph-like problems within the data.

        This node can be used for operations like finding network paths,
        calculating connected components, or performing other graph algorithms
        on relational data that represents nodes and edges.

        Args:
            graph_solver_settings: The settings object defining the graph inputs
                and the specific algorithm to apply.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.solve_graph(graph_solver_settings.graph_solver_input)

        self.add_node_step(
            node_id=graph_solver_settings.node_id,
            function=_func,
            node_type="graph_solver",
            setting_input=graph_solver_settings,
            input_node_ids=[graph_solver_settings.depending_on_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_formula(self, function_settings: input_schema.NodeFormula):
        """Adds a node that applies a formula to create or modify a column.

        Args:
            function_settings: The settings for the formula operation.
        """

        error = ""
        if function_settings.function.field.data_type not in (None, transform_schema.AUTO_DATA_TYPE):
            output_type = cast_str_to_polars_type(function_settings.function.field.data_type)
        else:
            output_type = None
        if output_type not in (None, transform_schema.AUTO_DATA_TYPE):
            new_col = [
                FlowfileColumn.from_input(column_name=function_settings.function.field.name, data_type=str(output_type))
            ]
        else:
            new_col = [FlowfileColumn.from_input(function_settings.function.field.name, "String")]

        def _func(fl: FlowDataEngine):
            return fl.apply_sql_formula(
                func=function_settings.function.function,
                col_name=function_settings.function.field.name,
                output_data_type=output_type,
            )

        self.add_node_step(
            function_settings.node_id,
            _func,
            output_schema=new_col,
            node_type="formula",
            renew_schema=False,
            setting_input=function_settings,
            input_node_ids=[function_settings.depending_on_id],
        )
        if error != "":
            node = self.get_node(function_settings.node_id)
            node.results.errors = error
            return False, error
        else:
            return True, ""

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_cross_join(self, cross_join_settings: input_schema.NodeCrossJoin) -> "FlowGraph":
        """Adds a cross join node to the graph.

        Args:
            cross_join_settings: The settings for the cross join operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
            for left_select in cross_join_settings.cross_join_input.left_select.renames:
                left_select.is_available = True if left_select.old_name in main.schema else False
            for right_select in cross_join_settings.cross_join_input.right_select.renames:
                right_select.is_available = True if right_select.old_name in right.schema else False
            return main.do_cross_join(
                cross_join_input=cross_join_settings.cross_join_input,
                auto_generate_selection=cross_join_settings.auto_generate_selection,
                verify_integrity=False,
                other=right,
            )

        def schema_callback():
            cj_copy = CrossJoinInputManager(cross_join_settings.cross_join_input)
            node = self.get_node(node_id=cross_join_settings.node_id)
            return calculate_cross_join_schema(
                cj_copy,
                left_schema=node.node_inputs.main_inputs[0].schema,
                right_schema=node.node_inputs.right_input.schema,
            )

        self.add_node_step(
            node_id=cross_join_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="cross_join",
            setting_input=cross_join_settings,
            input_node_ids=cross_join_settings.depending_on_ids,
            schema_callback=schema_callback,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_join(self, join_settings: input_schema.NodeJoin) -> "FlowGraph":
        """Adds a join node to combine two data streams based on key columns.

        Args:
            join_settings: The settings for the join operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
            join_input = deepcopy(join_settings.join_input)
            for left_select in join_input.left_select.renames:
                left_select.is_available = True if left_select.old_name in main.schema else False
            for right_select in join_input.right_select.renames:
                right_select.is_available = True if right_select.old_name in right.schema else False
            return main.join(
                join_input=join_input,
                auto_generate_selection=join_settings.auto_generate_selection,
                verify_integrity=False,
                other=right,
            )

        def schema_callback():
            j_copy = JoinInputManager(join_settings.join_input)
            node = self.get_node(node_id=join_settings.node_id)
            return calculate_join_schema(
                j_copy,
                left_schema=node.node_inputs.main_inputs[0].schema,
                right_schema=node.node_inputs.right_input.schema,
                auto_generate_selection=join_settings.auto_generate_selection,
            )

        self.add_node_step(
            node_id=join_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="join",
            setting_input=join_settings,
            input_node_ids=join_settings.depending_on_ids,
            schema_callback=schema_callback,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_train_model(self, train_settings: input_schema.NodeTrainModel) -> "FlowGraph":
        """Adds a Train Model node.

        Fits a regression model on the worker, stores the serialised artifact
        in the global catalog (via :class:`ArtifactService`), and passes the
        input data through unchanged so downstream nodes can keep transforming.

        Args:
            train_settings: Settings (model name, target/features, model_type, params).

        Returns:
            The :class:`FlowGraph` instance for chaining.
        """

        def _func(data: FlowDataEngine) -> FlowDataEngine:
            # Imports are deferred to runtime: importing flowfile_core.artifacts
            # at module top would trigger Alembic migrations and SQLAlchemy
            # engine setup (~3.5s), slowing every flow_graph import — including
            # CLI startup. Keep these inside _func.
            import shutil

            from flowfile_core.artifacts import get_storage_backend
            from flowfile_core.artifacts.service import ArtifactService
            from flowfile_core.auth.utils import get_local_user_id
            from flowfile_core.flowfile.catalog_helpers import (
                auto_register_flow,
                resolve_source_registration_id,
            )
            from flowfile_core.schemas.artifact_schema import PrepareUploadRequest
            from shared.ml.trainers import get_trainer

            settings = train_settings.train_input
            if not settings.target_column:
                raise ValueError("Train Model requires a 'target_column'.")
            if not settings.feature_columns:
                raise ValueError("Train Model requires at least one 'feature_columns' entry.")
            if settings.publish_to_catalog and not settings.model_name:
                raise ValueError("Train Model: 'model_name' is required when 'publish_to_catalog' is enabled.")

            # Validate model_type and hyperparameters early so the user gets
            # a clear error from core, not a worker-side stack trace.
            trainer = get_trainer(settings.model_type)
            try:
                trainer.params_class(**settings.params)
            except Exception as e:
                raise ValueError(f"Train Model: invalid params for model_type={settings.model_type!r}: {e}") from e

            # Always write the model to a flow-scoped path keyed off this node's id;
            # downstream Apply Model nodes in this flow read from there, no catalog
            # required. The same path is used as the staging path when publishing
            # so we only fit once.
            flow_path = ml_flow_model_path(self.flow_id, train_settings.node_id)
            flow_path.parent.mkdir(parents=True, exist_ok=True)

            prepared = None
            owner_id = train_settings.user_id or get_local_user_id() or 1
            staging_path = flow_path
            storage_backend = get_storage_backend()

            if settings.publish_to_catalog:
                # If the flow has a path on disk but no registration yet,
                # auto-register it (idempotently — same mechanism the open/save
                # routes use). This routes scratch flows under "General >
                # Unnamed Flows" / "Local Flows" so artifacts have a stable
                # lineage without forcing the user to explicitly register first.
                registration_id = self._flow_settings.source_registration_id
                if registration_id is None and self._flow_settings.path:
                    auto_register_flow(
                        self._flow_settings.path,
                        self._flow_settings.name or "",
                        owner_id,
                    )
                    resolve_source_registration_id(self)
                    registration_id = self._flow_settings.source_registration_id
                if registration_id is None:
                    raise ValueError(
                        "Publishing to catalog requires the flow to be registered. "
                        "Save the flow first, or disable 'Publish to catalog'."
                    )

                tags = list({"ml", trainer.task_type, settings.model_type, *settings.catalog_tags})
                with get_db_context() as _ns_db:
                    effective_namespace_id = _effective_namespace_id(
                        CatalogService(SQLAlchemyCatalogRepository(_ns_db)), settings
                    )
                    # A published model is a new catalog artifact — gate the target
                    # namespace on the executing principal, mirroring the catalog writer.
                    _authorize_catalog_write(
                        _ns_db, train_settings.user_id, existing=None, namespace_id=effective_namespace_id
                    )
                prepare_request = PrepareUploadRequest(
                    name=settings.model_name,
                    source_registration_id=registration_id,
                    namespace_id=effective_namespace_id,
                    serialization_format=trainer.serialization_format,
                    description=settings.catalog_description
                    or f"Trained via Flowfile node {train_settings.node_id} ({settings.model_type})",
                    tags=tags,
                    source_flow_id=self.flow_id,
                    source_node_id=train_settings.node_id,
                    python_type=f"flowfile.ml.{settings.model_type}",
                    python_module="flowfile.ml",
                )
                with get_db_context() as db:
                    prepared = ArtifactService(db, storage_backend).prepare_upload(prepare_request, owner_id=owner_id)
                if prepared.method != "file":
                    # v1 only supports the shared-filesystem backend; S3 needs a
                    # presigned-URL path on the worker which we haven't wired yet.
                    with get_db_context() as db:
                        ArtifactService(db, storage_backend).delete_artifact(prepared.artifact_id)
                    raise ValueError(
                        "Train Model currently requires the filesystem artifact backend "
                        "(FLOWFILE_ARTIFACT_STORAGE=filesystem). S3 support is not implemented."
                    )
                # Train into the catalog staging path; we'll copy to the flow
                # path after success so finalize_upload (which moves the
                # staging file to the permanent location) still works.
                staging_path = Path(prepared.path)

            node = self.get_node(node_id=train_settings.node_id)
            flow_path_written = False
            try:
                fetcher = MLTrainFetcher(
                    lf=data.data_frame,
                    staging_path=str(staging_path),
                    model_type=settings.model_type,
                    target_column=settings.target_column,
                    feature_columns=settings.feature_columns,
                    params=settings.params,
                    flow_id=self.flow_id,
                    node_id=train_settings.node_id,
                    file_ref=node.hash,
                    wait_on_completion=False,
                )
                node._fetch_cached_df = fetcher
                result = fetcher.get_result()
                if not isinstance(result, dict) or "sha256" not in result or "size_bytes" not in result:
                    raise RuntimeError(f"Worker did not return expected sha256/size_bytes payload, got: {result!r}")

                if prepared is not None:
                    # The staging file is also our flow-scoped copy. Atomically
                    # replace flow_path (write to .tmp, then os.replace) so a
                    # concurrent Apply Model reader can't see a half-written
                    # file. Done before finalize_upload (which moves the
                    # staging file away).
                    flow_tmp = flow_path.with_suffix(flow_path.suffix + ".tmp")
                    shutil.copyfile(staging_path, flow_tmp)
                    os.replace(flow_tmp, flow_path)
                    flow_path_written = True
                    with get_db_context() as db:
                        ArtifactService(db, storage_backend).finalize_upload(
                            artifact_id=prepared.artifact_id,
                            storage_key=prepared.storage_key,
                            sha256=result["sha256"],
                            size_bytes=result["size_bytes"],
                        )
            except Exception:
                if prepared is not None:
                    # Roll back the pending row on any failure so the user
                    # doesn't see ghost artifacts; subsequent re-runs auto-clean
                    # pending rows too.
                    with get_db_context() as db:
                        try:
                            ArtifactService(db, storage_backend).delete_artifact(prepared.artifact_id)
                        except Exception:
                            logger.exception("Failed to roll back pending artifact %s", prepared.artifact_id)
                    # Also roll back the flow_path copy if we wrote it; otherwise
                    # the next Apply Model run could quietly use the artifact
                    # whose catalog row we just deleted.
                    if flow_path_written:
                        try:
                            flow_path.unlink(missing_ok=True)
                        except Exception:
                            logger.exception("Failed to roll back flow_path copy %s", flow_path)
                raise

            if prepared is not None:
                self.flow_logger.info(
                    f"Train Model: stored '{settings.model_name}' v{prepared.version} "
                    f"(artifact_id={prepared.artifact_id}, size={result['size_bytes']}B); "
                    f"flow copy at {flow_path}"
                )
                artifact_name = f"{settings.model_name} v{prepared.version}"
            else:
                self.flow_logger.info(f"Train Model: wrote {result['size_bytes']}B to flow path {flow_path}")
                artifact_name = f"{settings.model_type} (flow only)"

            # Surface the trained model in the node's Artifacts tab + canvas badge.
            # Re-runs replace any prior entry rather than accumulating duplicates.
            self.artifact_context.clear_nodes({train_settings.node_id})
            self.artifact_context.record_published(
                node_id=train_settings.node_id,
                kernel_id="",
                artifacts=[
                    {
                        "name": artifact_name,
                        "type_name": f"flowfile.ml.{settings.model_type}",
                        "module": "flowfile.ml",
                        "size_bytes": result["size_bytes"],
                    }
                ],
            )
            return data

        def schema_callback():
            input_node: FlowNode = self.get_node(train_settings.node_id).node_inputs.main_inputs[0]
            return input_node.schema

        depending_on_id = train_settings.depending_on_id if hasattr(train_settings, "depending_on_id") else None
        self.add_node_step(
            node_id=train_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="train_model",
            setting_input=train_settings,
            schema_callback=schema_callback,
            input_node_ids=[depending_on_id] if depending_on_id is not None else None,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_apply_model(self, apply_settings: input_schema.NodeApplyModel) -> "FlowGraph":
        """Adds an Apply Model node.

        Fetches the artifact from the catalog and asks the worker to score the
        input data, returning a LazyFrame with one extra ``Float64`` column.

        Args:
            apply_settings: Settings (model_name, optional version, output_column).

        Returns:
            The :class:`FlowGraph` instance for chaining.
        """

        def _func(data: FlowDataEngine) -> FlowDataEngine:
            from flowfile_core.artifacts import get_storage_backend
            from flowfile_core.artifacts.service import ArtifactService

            settings = apply_settings.apply_input
            if not settings.output_column:
                raise ValueError("Apply Model requires an 'output_column'.")

            model_path: str
            origin_label: str

            if settings.source == "upstream":
                if settings.upstream_node_id is None:
                    raise ValueError(
                        "Apply Model: 'upstream_node_id' is required when source='upstream'. "
                        "Pick a Train Model node in the drawer or switch to 'catalog' source."
                    )
                upstream = self.get_node(node_id=settings.upstream_node_id)
                if upstream is None or upstream.node_type != "train_model":
                    raise ValueError(
                        f"Apply Model: upstream node {settings.upstream_node_id} is not a Train Model node."
                    )
                flow_path = ml_flow_model_path(self.flow_id, settings.upstream_node_id)
                if not flow_path.exists():
                    raise ValueError(
                        f"Apply Model: upstream Train Model (node {settings.upstream_node_id}) "
                        "has not produced a model yet. Make sure it runs before this node "
                        "(e.g. with a Wait For barrier)."
                    )
                model_path = str(flow_path)
                origin_label = f"upstream node {settings.upstream_node_id}"
            else:
                if not settings.model_name:
                    raise ValueError("Apply Model: 'model_name' is required when source='catalog'.")
                storage_backend = get_storage_backend()
                with get_db_context() as db:
                    effective_namespace_id = _effective_namespace_id(
                        CatalogService(SQLAlchemyCatalogRepository(db)), settings
                    )
                    artifact = ArtifactService(db, storage_backend).get_artifact_by_name(
                        name=settings.model_name,
                        namespace_id=effective_namespace_id,
                        version=settings.model_version,
                    )
                if artifact.download_source is None or artifact.download_source.method != "file":
                    raise ValueError(
                        "Apply Model currently requires the filesystem artifact backend "
                        "(FLOWFILE_ARTIFACT_STORAGE=filesystem). S3 support is not implemented."
                    )
                model_path = artifact.download_source.path
                if not os.path.exists(model_path):
                    raise ValueError(
                        f"Apply Model: data for catalog model '{settings.model_name}' "
                        f"v{artifact.version} (namespace {artifact.namespace_id}) is missing "
                        f"at {model_path}. If running in Docker, ensure the shared artifacts "
                        "volume is mounted into both core and the worker."
                    )
                origin_label = f"catalog '{settings.model_name}' v{artifact.version}"

            node = self.get_node(node_id=apply_settings.node_id)
            fetcher = MLApplyFetcher(
                lf=data.data_frame,
                model_path=model_path,
                output_column=settings.output_column,
                flow_id=self.flow_id,
                node_id=apply_settings.node_id,
                file_ref=node.hash,
                wait_on_completion=False,
            )
            node._fetch_cached_df = fetcher
            result_lf = fetcher.get_result()
            self.flow_logger.info(f"Apply Model: scored using {origin_label} -> column '{settings.output_column}'")
            return FlowDataEngine(result_lf)

        def schema_callback():
            input_node: FlowNode = self.get_node(apply_settings.node_id).node_inputs.main_inputs[0]
            input_schema_cols = list(input_node.schema)
            s = apply_settings.apply_input
            output_column = s.output_column or "prediction"
            # source='upstream' lets us read the trainer's declared output_dtype
            # so a future non-Float64 trainer (e.g. classification) gets the
            # right schema. source='catalog' falls back to Float64 — resolving
            # via the catalog DB at schema-resolve time would be too eager.
            output_dtype = "Float64"
            if s.source == "upstream" and s.upstream_node_id is not None:
                upstream = self.get_node(s.upstream_node_id)
                train_input = getattr(getattr(upstream, "setting_input", None), "train_input", None)
                model_type = getattr(train_input, "model_type", None)
                if model_type:
                    try:
                        from shared.ml.trainers import get_trainer

                        output_dtype = get_trainer(model_type).output_dtype
                    except ValueError:
                        pass
            return input_schema_cols + [FlowfileColumn.from_input(output_column, output_dtype)]

        depending_on_id = apply_settings.depending_on_id if hasattr(apply_settings, "depending_on_id") else None
        self.add_node_step(
            node_id=apply_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="apply_model",
            setting_input=apply_settings,
            schema_callback=schema_callback,
            input_node_ids=[depending_on_id] if depending_on_id is not None else None,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_evaluate_model(self, evaluate_settings: input_schema.NodeEvaluateModel) -> "FlowGraph":
        """Adds an Evaluate Model node.

        Compares the *actual* and *predicted* columns already present on the
        input dataframe and emits a long-form ``(metric, value)`` frame.
        Pure polars — no worker offload, no model file read.

        ``task_type="auto"`` resolves the metric set from the configured
        upstream Train Model node's trainer; otherwise uses the explicit
        ``regression`` / ``classification`` choice from settings.
        """

        def _resolve_task_type() -> str:
            s = evaluate_settings.evaluate_input
            if s.task_type != "auto":
                return s.task_type
            if s.upstream_train_node_id is not None:
                upstream = self.get_node(s.upstream_train_node_id)
                train_input = getattr(getattr(upstream, "setting_input", None), "train_input", None)
                model_type = getattr(train_input, "model_type", None)
                if model_type:
                    try:
                        from shared.ml.trainers import get_trainer

                        return get_trainer(model_type).task_type
                    except ValueError:
                        pass
            return "regression"

        def _func(data: FlowDataEngine) -> FlowDataEngine:
            from shared.ml.metrics import compute_metrics

            settings = evaluate_settings.evaluate_input
            if not settings.actual_column:
                raise ValueError("Evaluate Model requires an 'actual_column'.")
            if not settings.predicted_column:
                raise ValueError("Evaluate Model requires a 'predicted_column'.")

            task_type = _resolve_task_type()
            metrics_lf = compute_metrics(
                data.data_frame,
                actual_column=settings.actual_column,
                predicted_column=settings.predicted_column,
                task_type=task_type,
            )
            self.flow_logger.info(
                f"Evaluate Model: {settings.predicted_column} vs {settings.actual_column} " f"(task_type={task_type})"
            )
            return FlowDataEngine(metrics_lf)

        def schema_callback():
            return [
                FlowfileColumn.from_input(column_name="metric", data_type="String"),
                FlowfileColumn.from_input(column_name="value", data_type="Float64"),
            ]

        depending_on_id = evaluate_settings.depending_on_id if hasattr(evaluate_settings, "depending_on_id") else None
        self.add_node_step(
            node_id=evaluate_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="evaluate_model",
            setting_input=evaluate_settings,
            schema_callback=schema_callback,
            input_node_ids=[depending_on_id] if depending_on_id is not None else None,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_wait_for(self, settings: input_schema.NodeWaitFor) -> "FlowGraph":
        """Adds a Wait For node — passes the left input through and waits on the right.

        Two distinct input handles like Join: connect the data path to the left
        and the dependency node (e.g. Train Model) to the right. The right
        input's data is discarded; only its completion gates this node.
        """

        def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
            # *right* is intentionally unused — its only job is to make sure
            # the framework waits for the dependency node to finish.
            del right
            return main

        def schema_callback():
            node = self.get_node(settings.node_id)
            if node.node_inputs.main_inputs:
                return node.node_inputs.main_inputs[0].schema
            return []

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            input_columns=[],
            node_type="wait_for",
            setting_input=settings,
            schema_callback=schema_callback,
            input_node_ids=settings.depending_on_ids,
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_fuzzy_match(self, fuzzy_settings: input_schema.NodeFuzzyMatch) -> "FlowGraph":
        """Adds a fuzzy matching node to join data on approximate string matches.

        Args:
            fuzzy_settings: The settings for the fuzzy match operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
            node = self.get_node(node_id=fuzzy_settings.node_id)
            if self.execution_location == "local":
                return main.fuzzy_join(
                    fuzzy_match_input=deepcopy(fuzzy_settings.join_input),
                    other=right,
                    node_logger=self.flow_logger.get_node_logger(fuzzy_settings.node_id),
                )

            f = main.start_fuzzy_join(
                fuzzy_match_input=deepcopy(fuzzy_settings.join_input),
                other=right,
                file_ref=node.hash,
                flow_id=self.flow_id,
                node_id=fuzzy_settings.node_id,
            )
            logger.info("Started the fuzzy match action")
            node._fetch_cached_df = f  # Add to the node so it can be cancelled and fetch later if needed
            return FlowDataEngine(f.get_result())

        def schema_callback():
            fm_input_copy = FuzzyMatchInputManager(
                fuzzy_settings.join_input
            )  # Deepcopy create an unique object per func
            node = self.get_node(node_id=fuzzy_settings.node_id)
            return calculate_fuzzy_match_schema(
                fm_input_copy,
                left_schema=node.node_inputs.main_inputs[0].schema,
                right_schema=node.node_inputs.right_input.schema,
            )

        self.add_node_step(
            node_id=fuzzy_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="fuzzy_match",
            setting_input=fuzzy_settings,
            input_node_ids=fuzzy_settings.depending_on_ids,
            schema_callback=schema_callback,
        )

        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_text_to_rows(self, node_text_to_rows: input_schema.NodeTextToRows) -> "FlowGraph":
        """Adds a node that splits cell values into multiple rows.

        This is useful for un-nesting data where a single field contains multiple
        values separated by a delimiter.

        Args:
            node_text_to_rows: The settings object that specifies the column to split
                and the delimiter to use.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            return table.split(node_text_to_rows.text_to_rows_input)

        self.add_node_step(
            node_id=node_text_to_rows.node_id,
            function=_func,
            node_type="text_to_rows",
            setting_input=node_text_to_rows,
            input_node_ids=[node_text_to_rows.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_window_functions(self, settings: input_schema.NodeWindowFunctions) -> "FlowGraph":
        """Adds a window-functions node (rolling, cumulative, rank, tile).

        Args:
            settings: The settings for the window-functions operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(fl: FlowDataEngine) -> FlowDataEngine:
            return fl.do_window_functions(settings.window_input, False)

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            node_type="window_functions",
            setting_input=settings,
            input_node_ids=[settings.depending_on_id],
        )

        node = self.get_node(settings.node_id)

        def schema_callback():
            depends_on = node.node_inputs.main_inputs[0]
            input_schema_list = list(depends_on.schema)
            input_types = {s.name: s.data_type for s in depends_on.schema}
            output_schema = list(input_schema_list)
            for w in settings.window_input.window_functions:
                src_type = input_types.get(w.column) if w.column else None
                out_type = w.output_type or transform_schema.get_window_output_type(w.function, src_type)
                if out_type is None:
                    out_type = src_type or "Float64"
                output_schema.append(FlowfileColumn.from_input(data_type=out_type, column_name=w.new_column_name))
            return output_schema

        node.schema_callback = schema_callback
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_sort(self, sort_settings: input_schema.NodeSort) -> "FlowGraph":
        """Adds a node to sort the data based on one or more columns.

        Args:
            sort_settings: The settings for the sort operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            return table.do_sort(sort_settings.sort_input)

        self.add_node_step(
            node_id=sort_settings.node_id,
            function=_func,
            node_type="sort",
            setting_input=sort_settings,
            input_node_ids=[sort_settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_sample(self, sample_settings: input_schema.NodeSample) -> "FlowGraph":
        """Adds a node to take a random or top-N sample of the data.

        Every method stays lazy, so the node needs no local/remote branch: the
        sample is part of the plan the worker receives, not a materialised frame.

        Args:
            sample_settings: The settings object specifying the sampling method,
                the size or fraction to keep, and an optional seed.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            if sample_settings.sample_method == "random":
                return table.random_sample(n=sample_settings.sample_size, seed=sample_settings.seed)
            if sample_settings.sample_method == "random_fraction":
                return table.random_sample(fraction=sample_settings.fraction / 100.0, seed=sample_settings.seed)
            return table.get_sample(sample_settings.sample_size)

        self.add_node_step(
            node_id=sample_settings.node_id,
            function=_func,
            node_type="sample",
            setting_input=sample_settings,
            input_node_ids=[sample_settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_random_split(self, settings: input_schema.NodeRandomSplit) -> "FlowGraph":
        """Adds a node that randomly partitions rows into N labeled outputs.

        Returns a ``NamedOutputs``; the framework unpacks it into
        ``_named_outputs`` so each split is reachable via its own output handle.

        Args:
            settings: The settings object specifying the splits and optional seed.

        Returns:
            The `FlowGraph` instance for method chaining.
        """
        from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

        def _func(table: FlowDataEngine) -> NamedOutputs:
            split_pairs = [(s.name, s.percentage) for s in settings.splits]
            if self.execution_location == "local":
                return table.random_split(split_pairs, settings.seed)
            return table.random_split_external(
                split_pairs,
                settings.seed,
                flow_id=self.flow_id,
                node_id=settings.node_id,
            )

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            node_type="random_split",
            setting_input=settings,
            input_node_ids=[settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_record_id(self, record_id_settings: input_schema.NodeRecordId) -> "FlowGraph":
        """Adds a node to create a new column with a unique ID for each record.

        Args:
            record_id_settings: The settings object specifying the name of the
                new record ID column.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            return table.add_record_id(record_id_settings.record_id_input)

        self.add_node_step(
            node_id=record_id_settings.node_id,
            function=_func,
            node_type="record_id",
            setting_input=record_id_settings,
            input_node_ids=[record_id_settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_dynamic_rename(self, settings: input_schema.NodeDynamicRename) -> "FlowGraph":
        """Adds a node that renames many columns at once via a single rule.

        Supports prefix, suffix, formula-based, and first-row renaming across all
        columns, a specific list of columns, or every column of a given data type.
        In `first_row` mode the first row is dropped from the output after its
        values are promoted to column headers.

        Args:
            settings: The dynamic rename configuration.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            return table.apply_dynamic_rename(settings.dynamic_rename_input)

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            node_type="dynamic_rename",
            setting_input=settings,
            input_node_ids=[settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_select(self, select_settings: input_schema.NodeSelect) -> "FlowGraph":
        """Adds a node to select, rename, reorder, or drop columns.

        Args:
            select_settings: The settings for the select operation.

        Returns:
            The `FlowGraph` instance for method chaining.
        """

        select_cols = select_settings.select_input
        drop_cols = tuple(s.old_name for s in select_settings.select_input)

        def _func(table: FlowDataEngine) -> FlowDataEngine:
            input_cols = set(f.name for f in table.schema)
            ids_to_remove = []
            for i, select_col in enumerate(select_cols):
                if select_col.old_name not in input_cols:
                    select_col.is_available = False
                    if not select_col.keep:
                        ids_to_remove.append(i)
                    continue
                select_col.is_available = True
                if select_col.data_type is None:
                    select_col.data_type = table.get_schema_column(select_col.old_name).data_type
            ids_to_remove.reverse()
            for i in ids_to_remove:
                select_cols.pop(i)
            return table.do_select(
                select_inputs=transform_schema.SelectInputs(select_cols), keep_missing=select_settings.keep_missing
            )

        self.add_node_step(
            node_id=select_settings.node_id,
            function=_func,
            input_columns=[],
            node_type="select",
            drop_columns=list(drop_cols),
            setting_input=select_settings,
            input_node_ids=[select_settings.depending_on_id],
        )
        return self

    @property
    def graph_has_functions(self) -> bool:
        """Checks if the graph has any nodes."""
        return len(self._node_ids) > 0

    def delete_node(self, node_id: int | str):
        """Deletes a node from the graph and updates all its connections.

        Args:
            node_id: The ID of the node to delete.

        Raises:
            Exception: If the node with the given ID does not exist.
        """
        logger.info(f"Starting deletion of node with ID: {node_id}")

        node = self._node_db.get(node_id)
        if node:
            logger.info(f"Found node: {node_id}, processing deletion")
            group_id = getattr(node.setting_input, "group_id", None)

            lead_to_steps: list[FlowNode] = node.leads_to_nodes
            logger.debug(f"Node {node_id} leads to {len(lead_to_steps)} other nodes")

            if len(lead_to_steps) > 0:
                for lead_to_step in lead_to_steps:
                    logger.debug(f"Deleting input node {node_id} from dependent node {lead_to_step}")
                    lead_to_step.delete_input_node(node_id, complete=True)

            if not node.is_start:
                depends_on: list[FlowNode] = node.node_inputs.get_all_inputs()
                logger.debug(f"Node {node_id} depends on {len(depends_on)} other nodes")

                for depend_on in depends_on:
                    logger.debug(f"Removing lead_to reference {node_id} from node {depend_on}")
                    depend_on.delete_lead_to_node(node_id)

            self._node_db.pop(node_id)
            logger.debug(f"Successfully removed node {node_id} from node_db")
            del node
            logger.info("Node object deleted")
            # Drop a group that just lost its last member (keep it if it still holds sub-groups).
            if (
                group_id is not None
                and group_id in self._groups
                and not self._member_node_ids(group_id)
                and not self._child_group_ids(group_id)
            ):
                self._groups.pop(group_id, None)
        else:
            logger.error(f"Failed to find node with id {node_id}")
            raise Exception(f"Node with id {node_id} does not exist")

    @property
    def graph_has_input_data(self) -> bool:
        """Checks if the graph has an initial input data source."""
        return self._input_data is not None

    def add_node_step(
        self,
        node_id: int | str,
        function: Callable,
        input_columns: list[str] = None,
        output_schema: list[FlowfileColumn] = None,
        node_type: str = None,
        drop_columns: list[str] = None,
        renew_schema: bool = True,
        setting_input: Any = None,
        cache_results: bool = None,
        schema_callback: Callable = None,
        input_node_ids: list[int] = None,
    ) -> FlowNode:
        """The core method for adding or updating a node in the graph.

        Args:
            node_id: The unique ID for the node.
            function: The core processing function for the node.
            input_columns: A list of input column names required by the function.
            output_schema: A predefined schema for the node's output.
            node_type: A string identifying the type of node (e.g., 'filter', 'join').
            drop_columns: A list of columns to be dropped after the function executes.
            renew_schema: If True, the schema is recalculated after execution.
            setting_input: A configuration object containing settings for the node.
            cache_results: If True, the node's results are cached for future runs.
            schema_callback: A function that dynamically calculates the output schema.
            input_node_ids: A list of IDs for the nodes that this node depends on.

        Returns:
            The created or updated FlowNode object.
        """
        output_field_config = getattr(setting_input, "output_field_config", None) if setting_input else None

        logger.info(
            f"add_node_step: node_id={node_id}, node_type={node_type}, "
            f"has_setting_input={setting_input is not None}, "
            f"has_output_field_config={output_field_config is not None}, "
            f"config_enabled={output_field_config.enabled if output_field_config else False}, "
            f"has_schema_callback={schema_callback is not None}"
        )

        # IMPORTANT: Always create wrapped callback if output_field_config exists (even if enabled=False)
        # This ensures nodes like PolarsCode get a schema callback when output_field_config is defined
        if output_field_config:
            if output_field_config.enabled:
                logger.info(
                    f"add_node_step: Creating/wrapping schema_callback for node {node_id} with output_field_config "
                    f"(validation_mode={output_field_config.validation_mode_behavior}, "
                    f"{len(output_field_config.fields)} fields, "
                    f"base_callback={'present' if schema_callback else 'None'})"
                )
            else:
                logger.debug(f"add_node_step: output_field_config present for node {node_id} but disabled")

            schema_callback = create_schema_callback_with_output_config(schema_callback, output_field_config)
            logger.info(
                f"add_node_step: schema_callback {'created' if schema_callback else 'failed'} for node {node_id}"
            )

        existing_node = self.get_node(node_id)
        if existing_node is not None:
            if existing_node.node_type != node_type:
                self.delete_node(existing_node.node_id)
                existing_node = None
        if existing_node:
            input_nodes = existing_node.all_inputs
        elif input_node_ids is not None:
            input_nodes = [self.get_node(node_id) for node_id in input_node_ids]
        else:
            input_nodes = None
        if isinstance(input_columns, str):
            input_columns = [input_columns]
        if (
            input_nodes is not None
            or function.__name__ in ("placeholder", "analysis_preparation")
            or node_type in ("cloud_storage_reader", "catalog_reader", "polars_lazy_frame", "input_data")
        ):
            if not existing_node:
                node = FlowNode(
                    node_id=node_id,
                    function=function,
                    output_schema=output_schema,
                    input_columns=input_columns,
                    drop_columns=drop_columns,
                    renew_schema=renew_schema,
                    setting_input=setting_input,
                    node_type=node_type,
                    name=function.__name__,
                    schema_callback=schema_callback,
                    parent_uuid=self.uuid,
                )
            else:
                existing_node.update_node(
                    function=function,
                    output_schema=output_schema,
                    input_columns=input_columns,
                    drop_columns=drop_columns,
                    setting_input=setting_input,
                    schema_callback=schema_callback,
                )
                node = existing_node
        else:
            raise Exception("No data initialized")
        self._node_db[node_id] = node
        self._node_ids.append(node_id)
        # Give the node a callable that returns the current flow parameters so
        # that lazy schema prediction (_predicted_data_getter) can substitute
        # ${...} refs. Using a callable (rather than a copy of the dict) means
        # the node always reads the LATEST parameters, whether they were set via
        # the flow_settings.setter or mutated directly on flow_settings.parameters.
        _graph = self

        def _get_params() -> dict[str, ParamValue]:
            return {p.name: p.typed_default() for p in (_graph.flow_settings.parameters or [])}

        node._params_getter = _get_params
        return node

    def add_include_cols(self, include_columns: list[str]):
        """Adds columns to both the input and output column lists.

        Args:
            include_columns: A list of column names to include.
        """
        for column in include_columns:
            if column not in self._input_cols:
                self._input_cols.append(column)
            if column not in self._output_cols:
                self._output_cols.append(column)
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_output(self, output_file: input_schema.NodeOutput):
        """Adds an output node to write the final data to a destination.

        Args:
            output_file: The settings for the output file.
        """

        def _func(df: FlowDataEngine):
            if self.execution_location == "local":
                df.output(
                    output_fs=output_file.output_settings,
                    flow_id=self.flow_id,
                    node_id=output_file.node_id,
                    execute_remote=False,
                )
                return df
            output_fs = output_file.output_settings
            node = self.get_node(output_file.node_id)
            writer = ExternalOutputWriter(
                lf=df.data_frame,
                data_type=output_fs.file_type,
                path=output_fs.abs_file_path,
                write_mode=output_fs.write_mode,
                sheet_name=output_fs.sheet_name,
                delimiter=output_fs.delimiter,
                compression=output_fs.compression,
                flow_id=self.flow_id,
                node_id=output_file.node_id,
                wait_on_completion=False,
            )
            node._fetch_cached_df = writer
            writer.get_result()
            return df

        def schema_callback():
            input_node: FlowNode = self.get_node(output_file.node_id).node_inputs.main_inputs[0]

            return input_node.schema

        input_node_id = output_file.depending_on_id if hasattr(output_file, "depending_on_id") else None
        self.add_node_step(
            node_id=output_file.node_id,
            function=_func,
            input_columns=[],
            node_type="output",
            setting_input=output_file,
            schema_callback=schema_callback,
            input_node_ids=[input_node_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_api_response(self, api_response: input_schema.NodeApiResponse):
        """Adds an API-response sink node.

        The node is a pass-through marker: its result equals its input. When the flow
        is published as an HTTP API endpoint, the endpoint reads this node's result
        and serializes it as the response body. Behaves like an output node so its
        result is always materialized locally.

        Args:
            api_response: The settings for the API-response node.
        """

        def _func(df: FlowDataEngine):
            return df

        def schema_callback():
            input_node: FlowNode = self.get_node(api_response.node_id).node_inputs.main_inputs[0]
            return input_node.schema

        input_node_id = api_response.depending_on_id if hasattr(api_response, "depending_on_id") else None
        self.add_node_step(
            node_id=api_response.node_id,
            function=_func,
            input_columns=[],
            node_type="api_response",
            setting_input=api_response,
            schema_callback=schema_callback,
            input_node_ids=[input_node_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_flow_output(self, settings: input_schema.NodeFlowOutput) -> "FlowGraph":
        """Adds a named subflow-output sink (passthrough, always materialized).

        When this flow runs inside another flow via a run_flow node, the parent
        reads this node's result as one of the subflow's outputs.
        """
        for other in self.nodes:
            if (
                other.node_type == "flow_output"
                and other.node_id != settings.node_id
                and isinstance(other.setting_input, input_schema.NodeFlowOutput)
                and other.setting_input.output_name == settings.output_name
            ):
                raise ValueError(f"flow_output name '{settings.output_name}' is already used by node {other.node_id}")

        def _func(df: FlowDataEngine):
            return df

        def schema_callback():
            node: FlowNode = self.get_node(settings.node_id)
            if node.node_inputs.main_inputs:
                return node.node_inputs.main_inputs[0].schema
            return []

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            input_columns=[],
            node_type="flow_output",
            setting_input=settings,
            schema_callback=schema_callback,
            input_node_ids=[settings.depending_on_id],
        )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_run_flow(self, settings: input_schema.NodeRunFlow) -> "FlowGraph":
        """Adds a node that executes a catalog-registered flow as a subflow.

        Inputs are keyed: handle input-0 carries optional parameter data; handles
        input-1..input-N feed the subflow's flow_input nodes (input_slots order).
        Outputs mirror the subflow's flow_output nodes (output_slots order).
        """
        from flowfile_core.flowfile import subflow

        subflow.stamp_flow_reference(settings)
        _graph = self

        def _func(*inputs: FlowDataEngine):
            param_input = inputs[0] if inputs else None
            return subflow.execute_run_flow_node(_graph, settings, param_input, tuple(inputs[1:]))

        def schema_callback():
            node = _graph.get_node(settings.node_id)
            named = subflow.predict_run_flow_named_schemas(settings)
            if node is not None and named:
                node._named_schemas = named
            return named.get(DEFAULT_OUTPUT_HANDLE, [])

        existing = self.get_node(settings.node_id)
        old_slots: list[str] | None = None
        if existing is not None and isinstance(existing.setting_input, input_schema.NodeRunFlow):
            old_slots = list(existing.setting_input.input_slots)

        self.add_node_step(
            node_id=settings.node_id,
            function=_func,
            input_columns=[],
            node_type="run_flow",
            setting_input=settings,
            schema_callback=schema_callback,
            input_node_ids=[],
        )

        if old_slots is not None and old_slots != settings.input_slots:
            node = self.get_node(settings.node_id)
            # Keyed edges follow their slot by NAME; vanished names drop their edge.
            mapping: dict[str, str | None] = {}
            for old_index, slot_name in enumerate(old_slots):
                old_handle = input_handle(old_index + 1)
                if slot_name in settings.input_slots:
                    mapping[old_handle] = input_handle(settings.input_slots.index(slot_name) + 1)
                else:
                    mapping[old_handle] = None
            result = node.remap_dynamic_inputs(mapping)
            if result["dropped"]:
                self.flow_logger.warning(
                    f"run_flow node {settings.node_id}: dropped connection(s) on {', '.join(result['dropped'])} "
                    "after the subflow interface changed"
                )
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_catalog_reader(self, node_catalog_reader: input_schema.NodeCatalogReader):
        """Adds a node that reads a table from the catalog.

        Resolves the catalog table by ID (or name + namespace) and reads
        the materialized Parquet file.  When ``sql_query`` is set, executes
        the SQL against all catalog Delta tables instead.
        """

        if node_catalog_reader.sql_query:
            is_virtual_optimized = self._add_catalog_sql_reader(node_catalog_reader)
        else:
            is_virtual_optimized = self._add_catalog_table_reader(node_catalog_reader)
        node_catalog_reader.is_virtual_optimized = is_virtual_optimized

    def _add_catalog_sql_reader(self, node_catalog_reader: input_schema.NodeCatalogReader) -> bool | None:
        """Execute a SQL query against all catalog tables (physical + virtual).

        Returns:
            Whether all referenced virtual tables are optimized, or None if no virtual tables.
        """

        sql_code = node_catalog_reader.sql_query
        resolved = _resolve_catalog_sql_tables(node_catalog_reader.node_id, node_catalog_reader.user_id)
        table_paths = resolved.table_paths
        virtual_tables = resolved.virtual_tables
        table_namespaces = resolved.table_namespaces

        # Resolve cloud storage options per source namespace at wiring time, memoized by namespace_id
        # (a SQL query can join tables from different catalogs, each with its own storage).
        storage_options_by_name: dict[str, dict | None] = {}
        _opts_by_namespace: dict[int | None, dict | None] = {}
        for _name, _path in table_paths.items():
            if not _is_cloud_uri(_path):
                continue
            _ns = table_namespaces.get(_name)
            if _ns not in _opts_by_namespace:
                _opts_by_namespace[_ns] = resolve_for_namespace(_ns).storage_options or None
            storage_options_by_name[_name] = _opts_by_namespace[_ns]

        def _func() -> FlowDataEngine:
            if not table_paths and not virtual_tables:
                raise ValueError("No catalog tables available to query")
            ctx = pl.SQLContext()
            for name, path in table_paths.items():
                if _is_cloud_uri(path):
                    ctx.register(name, pl.scan_delta(path, storage_options=storage_options_by_name.get(name)))
                else:
                    ctx.register(name, pl.scan_delta(path))
            for name, (is_opt, ser_lf, tid, stv) in virtual_tables.items():
                ctx.register(
                    name,
                    _resolve_virtual_table(
                        is_opt,
                        ser_lf,
                        tid,
                        node_logger=self.flow_logger.get_node_logger(node_catalog_reader.node_id),
                        run_location=self.execution_location,
                        source_table_versions=stv,
                        user_id=node_catalog_reader.user_id,
                    ),
                )
            return FlowDataEngine(ctx.execute(sql_code))

        # todo: There are quite some round-trips happening here because the Flowgraph tries to predict the schema.
        is_virtual_optimized: bool | None = None
        if virtual_tables:
            is_virtual_optimized = all(is_opt for is_opt, _, _, _ in virtual_tables.values())

        self.add_node_step(
            node_id=node_catalog_reader.node_id,
            function=_func,
            input_columns=[],
            node_type="catalog_reader",
            setting_input=node_catalog_reader,
        )
        node = self.get_node(node_catalog_reader.node_id)
        self.add_node_to_starting_list(node)

        try:
            validate_sql_query(sql_code)
        except Exception as e:
            node.results.errors = str(e)

        return is_virtual_optimized

    def _add_catalog_table_reader(self, node_catalog_reader: input_schema.NodeCatalogReader) -> bool | None:
        """Read a single table from the catalog (physical or virtual).

        Returns:
            Whether the virtual table is optimized, or None if not a virtual table.
        """

        info = _resolve_catalog_table_info(node_catalog_reader)

        # Back-fill id from a name-only reference so the settings form and read
        # lineage (both keyed on catalog_table_id) work.
        if node_catalog_reader.catalog_table_id is None and info.table_id is not None:
            node_catalog_reader.catalog_table_id = info.table_id
            if node_catalog_reader.catalog_namespace_id is None:
                node_catalog_reader.catalog_namespace_id = info.namespace_id
            if node_catalog_reader.catalog_table_name is None:
                node_catalog_reader.catalog_table_name = info.table_name

        is_virtual_optimized: bool | None = info.is_optimized if info.table_type == "virtual" else None

        resolved_path = info.file_path
        delta_version = node_catalog_reader.delta_version
        _table_type = info.table_type
        _serialized_lf = info.serialized_lf
        _is_optimized = info.is_optimized
        _catalog_table_id = node_catalog_reader.catalog_table_id
        _source_table_versions = info.source_table_versions
        _authorized = info.authorized
        _user_id = node_catalog_reader.user_id

        # Resolve cloud storage options once at wiring time; local tables don't touch the DB.
        _reader_storage_options = None
        if resolved_path and _is_cloud_uri(resolved_path):
            _reader_namespace_id = info.namespace_id or node_catalog_reader.catalog_namespace_id
            _reader_storage_options = resolve_for_namespace(_reader_namespace_id).storage_options or None

        _scd2_filter = _scd2_row_filter(
            info.scd2_config,
            node_catalog_reader.scd2_view,
            node_catalog_reader.scd2_as_of,
            node_logger=self.flow_logger.get_node_logger(node_catalog_reader.node_id),
        )

        def _apply_scd2_filter(lf: pl.LazyFrame) -> FlowDataEngine:
            return FlowDataEngine(lf if _scd2_filter is None else lf.filter(_scd2_filter))

        def _func() -> FlowDataEngine:
            if not _authorized:
                raise PermissionError(
                    f"Not authorized to read the catalog table for node {node_catalog_reader.node_id}"
                )
            if _table_type == "virtual":
                return FlowDataEngine(
                    _resolve_virtual_table(
                        _is_optimized,
                        _serialized_lf,
                        _catalog_table_id,
                        node_logger=self.flow_logger.get_node_logger(node_catalog_reader.node_id),
                        run_location=self.execution_location,
                        source_table_versions=_source_table_versions,
                        user_id=_user_id,
                    )
                )

            if not resolved_path:
                raise ValueError("Catalog table could not be resolved — no file path found")
            scan_kwargs = {}
            if delta_version is not None:
                scan_kwargs["version"] = delta_version
            if _is_cloud_uri(resolved_path):
                # Cloud catalog table: scan directly (stays lazy ⇒ no collect in core).
                return _apply_scd2_filter(
                    pl.scan_delta(resolved_path, storage_options=_reader_storage_options, **scan_kwargs)
                )
            if is_delta_table(resolved_path):
                return _apply_scd2_filter(pl.scan_delta(resolved_path, **scan_kwargs))
            return _apply_scd2_filter(pl.scan_parquet(resolved_path))

        self.add_node_step(
            node_id=node_catalog_reader.node_id,
            function=_func,
            input_columns=[],
            node_type="catalog_reader",
            setting_input=node_catalog_reader,
        )
        node = self.get_node(node_catalog_reader.node_id)
        self.add_node_to_starting_list(node)
        return is_virtual_optimized

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_catalog_writer(self, node_catalog_writer: input_schema.NodeCatalogWriter):
        """Adds a node that writes its input to the catalog as a Delta table or virtual table."""

        def _func(df: FlowDataEngine) -> FlowDataEngine:
            settings = node_catalog_writer.catalog_write_settings
            if not settings.table_name:
                raise ValueError("Catalog writer requires a table name")
            if settings.write_mode == "virtual":
                return _handle_virtual_table_write(self, node_catalog_writer, df)
            return _handle_physical_table_write(self, node_catalog_writer, df)

        def schema_callback():
            input_node: FlowNode = self.get_node(node_catalog_writer.node_id).node_inputs.main_inputs[0]
            return input_node.schema

        input_node_id = node_catalog_writer.depending_on_id if hasattr(node_catalog_writer, "depending_on_id") else None
        self.add_node_step(
            node_id=node_catalog_writer.node_id,
            function=_func,
            input_columns=[],
            node_type="catalog_writer",
            setting_input=node_catalog_writer,
            schema_callback=schema_callback,
            input_node_ids=[input_node_id],
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_database_writer(self, node_database_writer: input_schema.NodeDatabaseWriter):
        """Adds a node to write data to a database.

        Args:
            node_database_writer: The settings for the database writer node.
        """

        node_type = "database_writer"
        database_settings: input_schema.DatabaseWriteSettings = node_database_writer.database_write_settings

        def _func(df: FlowDataEngine):
            database_connection, encrypted_password, database_reference_settings = _resolve_database_credentials(
                database_settings, node_database_writer.user_id
            )
            df.lazy = True
            table_name = (
                database_settings.schema_name + "." + database_settings.table_name
                if database_settings.schema_name
                else database_settings.table_name
            )

            if self.execution_location == "local":
                df.to_database_obj(
                    database_type=database_connection.database_type,
                    uri=sql_utils.construct_sql_uri(
                        database_type=database_connection.database_type,
                        host=database_connection.host,
                        port=database_connection.port,
                        database=database_connection.database,
                        username=database_connection.username,
                        password=decrypt_secret(encrypted_password) if encrypted_password else None,
                        ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                        connect_timeout=10,
                    ),
                    table_name=table_name,
                    if_exists=database_settings.if_exists or "append",
                )
                return df

            database_external_write_settings = (
                sql_models.DatabaseExternalWriteSettings.create_from_from_node_database_writer(
                    node_database_writer=node_database_writer,
                    password=encrypted_password,
                    table_name=table_name,
                    database_reference_settings=(
                        database_reference_settings if database_settings.connection_mode == "reference" else None
                    ),
                    lf=df.data_frame,
                )
            )
            external_database_writer = ExternalDatabaseWriter(
                database_external_write_settings, wait_on_completion=False
            )
            node._fetch_cached_df = external_database_writer
            external_database_writer.get_result()
            return df

        def schema_callback():
            input_node: FlowNode = self.get_node(node_database_writer.node_id).node_inputs.main_inputs[0]
            return input_node.schema

        self.add_node_step(
            node_id=node_database_writer.node_id,
            function=_func,
            input_columns=[],
            node_type=node_type,
            setting_input=node_database_writer,
            schema_callback=schema_callback,
            input_node_ids=[node_database_writer.depending_on_id],
        )
        node = self.get_node(node_database_writer.node_id)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_database_reader(self, node_database_reader: input_schema.NodeDatabaseReader):
        """Adds a node to read data from a database.

        Args:
            node_database_reader: The settings for the database reader node.
        """

        logger.info("Adding database reader")
        node_type = "database_reader"
        database_settings: input_schema.DatabaseSettings = node_database_reader.database_settings

        # Resolve the connection lazily so opening/undoing a flow never requires
        # the current session to own the connection. Memoized so ``_func`` and
        # ``schema_callback`` share a single lookup; the lock matters because the
        # schema callback runs on a background thread (``SingleExecutionFuture``)
        # while ``_func`` runs on the execution thread. Runs under the node's
        # ``user_id`` (the flow owner at execution time).
        _creds: dict = {}
        _creds_lock = threading.Lock()

        def _get_creds():
            with _creds_lock:
                if "v" not in _creds:
                    _creds["v"] = _resolve_database_credentials(database_settings, node_database_reader.user_id)
                return _creds["v"]

        def _func():
            database_connection, encrypted_password, database_reference_settings = _get_creds()
            sql_source = BaseSqlSource(
                query=None if database_settings.query_mode == "table" else database_settings.query,
                table_name=database_settings.table_name,
                schema_name=database_settings.schema_name,
                fields=node_database_reader.fields,
            )

            # Local and worker reads share shared.db_reader.read_sql_with_fallback
            # (via SqlSource here, via read_sql_source in the worker).
            if self.execution_location == "local":
                local_source = SqlSource(
                    connection_string=sql_utils.construct_sql_uri(
                        database_type=database_connection.database_type,
                        host=database_connection.host,
                        port=database_connection.port,
                        database=database_connection.database,
                        username=database_connection.username,
                        password=decrypt_secret(encrypted_password) if encrypted_password else None,
                        ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                        connect_timeout=10,
                    ),
                    query=None if database_settings.query_mode == "table" else database_settings.query,
                    table_name=database_settings.table_name,
                    schema_name=database_settings.schema_name,
                    fields=node_database_reader.fields,
                    cancel_check=lambda: self.flow_settings.is_canceled or node._execution_state.is_canceled,
                    database_type=database_connection.database_type,
                )
                fl = FlowDataEngine(local_source.get_pl_df())
                fl.lazy = True
                node_database_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
                return fl

            database_external_read_settings = (
                sql_models.DatabaseExternalReadSettings.create_from_from_node_database_reader(
                    node_database_reader=node_database_reader,
                    password=encrypted_password,
                    query=sql_source.query,
                    database_reference_settings=(
                        database_reference_settings if database_settings.connection_mode == "reference" else None
                    ),
                )
            )

            external_database_fetcher = ExternalDatabaseFetcher(
                database_external_read_settings, wait_on_completion=False
            )
            node._fetch_cached_df = external_database_fetcher
            fl = FlowDataEngine(external_database_fetcher.get_result())
            node_database_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
            return fl

        def schema_callback():
            # Prefer the schema cached on the node so opening a saved flow renders
            # columns without a live connection. Fall back to the connection only
            # when fields were never captured (failures here are caught per-node).
            if node_database_reader.fields:
                return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_database_reader.fields]
            database_connection, encrypted_password, _ = _get_creds()
            sql_source = SqlSource(
                connection_string=sql_utils.construct_sql_uri(
                    database_type=database_connection.database_type,
                    host=database_connection.host,
                    port=database_connection.port,
                    database=database_connection.database,
                    username=database_connection.username,
                    password=decrypt_secret(encrypted_password) if encrypted_password else None,
                    ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                    connect_timeout=10,
                ),
                query=None if database_settings.query_mode == "table" else database_settings.query,
                table_name=database_settings.table_name,
                schema_name=database_settings.schema_name,
                fields=node_database_reader.fields,
                database_type=database_connection.database_type,
            )
            return sql_source.get_schema()

        node = self.get_node(node_database_reader.node_id)
        if node:
            # Persist so the lightweight callback survives the reset() that setting_input triggers.
            node.user_provided_schema_callback = schema_callback
            node.schema_callback = schema_callback
            node.node_type = node_type
            node.name = node_type
            node.function = _func
            node.setting_input = node_database_reader
            node.node_settings.cache_results = node_database_reader.cache_results
            self.add_node_to_starting_list(node)
        else:
            node = FlowNode(
                node_database_reader.node_id,
                function=_func,
                setting_input=node_database_reader,
                name=node_type,
                node_type=node_type,
                parent_uuid=self.uuid,
                schema_callback=schema_callback,
            )
            self._node_db[node_database_reader.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(node_database_reader.node_id)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_kafka_source(self, node_kafka_source: input_schema.NodeKafkaSource):
        """Adds a node to read data from a Kafka or Redpanda topic.

        Follows the same pattern as add_database_reader: offloads consumption
        to the worker, which writes an IPC temp file and returns a serialized
        LazyFrame reference. Offset tracking is handled by Kafka consumer groups.

        Args:
            node_kafka_source: The settings for the Kafka source node.
        """

        logger.info("Adding kafka source")
        node_type = "kafka_source"
        kafka_settings = node_kafka_source.kafka_settings

        # Settings updates may echo back ``fields`` cached from a previous topic /
        # format / connection (the UI clears them, but programmatic callers may
        # not). Stale fields would make ``schema_callback`` report the old topic's
        # columns, so drop them whenever a schema-affecting setting changed.
        # Open/undo replays keep their fields: the replayed settings match the
        # node's previous ones (or the prior node is just a promise).
        prior_settings = getattr(self.get_node(node_kafka_source.node_id), "setting_input", None)
        if node_kafka_source.fields and isinstance(prior_settings, input_schema.NodeKafkaSource):
            prior_kafka = prior_settings.kafka_settings
            if (
                prior_kafka.topic_name != kafka_settings.topic_name
                or prior_kafka.value_format != kafka_settings.value_format
                or prior_kafka.kafka_connection_id != kafka_settings.kafka_connection_id
                or prior_kafka.kafka_connection_name != kafka_settings.kafka_connection_name
            ):
                node_kafka_source.fields = None

        # Resolve the connection lazily so opening/undoing a flow never requires
        # the current session to own the connection. Memoized so ``_func`` and
        # ``schema_callback`` share a single lookup; the lock matters because the
        # schema callback runs on a background thread (``SingleExecutionFuture``)
        # while ``_func`` runs on the execution thread. Runs under the node's
        # ``user_id`` (the flow owner at execution time).
        _read_settings: dict = {}
        _read_settings_lock = threading.Lock()

        def _get_kafka_read_settings() -> KafkaReadSettings:
            with _read_settings_lock:
                if "v" not in _read_settings:
                    with get_db_context() as db:
                        db_conn = get_kafka_connection(
                            db, kafka_settings.kafka_connection_id, node_kafka_source.user_id
                        )
                        if db_conn is None:
                            if kafka_settings.kafka_connection_name:
                                db_conn = get_kafka_connection_by_name(
                                    db, kafka_settings.kafka_connection_name, node_kafka_source.user_id
                                )
                            if db_conn is None:
                                raise HTTPException(status_code=400, detail="Kafka connection not found")
                        consumer_config = build_consumer_config(db, db_conn, node_kafka_source.user_id)
                    _read_settings["v"] = KafkaReadSettings.from_consumer_config(
                        consumer_config,
                        topic=kafka_settings.topic_name,
                        value_format=kafka_settings.value_format,
                        group_id=kafka_settings.sync_name
                        or f"flowfile-{node_kafka_source.flow_id}-node-{node_kafka_source.node_id}",
                        start_offset=kafka_settings.start_offset,
                        max_messages=kafka_settings.max_messages,
                        poll_timeout_seconds=kafka_settings.poll_timeout_seconds,
                        flowfile_flow_id=node_kafka_source.flow_id,
                        flowfile_node_id=node_kafka_source.node_id,
                    )
                return _read_settings["v"]

        def _func():
            kafka_read_settings = _get_kafka_read_settings()
            if self.execution_location == "local":
                # Local execution — consume directly in-process with spill-to-IPC
                import tempfile

                fd, spill_file = tempfile.mkstemp(suffix=".arrow", prefix="kafka_")
                os.close(fd)
                result, kafka_result = read_kafka_source(
                    kafka_read_settings,
                    commit=False,
                    decrypt_fn=_decrypt_fn,
                    spill_path=spill_file,
                )
                lf = result if isinstance(result, pl.LazyFrame) else result.lazy()
                fl = FlowDataEngine(lf)
                if kafka_result.messages_consumed > 0:
                    node._on_flow_complete = make_kafka_commit_callback(
                        kafka_read_settings,
                        kafka_result.new_offsets,
                        node_kafka_source.node_id,
                        self.flow_logger,
                        _decrypt_fn,
                    )
            else:
                # Remote execution — offload to worker (worker uses commit=False + sidecar)
                external_kafka_fetcher = ExternalKafkaFetcher(kafka_read_settings, wait_on_completion=False)
                node._fetch_cached_df = external_kafka_fetcher
                fl = FlowDataEngine(external_kafka_fetcher.get_result())
                offsets_data = fetch_kafka_offsets(external_kafka_fetcher.file_ref)
                if offsets_data and offsets_data.get("messages_consumed", 0) > 0:
                    node._on_flow_complete = make_kafka_commit_callback(
                        kafka_read_settings,
                        offsets_data["new_offsets"],
                        node_kafka_source.node_id,
                        self.flow_logger,
                        _decrypt_fn,
                    )
            # The worker DataFrame may have fewer columns than the inferred
            # schema (e.g. empty topic or starting at "latest"). Align to
            # the schema_callback result so downstream nodes see stable columns.
            expected_columns = schema_callback()
            fl = fl.align_to_schema(expected_columns)
            node_kafka_source.fields = [c.get_minimal_field_info() for c in fl.schema]
            return fl

        def _decrypt_fn(encrypted: str) -> str:
            return decrypt_secret(encrypted).get_secret_value()

        def schema_callback():
            # Prefer the schema cached on the node so opening a saved flow renders
            # columns without sampling the topic (a live connection). Sampling only
            # runs when fields were never captured (failures are caught per-node).
            if node_kafka_source.fields:
                return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_kafka_source.fields]
            schema_pairs = infer_topic_schema(_get_kafka_read_settings(), sample_size=10, decrypt_fn=_decrypt_fn)
            # Since the schema callback takes quite some time, we only run the function once.
            if not schema_pairs:
                result = [
                    FlowfileColumn.from_input(column_name="_kafka_key", data_type="String"),
                    FlowfileColumn.from_input(column_name="_kafka_partition", data_type="Int64"),
                    FlowfileColumn.from_input(column_name="_kafka_offset", data_type="Int64"),
                    FlowfileColumn.from_input(column_name="_kafka_timestamp", data_type="Datetime"),
                ]
            else:
                result = [FlowfileColumn.create_from_polars_dtype(column_name=n, data_type=t) for n, t in schema_pairs]
            return result

        node = self.get_node(node_kafka_source.node_id)
        if node:
            node.user_provided_schema_callback = schema_callback
            node.schema_callback = schema_callback
            node.node_type = node_type
            node.name = node_type
            node.function = _func
            node.setting_input = node_kafka_source
            node.node_settings.cache_results = node_kafka_source.cache_results
            self.add_node_to_starting_list(node)
        else:
            node = FlowNode(
                node_kafka_source.node_id,
                function=_func,
                setting_input=node_kafka_source,
                name=node_type,
                node_type=node_type,
                parent_uuid=self.uuid,
                schema_callback=schema_callback,
            )
            self._node_db[node_kafka_source.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(node_kafka_source.node_id)

    def add_sql_source(self, external_source_input: input_schema.NodeExternalSource):
        """Adds a node that reads data from a SQL source.

        This is a convenience alias for `add_external_source`.

        Args:
            external_source_input: The settings for the external SQL source node.
        """
        logger.info("Adding sql source")
        self.add_external_source(external_source_input)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_google_analytics_reader(self, node_ga_reader: input_schema.NodeGoogleAnalyticsReader) -> None:
        """Adds a node that reads from a Google Analytics 4 property.

        The actual API fetch (OAuth token refresh, ``run_report`` calls,
        pagination) is offloaded to the worker via ``ExternalGoogleAnalyticsFetcher``,
        so the core's event loop stays responsive. The ``schema_callback`` is
        derived locally from the selected metrics/dimensions — no network call
        is made during schema prediction, keeping downstream nodes lazy.
        """
        logger.info("Adding google analytics reader")
        node_type = "google_analytics_reader"
        ga_settings = node_ga_reader.google_analytics_settings

        def _build_worker_settings() -> WorkerGoogleAnalyticsReadSettings:
            # Connection resolution is deferred to run time so that *opening* or
            # *undoing* a flow never requires the current session to own the
            # connection (mirrors ``add_cloud_storage_reader``). It runs under
            # ``node_ga_reader.user_id`` — the flow owner at execution time.
            with get_db_context() as db:
                db_conn = get_ga_connection(db, ga_settings.ga_connection_name, node_ga_reader.user_id)
                if db_conn is None:
                    raise HTTPException(
                        status_code=400,
                        detail=(
                            f"Google Analytics connection '{ga_settings.ga_connection_name}' not found "
                            "or has not completed sign-in"
                        ),
                    )
                auth_method = db_conn.auth_method
                encrypted_credential = get_encrypted_credential(
                    db, ga_settings.ga_connection_name, node_ga_reader.user_id
                )
                if encrypted_credential is None:
                    raise HTTPException(
                        status_code=400,
                        detail=(
                            f"Google Analytics connection '{ga_settings.ga_connection_name}' has no stored credential"
                        ),
                    )
                # OAuth needs the per-instance client config; service accounts don't.
                # Resolved from the CONNECTION OWNER, not the run user: a group-shared
                # OAuth connection must use the owner's Google client config.
                oauth_cfg = get_google_oauth_config(db, db_conn.user_id) if auth_method == "oauth" else None

            common_kwargs = dict(
                property_id=ga_settings.property_id,
                start_date=ga_settings.start_date,
                end_date=ga_settings.end_date,
                metrics=ga_settings.metrics,
                dimensions=ga_settings.dimensions,
                limit=ga_settings.limit,
                filters=[
                    WorkerGoogleAnalyticsFilter(
                        field=f.field,
                        operator=f.operator,
                        value=f.value,
                        case_sensitive=f.case_sensitive,
                    )
                    for f in ga_settings.filters
                ],
                order_bys=[
                    WorkerGoogleAnalyticsOrderBy(field=ob.field, descending=ob.descending)
                    for ob in ga_settings.order_bys
                ],
                flowfile_flow_id=node_ga_reader.flow_id,
                flowfile_node_id=node_ga_reader.node_id,
            )

            if auth_method == "service_account":
                return WorkerGoogleAnalyticsReadSettings(
                    auth_method="service_account",
                    service_account_key_encrypted=encrypted_credential,
                    **common_kwargs,
                )
            # ``oauth_cfg`` is only fetched for the oauth auth method; guard against
            # an unknown auth_method value reaching this branch with ``None``.
            if not oauth_cfg or not oauth_cfg["client_id"] or not oauth_cfg["client_secret"]:
                raise HTTPException(
                    status_code=500,
                    detail=(
                        "Google OAuth is not configured on this instance. Open Admin → Google OAuth "
                        "and paste your OAuth client credentials before running this flow."
                    ),
                )
            return WorkerGoogleAnalyticsReadSettings(
                auth_method="oauth",
                refresh_token_encrypted=encrypted_credential,
                oauth_client_id=oauth_cfg["client_id"],
                oauth_client_secret_encrypted=_encrypt_with_master_key(oauth_cfg["client_secret"]),
                **common_kwargs,
            )

        # Stamp the predicted schema onto the setting object now, so downstream
        # nodes can introspect columns without ever invoking ``_func`` (which
        # would trigger a worker → Google round-trip). ``derive_schema`` is
        # pure-Python and runs against the chosen metrics/dimensions only — no DB,
        # so it stays eager and keeps flow-open connection-free.
        predicted_columns = derive_schema(metrics=ga_settings.metrics, dimensions=ga_settings.dimensions)
        node_ga_reader.fields = [c.get_minimal_field_info() for c in predicted_columns]

        def _func() -> FlowDataEngine:
            fetcher = ExternalGoogleAnalyticsFetcher(_build_worker_settings(), wait_on_completion=False)
            node._fetch_cached_df = fetcher
            # ``get_result()`` returns a ``pl.LazyFrame`` deserialised from the
            # worker's Arrow IPC file — never collect on the core service.
            fl = FlowDataEngine(fetcher.get_result())
            # Align to the predicted schema so downstream nodes see stable columns
            # even when the report is empty. ``align_to_schema`` lowers to lazy
            # ``with_columns``/``select`` calls, so this stays lazy.
            return fl.align_to_schema(schema_callback())

        def schema_callback() -> list[FlowfileColumn]:
            # Prefer the cached placeholder so repeated schema lookups don't
            # re-walk the heuristic table. ``derive_schema`` is the fallback
            # for the (rare) case where ``fields`` got cleared.
            if node_ga_reader.fields:
                return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_ga_reader.fields]
            return derive_schema(metrics=ga_settings.metrics, dimensions=ga_settings.dimensions)

        node = self.get_node(node_ga_reader.node_id)
        if node:
            node.schema_callback = schema_callback
            node.user_provided_schema_callback = schema_callback
            node.node_type = node_type
            node.name = node_type
            node.function = _func
            node.setting_input = node_ga_reader
            node.node_settings.cache_results = node_ga_reader.cache_results
            self.add_node_to_starting_list(node)
        else:
            node = FlowNode(
                node_ga_reader.node_id,
                function=_func,
                setting_input=node_ga_reader,
                name=node_type,
                node_type=node_type,
                parent_uuid=self.uuid,
                schema_callback=schema_callback,
            )
            node.user_provided_schema_callback = schema_callback
            self._node_db[node_ga_reader.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(node_ga_reader.node_id)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_rest_api_reader(self, node_rest_api_reader: input_schema.NodeRestApiReader) -> None:
        """Adds a node that reads from a REST API.

        All network I/O (HTTP round-trips, pagination, retries) is offloaded to
        the worker via ``ExternalRestApiFetcher`` — the core never makes the
        external call. The credential is resolved to an encrypted token here
        (from the user's secret store, or an inline plaintext) and the worker
        decrypts it just-in-time. A generic API's columns are unknown until a
        response is fetched, so ``schema_callback`` returns the columns cached on
        the node by the "Fetch sample" action — empty until the user samples or
        runs, in which case the fetched frame defines the schema.
        """
        logger.info("Adding rest api reader")
        node_type = "rest_api_reader"
        auth = node_rest_api_reader.rest_api_settings.auth

        # Encrypt any *inline* plaintext credential eagerly and null it out so it is
        # never persisted on the node (a security guarantee, independent of who owns
        # the flow). The *by-name* secret-store lookup is deferred to run time so
        # opening/undoing a flow never requires the current session to own the
        # secret — it resolves under the node's ``user_id`` (the flow owner).
        _inline_encrypted = _encrypt_with_master_key(auth.secret) if (auth.secret and not auth.secret_name) else None
        auth.secret = None

        def _resolve_secret_encrypted() -> str | None:
            if _inline_encrypted is not None:
                return _inline_encrypted
            return resolve_auth_secret_encrypted(auth, node_rest_api_reader.user_id)

        def _func() -> FlowDataEngine:
            encrypted = _resolve_secret_encrypted()
            worker_settings = build_rest_api_worker_settings(node_rest_api_reader, encrypted)
            if self.execution_location == "local":
                # No worker service in local runs — fetch in-process (cf. add_database_reader).
                from shared.rest_api.fetch import fetch_rest_api

                secret = decrypt_secret(encrypted).get_secret_value() if encrypted else None
                fl = FlowDataEngine(fetch_rest_api(worker_settings, secret=secret).lazy())
            else:
                fetcher = ExternalRestApiFetcher(worker_settings, wait_on_completion=False)
                node._fetch_cached_df = fetcher
                fl = FlowDataEngine(fetcher.get_result())
            cols = schema_callback()
            # Align to the sampled schema (if any) so downstream nodes see stable
            # columns; with no sample yet, the fetched frame defines the schema.
            if cols:
                return fl.align_to_schema(cols)
            node_rest_api_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
            return fl

        def schema_callback() -> list[FlowfileColumn]:
            if node_rest_api_reader.fields:
                return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_rest_api_reader.fields]
            return []

        node = self.get_node(node_rest_api_reader.node_id)
        if node:
            node.schema_callback = schema_callback
            node.user_provided_schema_callback = schema_callback
            node.node_type = node_type
            node.name = node_type
            node.function = _func
            node.setting_input = node_rest_api_reader
            node.node_settings.cache_results = node_rest_api_reader.cache_results
            self.add_node_to_starting_list(node)
        else:
            node = FlowNode(
                node_rest_api_reader.node_id,
                function=_func,
                setting_input=node_rest_api_reader,
                name=node_type,
                node_type=node_type,
                parent_uuid=self.uuid,
                schema_callback=schema_callback,
            )
            node.user_provided_schema_callback = schema_callback
            self._node_db[node_rest_api_reader.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(node_rest_api_reader.node_id)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_cloud_storage_writer(self, node_cloud_storage_writer: input_schema.NodeCloudStorageWriter) -> None:
        """Adds a node to write data to a cloud storage provider.

        Args:
            node_cloud_storage_writer: The settings for the cloud storage writer node.
        """

        node_type = "cloud_storage_writer"

        def _func(df: FlowDataEngine):
            df.lazy = True
            execute_remote = self.execution_location != "local"
            cloud_connection_settings = get_cloud_connection_settings(
                connection_name=node_cloud_storage_writer.cloud_storage_settings.connection_name,
                user_id=node_cloud_storage_writer.user_id,
                auth_mode=node_cloud_storage_writer.cloud_storage_settings.auth_mode,
            )
            full_cloud_storage_connection = cloud_connection_settings
            if execute_remote:
                settings = get_cloud_storage_write_settings_worker_interface(
                    write_settings=node_cloud_storage_writer.cloud_storage_settings,
                    connection=full_cloud_storage_connection,
                    lf=df.data_frame,
                    user_id=node_cloud_storage_writer.user_id,
                    flowfile_node_id=node_cloud_storage_writer.node_id,
                    flowfile_flow_id=self.flow_id,
                )
                external_database_writer = ExternalCloudWriter(settings, wait_on_completion=False)
                node._fetch_cached_df = external_database_writer
                external_database_writer.get_result()
            else:
                cloud_storage_write_settings_internal = CloudStorageWriteSettingsInternal(
                    connection=full_cloud_storage_connection,
                    write_settings=node_cloud_storage_writer.cloud_storage_settings,
                )
                df.to_cloud_storage_obj(cloud_storage_write_settings_internal)
            return df

        def schema_callback():
            logger.info("Starting to run the schema callback for cloud storage writer")
            if self.get_node(node_cloud_storage_writer.node_id).is_correct:
                return self.get_node(node_cloud_storage_writer.node_id).node_inputs.main_inputs[0].schema
            else:
                return [FlowfileColumn.from_input(column_name="__error__", data_type="String")]

        self.add_node_step(
            node_id=node_cloud_storage_writer.node_id,
            function=_func,
            input_columns=[],
            node_type=node_type,
            setting_input=node_cloud_storage_writer,
            schema_callback=schema_callback,
            input_node_ids=[node_cloud_storage_writer.depending_on_id],
        )

        node = self.get_node(node_cloud_storage_writer.node_id)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_cloud_storage_reader(self, node_cloud_storage_reader: input_schema.NodeCloudStorageReader) -> None:
        """Adds a cloud storage read node to the flow graph.

        Args:
            node_cloud_storage_reader: The settings for the cloud storage read node.
        """
        node_type = "cloud_storage_reader"
        logger.info("Adding cloud storage reader")
        cloud_storage_read_settings = node_cloud_storage_reader.cloud_storage_settings

        def _func():
            logger.info("Starting to run the schema callback for cloud storage reader")
            self.flow_logger.info("Starting to run the schema callback for cloud storage reader")
            settings = CloudStorageReadSettingsInternal(
                read_settings=cloud_storage_read_settings,
                connection=get_cloud_connection_settings(
                    connection_name=cloud_storage_read_settings.connection_name,
                    user_id=node_cloud_storage_reader.user_id,
                    auth_mode=cloud_storage_read_settings.auth_mode,
                ),
            )
            fl = FlowDataEngine.from_cloud_storage_obj(settings)
            return fl

        node = self.add_node_step(
            node_id=node_cloud_storage_reader.node_id,
            function=_func,
            cache_results=node_cloud_storage_reader.cache_results,
            setting_input=node_cloud_storage_reader,
            node_type=node_type,
        )
        self.add_node_to_starting_list(node)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_external_source(self, external_source_input: input_schema.NodeExternalSource):
        """Adds a node for a custom external data source.

        Args:
            external_source_input: The settings for the external source node.
        """

        node_type = "external_source"
        external_source_script = getattr(external_sources.custom_external_sources, external_source_input.identifier)
        source_settings = getattr(
            input_schema, snake_case_to_camel_case(external_source_input.identifier)
        ).model_validate(external_source_input.source_settings)
        if hasattr(external_source_script, "initial_getter"):
            initial_getter = external_source_script.initial_getter(source_settings)
        else:
            initial_getter = None
        data_getter = external_source_script.getter(source_settings)
        external_source = data_source_factory(
            source_type="custom",
            data_getter=data_getter,
            initial_data_getter=initial_getter,
            orientation=external_source_input.source_settings.orientation,
            schema=None,
        )

        def _func():
            logger.info("Calling external source")
            fl = FlowDataEngine.create_from_external_source(external_source=external_source)
            external_source_input.source_settings.fields = [c.get_minimal_field_info() for c in fl.schema]
            return fl

        node = self.get_node(external_source_input.node_id)
        if node:
            node.node_type = node_type
            node.name = node_type
            node.function = _func
            node.setting_input = external_source_input
            node.node_settings.cache_results = external_source_input.cache_results
            self.add_node_to_starting_list(node)

        else:
            node = FlowNode(
                external_source_input.node_id,
                function=_func,
                setting_input=external_source_input,
                name=node_type,
                node_type=node_type,
                parent_uuid=self.uuid,
            )
            self._node_db[external_source_input.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(external_source_input.node_id)
        if external_source_input.source_settings.fields and len(external_source_input.source_settings.fields) > 0:
            logger.info("Using provided schema in the node")

            def schema_callback():
                return [
                    FlowfileColumn.from_input(f.name, f.data_type) for f in external_source_input.source_settings.fields
                ]

            node.schema_callback = schema_callback
            node.user_provided_schema_callback = schema_callback
        else:
            logger.warning("Removing schema")
            node._schema_callback = None
        self.add_node_step(
            node_id=external_source_input.node_id,
            function=_func,
            input_columns=[],
            node_type=node_type,
            setting_input=external_source_input,
        )

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_read(self, input_file: input_schema.NodeRead):
        """Adds a node to read data from a local file (e.g., CSV, Parquet, Excel).

        Args:
            input_file: The settings for the read operation.
        """
        if (
            input_file.received_file.file_type in ("xlsx", "excel")
            and input_file.received_file.table_settings.sheet_name == ""
        ):
            sheet_name = fastexcel.read_excel(input_file.received_file.path).sheet_names[0]
            input_file.received_file.table_settings.sheet_name = sheet_name

        received_file = input_file.received_file
        input_file.received_file.set_absolute_filepath()

        def _func():
            input_file.received_file.set_absolute_filepath()
            if self.execution_location == "local":
                input_data = FlowDataEngine.create_from_path(input_file.received_file)
            elif input_file.received_file.file_type in ("parquet", "ipc", "ndjson"):
                input_data = FlowDataEngine.create_from_path(input_file.received_file)
            elif (
                input_file.received_file.file_type == "csv"
                and "utf" in input_file.received_file.table_settings.encoding
            ):
                input_data = FlowDataEngine.create_from_path(input_file.received_file)
            else:
                input_data = FlowDataEngine.create_from_path_worker(
                    input_file.received_file, node_id=input_file.node_id, flow_id=self.flow_id
                )
            input_data.name = input_file.received_file.name
            return input_data

        node = self.get_node(input_file.node_id)
        schema_callback = None
        if node:
            start_hash = node.hash
            node.node_type = "read"
            node.name = "read"
            node.function = _func
            node.setting_input = input_file
            self.add_node_to_starting_list(node)

            if start_hash != node.hash:
                logger.info("Hash changed, updating schema")
                if len(received_file.fields) > 0:

                    def schema_callback():
                        return [FlowfileColumn.from_input(f.name, f.data_type) for f in received_file.fields]

                elif input_file.received_file.file_type in ("csv", "json", "parquet", "ipc", "ndjson"):

                    def schema_callback():
                        input_data = FlowDataEngine.create_from_path(input_file.received_file)
                        return input_data.schema

                elif input_file.received_file.file_type in ("xlsx", "excel"):
                    schema_callback = get_xlsx_schema_callback(
                        engine="openpyxl",
                        file_path=received_file.file_path,
                        sheet_name=received_file.table_settings.sheet_name,
                        start_row=received_file.table_settings.start_row,
                        end_row=received_file.table_settings.end_row,
                        start_column=received_file.table_settings.start_column,
                        end_column=received_file.table_settings.end_column,
                        has_headers=received_file.table_settings.has_headers,
                    )
                else:
                    schema_callback = None
        else:
            node = FlowNode(
                input_file.node_id,
                function=_func,
                setting_input=input_file,
                name="read",
                node_type="read",
                parent_uuid=self.uuid,
            )
            self._node_db[input_file.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(input_file.node_id)

        if schema_callback is not None:
            node.schema_callback = schema_callback
            node.user_provided_schema_callback = schema_callback
        return self

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_datasource(self, input_file: input_schema.NodeDatasource | input_schema.NodeManualInput) -> "FlowGraph":
        """Adds a data source node to the graph.

        This method serves as a factory for creating starting nodes, handling both
        file-based sources and direct manual data entry.

        Args:
            input_file: The configuration object for the data source.

        Returns:
            The `FlowGraph` instance for method chaining.
        """
        if isinstance(input_file, input_schema.NodeManualInput):
            input_data = FlowDataEngine(input_file.raw_data_format)
            ref = "manual_input"
        else:
            input_data = FlowDataEngine(path_ref=input_file.file_ref)
            ref = "datasource"
        node = self.get_node(input_file.node_id)
        if node:
            node.node_type = ref
            node.name = ref
            node.function = input_data
            node.setting_input = input_file
            self.add_node_to_starting_list(node)

        else:
            input_data.collect()
            node = FlowNode(
                input_file.node_id,
                function=input_data,
                setting_input=input_file,
                name=ref,
                node_type=ref,
                parent_uuid=self.uuid,
            )
            self._node_db[input_file.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(input_file.node_id)
        return self

    def add_manual_input(self, input_file: input_schema.NodeManualInput):
        """Adds a node for manual data entry.

        This is a convenience alias for `add_datasource`.

        Args:
            input_file: The settings and data for the manual input node.
        """
        self.add_datasource(input_file)

    @with_history_capture(HistoryActionType.UPDATE_SETTINGS)
    def add_flow_input(self, settings: input_schema.NodeFlowInput) -> "FlowGraph":
        """Adds a named subflow-input placeholder source.

        Standalone runs serve the optional sample data (empty frame otherwise);
        a parent run_flow node overwrites ``node.function`` with real data.
        """
        for other in self.nodes:
            if (
                other.node_type == "flow_input"
                and other.node_id != settings.node_id
                and isinstance(other.setting_input, input_schema.NodeFlowInput)
                and other.setting_input.input_name == settings.input_name
            ):
                raise ValueError(f"flow_input name '{settings.input_name}' is already used by node {other.node_id}")
        if settings.raw_data_format is not None and settings.raw_data_format.columns:
            input_data = FlowDataEngine(settings.raw_data_format)
        else:
            input_data = FlowDataEngine()
        node = self.get_node(settings.node_id)
        if node:
            node.node_type = "flow_input"
            node.name = "flow_input"
            node.function = input_data
            node.setting_input = settings
            self.add_node_to_starting_list(node)
        else:
            node = FlowNode(
                settings.node_id,
                function=input_data,
                setting_input=settings,
                name="flow_input",
                node_type="flow_input",
                parent_uuid=self.uuid,
            )
            self._node_db[settings.node_id] = node
            self.add_node_to_starting_list(node)
            self._node_ids.append(settings.node_id)
        return self

    @property
    def nodes(self) -> list[FlowNode]:
        """Gets a list of all FlowNode objects in the graph."""

        return list(self._node_db.values())

    def check_flow_laziness(self) -> tuple[bool, list[str]]:
        """Check whether the flow supports lazy execution for virtual tables.

        Finds all catalog-writer nodes in the graph and checks whether their
        upstream dependencies are fully lazy.  Only the nodes that actually
        feed into a catalog writer matter — unrelated branches (e.g. an
        Explore Data node on a separate path) are ignored.

        Returns a tuple of (is_optimizable, reasons_if_not).
        """
        catalog_writers = [n for n in self.nodes if n.node_type == "catalog_writer"]
        if not catalog_writers:
            # No catalog writer → nothing to optimise; treat as non-lazy
            return False, ["No catalog writer node found in the flow"]
        all_reasons: list[str] = []
        for writer in catalog_writers:
            _, reasons = writer.check_upstream_laziness()
            all_reasons.extend(reasons)
        seen: set[str] = set()
        unique: list[str] = []
        for r in all_reasons:
            if r not in seen:
                seen.add(r)
                unique.append(r)
        return len(unique) == 0, unique

    @property
    def execution_mode(self) -> schemas.ExecutionModeLiteral:
        """Gets the current execution mode ('Development' or 'Performance')."""
        return self.flow_settings.execution_mode

    def get_implicit_starter_nodes(self) -> list[FlowNode]:
        """Finds nodes that can act as starting points but are not explicitly defined as such.

        Some nodes, like the Polars Code node, can function without an input. This
        method identifies such nodes if they have no incoming connections.

        Returns:
            A list of `FlowNode` objects that are implicit starting nodes.
        """
        starting_node_ids = [node.node_id for node in self._flow_starts]
        implicit_starting_nodes = []
        for node in self.nodes:
            if node.node_template.can_be_start and not node.has_input and node.node_id not in starting_node_ids:
                implicit_starting_nodes.append(node)
        return implicit_starting_nodes

    @execution_mode.setter
    def execution_mode(self, mode: schemas.ExecutionModeLiteral):
        """Sets the execution mode for the flow.

        Args:
            mode: The execution mode to set.
        """
        self.flow_settings.execution_mode = mode

    @property
    def execution_location(self) -> schemas.ExecutionLocationsLiteral:
        """Gets the current execution location."""
        return self.flow_settings.execution_location

    @execution_location.setter
    def execution_location(self, execution_location: schemas.ExecutionLocationsLiteral):
        """Sets the execution location for the flow.

        Args:
            execution_location: The execution location to set.
        """
        if self.flow_settings.execution_location != execution_location:
            self.reset()
        self.flow_settings.execution_location = execution_location

    def validate_if_node_can_be_fetched(self, node_id: int) -> None:
        flow_node = self._node_db.get(node_id)
        if not flow_node:
            raise Exception("Node not found found")
        execution_plan = compute_execution_plan(
            nodes=self.nodes, flow_starts=self._flow_starts + self.get_implicit_starter_nodes()
        )
        if flow_node.node_id in [skip_node.node_id for skip_node in execution_plan.skip_nodes]:
            raise Exception("Node can not be executed because it does not have it's inputs")

    def create_initial_run_information(self, number_of_nodes: int, run_type: Literal["fetch_one", "full_run"]):
        return RunInformation(
            flow_id=self.flow_id,
            start_time=datetime.datetime.now(),
            end_time=None,
            success=None,
            is_running=True,
            execution_mode=self.flow_settings.execution_mode,
            number_of_nodes=number_of_nodes,
            node_step_result=[],
            run_type=run_type,
        )

    def create_empty_run_information(self) -> RunInformation:
        return RunInformation(
            flow_id=self.flow_id,
            start_time=None,
            end_time=None,
            success=None,
            is_running=False,
            execution_mode=self.flow_settings.execution_mode,
            number_of_nodes=0,
            node_step_result=[],
            run_type="init",
        )

    def try_claim_run(self) -> bool:
        """Atomically claim the flow's single-run slot; False when a run is already in flight."""
        with self._run_claim_lock:
            if self.flow_settings.is_running:
                return False
            self.flow_settings.is_running = True
            return True

    def release_run(self) -> None:
        """Release the single-run slot claimed by try_claim_run (idempotent)."""
        with self._run_claim_lock:
            self.flow_settings.is_running = False

    def trigger_fetch_node(
        self,
        node_id: int,
        *,
        performance_mode: bool = False,
        reset_cache: bool = True,
    ) -> RunInformation | None:
        """Executes a specific node in the graph by its ID.

        The defaults are the data-preview contract: a non-performance run, so the
        node stores its result and can serve the 100-row example grid. Callers
        that only need the node's query plan (the Explore Data drawer) pass
        ``performance_mode=True``, which skips that store entirely, and
        ``reset_cache=False`` so exploring doesn't evict a useful cache.
        """
        if not self.try_claim_run():
            raise Exception("Flow is already running")
        flow_node = self.get_node(node_id)
        self.flow_settings.is_canceled = False
        self.flow_logger.clear_log_file()
        self.latest_run_info = self.create_initial_run_information(1, "fetch_one")
        node_logger = self.flow_logger.get_node_logger(flow_node.node_id)
        node_result = NodeResult(
            node_id=flow_node.node_id,
            node_name=flow_node.name,
            description=flow_node.get_node_information().description,
        )
        logger.info(f"Starting to run: node {flow_node.node_id}, start time: {node_result.start_timestamp}")
        try:
            self.latest_run_info.node_step_result.append(node_result)
            flow_node.execute_node(
                run_location=self.flow_settings.execution_location,
                performance_mode=performance_mode,
                node_logger=node_logger,
                optimize_for_downstream=False,
                reset_cache=reset_cache,
            )
            node_result.error = str(flow_node.results.errors)
            if self.flow_settings.is_canceled:
                node_result.success = None
                node_result.success = None
                node_result.is_running = False
            node_result.success = flow_node.results.errors is None
            node_result.end_timestamp = time()
            node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
            node_result.is_running = False
            self.latest_run_info.nodes_completed += 1
            self.latest_run_info.end_time = datetime.datetime.now()
            self.release_run()
            return self.get_run_info()
        except Exception as e:
            node_result.error = "Node did not run"
            node_result.success = False
            node_result.end_timestamp = time()
            node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
            node_result.is_running = False
            node_logger.error(f"Error in node {flow_node.node_id}: {e}")
        finally:
            self.release_run()

    # Artifact helpers

    @staticmethod
    def _resolve_input_names(node: FlowNode | None, table_count: int) -> list[str] | None:
        """Derive named input keys from connected source nodes.

        Uses the source node's ``node_reference`` when set, otherwise
        falls back to ``df_{node_id}``.  Returns ``None`` when no node
        is available, there are no input tables, or the number of
        connected sources doesn't match ``table_count`` (original
        unnamed behaviour).
        """
        if node is None or table_count == 0:
            return None
        input_names: list[str] = []
        for source_node in node.all_inputs:
            ref = getattr(source_node.setting_input, "node_reference", None)
            name = ref if ref else f"df_{source_node.node_id}"
            input_names.append(name)
        if len(input_names) != table_count:
            return None
        return input_names

    def _get_upstream_node_ids(self, node_id: int) -> list[int]:
        """Get all upstream node IDs (direct and transitive) for *node_id*.

        Traverses the ``all_inputs`` links recursively and returns a
        deduplicated list in breadth-first order.
        """
        node = self.get_node(node_id)
        if node is None:
            return []

        visited: set[int] = set()
        result: list[int] = []
        queue = list(node.all_inputs)
        while queue:
            current = queue.pop(0)
            cid = current.node_id
            if cid in visited:
                continue
            visited.add(cid)
            result.append(cid)
            queue.extend(current.all_inputs)
        return result

    def _get_required_kernel_ids(self) -> set[str]:
        """Return the set of kernel IDs used by ``python_script`` nodes."""
        kernel_ids: set[str] = set()
        for node in self.nodes:
            if node.node_type == "python_script" and node.setting_input is not None:
                kid = getattr(
                    getattr(node.setting_input, "python_script_input", None),
                    "kernel_id",
                    None,
                )
                if kid:
                    kernel_ids.add(kid)
        return kernel_ids

    def _compute_rerun_python_script_node_ids(
        self,
        plan_skip_ids: set[str | int],
    ) -> set[int]:
        """Return node IDs for ``python_script`` nodes that will re-execute.

        A python_script node will re-execute (and thus needs its old
        artifacts cleared) when:

        * It is NOT in the execution-plan skip set, **and**
        * Its execution state indicates it has NOT already run with the
          current setup (i.e. its cache is stale or it never ran).
        """
        rerun: set[int] = set()
        for node in self.nodes:
            if node.node_type != "python_script":
                continue
            if node.node_id in plan_skip_ids:
                continue
            if not node._execution_state.has_run_with_current_setup:
                rerun.add(node.node_id)
        return rerun

    def _group_rerun_nodes_by_kernel(
        self,
        rerun_node_ids: set[int],
    ) -> dict[str, set[int]]:
        """Group *rerun_node_ids* by their kernel ID.

        Returns a mapping ``kernel_id → {node_id, …}``.
        """
        kernel_nodes: dict[str, set[int]] = {}
        for node in self.nodes:
            if node.node_id not in rerun_node_ids:
                continue
            if node.node_type == "python_script" and node.setting_input is not None:
                kid = getattr(
                    getattr(node.setting_input, "python_script_input", None),
                    "kernel_id",
                    None,
                )
                if kid:
                    kernel_nodes.setdefault(kid, set()).add(node.node_id)
        return kernel_nodes

    def _execute_single_node(
        self,
        node: FlowNode,
        performance_mode: bool,
        run_info_lock: threading.Lock,
        params: dict[str, ParamValue] | None = None,
    ) -> tuple[NodeResult, FlowNode]:
        """Executes a single node, records its result, and returns both.

        Thread-safe: uses run_info_lock when mutating shared run information.

        Args:
            node: The node to execute.
            performance_mode: Whether to run in performance mode.
            run_info_lock: Lock protecting shared RunInformation state.
            params: Optional parameter dict for ${name} substitution in node settings.

        Returns:
            A (NodeResult, FlowNode) tuple for post-stage failure propagation.
        """
        node_logger = self.flow_logger.get_node_logger(node.node_id)
        node_result = NodeResult(
            node_id=node.node_id,
            node_name=node.name,
            description=node.get_node_information().description,
        )

        with run_info_lock:
            self.latest_run_info.node_step_result.append(node_result)

        # Temporarily substitute parameters into node settings (in-place so closures see the values)
        restorations = []
        # Save the node's hash before substitution. executor.execute() calls node.reset()
        # while setting_input is mutated, which recomputes _hash from the resolved path.
        # After restore_parameters the path returns to the original ${...} form but _hash
        # still holds the resolved-path hash → needs_reset() returns True on the next
        # setting_input write → clears example_data_generator / has_completed_last_run.
        # Restoring _hash after restore_parameters keeps the hash consistent with the
        # restored setting_input and prevents that spurious reset.
        saved_hash = node._hash
        if params:
            try:
                restorations = apply_parameters_in_place(node.setting_input, params)
            except ValueError as e:
                node_result.error = str(e)
                node_result.success = False
                node_result.end_timestamp = time()
                node_result.run_time_ms = 0
                node_result.is_running = False
                node_logger.error(f"Parameter resolution failed for node {node.node_id}: {e}")
                return node_result, node

        logger.info(f"Starting to run: node {node.node_id}, start time: {node_result.start_timestamp}")
        try:
            node.execute_node(
                run_location=self.flow_settings.execution_location,
                performance_mode=performance_mode,
                node_logger=node_logger,
            )
        finally:
            # Restore original ${...} refs so the saved flow is unchanged
            if restorations:
                restore_parameters(restorations)
            # Restore the hash to match the restored setting_input so that
            # subsequent get_node_data / setting_input writes don't trigger
            # a spurious reset (and lose example_data_generator / has_completed_last_run).
            node._hash = saved_hash
        try:
            node_result.error = "" if node.results.errors is None else str(node.results.errors)
            if self.flow_settings.is_canceled:
                node_result.success = None
                node_result.is_running = False
                return node_result, node
            node_result.success = node.results.errors is None
            node_result.end_timestamp = time()
            node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
            node_result.is_running = False
        except Exception as e:
            node_result.error = "Node did not run"
            node_result.success = False
            node_result.end_timestamp = time()
            node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
            node_result.is_running = False
            node_logger.error(f"Error in node {node.node_id}: {e}")

        node_logger.info(f"Completed node with success: {node_result.success}")
        with run_info_lock:
            self.latest_run_info.nodes_completed += 1

        return node_result, node

    def _prepare_rerun_artifacts(self, plan_skip_ids: set[str | int]) -> None:
        """Prepare artifact state for nodes that will re-run.

        Computes which python_script nodes need re-execution, expands the set
        to include producer nodes whose artifacts were deleted, marks them
        stale, and clears both metadata and kernel-side artifacts.
        """
        rerun_node_ids = self._compute_rerun_python_script_node_ids(plan_skip_ids)

        # Expand re-run set: if a re-running node previously deleted
        # artifacts, the original producer nodes must also re-run so
        # those artifacts are available again in the kernel store.
        while True:
            deleted_producers = self.artifact_context.get_producer_nodes_for_deletions(
                rerun_node_ids,
            )
            new_ids = deleted_producers - rerun_node_ids
            if not new_ids:
                break
            rerun_node_ids |= new_ids

        # Force producer nodes (added due to artifact deletions) to
        # actually re-execute by marking their execution state stale.
        for nid in rerun_node_ids:
            node = self.get_node(nid)
            if node is not None and node._execution_state.has_run_with_current_setup:
                node._execution_state.has_run_with_current_setup = False

        # Also purge stale metadata for nodes not in this graph
        # (e.g. injected externally or left over from removed nodes).
        graph_node_ids = set(self._node_db.keys())
        stale_node_ids = {nid for nid in self.artifact_context._node_states if nid not in graph_node_ids}
        nodes_to_clear = rerun_node_ids | stale_node_ids
        if nodes_to_clear:
            self.artifact_context.clear_nodes(nodes_to_clear)

        if rerun_node_ids:
            kernel_node_map = self._group_rerun_nodes_by_kernel(rerun_node_ids)
            for kid, node_ids_for_kernel in kernel_node_map.items():
                try:
                    manager = get_kernel_manager()
                    manager.clear_node_artifacts_sync(
                        kid, list(node_ids_for_kernel), flow_id=self.flow_id, flow_logger=self.flow_logger
                    )
                except Exception:
                    logger.debug(
                        "Could not clear node artifacts for kernel '%s', nodes %s",
                        kid,
                        sorted(node_ids_for_kernel),
                    )

    def _execute_stages(
        self,
        execution_plan: ExecutionPlan,
        performance_mode: bool,
        params: dict[str, ParamValue],
        skip_node_ids: set[str | int],
    ) -> set[str | int]:
        """Execute all stages in the plan, running independent nodes in parallel.

        Iterates through stages sequentially. Within each stage, independent
        nodes are executed in parallel (or sequentially if parallelism is
        disabled). Failed nodes cause their dependents to be skipped.

        Returns:
            Set of node IDs that failed during execution.
        """
        run_info_lock = threading.Lock()
        failed_node_ids: set[str | int] = set()

        for stage in execution_plan.stages:
            if self.flow_settings.is_canceled:
                self.flow_logger.info("Flow canceled")
                break

            nodes_to_run = [n for n in stage.nodes if n.node_id not in skip_node_ids]

            for skipped in stage.nodes:
                if skipped.node_id in skip_node_ids:
                    node_logger = self.flow_logger.get_node_logger(skipped.node_id)
                    node_logger.info(f"Skipping node {skipped.node_id}")

            if not nodes_to_run:
                continue

            is_local = self.flow_settings.execution_location == "local"
            max_workers = 1 if is_local else self.flow_settings.max_parallel_workers
            if len(nodes_to_run) == 1 or max_workers == 1:
                stage_results = [
                    self._execute_single_node(node, performance_mode, run_info_lock, params or None)
                    for node in nodes_to_run
                ]
            else:
                stage_results: list[tuple[NodeResult, FlowNode]] = []
                workers = min(max_workers, len(nodes_to_run))
                with ThreadPoolExecutor(max_workers=workers) as executor:
                    futures = {
                        executor.submit(
                            self._execute_single_node, node, performance_mode, run_info_lock, params or None
                        ): node
                        for node in nodes_to_run
                    }
                    for future in as_completed(futures):
                        stage_results.append(future.result())

            for node_result, node in stage_results:
                if not node_result.success:
                    failed_node_ids.add(node.node_id)
                    skip_node_ids.add(node.node_id)
                    for dep in node.get_all_dependent_nodes():
                        skip_node_ids.add(dep.node_id)

        return failed_node_ids

    def _run_post_execution_callbacks(
        self,
        failed_node_ids: set[str | int],
        skip_node_ids: set[str | int],
    ) -> None:
        """Invoke _on_flow_complete callbacks registered by source nodes.

        Each callback receives ``success=True`` when the node and all its
        downstream dependents completed without failure or skip.
        Used e.g. by Kafka sources to commit offsets only on full success.

        Note: the caller must guard against cancellation — this method is
        only invoked when ``is_canceled`` is False.
        """
        incomplete_node_ids = failed_node_ids | skip_node_ids

        for n in self.nodes:
            callback = n._on_flow_complete
            if callback is None:
                continue
            downstream_incomplete = n.node_id in incomplete_node_ids or any(
                dep.node_id in incomplete_node_ids for dep in n.get_all_dependent_nodes()
            )
            success = not downstream_incomplete
            try:
                callback(success)
            except Exception as e:
                self.flow_logger.error(f"Post-execution callback failed for node {n.node_id}: {e}")
            n._on_flow_complete = None

    def _refresh_catalog_reader_freshness(self) -> None:
        """Invalidate catalog_reader nodes whose Delta sources changed since their last run.

        The node hash is source-blind (settings + upstream hashes only), so in
        Development mode an unchanged-settings reader is skipped and downstream
        keeps reading a frozen worker snapshot. This probes the live Delta
        versions once per run and bumps the node's cache epoch on drift — which
        rotates the hash, defeats the dev-mode skip AND the explicit
        cache_results worker lookup, and cascades resets downstream.

        Pinned ``delta_version`` readers are deliberate time travel and are
        never probed. Probe failures fail open (invalidate) so the real error
        surfaces on the canvas instead of a silently-served stale snapshot.
        """
        version_cache: dict[str, int] = {}
        opts_by_namespace: dict[int | None, dict | None] = {}
        for node in self.nodes:
            if node.node_type != "catalog_reader":
                continue
            settings = node.setting_input
            if not isinstance(settings, input_schema.NodeCatalogReader):
                continue
            if not settings.sql_query and settings.delta_version is not None:
                continue
            try:
                fingerprint, force = _catalog_reader_source_fingerprint(settings, version_cache, opts_by_namespace)
            except Exception:
                self.flow_logger.warning(
                    f"Node {node.node_id}: could not probe catalog source freshness; re-running to be safe"
                )
                fingerprint, force = None, True
            recorded = node._execution_state.source_version_info
            if force or (recorded is not None and fingerprint != recorded):
                node.invalidate_cache()
                self.flow_logger.info(f"Node {node.node_id}: catalog source changed; invalidating cached result")
            node._execution_state.source_version_info = fingerprint

    def run_graph(self) -> RunInformation | None:
        """Executes the entire data flow graph from start to finish.

        Independent nodes within the same execution stage are run in parallel
        using threads. Stages are processed sequentially so that all dependencies
        are satisfied before a stage begins.

        Returns:
            A RunInformation object summarizing the execution results.

        Raises:
            Exception: If the flow is already running.
        """
        if not self.try_claim_run():
            raise Exception("Flow is already running")
        try:
            self.flow_settings.is_canceled = False
            self.flow_logger.clear_log_file()
            self.flow_logger.info("Starting to run flowfile flow...")

            self._refresh_catalog_reader_freshness()

            execution_plan = compute_execution_plan(
                nodes=self.nodes, flow_starts=self._flow_starts + self.get_implicit_starter_nodes()
            )

            plan_skip_ids: set[str | int] = {n.node_id for n in execution_plan.skip_nodes}
            self._prepare_rerun_artifacts(plan_skip_ids)

            self.latest_run_info = self.create_initial_run_information(execution_plan.node_count, "full_run")
            skip_node_message(self.flow_logger, execution_plan.skip_nodes)
            execution_order_message(self.flow_logger, execution_plan.stages)

            performance_mode = self.flow_settings.execution_mode == "Performance"
            params: dict[str, ParamValue] = {p.name: p.typed_default() for p in self.flow_settings.parameters}

            failed_node_ids = self._execute_stages(execution_plan, performance_mode, params, plan_skip_ids)
            if not self.flow_settings.is_canceled:
                self._run_post_execution_callbacks(failed_node_ids, plan_skip_ids)

            self.latest_run_info.end_time = datetime.datetime.now()
            self.flow_logger.info("Flow completed!")
            self.end_datetime = datetime.datetime.now()
            self.release_run()
            if self.flow_settings.is_canceled:
                self.flow_logger.info("Flow canceled")
            return self.get_run_info()
        except Exception as e:
            raise e
        finally:
            self.release_run()

    def get_run_info(self) -> RunInformation:
        """Gets a summary of the most recent graph execution.

        Returns:
            A RunInformation object with details about the last run.
        """
        is_running = self.flow_settings.is_running
        if self.latest_run_info is None:
            return self.create_empty_run_information()

        run_info = self.latest_run_info
        run_info.is_running = is_running
        run_info.execution_mode = self.flow_settings.execution_mode
        if not is_running and run_info.success is None:
            run_info.success = all(nr.success for nr in run_info.node_step_result)
        return run_info

    @property
    def node_connections(self) -> list[tuple[int, int]]:
        """Computes and returns a list of all connections in the graph.

        Returns:
            A list of tuples, where each tuple is a (source_id, target_id) pair.
        """
        connections = set()
        for node in self.nodes:
            outgoing_connections = [(node.node_id, ltn.node_id) for ltn in node.leads_to_nodes]
            incoming_connections = [(don.node_id, node.node_id) for don in node.all_inputs]
            node_connections = [
                c for c in outgoing_connections + incoming_connections if (c[0] is not None and c[1] is not None)
            ]
            for node_connection in node_connections:
                if node_connection not in connections:
                    connections.add(node_connection)
        return list(connections)

    def get_node_data(self, node_id: int, include_example: bool = True) -> NodeData:
        """Retrieves all data needed to render a node in the UI.

        Args:
            node_id: The ID of the node.
            include_example: Whether to include data samples in the result.

        Returns:
            A NodeData object, or None if the node is not found.
        """
        node = self._node_db[node_id]
        return node.get_node_data(flow_id=self.flow_id, include_example=include_example)

    def get_flowfile_data(self) -> schemas.FlowfileData:
        start_node_ids = {v.node_id for v in self._flow_starts}

        nodes = []
        for node in self.nodes:
            node_info = node.get_node_information()
            flowfile_node = schemas.FlowfileNode(
                id=node_info.id,
                type=node_info.type,
                is_start_node=node.node_id in start_node_ids,
                description=node_info.description,
                node_reference=node_info.node_reference,
                x_position=int(node_info.x_position),
                y_position=int(node_info.y_position),
                group_id=node_info.group_id,
                left_input_id=node_info.left_input_id,
                right_input_id=node_info.right_input_id,
                input_ids=node_info.input_ids,
                outputs=node_info.outputs,
                output_handles=node_info.output_handles,
                input_connections=node_info.input_connections,
                setting_input=node_info.setting_input,
            )
            nodes.append(flowfile_node)

        settings = schemas.FlowfileSettings(
            description=self.flow_settings.description,
            execution_mode=self.flow_settings.execution_mode,
            execution_location=self.flow_settings.execution_location,
            auto_save=self.flow_settings.auto_save,
            show_detailed_progress=self.flow_settings.show_detailed_progress,
            validate_settings=self.flow_settings.validate_settings,
            max_parallel_workers=self.flow_settings.max_parallel_workers,
            source_registration_id=self.flow_settings.source_registration_id,
            parameters=self.flow_settings.parameters,
        )
        # Persist only groups that still have members (prune orphans).
        groups = [
            schemas.FlowfileGroup(**self._groups[group_id].model_dump())
            for group_id in self._groups
            if self._member_node_ids(group_id) or self._child_group_ids(group_id)
        ]
        return schemas.FlowfileData(
            flowfile_version=__version__,
            flowfile_id=self.flow_id,
            flowfile_name=self.__name__,
            flowfile_settings=settings,
            nodes=nodes,
            groups=groups,
        )

    def get_node_storage(self) -> schemas.FlowInformation:
        """Serializes the entire graph's state into a storable format.

        Returns:
            A FlowInformation object representing the complete graph.
        """
        node_information = {
            node.node_id: node.get_node_information() for node in self.nodes if node.is_setup and node.is_correct
        }

        return schemas.FlowInformation(
            flow_id=self.flow_id,
            flow_name=self.__name__,
            flow_settings=self.flow_settings,
            data=node_information,
            node_starts=[v.node_id for v in self._flow_starts],
            node_connections=self.node_connections,
        )

    def cancel(self):
        """Cancels an ongoing graph execution."""

        if not self.flow_settings.is_running:
            return
        self.flow_settings.is_canceled = True
        for node in self.nodes:
            node.cancel()

    def close_flow(self):
        """Performs cleanup operations, such as clearing node caches."""

        for node in self.nodes:
            node.remove_cache()

    def _handle_flow_renaming(self, new_name: str, new_path: Path):
        """Adopt the target file's stem as the flow name, but only when a save relocates the flow.

        A same-path save must never rename — the name can have been set from the
        catalog (``POST /editor/rename_flow/``) and the file stem is not authoritative.
        """
        if not self.flow_settings:
            return
        if self.flow_settings.path and Path(self.flow_settings.path).absolute() != new_path.absolute():
            self.__name__ = new_name
            self.flow_settings.save_location = str(new_path.absolute())
            self.flow_settings.name = new_name
        elif not self.flow_settings.save_location:
            # Back-fill where the flow lives, never what it is called.
            self.flow_settings.save_location = str(new_path.absolute())
            if not self.flow_settings.name:
                self.__name__ = new_name
                self.flow_settings.name = new_name

    def save_flow(self, flow_path: str):
        """Saves the current state of the flow graph to a file.

        Supports multiple formats based on file extension:
        - .yaml / .yml: New YAML format
        - .json: JSON format

        Args:
            flow_path: The path where the flow file will be saved.
        """
        logger.info("Saving flow to %s", flow_path)
        path = Path(flow_path)
        os.makedirs(path.parent, exist_ok=True)
        suffix = path.suffix.lower()
        new_flow_name = path.name.replace(suffix, "")
        self._handle_flow_renaming(new_flow_name, path)
        self.flow_settings.modified_on = datetime.datetime.now().timestamp()
        self._validate_registration_ownership(flow_path)
        try:
            if suffix == ".flowfile":
                raise DeprecationWarning(
                    "The .flowfile format is deprecated. Please use .yaml or .json formats.\n\n"
                    "Or stay on.1 if you still need .flowfile support.\n\n"
                )
            elif suffix in (".yaml", ".yml"):
                flowfile_data = self.get_flowfile_data()
                data = flowfile_data.model_dump(mode="json")
                with open(flow_path, "w", encoding="utf-8") as f:
                    yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)
            elif suffix == ".json":
                flowfile_data = self.get_flowfile_data()
                data = flowfile_data.model_dump(mode="json")
                with open(flow_path, "w", encoding="utf-8") as f:
                    json.dump(data, f, indent=2, ensure_ascii=False)

            else:
                flowfile_data = self.get_flowfile_data()
                logger.warning(f"Unknown file extension {suffix}. Defaulting to YAML format.")
                data = flowfile_data.model_dump(mode="json")
                with open(flow_path, "w", encoding="utf-8") as f:
                    yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)

        except Exception as e:
            logger.error(f"Error saving flow: {e}")
            raise

        self.flow_settings.path = flow_path
        self._sync_catalog_read_links()
        # Record the current state as the clean baseline for dirty tracking
        self.mark_as_saved()

    def _owns_registration(self, repo: SQLAlchemyCatalogRepository, registration_id: int, own_path: str | None) -> bool:
        """Whether ``registration_id`` really points at ``own_path``, this flow's file.

        Registration ids are machine-local and can alias another flow: SQLite reuses
        rowids after a registration is deleted, and a copied YAML carries the original's
        id. Since the read-link sync prunes, syncing under an aliased id would wipe the
        other flow's lineage. Renames never move ``flow_path``, so path identity is the
        authoritative check.
        """
        registration = repo.get_flow(registration_id)
        if registration is None or not registration.flow_path or not own_path:
            return False
        return os.path.realpath(registration.flow_path) == os.path.realpath(own_path)

    def _validate_registration_ownership(self, flow_path: str) -> None:
        """Re-point (or drop) a ``source_registration_id`` that isn't this file's own.

        Runs before the save serializes settings, so the corrected id is what lands in the
        file. Clearing it after the write would be resurrected on the next open —
        ``resolve_source_registration_id`` keeps any non-None stored id — leaving the flow's
        read-link sync permanently wedged. A transient DB failure never drops the id.
        """
        registration_id = getattr(self._flow_settings, "source_registration_id", None)
        if not registration_id:
            return
        try:
            with get_db_context() as db:
                repo = SQLAlchemyCatalogRepository(db)
                if self._owns_registration(repo, registration_id, flow_path):
                    return
                own_id = CatalogService(repo).resolve_registration_id(flow_path)
            logger.warning(
                "Registration %s does not belong to flow '%s' (reused id or copied flow file); re-resolved to %s",
                registration_id,
                flow_path,
                own_id,
            )
            self._flow_settings.source_registration_id = own_id
        except Exception:
            logger.warning("Could not validate the catalog registration of flow '%s'", flow_path, exc_info=True)

    def _sync_catalog_read_links(self):
        """Record which catalog tables this flow reads from.

        Scans all nodes for catalog_reader types and replaces the flow's read
        links with exactly that set, so removing a reader drops its link. Runs
        at save time so that source_registration_id is guaranteed to be set.
        """
        registration_id = self._flow_settings.source_registration_id
        logger.debug("Found registration_id %s", registration_id)
        if not registration_id:
            return

        table_ids = set()
        for node in self.nodes:
            if node.node_type != "catalog_reader":
                continue
            setting = node.setting_input
            table_id = getattr(setting, "catalog_table_id", None)
            if table_id:
                table_ids.add(table_id)

        try:
            with get_db_context() as db:
                repo = SQLAlchemyCatalogRepository(db)
                own_path = self._flow_settings.path or self._flow_settings.save_location
                if not self._owns_registration(repo, registration_id, own_path):
                    # Backstop for callers that set the id without going through save_flow.
                    logger.warning(
                        "Registration %s does not belong to flow '%s' (reused id or copied flow file); "
                        "clearing it and skipping the catalog read-link sync",
                        registration_id,
                        own_path,
                    )
                    self._flow_settings.source_registration_id = None
                    return
                repo.replace_read_links(registration_id, table_ids)
        except Exception:
            logger.warning(
                "Failed to record catalog read links for tables %s",
                table_ids,
                exc_info=True,
            )

    def get_frontend_data(self) -> dict:
        """Formats the graph structure into a JSON-like dictionary for a specific legacy frontend.

        This method transforms the graph's state into a format compatible with the
        Drawflow.js library.

        Returns:
            A dictionary representing the graph in Drawflow format.
        """
        result = {"Home": {"data": {}}}
        flow_info: schemas.FlowInformation = self.get_node_storage()

        for node_id, node_info in flow_info.data.items():
            if node_info.is_setup:
                try:
                    pos_x = node_info.data.pos_x
                    pos_y = node_info.data.pos_y
                    result["Home"]["data"][str(node_id)] = {
                        "id": node_info.id,
                        "name": node_info.type,
                        "data": {},
                        "class": node_info.type,
                        "html": node_info.type,
                        "typenode": "vue",
                        "inputs": {},
                        "outputs": {},
                        "pos_x": pos_x,
                        "pos_y": pos_y,
                    }
                except Exception as e:
                    logger.error(e)
            if node_info.outputs:
                outputs = {o: 0 for o in node_info.outputs}
                for o in node_info.outputs:
                    outputs[o] += 1
                connections = []
                for output_node_id, _n_connections in outputs.items():
                    leading_to_node = self.get_node(output_node_id)
                    input_types = leading_to_node.get_input_type(node_info.id)
                    for input_type in input_types:
                        if input_type == "main":
                            input_frontend_id = "input_1"
                        elif input_type == "right":
                            input_frontend_id = "input_2"
                        elif input_type == "left":
                            input_frontend_id = "input_3"
                        else:
                            input_frontend_id = "input_1"
                        connection = {"node": str(output_node_id), "input": input_frontend_id}
                        connections.append(connection)

                result["Home"]["data"][str(node_id)]["outputs"]["output_1"] = {"connections": connections}
            else:
                result["Home"]["data"][str(node_id)]["outputs"] = {"output_1": {"connections": []}}

            if (
                node_info.left_input_id is not None
                or node_info.right_input_id is not None
                or node_info.input_ids is not None
            ):
                main_inputs = node_info.main_input_ids
                result["Home"]["data"][str(node_id)]["inputs"]["input_1"] = {
                    "connections": [{"node": str(main_node_id), "input": "output_1"} for main_node_id in main_inputs]
                }
                if node_info.right_input_id is not None:
                    result["Home"]["data"][str(node_id)]["inputs"]["input_2"] = {
                        "connections": [{"node": str(node_info.right_input_id), "input": "output_1"}]
                    }
                if node_info.left_input_id is not None:
                    result["Home"]["data"][str(node_id)]["inputs"]["input_3"] = {
                        "connections": [{"node": str(node_info.left_input_id), "input": "output_1"}]
                    }
        return result

    def get_vue_flow_input(self) -> schemas.VueFlowInput:
        """Formats the graph's nodes and edges into a schema suitable for the VueFlow frontend.

        Returns:
            A VueFlowInput object.
        """
        edges: list[schemas.NodeEdge] = []
        nodes: list[schemas.NodeInput] = []
        for node in self.nodes:
            nodes.append(node.get_node_input())
            edges.extend(node.get_edge_input())
        groups = [
            schemas.FlowfileGroup(**self._groups[group_id].model_dump())
            for group_id in self._groups
            if self._member_node_ids(group_id) or self._child_group_ids(group_id)
        ]
        return schemas.VueFlowInput(node_edges=edges, node_inputs=nodes, groups=groups)

    def reset(self):
        """Forces a deep reset on all nodes in the graph."""

        for node in self.nodes:
            node.reset(True)

    def copy_node(
        self, new_node_settings: input_schema.NodePromise, existing_setting_input: Any, node_type: str
    ) -> None:
        """Creates a copy of an existing node.

        Args:
            new_node_settings: The promise containing new settings (like ID and position).
            existing_setting_input: The settings object from the node being copied.
            node_type: The type of the node being copied.
        """
        # A custom node whose type isn't installed needs a placeholder template before
        # the promise can be placed (mirrors the flow-restore path).
        if getattr(existing_setting_input, "is_user_defined", False) and node_type not in CUSTOM_NODE_STORE:
            register_missing_node_template(node_type)
        self.add_node_promise(new_node_settings)

        if isinstance(existing_setting_input, input_schema.NodePromise):
            return

        combined_settings = combine_existing_settings_and_new_settings(existing_setting_input, new_node_settings)
        # Subflow port names must stay unique; auto-rename the copy so it doesn't collide with the source.
        if node_type == "flow_output" and isinstance(combined_settings, input_schema.NodeFlowOutput):
            combined_settings.output_name = self._unique_subflow_port_name(
                combined_settings.output_name, node_type, combined_settings.node_id
            )
        elif node_type == "flow_input" and isinstance(combined_settings, input_schema.NodeFlowInput):
            combined_settings.input_name = self._unique_subflow_port_name(
                combined_settings.input_name, node_type, combined_settings.node_id
            )
        try:
            if getattr(existing_setting_input, "is_user_defined", False):
                self._place_user_defined_node(node_type, combined_settings)
            else:
                getattr(self, f"add_{node_type}")(combined_settings)
        except Exception:
            # A failed copy must not leave the pre-added promise dangling in the graph.
            if self.get_node(new_node_settings.node_id) is not None:
                self.delete_node(new_node_settings.node_id)
            raise

    def _unique_subflow_port_name(self, desired_name: str, node_type: str, exclude_node_id: int) -> str:
        """Return a subflow port name not already used by another flow_input/flow_output node.

        Used when copying: duplicating a 'result' output yields 'result_1', 'result_2', …
        (a trailing '_<n>' is stripped first so copies of copies keep incrementing the base).
        """
        attr = "output_name" if node_type == "flow_output" else "input_name"
        taken = {
            getattr(node.setting_input, attr, None)
            for node in self.nodes
            if node.node_type == node_type and node.node_id != exclude_node_id
        }
        taken.discard(None)
        if desired_name not in taken:
            return desired_name
        base, _, suffix = desired_name.rpartition("_")
        base = base if base and suffix.isdigit() else desired_name
        counter = 1
        while f"{base}_{counter}" in taken:
            counter += 1
        return f"{base}_{counter}"

    def generate_code(self):
        """Generates code for the flow graph.
        This method exports the flow graph to a Polars-compatible format.
        """
        from flowfile_core.flowfile.code_generator.code_generator import export_flow_to_polars

        print(export_flow_to_polars(self))

execution_location property writable

Gets the current execution location.

execution_mode property writable

Gets the current execution mode ('Development' or 'Performance').

flow_id property writable

Gets the unique identifier of the flow.

graph_has_functions property

Checks if the graph has any nodes.

graph_has_input_data property

Checks if the graph has an initial input data source.

node_connections property

Computes and returns a list of all connections in the graph.

Returns:

Type	Description
list[tuple[int, int]]	A list of tuples, where each tuple is a (source_id, target_id) pair.

nodes property

Gets a list of all FlowNode objects in the graph.

__init__(flow_settings, name=None, input_cols=None, output_cols=None, path_ref=None, input_flow=None, cache_results=False)

Initializes a new FlowGraph instance.

Parameters:

Name	Type	Description	Default
flow_settings	FlowSettings | FlowGraphConfig	The configuration settings for the flow.	required
name	str	The name of the flow.	None
input_cols	list[str]	A list of input column names.	None
output_cols	list[str]	A list of output column names.	None
path_ref	str	An optional path to an initial data source.	None
input_flow	Union[ParquetFile, FlowDataEngine, FlowGraph]	An optional existing data object to start the flow with.	None
cache_results	bool	A global flag to enable or disable result caching.	False

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def __init__(
    self,
    flow_settings: schemas.FlowSettings | schemas.FlowGraphConfig,
    name: str = None,
    input_cols: list[str] = None,
    output_cols: list[str] = None,
    path_ref: str = None,
    input_flow: Union[ParquetFile, FlowDataEngine, "FlowGraph"] = None,
    cache_results: bool = False,
):
    """Initializes a new FlowGraph instance.

    Args:
        flow_settings: The configuration settings for the flow.
        name: The name of the flow.
        input_cols: A list of input column names.
        output_cols: A list of output column names.
        path_ref: An optional path to an initial data source.
        input_flow: An optional existing data object to start the flow with.
        cache_results: A global flag to enable or disable result caching.
    """
    if isinstance(flow_settings, schemas.FlowGraphConfig):
        flow_settings = schemas.FlowSettings.from_flow_settings_input(flow_settings)

    self._flow_settings = flow_settings
    self.uuid = str(uuid1())
    self.start_datetime = None
    self.end_datetime = None
    self.latest_run_info = None
    self._flow_id = flow_settings.flow_id
    self.flow_logger = FlowLogger(flow_settings.flow_id)
    self._flow_starts: list[FlowNode] = []
    self._results = None
    self.schema = None
    self.has_over_row_function = False
    self._input_cols = [] if input_cols is None else input_cols
    self._output_cols = [] if output_cols is None else output_cols
    self._node_ids = []
    self._node_db = {}
    # Visual node groups: organizational only, never read by the executor.
    # Membership lives on each node's setting_input.group_id; this is the box registry.
    self._groups: dict[int, schemas.GroupInformation] = {}
    self._group_id_seq: int = 0  # monotonic group-id allocator; never reuses a freed id
    self._active_group_id: int | None = None
    # Serializes claiming flow_settings.is_running: the bare check-then-set in the
    # run entry points raced when callers arrive from non-asyncio threads.
    self._run_claim_lock = threading.Lock()
    self.cache_results = cache_results
    self.__name__ = name if name else "flow_" + str(id(self))
    self.depends_on = {}
    self.artifact_context = ArtifactContext()
    # Subflow recursion guards: resolved paths of every ancestor flow file and
    # this graph's nesting depth. Attributes (not contextvars) because stages
    # execute on ThreadPoolExecutor threads.
    self._subflow_ancestry: frozenset[str] = frozenset()
    self._subflow_depth: int = 0
    # Last user_id seen on any node settings (stamped by the editor routes /
    # open_flow). Lets restore_from_snapshot re-stamp the owner even when the
    # live graph is empty at undo time (snapshots intentionally omit user_id).
    self._owner_user_id: int | None = None

    from flowfile_core.flowfile.history_manager import HistoryManager
    from flowfile_core.schemas.history_schema import HistoryConfig

    history_config = HistoryConfig(enabled=flow_settings.track_history)
    self._history_manager = HistoryManager(config=history_config)

    if path_ref is not None:
        self.add_datasource(input_schema.NodeDatasource(file_path=path_ref))
    elif input_flow is not None:
        self.add_datasource(input_file=input_flow)

    # Mark the empty initial state as the saved baseline so an unmodified
    # flow is not considered dirty.
    self._history_manager.mark_saved(self)

__repr__()

Provides the official string representation of the FlowGraph instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def __repr__(self):
    """Provides the official string representation of the FlowGraph instance."""
    settings_str = "  -" + "\n  -".join(f"{k}: {v}" for k, v in self.flow_settings)
    return f"FlowGraph(\nNodes: {self._node_db}\n\nSettings:\n{settings_str}"

add_api_response(api_response)

Adds an API-response sink node.

The node is a pass-through marker: its result equals its input. When the flow is published as an HTTP API endpoint, the endpoint reads this node's result and serializes it as the response body. Behaves like an output node so its result is always materialized locally.

Parameters:

Name	Type	Description	Default
api_response	NodeApiResponse	The settings for the API-response node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_api_response(self, api_response: input_schema.NodeApiResponse):
    """Adds an API-response sink node.

    The node is a pass-through marker: its result equals its input. When the flow
    is published as an HTTP API endpoint, the endpoint reads this node's result
    and serializes it as the response body. Behaves like an output node so its
    result is always materialized locally.

    Args:
        api_response: The settings for the API-response node.
    """

    def _func(df: FlowDataEngine):
        return df

    def schema_callback():
        input_node: FlowNode = self.get_node(api_response.node_id).node_inputs.main_inputs[0]
        return input_node.schema

    input_node_id = api_response.depending_on_id if hasattr(api_response, "depending_on_id") else None
    self.add_node_step(
        node_id=api_response.node_id,
        function=_func,
        input_columns=[],
        node_type="api_response",
        setting_input=api_response,
        schema_callback=schema_callback,
        input_node_ids=[input_node_id],
    )

add_apply_model(apply_settings)

Adds an Apply Model node.

Fetches the artifact from the catalog and asks the worker to score the input data, returning a LazyFrame with one extra Float64 column.

Parameters:

Name	Type	Description	Default
apply_settings	NodeApplyModel	Settings (model_name, optional version, output_column).	required

Returns:

Name	Type	Description
The	FlowGraph	class:FlowGraph instance for chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_apply_model(self, apply_settings: input_schema.NodeApplyModel) -> "FlowGraph":
    """Adds an Apply Model node.

    Fetches the artifact from the catalog and asks the worker to score the
    input data, returning a LazyFrame with one extra ``Float64`` column.

    Args:
        apply_settings: Settings (model_name, optional version, output_column).

    Returns:
        The :class:`FlowGraph` instance for chaining.
    """

    def _func(data: FlowDataEngine) -> FlowDataEngine:
        from flowfile_core.artifacts import get_storage_backend
        from flowfile_core.artifacts.service import ArtifactService

        settings = apply_settings.apply_input
        if not settings.output_column:
            raise ValueError("Apply Model requires an 'output_column'.")

        model_path: str
        origin_label: str

        if settings.source == "upstream":
            if settings.upstream_node_id is None:
                raise ValueError(
                    "Apply Model: 'upstream_node_id' is required when source='upstream'. "
                    "Pick a Train Model node in the drawer or switch to 'catalog' source."
                )
            upstream = self.get_node(node_id=settings.upstream_node_id)
            if upstream is None or upstream.node_type != "train_model":
                raise ValueError(
                    f"Apply Model: upstream node {settings.upstream_node_id} is not a Train Model node."
                )
            flow_path = ml_flow_model_path(self.flow_id, settings.upstream_node_id)
            if not flow_path.exists():
                raise ValueError(
                    f"Apply Model: upstream Train Model (node {settings.upstream_node_id}) "
                    "has not produced a model yet. Make sure it runs before this node "
                    "(e.g. with a Wait For barrier)."
                )
            model_path = str(flow_path)
            origin_label = f"upstream node {settings.upstream_node_id}"
        else:
            if not settings.model_name:
                raise ValueError("Apply Model: 'model_name' is required when source='catalog'.")
            storage_backend = get_storage_backend()
            with get_db_context() as db:
                effective_namespace_id = _effective_namespace_id(
                    CatalogService(SQLAlchemyCatalogRepository(db)), settings
                )
                artifact = ArtifactService(db, storage_backend).get_artifact_by_name(
                    name=settings.model_name,
                    namespace_id=effective_namespace_id,
                    version=settings.model_version,
                )
            if artifact.download_source is None or artifact.download_source.method != "file":
                raise ValueError(
                    "Apply Model currently requires the filesystem artifact backend "
                    "(FLOWFILE_ARTIFACT_STORAGE=filesystem). S3 support is not implemented."
                )
            model_path = artifact.download_source.path
            if not os.path.exists(model_path):
                raise ValueError(
                    f"Apply Model: data for catalog model '{settings.model_name}' "
                    f"v{artifact.version} (namespace {artifact.namespace_id}) is missing "
                    f"at {model_path}. If running in Docker, ensure the shared artifacts "
                    "volume is mounted into both core and the worker."
                )
            origin_label = f"catalog '{settings.model_name}' v{artifact.version}"

        node = self.get_node(node_id=apply_settings.node_id)
        fetcher = MLApplyFetcher(
            lf=data.data_frame,
            model_path=model_path,
            output_column=settings.output_column,
            flow_id=self.flow_id,
            node_id=apply_settings.node_id,
            file_ref=node.hash,
            wait_on_completion=False,
        )
        node._fetch_cached_df = fetcher
        result_lf = fetcher.get_result()
        self.flow_logger.info(f"Apply Model: scored using {origin_label} -> column '{settings.output_column}'")
        return FlowDataEngine(result_lf)

    def schema_callback():
        input_node: FlowNode = self.get_node(apply_settings.node_id).node_inputs.main_inputs[0]
        input_schema_cols = list(input_node.schema)
        s = apply_settings.apply_input
        output_column = s.output_column or "prediction"
        # source='upstream' lets us read the trainer's declared output_dtype
        # so a future non-Float64 trainer (e.g. classification) gets the
        # right schema. source='catalog' falls back to Float64 — resolving
        # via the catalog DB at schema-resolve time would be too eager.
        output_dtype = "Float64"
        if s.source == "upstream" and s.upstream_node_id is not None:
            upstream = self.get_node(s.upstream_node_id)
            train_input = getattr(getattr(upstream, "setting_input", None), "train_input", None)
            model_type = getattr(train_input, "model_type", None)
            if model_type:
                try:
                    from shared.ml.trainers import get_trainer

                    output_dtype = get_trainer(model_type).output_dtype
                except ValueError:
                    pass
        return input_schema_cols + [FlowfileColumn.from_input(output_column, output_dtype)]

    depending_on_id = apply_settings.depending_on_id if hasattr(apply_settings, "depending_on_id") else None
    self.add_node_step(
        node_id=apply_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="apply_model",
        setting_input=apply_settings,
        schema_callback=schema_callback,
        input_node_ids=[depending_on_id] if depending_on_id is not None else None,
    )
    return self

add_catalog_reader(node_catalog_reader)

Adds a node that reads a table from the catalog.

Resolves the catalog table by ID (or name + namespace) and reads the materialized Parquet file. When sql_query is set, executes the SQL against all catalog Delta tables instead.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_catalog_reader(self, node_catalog_reader: input_schema.NodeCatalogReader):
    """Adds a node that reads a table from the catalog.

    Resolves the catalog table by ID (or name + namespace) and reads
    the materialized Parquet file.  When ``sql_query`` is set, executes
    the SQL against all catalog Delta tables instead.
    """

    if node_catalog_reader.sql_query:
        is_virtual_optimized = self._add_catalog_sql_reader(node_catalog_reader)
    else:
        is_virtual_optimized = self._add_catalog_table_reader(node_catalog_reader)
    node_catalog_reader.is_virtual_optimized = is_virtual_optimized

add_catalog_writer(node_catalog_writer)

Adds a node that writes its input to the catalog as a Delta table or virtual table.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_catalog_writer(self, node_catalog_writer: input_schema.NodeCatalogWriter):
    """Adds a node that writes its input to the catalog as a Delta table or virtual table."""

    def _func(df: FlowDataEngine) -> FlowDataEngine:
        settings = node_catalog_writer.catalog_write_settings
        if not settings.table_name:
            raise ValueError("Catalog writer requires a table name")
        if settings.write_mode == "virtual":
            return _handle_virtual_table_write(self, node_catalog_writer, df)
        return _handle_physical_table_write(self, node_catalog_writer, df)

    def schema_callback():
        input_node: FlowNode = self.get_node(node_catalog_writer.node_id).node_inputs.main_inputs[0]
        return input_node.schema

    input_node_id = node_catalog_writer.depending_on_id if hasattr(node_catalog_writer, "depending_on_id") else None
    self.add_node_step(
        node_id=node_catalog_writer.node_id,
        function=_func,
        input_columns=[],
        node_type="catalog_writer",
        setting_input=node_catalog_writer,
        schema_callback=schema_callback,
        input_node_ids=[input_node_id],
    )

add_cloud_storage_reader(node_cloud_storage_reader)

Adds a cloud storage read node to the flow graph.

Parameters:

Name	Type	Description	Default
node_cloud_storage_reader	NodeCloudStorageReader	The settings for the cloud storage read node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_cloud_storage_reader(self, node_cloud_storage_reader: input_schema.NodeCloudStorageReader) -> None:
    """Adds a cloud storage read node to the flow graph.

    Args:
        node_cloud_storage_reader: The settings for the cloud storage read node.
    """
    node_type = "cloud_storage_reader"
    logger.info("Adding cloud storage reader")
    cloud_storage_read_settings = node_cloud_storage_reader.cloud_storage_settings

    def _func():
        logger.info("Starting to run the schema callback for cloud storage reader")
        self.flow_logger.info("Starting to run the schema callback for cloud storage reader")
        settings = CloudStorageReadSettingsInternal(
            read_settings=cloud_storage_read_settings,
            connection=get_cloud_connection_settings(
                connection_name=cloud_storage_read_settings.connection_name,
                user_id=node_cloud_storage_reader.user_id,
                auth_mode=cloud_storage_read_settings.auth_mode,
            ),
        )
        fl = FlowDataEngine.from_cloud_storage_obj(settings)
        return fl

    node = self.add_node_step(
        node_id=node_cloud_storage_reader.node_id,
        function=_func,
        cache_results=node_cloud_storage_reader.cache_results,
        setting_input=node_cloud_storage_reader,
        node_type=node_type,
    )
    self.add_node_to_starting_list(node)

add_cloud_storage_writer(node_cloud_storage_writer)

Adds a node to write data to a cloud storage provider.

Parameters:

Name	Type	Description	Default
node_cloud_storage_writer	NodeCloudStorageWriter	The settings for the cloud storage writer node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_cloud_storage_writer(self, node_cloud_storage_writer: input_schema.NodeCloudStorageWriter) -> None:
    """Adds a node to write data to a cloud storage provider.

    Args:
        node_cloud_storage_writer: The settings for the cloud storage writer node.
    """

    node_type = "cloud_storage_writer"

    def _func(df: FlowDataEngine):
        df.lazy = True
        execute_remote = self.execution_location != "local"
        cloud_connection_settings = get_cloud_connection_settings(
            connection_name=node_cloud_storage_writer.cloud_storage_settings.connection_name,
            user_id=node_cloud_storage_writer.user_id,
            auth_mode=node_cloud_storage_writer.cloud_storage_settings.auth_mode,
        )
        full_cloud_storage_connection = cloud_connection_settings
        if execute_remote:
            settings = get_cloud_storage_write_settings_worker_interface(
                write_settings=node_cloud_storage_writer.cloud_storage_settings,
                connection=full_cloud_storage_connection,
                lf=df.data_frame,
                user_id=node_cloud_storage_writer.user_id,
                flowfile_node_id=node_cloud_storage_writer.node_id,
                flowfile_flow_id=self.flow_id,
            )
            external_database_writer = ExternalCloudWriter(settings, wait_on_completion=False)
            node._fetch_cached_df = external_database_writer
            external_database_writer.get_result()
        else:
            cloud_storage_write_settings_internal = CloudStorageWriteSettingsInternal(
                connection=full_cloud_storage_connection,
                write_settings=node_cloud_storage_writer.cloud_storage_settings,
            )
            df.to_cloud_storage_obj(cloud_storage_write_settings_internal)
        return df

    def schema_callback():
        logger.info("Starting to run the schema callback for cloud storage writer")
        if self.get_node(node_cloud_storage_writer.node_id).is_correct:
            return self.get_node(node_cloud_storage_writer.node_id).node_inputs.main_inputs[0].schema
        else:
            return [FlowfileColumn.from_input(column_name="__error__", data_type="String")]

    self.add_node_step(
        node_id=node_cloud_storage_writer.node_id,
        function=_func,
        input_columns=[],
        node_type=node_type,
        setting_input=node_cloud_storage_writer,
        schema_callback=schema_callback,
        input_node_ids=[node_cloud_storage_writer.depending_on_id],
    )

    node = self.get_node(node_cloud_storage_writer.node_id)

add_cross_join(cross_join_settings)

Adds a cross join node to the graph.

Parameters:

Name	Type	Description	Default
cross_join_settings	NodeCrossJoin	The settings for the cross join operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_cross_join(self, cross_join_settings: input_schema.NodeCrossJoin) -> "FlowGraph":
    """Adds a cross join node to the graph.

    Args:
        cross_join_settings: The settings for the cross join operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
        for left_select in cross_join_settings.cross_join_input.left_select.renames:
            left_select.is_available = True if left_select.old_name in main.schema else False
        for right_select in cross_join_settings.cross_join_input.right_select.renames:
            right_select.is_available = True if right_select.old_name in right.schema else False
        return main.do_cross_join(
            cross_join_input=cross_join_settings.cross_join_input,
            auto_generate_selection=cross_join_settings.auto_generate_selection,
            verify_integrity=False,
            other=right,
        )

    def schema_callback():
        cj_copy = CrossJoinInputManager(cross_join_settings.cross_join_input)
        node = self.get_node(node_id=cross_join_settings.node_id)
        return calculate_cross_join_schema(
            cj_copy,
            left_schema=node.node_inputs.main_inputs[0].schema,
            right_schema=node.node_inputs.right_input.schema,
        )

    self.add_node_step(
        node_id=cross_join_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="cross_join",
        setting_input=cross_join_settings,
        input_node_ids=cross_join_settings.depending_on_ids,
        schema_callback=schema_callback,
    )
    return self

add_database_reader(node_database_reader)

Adds a node to read data from a database.

Parameters:

Name	Type	Description	Default
node_database_reader	NodeDatabaseReader	The settings for the database reader node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_database_reader(self, node_database_reader: input_schema.NodeDatabaseReader):
    """Adds a node to read data from a database.

    Args:
        node_database_reader: The settings for the database reader node.
    """

    logger.info("Adding database reader")
    node_type = "database_reader"
    database_settings: input_schema.DatabaseSettings = node_database_reader.database_settings

    # Resolve the connection lazily so opening/undoing a flow never requires
    # the current session to own the connection. Memoized so ``_func`` and
    # ``schema_callback`` share a single lookup; the lock matters because the
    # schema callback runs on a background thread (``SingleExecutionFuture``)
    # while ``_func`` runs on the execution thread. Runs under the node's
    # ``user_id`` (the flow owner at execution time).
    _creds: dict = {}
    _creds_lock = threading.Lock()

    def _get_creds():
        with _creds_lock:
            if "v" not in _creds:
                _creds["v"] = _resolve_database_credentials(database_settings, node_database_reader.user_id)
            return _creds["v"]

    def _func():
        database_connection, encrypted_password, database_reference_settings = _get_creds()
        sql_source = BaseSqlSource(
            query=None if database_settings.query_mode == "table" else database_settings.query,
            table_name=database_settings.table_name,
            schema_name=database_settings.schema_name,
            fields=node_database_reader.fields,
        )

        # Local and worker reads share shared.db_reader.read_sql_with_fallback
        # (via SqlSource here, via read_sql_source in the worker).
        if self.execution_location == "local":
            local_source = SqlSource(
                connection_string=sql_utils.construct_sql_uri(
                    database_type=database_connection.database_type,
                    host=database_connection.host,
                    port=database_connection.port,
                    database=database_connection.database,
                    username=database_connection.username,
                    password=decrypt_secret(encrypted_password) if encrypted_password else None,
                    ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                    connect_timeout=10,
                ),
                query=None if database_settings.query_mode == "table" else database_settings.query,
                table_name=database_settings.table_name,
                schema_name=database_settings.schema_name,
                fields=node_database_reader.fields,
                cancel_check=lambda: self.flow_settings.is_canceled or node._execution_state.is_canceled,
                database_type=database_connection.database_type,
            )
            fl = FlowDataEngine(local_source.get_pl_df())
            fl.lazy = True
            node_database_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
            return fl

        database_external_read_settings = (
            sql_models.DatabaseExternalReadSettings.create_from_from_node_database_reader(
                node_database_reader=node_database_reader,
                password=encrypted_password,
                query=sql_source.query,
                database_reference_settings=(
                    database_reference_settings if database_settings.connection_mode == "reference" else None
                ),
            )
        )

        external_database_fetcher = ExternalDatabaseFetcher(
            database_external_read_settings, wait_on_completion=False
        )
        node._fetch_cached_df = external_database_fetcher
        fl = FlowDataEngine(external_database_fetcher.get_result())
        node_database_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
        return fl

    def schema_callback():
        # Prefer the schema cached on the node so opening a saved flow renders
        # columns without a live connection. Fall back to the connection only
        # when fields were never captured (failures here are caught per-node).
        if node_database_reader.fields:
            return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_database_reader.fields]
        database_connection, encrypted_password, _ = _get_creds()
        sql_source = SqlSource(
            connection_string=sql_utils.construct_sql_uri(
                database_type=database_connection.database_type,
                host=database_connection.host,
                port=database_connection.port,
                database=database_connection.database,
                username=database_connection.username,
                password=decrypt_secret(encrypted_password) if encrypted_password else None,
                ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                connect_timeout=10,
            ),
            query=None if database_settings.query_mode == "table" else database_settings.query,
            table_name=database_settings.table_name,
            schema_name=database_settings.schema_name,
            fields=node_database_reader.fields,
            database_type=database_connection.database_type,
        )
        return sql_source.get_schema()

    node = self.get_node(node_database_reader.node_id)
    if node:
        # Persist so the lightweight callback survives the reset() that setting_input triggers.
        node.user_provided_schema_callback = schema_callback
        node.schema_callback = schema_callback
        node.node_type = node_type
        node.name = node_type
        node.function = _func
        node.setting_input = node_database_reader
        node.node_settings.cache_results = node_database_reader.cache_results
        self.add_node_to_starting_list(node)
    else:
        node = FlowNode(
            node_database_reader.node_id,
            function=_func,
            setting_input=node_database_reader,
            name=node_type,
            node_type=node_type,
            parent_uuid=self.uuid,
            schema_callback=schema_callback,
        )
        self._node_db[node_database_reader.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(node_database_reader.node_id)

add_database_writer(node_database_writer)

Adds a node to write data to a database.

Parameters:

Name	Type	Description	Default
node_database_writer	NodeDatabaseWriter	The settings for the database writer node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_database_writer(self, node_database_writer: input_schema.NodeDatabaseWriter):
    """Adds a node to write data to a database.

    Args:
        node_database_writer: The settings for the database writer node.
    """

    node_type = "database_writer"
    database_settings: input_schema.DatabaseWriteSettings = node_database_writer.database_write_settings

    def _func(df: FlowDataEngine):
        database_connection, encrypted_password, database_reference_settings = _resolve_database_credentials(
            database_settings, node_database_writer.user_id
        )
        df.lazy = True
        table_name = (
            database_settings.schema_name + "." + database_settings.table_name
            if database_settings.schema_name
            else database_settings.table_name
        )

        if self.execution_location == "local":
            df.to_database_obj(
                database_type=database_connection.database_type,
                uri=sql_utils.construct_sql_uri(
                    database_type=database_connection.database_type,
                    host=database_connection.host,
                    port=database_connection.port,
                    database=database_connection.database,
                    username=database_connection.username,
                    password=decrypt_secret(encrypted_password) if encrypted_password else None,
                    ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)),
                    connect_timeout=10,
                ),
                table_name=table_name,
                if_exists=database_settings.if_exists or "append",
            )
            return df

        database_external_write_settings = (
            sql_models.DatabaseExternalWriteSettings.create_from_from_node_database_writer(
                node_database_writer=node_database_writer,
                password=encrypted_password,
                table_name=table_name,
                database_reference_settings=(
                    database_reference_settings if database_settings.connection_mode == "reference" else None
                ),
                lf=df.data_frame,
            )
        )
        external_database_writer = ExternalDatabaseWriter(
            database_external_write_settings, wait_on_completion=False
        )
        node._fetch_cached_df = external_database_writer
        external_database_writer.get_result()
        return df

    def schema_callback():
        input_node: FlowNode = self.get_node(node_database_writer.node_id).node_inputs.main_inputs[0]
        return input_node.schema

    self.add_node_step(
        node_id=node_database_writer.node_id,
        function=_func,
        input_columns=[],
        node_type=node_type,
        setting_input=node_database_writer,
        schema_callback=schema_callback,
        input_node_ids=[node_database_writer.depending_on_id],
    )
    node = self.get_node(node_database_writer.node_id)

add_datasource(input_file)

Adds a data source node to the graph.

This method serves as a factory for creating starting nodes, handling both file-based sources and direct manual data entry.

Parameters:

Name	Type	Description	Default
input_file	NodeDatasource | NodeManualInput	The configuration object for the data source.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_datasource(self, input_file: input_schema.NodeDatasource | input_schema.NodeManualInput) -> "FlowGraph":
    """Adds a data source node to the graph.

    This method serves as a factory for creating starting nodes, handling both
    file-based sources and direct manual data entry.

    Args:
        input_file: The configuration object for the data source.

    Returns:
        The `FlowGraph` instance for method chaining.
    """
    if isinstance(input_file, input_schema.NodeManualInput):
        input_data = FlowDataEngine(input_file.raw_data_format)
        ref = "manual_input"
    else:
        input_data = FlowDataEngine(path_ref=input_file.file_ref)
        ref = "datasource"
    node = self.get_node(input_file.node_id)
    if node:
        node.node_type = ref
        node.name = ref
        node.function = input_data
        node.setting_input = input_file
        self.add_node_to_starting_list(node)

    else:
        input_data.collect()
        node = FlowNode(
            input_file.node_id,
            function=input_data,
            setting_input=input_file,
            name=ref,
            node_type=ref,
            parent_uuid=self.uuid,
        )
        self._node_db[input_file.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(input_file.node_id)
    return self

add_dependency_on_polars_lazy_frame(lazy_frame, node_id)

Adds a special node that directly injects a Polars LazyFrame into the graph.

Note: This is intended for backend use and will not work in the UI editor.

Parameters:

Name	Type	Description	Default
lazy_frame	LazyFrame	The Polars LazyFrame to inject.	required
node_id	int	The ID for the new node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_dependency_on_polars_lazy_frame(self, lazy_frame: pl.LazyFrame, node_id: int):
    """Adds a special node that directly injects a Polars LazyFrame into the graph.

    Note: This is intended for backend use and will not work in the UI editor.

    Args:
        lazy_frame: The Polars LazyFrame to inject.
        node_id: The ID for the new node.
    """

    def _func():
        return FlowDataEngine(lazy_frame)

    node_promise = input_schema.NodePromise(
        flow_id=self.flow_id, node_id=node_id, node_type="polars_lazy_frame", is_setup=True
    )
    self.add_node_step(
        node_id=node_promise.node_id, node_type=node_promise.node_type, function=_func, setting_input=node_promise
    )

add_dynamic_rename(settings)

Adds a node that renames many columns at once via a single rule.

Supports prefix, suffix, formula-based, and first-row renaming across all columns, a specific list of columns, or every column of a given data type. In first_row mode the first row is dropped from the output after its values are promoted to column headers.

Parameters:

Name	Type	Description	Default
settings	NodeDynamicRename	The dynamic rename configuration.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_dynamic_rename(self, settings: input_schema.NodeDynamicRename) -> "FlowGraph":
    """Adds a node that renames many columns at once via a single rule.

    Supports prefix, suffix, formula-based, and first-row renaming across all
    columns, a specific list of columns, or every column of a given data type.
    In `first_row` mode the first row is dropped from the output after its
    values are promoted to column headers.

    Args:
        settings: The dynamic rename configuration.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        return table.apply_dynamic_rename(settings.dynamic_rename_input)

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        node_type="dynamic_rename",
        setting_input=settings,
        input_node_ids=[settings.depending_on_id],
    )
    return self

add_evaluate_model(evaluate_settings)

Adds an Evaluate Model node.

Compares the actual and predicted columns already present on the input dataframe and emits a long-form (metric, value) frame. Pure polars — no worker offload, no model file read.

task_type="auto" resolves the metric set from the configured upstream Train Model node's trainer; otherwise uses the explicit regression / classification choice from settings.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_evaluate_model(self, evaluate_settings: input_schema.NodeEvaluateModel) -> "FlowGraph":
    """Adds an Evaluate Model node.

    Compares the *actual* and *predicted* columns already present on the
    input dataframe and emits a long-form ``(metric, value)`` frame.
    Pure polars — no worker offload, no model file read.

    ``task_type="auto"`` resolves the metric set from the configured
    upstream Train Model node's trainer; otherwise uses the explicit
    ``regression`` / ``classification`` choice from settings.
    """

    def _resolve_task_type() -> str:
        s = evaluate_settings.evaluate_input
        if s.task_type != "auto":
            return s.task_type
        if s.upstream_train_node_id is not None:
            upstream = self.get_node(s.upstream_train_node_id)
            train_input = getattr(getattr(upstream, "setting_input", None), "train_input", None)
            model_type = getattr(train_input, "model_type", None)
            if model_type:
                try:
                    from shared.ml.trainers import get_trainer

                    return get_trainer(model_type).task_type
                except ValueError:
                    pass
        return "regression"

    def _func(data: FlowDataEngine) -> FlowDataEngine:
        from shared.ml.metrics import compute_metrics

        settings = evaluate_settings.evaluate_input
        if not settings.actual_column:
            raise ValueError("Evaluate Model requires an 'actual_column'.")
        if not settings.predicted_column:
            raise ValueError("Evaluate Model requires a 'predicted_column'.")

        task_type = _resolve_task_type()
        metrics_lf = compute_metrics(
            data.data_frame,
            actual_column=settings.actual_column,
            predicted_column=settings.predicted_column,
            task_type=task_type,
        )
        self.flow_logger.info(
            f"Evaluate Model: {settings.predicted_column} vs {settings.actual_column} " f"(task_type={task_type})"
        )
        return FlowDataEngine(metrics_lf)

    def schema_callback():
        return [
            FlowfileColumn.from_input(column_name="metric", data_type="String"),
            FlowfileColumn.from_input(column_name="value", data_type="Float64"),
        ]

    depending_on_id = evaluate_settings.depending_on_id if hasattr(evaluate_settings, "depending_on_id") else None
    self.add_node_step(
        node_id=evaluate_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="evaluate_model",
        setting_input=evaluate_settings,
        schema_callback=schema_callback,
        input_node_ids=[depending_on_id] if depending_on_id is not None else None,
    )
    return self

add_explore_data(node_analysis)

Adds a specialized node for data exploration and visualization.

Parameters:

Name	Type	Description	Default
node_analysis	NodeExploreData	The settings for the data exploration node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_explore_data(self, node_analysis: input_schema.NodeExploreData):
    """Adds a specialized node for data exploration and visualization.

    Args:
        node_analysis: The settings for the data exploration node.
    """

    def analysis_preparation(flowfile_table: FlowDataEngine) -> FlowDataEngine:
        """Pass-through: Graphic Walker aggregates on the worker, not in the browser.

        Charts read the node's result plan through ``/analysis_data/compute``,
        so the run itself owes the explorer nothing.
        """
        return flowfile_table

    def schema_callback():
        node = self.get_node(node_analysis.node_id)
        if len(node.all_inputs) == 1:
            input_node = node.all_inputs[0]
            return input_node.schema
        else:
            return [FlowfileColumn.from_input("col_1", "na")]

    self.add_node_step(
        node_id=node_analysis.node_id,
        node_type="explore_data",
        function=analysis_preparation,
        setting_input=node_analysis,
        schema_callback=schema_callback,
    )

add_external_source(external_source_input)

Adds a node for a custom external data source.

Parameters:

Name	Type	Description	Default
external_source_input	NodeExternalSource	The settings for the external source node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_external_source(self, external_source_input: input_schema.NodeExternalSource):
    """Adds a node for a custom external data source.

    Args:
        external_source_input: The settings for the external source node.
    """

    node_type = "external_source"
    external_source_script = getattr(external_sources.custom_external_sources, external_source_input.identifier)
    source_settings = getattr(
        input_schema, snake_case_to_camel_case(external_source_input.identifier)
    ).model_validate(external_source_input.source_settings)
    if hasattr(external_source_script, "initial_getter"):
        initial_getter = external_source_script.initial_getter(source_settings)
    else:
        initial_getter = None
    data_getter = external_source_script.getter(source_settings)
    external_source = data_source_factory(
        source_type="custom",
        data_getter=data_getter,
        initial_data_getter=initial_getter,
        orientation=external_source_input.source_settings.orientation,
        schema=None,
    )

    def _func():
        logger.info("Calling external source")
        fl = FlowDataEngine.create_from_external_source(external_source=external_source)
        external_source_input.source_settings.fields = [c.get_minimal_field_info() for c in fl.schema]
        return fl

    node = self.get_node(external_source_input.node_id)
    if node:
        node.node_type = node_type
        node.name = node_type
        node.function = _func
        node.setting_input = external_source_input
        node.node_settings.cache_results = external_source_input.cache_results
        self.add_node_to_starting_list(node)

    else:
        node = FlowNode(
            external_source_input.node_id,
            function=_func,
            setting_input=external_source_input,
            name=node_type,
            node_type=node_type,
            parent_uuid=self.uuid,
        )
        self._node_db[external_source_input.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(external_source_input.node_id)
    if external_source_input.source_settings.fields and len(external_source_input.source_settings.fields) > 0:
        logger.info("Using provided schema in the node")

        def schema_callback():
            return [
                FlowfileColumn.from_input(f.name, f.data_type) for f in external_source_input.source_settings.fields
            ]

        node.schema_callback = schema_callback
        node.user_provided_schema_callback = schema_callback
    else:
        logger.warning("Removing schema")
        node._schema_callback = None
    self.add_node_step(
        node_id=external_source_input.node_id,
        function=_func,
        input_columns=[],
        node_type=node_type,
        setting_input=external_source_input,
    )

add_filter(filter_settings)

Adds a filter node to the graph.

Parameters:

Name	Type	Description	Default
filter_settings	NodeFilter	The settings for the filter operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_filter(self, filter_settings: input_schema.NodeFilter):
    """Adds a filter node to the graph.

    Args:
        filter_settings: The settings for the filter operation.
    """

    def _func(fl: FlowDataEngine):
        is_advanced = filter_settings.filter_input.is_advanced()

        if is_advanced:
            expression = filter_settings.filter_input.advanced_filter
        else:
            basic_filter = filter_settings.filter_input.basic_filter
            if basic_filter is None:
                logger.warning("Basic filter is None, returning unfiltered data")
                return fl

            try:
                field_data_type = fl.get_schema_column(basic_filter.field).generic_datatype()
            except Exception:
                field_data_type = None

            expression = build_filter_expression(basic_filter, field_data_type)
            filter_settings.filter_input.advanced_filter = expression

        if filter_settings.split_mode:
            return fl.filter_split(expression)
        return fl.do_filter(expression)

    self.add_node_step(
        filter_settings.node_id,
        _func,
        node_type="filter",
        renew_schema=False,
        setting_input=filter_settings,
        input_node_ids=[filter_settings.depending_on_id],
    )

add_flow_input(settings)

Adds a named subflow-input placeholder source.

Standalone runs serve the optional sample data (empty frame otherwise); a parent run_flow node overwrites node.function with real data.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_flow_input(self, settings: input_schema.NodeFlowInput) -> "FlowGraph":
    """Adds a named subflow-input placeholder source.

    Standalone runs serve the optional sample data (empty frame otherwise);
    a parent run_flow node overwrites ``node.function`` with real data.
    """
    for other in self.nodes:
        if (
            other.node_type == "flow_input"
            and other.node_id != settings.node_id
            and isinstance(other.setting_input, input_schema.NodeFlowInput)
            and other.setting_input.input_name == settings.input_name
        ):
            raise ValueError(f"flow_input name '{settings.input_name}' is already used by node {other.node_id}")
    if settings.raw_data_format is not None and settings.raw_data_format.columns:
        input_data = FlowDataEngine(settings.raw_data_format)
    else:
        input_data = FlowDataEngine()
    node = self.get_node(settings.node_id)
    if node:
        node.node_type = "flow_input"
        node.name = "flow_input"
        node.function = input_data
        node.setting_input = settings
        self.add_node_to_starting_list(node)
    else:
        node = FlowNode(
            settings.node_id,
            function=input_data,
            setting_input=settings,
            name="flow_input",
            node_type="flow_input",
            parent_uuid=self.uuid,
        )
        self._node_db[settings.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(settings.node_id)
    return self

add_flow_output(settings)

Adds a named subflow-output sink (passthrough, always materialized).

When this flow runs inside another flow via a run_flow node, the parent reads this node's result as one of the subflow's outputs.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_flow_output(self, settings: input_schema.NodeFlowOutput) -> "FlowGraph":
    """Adds a named subflow-output sink (passthrough, always materialized).

    When this flow runs inside another flow via a run_flow node, the parent
    reads this node's result as one of the subflow's outputs.
    """
    for other in self.nodes:
        if (
            other.node_type == "flow_output"
            and other.node_id != settings.node_id
            and isinstance(other.setting_input, input_schema.NodeFlowOutput)
            and other.setting_input.output_name == settings.output_name
        ):
            raise ValueError(f"flow_output name '{settings.output_name}' is already used by node {other.node_id}")

    def _func(df: FlowDataEngine):
        return df

    def schema_callback():
        node: FlowNode = self.get_node(settings.node_id)
        if node.node_inputs.main_inputs:
            return node.node_inputs.main_inputs[0].schema
        return []

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        input_columns=[],
        node_type="flow_output",
        setting_input=settings,
        schema_callback=schema_callback,
        input_node_ids=[settings.depending_on_id],
    )
    return self

add_formula(function_settings)

Adds a node that applies a formula to create or modify a column.

Parameters:

Name	Type	Description	Default
function_settings	NodeFormula	The settings for the formula operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_formula(self, function_settings: input_schema.NodeFormula):
    """Adds a node that applies a formula to create or modify a column.

    Args:
        function_settings: The settings for the formula operation.
    """

    error = ""
    if function_settings.function.field.data_type not in (None, transform_schema.AUTO_DATA_TYPE):
        output_type = cast_str_to_polars_type(function_settings.function.field.data_type)
    else:
        output_type = None
    if output_type not in (None, transform_schema.AUTO_DATA_TYPE):
        new_col = [
            FlowfileColumn.from_input(column_name=function_settings.function.field.name, data_type=str(output_type))
        ]
    else:
        new_col = [FlowfileColumn.from_input(function_settings.function.field.name, "String")]

    def _func(fl: FlowDataEngine):
        return fl.apply_sql_formula(
            func=function_settings.function.function,
            col_name=function_settings.function.field.name,
            output_data_type=output_type,
        )

    self.add_node_step(
        function_settings.node_id,
        _func,
        output_schema=new_col,
        node_type="formula",
        renew_schema=False,
        setting_input=function_settings,
        input_node_ids=[function_settings.depending_on_id],
    )
    if error != "":
        node = self.get_node(function_settings.node_id)
        node.results.errors = error
        return False, error
    else:
        return True, ""

add_fuzzy_match(fuzzy_settings)

Adds a fuzzy matching node to join data on approximate string matches.

Parameters:

Name	Type	Description	Default
fuzzy_settings	NodeFuzzyMatch	The settings for the fuzzy match operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_fuzzy_match(self, fuzzy_settings: input_schema.NodeFuzzyMatch) -> "FlowGraph":
    """Adds a fuzzy matching node to join data on approximate string matches.

    Args:
        fuzzy_settings: The settings for the fuzzy match operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
        node = self.get_node(node_id=fuzzy_settings.node_id)
        if self.execution_location == "local":
            return main.fuzzy_join(
                fuzzy_match_input=deepcopy(fuzzy_settings.join_input),
                other=right,
                node_logger=self.flow_logger.get_node_logger(fuzzy_settings.node_id),
            )

        f = main.start_fuzzy_join(
            fuzzy_match_input=deepcopy(fuzzy_settings.join_input),
            other=right,
            file_ref=node.hash,
            flow_id=self.flow_id,
            node_id=fuzzy_settings.node_id,
        )
        logger.info("Started the fuzzy match action")
        node._fetch_cached_df = f  # Add to the node so it can be cancelled and fetch later if needed
        return FlowDataEngine(f.get_result())

    def schema_callback():
        fm_input_copy = FuzzyMatchInputManager(
            fuzzy_settings.join_input
        )  # Deepcopy create an unique object per func
        node = self.get_node(node_id=fuzzy_settings.node_id)
        return calculate_fuzzy_match_schema(
            fm_input_copy,
            left_schema=node.node_inputs.main_inputs[0].schema,
            right_schema=node.node_inputs.right_input.schema,
        )

    self.add_node_step(
        node_id=fuzzy_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="fuzzy_match",
        setting_input=fuzzy_settings,
        input_node_ids=fuzzy_settings.depending_on_ids,
        schema_callback=schema_callback,
    )

    return self

add_google_analytics_reader(node_ga_reader)

Adds a node that reads from a Google Analytics 4 property.

The actual API fetch (OAuth token refresh, run_report calls, pagination) is offloaded to the worker via ExternalGoogleAnalyticsFetcher, so the core's event loop stays responsive. The schema_callback is derived locally from the selected metrics/dimensions — no network call is made during schema prediction, keeping downstream nodes lazy.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_google_analytics_reader(self, node_ga_reader: input_schema.NodeGoogleAnalyticsReader) -> None:
    """Adds a node that reads from a Google Analytics 4 property.

    The actual API fetch (OAuth token refresh, ``run_report`` calls,
    pagination) is offloaded to the worker via ``ExternalGoogleAnalyticsFetcher``,
    so the core's event loop stays responsive. The ``schema_callback`` is
    derived locally from the selected metrics/dimensions — no network call
    is made during schema prediction, keeping downstream nodes lazy.
    """
    logger.info("Adding google analytics reader")
    node_type = "google_analytics_reader"
    ga_settings = node_ga_reader.google_analytics_settings

    def _build_worker_settings() -> WorkerGoogleAnalyticsReadSettings:
        # Connection resolution is deferred to run time so that *opening* or
        # *undoing* a flow never requires the current session to own the
        # connection (mirrors ``add_cloud_storage_reader``). It runs under
        # ``node_ga_reader.user_id`` — the flow owner at execution time.
        with get_db_context() as db:
            db_conn = get_ga_connection(db, ga_settings.ga_connection_name, node_ga_reader.user_id)
            if db_conn is None:
                raise HTTPException(
                    status_code=400,
                    detail=(
                        f"Google Analytics connection '{ga_settings.ga_connection_name}' not found "
                        "or has not completed sign-in"
                    ),
                )
            auth_method = db_conn.auth_method
            encrypted_credential = get_encrypted_credential(
                db, ga_settings.ga_connection_name, node_ga_reader.user_id
            )
            if encrypted_credential is None:
                raise HTTPException(
                    status_code=400,
                    detail=(
                        f"Google Analytics connection '{ga_settings.ga_connection_name}' has no stored credential"
                    ),
                )
            # OAuth needs the per-instance client config; service accounts don't.
            # Resolved from the CONNECTION OWNER, not the run user: a group-shared
            # OAuth connection must use the owner's Google client config.
            oauth_cfg = get_google_oauth_config(db, db_conn.user_id) if auth_method == "oauth" else None

        common_kwargs = dict(
            property_id=ga_settings.property_id,
            start_date=ga_settings.start_date,
            end_date=ga_settings.end_date,
            metrics=ga_settings.metrics,
            dimensions=ga_settings.dimensions,
            limit=ga_settings.limit,
            filters=[
                WorkerGoogleAnalyticsFilter(
                    field=f.field,
                    operator=f.operator,
                    value=f.value,
                    case_sensitive=f.case_sensitive,
                )
                for f in ga_settings.filters
            ],
            order_bys=[
                WorkerGoogleAnalyticsOrderBy(field=ob.field, descending=ob.descending)
                for ob in ga_settings.order_bys
            ],
            flowfile_flow_id=node_ga_reader.flow_id,
            flowfile_node_id=node_ga_reader.node_id,
        )

        if auth_method == "service_account":
            return WorkerGoogleAnalyticsReadSettings(
                auth_method="service_account",
                service_account_key_encrypted=encrypted_credential,
                **common_kwargs,
            )
        # ``oauth_cfg`` is only fetched for the oauth auth method; guard against
        # an unknown auth_method value reaching this branch with ``None``.
        if not oauth_cfg or not oauth_cfg["client_id"] or not oauth_cfg["client_secret"]:
            raise HTTPException(
                status_code=500,
                detail=(
                    "Google OAuth is not configured on this instance. Open Admin → Google OAuth "
                    "and paste your OAuth client credentials before running this flow."
                ),
            )
        return WorkerGoogleAnalyticsReadSettings(
            auth_method="oauth",
            refresh_token_encrypted=encrypted_credential,
            oauth_client_id=oauth_cfg["client_id"],
            oauth_client_secret_encrypted=_encrypt_with_master_key(oauth_cfg["client_secret"]),
            **common_kwargs,
        )

    # Stamp the predicted schema onto the setting object now, so downstream
    # nodes can introspect columns without ever invoking ``_func`` (which
    # would trigger a worker → Google round-trip). ``derive_schema`` is
    # pure-Python and runs against the chosen metrics/dimensions only — no DB,
    # so it stays eager and keeps flow-open connection-free.
    predicted_columns = derive_schema(metrics=ga_settings.metrics, dimensions=ga_settings.dimensions)
    node_ga_reader.fields = [c.get_minimal_field_info() for c in predicted_columns]

    def _func() -> FlowDataEngine:
        fetcher = ExternalGoogleAnalyticsFetcher(_build_worker_settings(), wait_on_completion=False)
        node._fetch_cached_df = fetcher
        # ``get_result()`` returns a ``pl.LazyFrame`` deserialised from the
        # worker's Arrow IPC file — never collect on the core service.
        fl = FlowDataEngine(fetcher.get_result())
        # Align to the predicted schema so downstream nodes see stable columns
        # even when the report is empty. ``align_to_schema`` lowers to lazy
        # ``with_columns``/``select`` calls, so this stays lazy.
        return fl.align_to_schema(schema_callback())

    def schema_callback() -> list[FlowfileColumn]:
        # Prefer the cached placeholder so repeated schema lookups don't
        # re-walk the heuristic table. ``derive_schema`` is the fallback
        # for the (rare) case where ``fields`` got cleared.
        if node_ga_reader.fields:
            return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_ga_reader.fields]
        return derive_schema(metrics=ga_settings.metrics, dimensions=ga_settings.dimensions)

    node = self.get_node(node_ga_reader.node_id)
    if node:
        node.schema_callback = schema_callback
        node.user_provided_schema_callback = schema_callback
        node.node_type = node_type
        node.name = node_type
        node.function = _func
        node.setting_input = node_ga_reader
        node.node_settings.cache_results = node_ga_reader.cache_results
        self.add_node_to_starting_list(node)
    else:
        node = FlowNode(
            node_ga_reader.node_id,
            function=_func,
            setting_input=node_ga_reader,
            name=node_type,
            node_type=node_type,
            parent_uuid=self.uuid,
            schema_callback=schema_callback,
        )
        node.user_provided_schema_callback = schema_callback
        self._node_db[node_ga_reader.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(node_ga_reader.node_id)

add_graph_solver(graph_solver_settings)

Adds a node that solves graph-like problems within the data.

This node can be used for operations like finding network paths, calculating connected components, or performing other graph algorithms on relational data that represents nodes and edges.

Parameters:

Name	Type	Description	Default
graph_solver_settings	NodeGraphSolver	The settings object defining the graph inputs and the specific algorithm to apply.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_graph_solver(self, graph_solver_settings: input_schema.NodeGraphSolver):
    """Adds a node that solves graph-like problems within the data.

    This node can be used for operations like finding network paths,
    calculating connected components, or performing other graph algorithms
    on relational data that represents nodes and edges.

    Args:
        graph_solver_settings: The settings object defining the graph inputs
            and the specific algorithm to apply.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.solve_graph(graph_solver_settings.graph_solver_input)

    self.add_node_step(
        node_id=graph_solver_settings.node_id,
        function=_func,
        node_type="graph_solver",
        setting_input=graph_solver_settings,
        input_node_ids=[graph_solver_settings.depending_on_id],
    )

add_group_by(group_by_settings)

Adds a group-by aggregation node to the graph.

Parameters:

Name	Type	Description	Default
group_by_settings	NodeGroupBy	The settings for the group-by operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_group_by(self, group_by_settings: input_schema.NodeGroupBy):
    """Adds a group-by aggregation node to the graph.

    Args:
        group_by_settings: The settings for the group-by operation.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.do_group_by(group_by_settings.groupby_input, False)

    self.add_node_step(
        node_id=group_by_settings.node_id,
        function=_func,
        node_type="group_by",
        setting_input=group_by_settings,
        input_node_ids=[group_by_settings.depending_on_id],
    )

    node = self.get_node(group_by_settings.node_id)

    def schema_callback():
        output_columns = [(c.old_name, c.new_name, c.output_type) for c in group_by_settings.groupby_input.agg_cols]
        depends_on = node.node_inputs.main_inputs[0]
        input_schema_dict: dict[str, str] = {s.name: s.data_type for s in depends_on.schema}
        output_schema = []
        for old_name, new_name, data_type in output_columns:
            data_type = input_schema_dict[old_name] if data_type is None else data_type
            output_schema.append(FlowfileColumn.from_input(data_type=data_type, column_name=new_name))
        return output_schema

    node.schema_callback = schema_callback

add_include_cols(include_columns)

Adds columns to both the input and output column lists.

Parameters:

Name	Type	Description	Default
include_columns	list[str]	A list of column names to include.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_include_cols(self, include_columns: list[str]):
    """Adds columns to both the input and output column lists.

    Args:
        include_columns: A list of column names to include.
    """
    for column in include_columns:
        if column not in self._input_cols:
            self._input_cols.append(column)
        if column not in self._output_cols:
            self._output_cols.append(column)
    return self

add_initial_node_analysis(node_promise, track_history=True)

Adds a data exploration/analysis node based on a node promise.

Automatically captures history for undo/redo support.

Parameters:

Name	Type	Description	Default
node_promise	NodePromise	The promise representing the node to be analyzed.	required
track_history	bool	Whether to track this change in history (default True).	True

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_initial_node_analysis(self, node_promise: input_schema.NodePromise, track_history: bool = True):
    """Adds a data exploration/analysis node based on a node promise.

    Automatically captures history for undo/redo support.

    Args:
        node_promise: The promise representing the node to be analyzed.
        track_history: Whether to track this change in history (default True).
    """

    def _do_add():
        node_analysis = create_graphic_walker_node_from_node_promise(node_promise)
        self.add_explore_data(node_analysis)

    if track_history:
        self._execute_with_history(
            _do_add,
            HistoryActionType.ADD_NODE,
            f"Add {node_promise.node_type} node",
            node_id=node_promise.node_id,
        )
    else:
        _do_add()

add_join(join_settings)

Adds a join node to combine two data streams based on key columns.

Parameters:

Name	Type	Description	Default
join_settings	NodeJoin	The settings for the join operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_join(self, join_settings: input_schema.NodeJoin) -> "FlowGraph":
    """Adds a join node to combine two data streams based on key columns.

    Args:
        join_settings: The settings for the join operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
        join_input = deepcopy(join_settings.join_input)
        for left_select in join_input.left_select.renames:
            left_select.is_available = True if left_select.old_name in main.schema else False
        for right_select in join_input.right_select.renames:
            right_select.is_available = True if right_select.old_name in right.schema else False
        return main.join(
            join_input=join_input,
            auto_generate_selection=join_settings.auto_generate_selection,
            verify_integrity=False,
            other=right,
        )

    def schema_callback():
        j_copy = JoinInputManager(join_settings.join_input)
        node = self.get_node(node_id=join_settings.node_id)
        return calculate_join_schema(
            j_copy,
            left_schema=node.node_inputs.main_inputs[0].schema,
            right_schema=node.node_inputs.right_input.schema,
            auto_generate_selection=join_settings.auto_generate_selection,
        )

    self.add_node_step(
        node_id=join_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="join",
        setting_input=join_settings,
        input_node_ids=join_settings.depending_on_ids,
        schema_callback=schema_callback,
    )
    return self

add_kafka_source(node_kafka_source)

Adds a node to read data from a Kafka or Redpanda topic.

Follows the same pattern as add_database_reader: offloads consumption to the worker, which writes an IPC temp file and returns a serialized LazyFrame reference. Offset tracking is handled by Kafka consumer groups.

Parameters:

Name	Type	Description	Default
node_kafka_source	NodeKafkaSource	The settings for the Kafka source node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_kafka_source(self, node_kafka_source: input_schema.NodeKafkaSource):
    """Adds a node to read data from a Kafka or Redpanda topic.

    Follows the same pattern as add_database_reader: offloads consumption
    to the worker, which writes an IPC temp file and returns a serialized
    LazyFrame reference. Offset tracking is handled by Kafka consumer groups.

    Args:
        node_kafka_source: The settings for the Kafka source node.
    """

    logger.info("Adding kafka source")
    node_type = "kafka_source"
    kafka_settings = node_kafka_source.kafka_settings

    # Settings updates may echo back ``fields`` cached from a previous topic /
    # format / connection (the UI clears them, but programmatic callers may
    # not). Stale fields would make ``schema_callback`` report the old topic's
    # columns, so drop them whenever a schema-affecting setting changed.
    # Open/undo replays keep their fields: the replayed settings match the
    # node's previous ones (or the prior node is just a promise).
    prior_settings = getattr(self.get_node(node_kafka_source.node_id), "setting_input", None)
    if node_kafka_source.fields and isinstance(prior_settings, input_schema.NodeKafkaSource):
        prior_kafka = prior_settings.kafka_settings
        if (
            prior_kafka.topic_name != kafka_settings.topic_name
            or prior_kafka.value_format != kafka_settings.value_format
            or prior_kafka.kafka_connection_id != kafka_settings.kafka_connection_id
            or prior_kafka.kafka_connection_name != kafka_settings.kafka_connection_name
        ):
            node_kafka_source.fields = None

    # Resolve the connection lazily so opening/undoing a flow never requires
    # the current session to own the connection. Memoized so ``_func`` and
    # ``schema_callback`` share a single lookup; the lock matters because the
    # schema callback runs on a background thread (``SingleExecutionFuture``)
    # while ``_func`` runs on the execution thread. Runs under the node's
    # ``user_id`` (the flow owner at execution time).
    _read_settings: dict = {}
    _read_settings_lock = threading.Lock()

    def _get_kafka_read_settings() -> KafkaReadSettings:
        with _read_settings_lock:
            if "v" not in _read_settings:
                with get_db_context() as db:
                    db_conn = get_kafka_connection(
                        db, kafka_settings.kafka_connection_id, node_kafka_source.user_id
                    )
                    if db_conn is None:
                        if kafka_settings.kafka_connection_name:
                            db_conn = get_kafka_connection_by_name(
                                db, kafka_settings.kafka_connection_name, node_kafka_source.user_id
                            )
                        if db_conn is None:
                            raise HTTPException(status_code=400, detail="Kafka connection not found")
                    consumer_config = build_consumer_config(db, db_conn, node_kafka_source.user_id)
                _read_settings["v"] = KafkaReadSettings.from_consumer_config(
                    consumer_config,
                    topic=kafka_settings.topic_name,
                    value_format=kafka_settings.value_format,
                    group_id=kafka_settings.sync_name
                    or f"flowfile-{node_kafka_source.flow_id}-node-{node_kafka_source.node_id}",
                    start_offset=kafka_settings.start_offset,
                    max_messages=kafka_settings.max_messages,
                    poll_timeout_seconds=kafka_settings.poll_timeout_seconds,
                    flowfile_flow_id=node_kafka_source.flow_id,
                    flowfile_node_id=node_kafka_source.node_id,
                )
            return _read_settings["v"]

    def _func():
        kafka_read_settings = _get_kafka_read_settings()
        if self.execution_location == "local":
            # Local execution — consume directly in-process with spill-to-IPC
            import tempfile

            fd, spill_file = tempfile.mkstemp(suffix=".arrow", prefix="kafka_")
            os.close(fd)
            result, kafka_result = read_kafka_source(
                kafka_read_settings,
                commit=False,
                decrypt_fn=_decrypt_fn,
                spill_path=spill_file,
            )
            lf = result if isinstance(result, pl.LazyFrame) else result.lazy()
            fl = FlowDataEngine(lf)
            if kafka_result.messages_consumed > 0:
                node._on_flow_complete = make_kafka_commit_callback(
                    kafka_read_settings,
                    kafka_result.new_offsets,
                    node_kafka_source.node_id,
                    self.flow_logger,
                    _decrypt_fn,
                )
        else:
            # Remote execution — offload to worker (worker uses commit=False + sidecar)
            external_kafka_fetcher = ExternalKafkaFetcher(kafka_read_settings, wait_on_completion=False)
            node._fetch_cached_df = external_kafka_fetcher
            fl = FlowDataEngine(external_kafka_fetcher.get_result())
            offsets_data = fetch_kafka_offsets(external_kafka_fetcher.file_ref)
            if offsets_data and offsets_data.get("messages_consumed", 0) > 0:
                node._on_flow_complete = make_kafka_commit_callback(
                    kafka_read_settings,
                    offsets_data["new_offsets"],
                    node_kafka_source.node_id,
                    self.flow_logger,
                    _decrypt_fn,
                )
        # The worker DataFrame may have fewer columns than the inferred
        # schema (e.g. empty topic or starting at "latest"). Align to
        # the schema_callback result so downstream nodes see stable columns.
        expected_columns = schema_callback()
        fl = fl.align_to_schema(expected_columns)
        node_kafka_source.fields = [c.get_minimal_field_info() for c in fl.schema]
        return fl

    def _decrypt_fn(encrypted: str) -> str:
        return decrypt_secret(encrypted).get_secret_value()

    def schema_callback():
        # Prefer the schema cached on the node so opening a saved flow renders
        # columns without sampling the topic (a live connection). Sampling only
        # runs when fields were never captured (failures are caught per-node).
        if node_kafka_source.fields:
            return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_kafka_source.fields]
        schema_pairs = infer_topic_schema(_get_kafka_read_settings(), sample_size=10, decrypt_fn=_decrypt_fn)
        # Since the schema callback takes quite some time, we only run the function once.
        if not schema_pairs:
            result = [
                FlowfileColumn.from_input(column_name="_kafka_key", data_type="String"),
                FlowfileColumn.from_input(column_name="_kafka_partition", data_type="Int64"),
                FlowfileColumn.from_input(column_name="_kafka_offset", data_type="Int64"),
                FlowfileColumn.from_input(column_name="_kafka_timestamp", data_type="Datetime"),
            ]
        else:
            result = [FlowfileColumn.create_from_polars_dtype(column_name=n, data_type=t) for n, t in schema_pairs]
        return result

    node = self.get_node(node_kafka_source.node_id)
    if node:
        node.user_provided_schema_callback = schema_callback
        node.schema_callback = schema_callback
        node.node_type = node_type
        node.name = node_type
        node.function = _func
        node.setting_input = node_kafka_source
        node.node_settings.cache_results = node_kafka_source.cache_results
        self.add_node_to_starting_list(node)
    else:
        node = FlowNode(
            node_kafka_source.node_id,
            function=_func,
            setting_input=node_kafka_source,
            name=node_type,
            node_type=node_type,
            parent_uuid=self.uuid,
            schema_callback=schema_callback,
        )
        self._node_db[node_kafka_source.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(node_kafka_source.node_id)

add_manual_input(input_file)

Adds a node for manual data entry.

This is a convenience alias for add_datasource.

Parameters:

Name	Type	Description	Default
input_file	NodeManualInput	The settings and data for the manual input node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_manual_input(self, input_file: input_schema.NodeManualInput):
    """Adds a node for manual data entry.

    This is a convenience alias for `add_datasource`.

    Args:
        input_file: The settings and data for the manual input node.
    """
    self.add_datasource(input_file)

add_missing_user_defined_node(*, user_defined_node_settings, node_type, error)

Adds a placeholder for a custom node that cannot be loaded on this machine.

The stored settings are preserved verbatim (lossless re-save), the node renders with its connections, and running the flow fails this node with error instead of silently dropping it.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_missing_user_defined_node(
    self, *, user_defined_node_settings: input_schema.UserDefinedNode, node_type: str, error: str
):
    """Adds a placeholder for a custom node that cannot be loaded on this machine.

    The stored settings are preserved verbatim (lossless re-save), the node
    renders with its connections, and running the flow fails this node with
    ``error`` instead of silently dropping it.
    """
    register_missing_node_template(node_type)

    def _missing_custom_node(*_flow_data_engine: FlowDataEngine) -> FlowDataEngine:
        raise ValueError(error)

    self.add_node_step(
        node_id=user_defined_node_settings.node_id,
        function=_missing_custom_node,
        setting_input=user_defined_node_settings,
        input_node_ids=user_defined_node_settings.depending_on_ids,
        node_type=node_type,
    )
    node = self.get_node(user_defined_node_settings.node_id)
    node.results.errors = error

add_node_promise(node_promise, track_history=True)

Adds a placeholder node to the graph that is not yet fully configured.

Useful for building the graph structure before all settings are available. Automatically captures history for undo/redo support.

Parameters:

Name	Type	Description	Default
node_promise	NodePromise	A promise object containing basic node information.	required
track_history	bool	Whether to track this change in history (default True).	True

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_node_promise(self, node_promise: input_schema.NodePromise, track_history: bool = True):
    """Adds a placeholder node to the graph that is not yet fully configured.

    Useful for building the graph structure before all settings are available.
    Automatically captures history for undo/redo support.

    Args:
        node_promise: A promise object containing basic node information.
        track_history: Whether to track this change in history (default True).
    """

    def _do_add():
        def placeholder(n: FlowNode = None):
            if n is None:
                return FlowDataEngine()
            return n

        self.add_node_step(
            node_id=node_promise.node_id,
            node_type=node_promise.node_type,
            function=placeholder,
            setting_input=node_promise,
        )
        if node_promise.is_user_defined:
            node_needs_settings: bool
            custom_node = CUSTOM_NODE_STORE.get(node_promise.node_type)
            if custom_node is None:
                raise ValueError(missing_custom_node_error(node_promise.node_type))
            settings_schema = custom_node.model_fields["settings_schema"].default
            node_needs_settings = settings_schema is not None and not settings_schema.is_empty()
            if not node_needs_settings:
                user_defined_node_settings = input_schema.UserDefinedNode(settings={}, **node_promise.model_dump())
                initialized_model = custom_node()
                self.add_user_defined_node(
                    custom_node=initialized_model, user_defined_node_settings=user_defined_node_settings
                )

    if track_history:
        self._execute_with_history(
            _do_add,
            HistoryActionType.ADD_NODE,
            f"Add {node_promise.node_type} node",
            node_id=node_promise.node_id,
        )
    else:
        _do_add()

add_node_step(node_id, function, input_columns=None, output_schema=None, node_type=None, drop_columns=None, renew_schema=True, setting_input=None, cache_results=None, schema_callback=None, input_node_ids=None)

The core method for adding or updating a node in the graph.

Parameters:

Name	Type	Description	Default
node_id	int | str	The unique ID for the node.	required
function	Callable	The core processing function for the node.	required
input_columns	list[str]	A list of input column names required by the function.	None
output_schema	list[FlowfileColumn]	A predefined schema for the node's output.	None
node_type	str	A string identifying the type of node (e.g., 'filter', 'join').	None
drop_columns	list[str]	A list of columns to be dropped after the function executes.	None
renew_schema	bool	If True, the schema is recalculated after execution.	True
setting_input	Any	A configuration object containing settings for the node.	None
cache_results	bool	If True, the node's results are cached for future runs.	None
schema_callback	Callable	A function that dynamically calculates the output schema.	None
input_node_ids	list[int]	A list of IDs for the nodes that this node depends on.	None

Returns:

Type	Description
FlowNode	The created or updated FlowNode object.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_node_step(
    self,
    node_id: int | str,
    function: Callable,
    input_columns: list[str] = None,
    output_schema: list[FlowfileColumn] = None,
    node_type: str = None,
    drop_columns: list[str] = None,
    renew_schema: bool = True,
    setting_input: Any = None,
    cache_results: bool = None,
    schema_callback: Callable = None,
    input_node_ids: list[int] = None,
) -> FlowNode:
    """The core method for adding or updating a node in the graph.

    Args:
        node_id: The unique ID for the node.
        function: The core processing function for the node.
        input_columns: A list of input column names required by the function.
        output_schema: A predefined schema for the node's output.
        node_type: A string identifying the type of node (e.g., 'filter', 'join').
        drop_columns: A list of columns to be dropped after the function executes.
        renew_schema: If True, the schema is recalculated after execution.
        setting_input: A configuration object containing settings for the node.
        cache_results: If True, the node's results are cached for future runs.
        schema_callback: A function that dynamically calculates the output schema.
        input_node_ids: A list of IDs for the nodes that this node depends on.

    Returns:
        The created or updated FlowNode object.
    """
    output_field_config = getattr(setting_input, "output_field_config", None) if setting_input else None

    logger.info(
        f"add_node_step: node_id={node_id}, node_type={node_type}, "
        f"has_setting_input={setting_input is not None}, "
        f"has_output_field_config={output_field_config is not None}, "
        f"config_enabled={output_field_config.enabled if output_field_config else False}, "
        f"has_schema_callback={schema_callback is not None}"
    )

    # IMPORTANT: Always create wrapped callback if output_field_config exists (even if enabled=False)
    # This ensures nodes like PolarsCode get a schema callback when output_field_config is defined
    if output_field_config:
        if output_field_config.enabled:
            logger.info(
                f"add_node_step: Creating/wrapping schema_callback for node {node_id} with output_field_config "
                f"(validation_mode={output_field_config.validation_mode_behavior}, "
                f"{len(output_field_config.fields)} fields, "
                f"base_callback={'present' if schema_callback else 'None'})"
            )
        else:
            logger.debug(f"add_node_step: output_field_config present for node {node_id} but disabled")

        schema_callback = create_schema_callback_with_output_config(schema_callback, output_field_config)
        logger.info(
            f"add_node_step: schema_callback {'created' if schema_callback else 'failed'} for node {node_id}"
        )

    existing_node = self.get_node(node_id)
    if existing_node is not None:
        if existing_node.node_type != node_type:
            self.delete_node(existing_node.node_id)
            existing_node = None
    if existing_node:
        input_nodes = existing_node.all_inputs
    elif input_node_ids is not None:
        input_nodes = [self.get_node(node_id) for node_id in input_node_ids]
    else:
        input_nodes = None
    if isinstance(input_columns, str):
        input_columns = [input_columns]
    if (
        input_nodes is not None
        or function.__name__ in ("placeholder", "analysis_preparation")
        or node_type in ("cloud_storage_reader", "catalog_reader", "polars_lazy_frame", "input_data")
    ):
        if not existing_node:
            node = FlowNode(
                node_id=node_id,
                function=function,
                output_schema=output_schema,
                input_columns=input_columns,
                drop_columns=drop_columns,
                renew_schema=renew_schema,
                setting_input=setting_input,
                node_type=node_type,
                name=function.__name__,
                schema_callback=schema_callback,
                parent_uuid=self.uuid,
            )
        else:
            existing_node.update_node(
                function=function,
                output_schema=output_schema,
                input_columns=input_columns,
                drop_columns=drop_columns,
                setting_input=setting_input,
                schema_callback=schema_callback,
            )
            node = existing_node
    else:
        raise Exception("No data initialized")
    self._node_db[node_id] = node
    self._node_ids.append(node_id)
    # Give the node a callable that returns the current flow parameters so
    # that lazy schema prediction (_predicted_data_getter) can substitute
    # ${...} refs. Using a callable (rather than a copy of the dict) means
    # the node always reads the LATEST parameters, whether they were set via
    # the flow_settings.setter or mutated directly on flow_settings.parameters.
    _graph = self

    def _get_params() -> dict[str, ParamValue]:
        return {p.name: p.typed_default() for p in (_graph.flow_settings.parameters or [])}

    node._params_getter = _get_params
    return node

add_node_to_starting_list(node)

Adds a node to the list of starting nodes for the flow if not already present.

Parameters:

Name	Type	Description	Default
node	FlowNode	The FlowNode to add as a starting node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_node_to_starting_list(self, node: FlowNode) -> None:
    """Adds a node to the list of starting nodes for the flow if not already present.

    Args:
        node: The FlowNode to add as a starting node.
    """
    if node.node_id not in {self_node.node_id for self_node in self._flow_starts}:
        self._flow_starts.append(node)

add_nodes_to_group(group_id, node_ids)

Add nodes to an existing group and refit its bounds.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_nodes_to_group(self, group_id: int, node_ids: list[int]) -> schemas.GroupInformation:
    """Add nodes to an existing group and refit its bounds."""
    group = self._groups.get(group_id)
    if group is None:
        raise ValueError(f"Group {group_id} does not exist")

    def _do() -> schemas.GroupInformation:
        for node_id in node_ids:
            self._set_node_group(node_id, group_id)
        self._recompute_group_bounds(group_id)
        return group

    return self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP_MEMBERSHIP, "Add nodes to group")

add_output(output_file)

Adds an output node to write the final data to a destination.

Parameters:

Name	Type	Description	Default
output_file	NodeOutput	The settings for the output file.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_output(self, output_file: input_schema.NodeOutput):
    """Adds an output node to write the final data to a destination.

    Args:
        output_file: The settings for the output file.
    """

    def _func(df: FlowDataEngine):
        if self.execution_location == "local":
            df.output(
                output_fs=output_file.output_settings,
                flow_id=self.flow_id,
                node_id=output_file.node_id,
                execute_remote=False,
            )
            return df
        output_fs = output_file.output_settings
        node = self.get_node(output_file.node_id)
        writer = ExternalOutputWriter(
            lf=df.data_frame,
            data_type=output_fs.file_type,
            path=output_fs.abs_file_path,
            write_mode=output_fs.write_mode,
            sheet_name=output_fs.sheet_name,
            delimiter=output_fs.delimiter,
            compression=output_fs.compression,
            flow_id=self.flow_id,
            node_id=output_file.node_id,
            wait_on_completion=False,
        )
        node._fetch_cached_df = writer
        writer.get_result()
        return df

    def schema_callback():
        input_node: FlowNode = self.get_node(output_file.node_id).node_inputs.main_inputs[0]

        return input_node.schema

    input_node_id = output_file.depending_on_id if hasattr(output_file, "depending_on_id") else None
    self.add_node_step(
        node_id=output_file.node_id,
        function=_func,
        input_columns=[],
        node_type="output",
        setting_input=output_file,
        schema_callback=schema_callback,
        input_node_ids=[input_node_id],
    )

add_pivot(pivot_settings)

Adds a pivot node to the graph.

Parameters:

Name	Type	Description	Default
pivot_settings	NodePivot	The settings for the pivot operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_pivot(self, pivot_settings: input_schema.NodePivot):
    """Adds a pivot node to the graph.

    Args:
        pivot_settings: The settings for the pivot operation.
    """

    def _func(fl: FlowDataEngine):
        return fl.do_pivot(pivot_settings.pivot_input, self.flow_logger.get_node_logger(pivot_settings.node_id))

    self.add_node_step(
        node_id=pivot_settings.node_id,
        function=_func,
        node_type="pivot",
        setting_input=pivot_settings,
        input_node_ids=[pivot_settings.depending_on_id],
    )

    node = self.get_node(pivot_settings.node_id)
    node._prediction_requires_data = True

    def schema_callback():
        node._schema_prediction_blocked = None
        reason = kernel_block_reason(node, include_self=False)
        if reason:
            # Pivot columns need real data; never run a kernel implicitly for it.
            node._schema_prediction_blocked = reason
            node.results.warnings = reason
            return []
        input_data = node.singular_main_input.get_resulting_data()
        # Runs on a background thread: never mutate the shared memoized
        # engine (input_data.lazy = ...); build a local lazy frame instead.
        input_lf = input_data.data_frame.lazy()
        return pre_calculate_pivot_schema(input_data.schema, pivot_settings.pivot_input, input_lf=input_lf)

    node.schema_callback = schema_callback

add_polars_code(node_polars_code)

Adds a node that executes custom Polars code.

Parameters:

Name	Type	Description	Default
node_polars_code	NodePolarsCode	The settings for the Polars code node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_polars_code(self, node_polars_code: input_schema.NodePolarsCode):
    """Adds a node that executes custom Polars code.

    Args:
        node_polars_code: The settings for the Polars code node.
    """

    def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
        return execute_polars_code(*flowfile_tables, code=node_polars_code.polars_code_input.polars_code)

    self.add_node_step(
        node_id=node_polars_code.node_id,
        function=_func,
        node_type="polars_code",
        setting_input=node_polars_code,
        input_node_ids=node_polars_code.depending_on_ids,
    )

    try:
        polars_code_parser.validate_code(node_polars_code.polars_code_input.polars_code)
    except Exception as e:
        node = self.get_node(node_id=node_polars_code.node_id)
        node.results.errors = str(e)

add_python_script(node_python_script)

Adds a node that executes Python code on a kernel container.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_python_script(self, node_python_script: input_schema.NodePythonScript):
    """Adds a node that executes Python code on a kernel container."""

    def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
        kernel_id = node_python_script.python_script_input.kernel_id
        if not kernel_id:
            raise ValueError("No kernel selected for python_script node")
        result = self._execute_on_kernel(
            node_id=node_python_script.node_id,
            kernel_id=kernel_id,
            code=node_python_script.python_script_input.code,
            output_names=node_python_script.output_names,
            flow_data_engine=flowfile_tables,
        )
        return result or (flowfile_tables[0] if flowfile_tables else FlowDataEngine(pl.LazyFrame()))

    def schema_callback():
        """Best-effort schema prediction for python_script nodes.

        Returns the input node(s) schema as a reasonable default
        (most python_script nodes transform and pass through).
        If nothing is available, returns [] — never raises.
        """
        try:
            node = self.get_node(node_python_script.node_id)
            if node is None:
                return []

            main_inputs = node.node_inputs.main_inputs
            if main_inputs:
                first_input = main_inputs[0]
                input_node_schema = first_input.schema
                if input_node_schema:
                    return input_node_schema
            return []
        except Exception:
            return []

    self.add_node_step(
        node_id=node_python_script.node_id,
        function=_func,
        node_type="python_script",
        setting_input=node_python_script,
        input_node_ids=node_python_script.depending_on_ids,
        schema_callback=schema_callback,
    )

    node = self.get_node(node_python_script.node_id)
    if node is not None:
        node._executes_on_kernel = bool(node_python_script.python_script_input.kernel_id)
    output_names = node_python_script.output_names
    if len(output_names) > 1:
        if node is not None:
            node.node_template = node.node_template.model_copy(update={"output": len(output_names)})

add_random_split(settings)

Adds a node that randomly partitions rows into N labeled outputs.

Returns a NamedOutputs; the framework unpacks it into _named_outputs so each split is reachable via its own output handle.

Parameters:

Name	Type	Description	Default
settings	NodeRandomSplit	The settings object specifying the splits and optional seed.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_random_split(self, settings: input_schema.NodeRandomSplit) -> "FlowGraph":
    """Adds a node that randomly partitions rows into N labeled outputs.

    Returns a ``NamedOutputs``; the framework unpacks it into
    ``_named_outputs`` so each split is reachable via its own output handle.

    Args:
        settings: The settings object specifying the splits and optional seed.

    Returns:
        The `FlowGraph` instance for method chaining.
    """
    from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

    def _func(table: FlowDataEngine) -> NamedOutputs:
        split_pairs = [(s.name, s.percentage) for s in settings.splits]
        if self.execution_location == "local":
            return table.random_split(split_pairs, settings.seed)
        return table.random_split_external(
            split_pairs,
            settings.seed,
            flow_id=self.flow_id,
            node_id=settings.node_id,
        )

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        node_type="random_split",
        setting_input=settings,
        input_node_ids=[settings.depending_on_id],
    )
    return self

add_read(input_file)

Adds a node to read data from a local file (e.g., CSV, Parquet, Excel).

Parameters:

Name	Type	Description	Default
input_file	NodeRead	The settings for the read operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_read(self, input_file: input_schema.NodeRead):
    """Adds a node to read data from a local file (e.g., CSV, Parquet, Excel).

    Args:
        input_file: The settings for the read operation.
    """
    if (
        input_file.received_file.file_type in ("xlsx", "excel")
        and input_file.received_file.table_settings.sheet_name == ""
    ):
        sheet_name = fastexcel.read_excel(input_file.received_file.path).sheet_names[0]
        input_file.received_file.table_settings.sheet_name = sheet_name

    received_file = input_file.received_file
    input_file.received_file.set_absolute_filepath()

    def _func():
        input_file.received_file.set_absolute_filepath()
        if self.execution_location == "local":
            input_data = FlowDataEngine.create_from_path(input_file.received_file)
        elif input_file.received_file.file_type in ("parquet", "ipc", "ndjson"):
            input_data = FlowDataEngine.create_from_path(input_file.received_file)
        elif (
            input_file.received_file.file_type == "csv"
            and "utf" in input_file.received_file.table_settings.encoding
        ):
            input_data = FlowDataEngine.create_from_path(input_file.received_file)
        else:
            input_data = FlowDataEngine.create_from_path_worker(
                input_file.received_file, node_id=input_file.node_id, flow_id=self.flow_id
            )
        input_data.name = input_file.received_file.name
        return input_data

    node = self.get_node(input_file.node_id)
    schema_callback = None
    if node:
        start_hash = node.hash
        node.node_type = "read"
        node.name = "read"
        node.function = _func
        node.setting_input = input_file
        self.add_node_to_starting_list(node)

        if start_hash != node.hash:
            logger.info("Hash changed, updating schema")
            if len(received_file.fields) > 0:

                def schema_callback():
                    return [FlowfileColumn.from_input(f.name, f.data_type) for f in received_file.fields]

            elif input_file.received_file.file_type in ("csv", "json", "parquet", "ipc", "ndjson"):

                def schema_callback():
                    input_data = FlowDataEngine.create_from_path(input_file.received_file)
                    return input_data.schema

            elif input_file.received_file.file_type in ("xlsx", "excel"):
                schema_callback = get_xlsx_schema_callback(
                    engine="openpyxl",
                    file_path=received_file.file_path,
                    sheet_name=received_file.table_settings.sheet_name,
                    start_row=received_file.table_settings.start_row,
                    end_row=received_file.table_settings.end_row,
                    start_column=received_file.table_settings.start_column,
                    end_column=received_file.table_settings.end_column,
                    has_headers=received_file.table_settings.has_headers,
                )
            else:
                schema_callback = None
    else:
        node = FlowNode(
            input_file.node_id,
            function=_func,
            setting_input=input_file,
            name="read",
            node_type="read",
            parent_uuid=self.uuid,
        )
        self._node_db[input_file.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(input_file.node_id)

    if schema_callback is not None:
        node.schema_callback = schema_callback
        node.user_provided_schema_callback = schema_callback
    return self

add_record_count(node_number_of_records)

Adds a filter node to the graph.

Parameters:

Name	Type	Description	Default
node_number_of_records	NodeRecordCount	The settings for the record count operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_record_count(self, node_number_of_records: input_schema.NodeRecordCount):
    """Adds a filter node to the graph.

    Args:
        node_number_of_records: The settings for the record count operation.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.get_record_count()

    self.add_node_step(
        node_id=node_number_of_records.node_id,
        function=_func,
        node_type="record_count",
        setting_input=node_number_of_records,
        input_node_ids=[node_number_of_records.depending_on_id],
    )

add_record_id(record_id_settings)

Adds a node to create a new column with a unique ID for each record.

Parameters:

Name	Type	Description	Default
record_id_settings	NodeRecordId	The settings object specifying the name of the new record ID column.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_record_id(self, record_id_settings: input_schema.NodeRecordId) -> "FlowGraph":
    """Adds a node to create a new column with a unique ID for each record.

    Args:
        record_id_settings: The settings object specifying the name of the
            new record ID column.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        return table.add_record_id(record_id_settings.record_id_input)

    self.add_node_step(
        node_id=record_id_settings.node_id,
        function=_func,
        node_type="record_id",
        setting_input=record_id_settings,
        input_node_ids=[record_id_settings.depending_on_id],
    )
    return self

add_rest_api_reader(node_rest_api_reader)

Adds a node that reads from a REST API.

All network I/O (HTTP round-trips, pagination, retries) is offloaded to the worker via ExternalRestApiFetcher — the core never makes the external call. The credential is resolved to an encrypted token here (from the user's secret store, or an inline plaintext) and the worker decrypts it just-in-time. A generic API's columns are unknown until a response is fetched, so schema_callback returns the columns cached on the node by the "Fetch sample" action — empty until the user samples or runs, in which case the fetched frame defines the schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_rest_api_reader(self, node_rest_api_reader: input_schema.NodeRestApiReader) -> None:
    """Adds a node that reads from a REST API.

    All network I/O (HTTP round-trips, pagination, retries) is offloaded to
    the worker via ``ExternalRestApiFetcher`` — the core never makes the
    external call. The credential is resolved to an encrypted token here
    (from the user's secret store, or an inline plaintext) and the worker
    decrypts it just-in-time. A generic API's columns are unknown until a
    response is fetched, so ``schema_callback`` returns the columns cached on
    the node by the "Fetch sample" action — empty until the user samples or
    runs, in which case the fetched frame defines the schema.
    """
    logger.info("Adding rest api reader")
    node_type = "rest_api_reader"
    auth = node_rest_api_reader.rest_api_settings.auth

    # Encrypt any *inline* plaintext credential eagerly and null it out so it is
    # never persisted on the node (a security guarantee, independent of who owns
    # the flow). The *by-name* secret-store lookup is deferred to run time so
    # opening/undoing a flow never requires the current session to own the
    # secret — it resolves under the node's ``user_id`` (the flow owner).
    _inline_encrypted = _encrypt_with_master_key(auth.secret) if (auth.secret and not auth.secret_name) else None
    auth.secret = None

    def _resolve_secret_encrypted() -> str | None:
        if _inline_encrypted is not None:
            return _inline_encrypted
        return resolve_auth_secret_encrypted(auth, node_rest_api_reader.user_id)

    def _func() -> FlowDataEngine:
        encrypted = _resolve_secret_encrypted()
        worker_settings = build_rest_api_worker_settings(node_rest_api_reader, encrypted)
        if self.execution_location == "local":
            # No worker service in local runs — fetch in-process (cf. add_database_reader).
            from shared.rest_api.fetch import fetch_rest_api

            secret = decrypt_secret(encrypted).get_secret_value() if encrypted else None
            fl = FlowDataEngine(fetch_rest_api(worker_settings, secret=secret).lazy())
        else:
            fetcher = ExternalRestApiFetcher(worker_settings, wait_on_completion=False)
            node._fetch_cached_df = fetcher
            fl = FlowDataEngine(fetcher.get_result())
        cols = schema_callback()
        # Align to the sampled schema (if any) so downstream nodes see stable
        # columns; with no sample yet, the fetched frame defines the schema.
        if cols:
            return fl.align_to_schema(cols)
        node_rest_api_reader.fields = [c.get_minimal_field_info() for c in fl.schema]
        return fl

    def schema_callback() -> list[FlowfileColumn]:
        if node_rest_api_reader.fields:
            return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_rest_api_reader.fields]
        return []

    node = self.get_node(node_rest_api_reader.node_id)
    if node:
        node.schema_callback = schema_callback
        node.user_provided_schema_callback = schema_callback
        node.node_type = node_type
        node.name = node_type
        node.function = _func
        node.setting_input = node_rest_api_reader
        node.node_settings.cache_results = node_rest_api_reader.cache_results
        self.add_node_to_starting_list(node)
    else:
        node = FlowNode(
            node_rest_api_reader.node_id,
            function=_func,
            setting_input=node_rest_api_reader,
            name=node_type,
            node_type=node_type,
            parent_uuid=self.uuid,
            schema_callback=schema_callback,
        )
        node.user_provided_schema_callback = schema_callback
        self._node_db[node_rest_api_reader.node_id] = node
        self.add_node_to_starting_list(node)
        self._node_ids.append(node_rest_api_reader.node_id)

add_run_flow(settings)

Adds a node that executes a catalog-registered flow as a subflow.

Inputs are keyed: handle input-0 carries optional parameter data; handles input-1..input-N feed the subflow's flow_input nodes (input_slots order). Outputs mirror the subflow's flow_output nodes (output_slots order).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_run_flow(self, settings: input_schema.NodeRunFlow) -> "FlowGraph":
    """Adds a node that executes a catalog-registered flow as a subflow.

    Inputs are keyed: handle input-0 carries optional parameter data; handles
    input-1..input-N feed the subflow's flow_input nodes (input_slots order).
    Outputs mirror the subflow's flow_output nodes (output_slots order).
    """
    from flowfile_core.flowfile import subflow

    subflow.stamp_flow_reference(settings)
    _graph = self

    def _func(*inputs: FlowDataEngine):
        param_input = inputs[0] if inputs else None
        return subflow.execute_run_flow_node(_graph, settings, param_input, tuple(inputs[1:]))

    def schema_callback():
        node = _graph.get_node(settings.node_id)
        named = subflow.predict_run_flow_named_schemas(settings)
        if node is not None and named:
            node._named_schemas = named
        return named.get(DEFAULT_OUTPUT_HANDLE, [])

    existing = self.get_node(settings.node_id)
    old_slots: list[str] | None = None
    if existing is not None and isinstance(existing.setting_input, input_schema.NodeRunFlow):
        old_slots = list(existing.setting_input.input_slots)

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        input_columns=[],
        node_type="run_flow",
        setting_input=settings,
        schema_callback=schema_callback,
        input_node_ids=[],
    )

    if old_slots is not None and old_slots != settings.input_slots:
        node = self.get_node(settings.node_id)
        # Keyed edges follow their slot by NAME; vanished names drop their edge.
        mapping: dict[str, str | None] = {}
        for old_index, slot_name in enumerate(old_slots):
            old_handle = input_handle(old_index + 1)
            if slot_name in settings.input_slots:
                mapping[old_handle] = input_handle(settings.input_slots.index(slot_name) + 1)
            else:
                mapping[old_handle] = None
        result = node.remap_dynamic_inputs(mapping)
        if result["dropped"]:
            self.flow_logger.warning(
                f"run_flow node {settings.node_id}: dropped connection(s) on {', '.join(result['dropped'])} "
                "after the subflow interface changed"
            )
    return self

add_sample(sample_settings)

Adds a node to take a random or top-N sample of the data.

Every method stays lazy, so the node needs no local/remote branch: the sample is part of the plan the worker receives, not a materialised frame.

Parameters:

Name	Type	Description	Default
sample_settings	NodeSample	The settings object specifying the sampling method, the size or fraction to keep, and an optional seed.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_sample(self, sample_settings: input_schema.NodeSample) -> "FlowGraph":
    """Adds a node to take a random or top-N sample of the data.

    Every method stays lazy, so the node needs no local/remote branch: the
    sample is part of the plan the worker receives, not a materialised frame.

    Args:
        sample_settings: The settings object specifying the sampling method,
            the size or fraction to keep, and an optional seed.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        if sample_settings.sample_method == "random":
            return table.random_sample(n=sample_settings.sample_size, seed=sample_settings.seed)
        if sample_settings.sample_method == "random_fraction":
            return table.random_sample(fraction=sample_settings.fraction / 100.0, seed=sample_settings.seed)
        return table.get_sample(sample_settings.sample_size)

    self.add_node_step(
        node_id=sample_settings.node_id,
        function=_func,
        node_type="sample",
        setting_input=sample_settings,
        input_node_ids=[sample_settings.depending_on_id],
    )
    return self

add_select(select_settings)

Adds a node to select, rename, reorder, or drop columns.

Parameters:

Name	Type	Description	Default
select_settings	NodeSelect	The settings for the select operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_select(self, select_settings: input_schema.NodeSelect) -> "FlowGraph":
    """Adds a node to select, rename, reorder, or drop columns.

    Args:
        select_settings: The settings for the select operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    select_cols = select_settings.select_input
    drop_cols = tuple(s.old_name for s in select_settings.select_input)

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        input_cols = set(f.name for f in table.schema)
        ids_to_remove = []
        for i, select_col in enumerate(select_cols):
            if select_col.old_name not in input_cols:
                select_col.is_available = False
                if not select_col.keep:
                    ids_to_remove.append(i)
                continue
            select_col.is_available = True
            if select_col.data_type is None:
                select_col.data_type = table.get_schema_column(select_col.old_name).data_type
        ids_to_remove.reverse()
        for i in ids_to_remove:
            select_cols.pop(i)
        return table.do_select(
            select_inputs=transform_schema.SelectInputs(select_cols), keep_missing=select_settings.keep_missing
        )

    self.add_node_step(
        node_id=select_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="select",
        drop_columns=list(drop_cols),
        setting_input=select_settings,
        input_node_ids=[select_settings.depending_on_id],
    )
    return self

add_sort(sort_settings)

Adds a node to sort the data based on one or more columns.

Parameters:

Name	Type	Description	Default
sort_settings	NodeSort	The settings for the sort operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_sort(self, sort_settings: input_schema.NodeSort) -> "FlowGraph":
    """Adds a node to sort the data based on one or more columns.

    Args:
        sort_settings: The settings for the sort operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        return table.do_sort(sort_settings.sort_input)

    self.add_node_step(
        node_id=sort_settings.node_id,
        function=_func,
        node_type="sort",
        setting_input=sort_settings,
        input_node_ids=[sort_settings.depending_on_id],
    )
    return self

add_sql_query(node_sql_query)

Adds a node that executes a SQL query against connected data sources.

Parameters:

Name	Type	Description	Default
node_sql_query	NodeSqlQuery	The settings for the SQL query node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_sql_query(self, node_sql_query: input_schema.NodeSqlQuery):
    """Adds a node that executes a SQL query against connected data sources.

    Args:
        node_sql_query: The settings for the SQL query node.
    """

    def _func(*flowfile_tables: FlowDataEngine) -> FlowDataEngine:
        return execute_sql_query(*flowfile_tables, sql_code=node_sql_query.sql_query_input.sql_code)

    self.add_node_step(
        node_id=node_sql_query.node_id,
        function=_func,
        node_type="sql_query",
        setting_input=node_sql_query,
        input_node_ids=node_sql_query.depending_on_ids,
    )

    node = self.get_node(node_id=node_sql_query.node_id)

    def schema_callback() -> list[FlowfileColumn]:
        # Resolve the output schema by running the query plan lazily over
        # 0-row upstream frames (input_1..N); no data is collected.
        inputs = [
            v.get_predicted_resulting_data(src_handle) if v is not None else FlowDataEngine()
            for v, src_handle in node._slot_input_pairs()
        ]
        return execute_sql_query(*inputs, sql_code=node_sql_query.sql_query_input.sql_code).schema

    node.schema_callback = schema_callback

    try:
        validate_sql_query(node_sql_query.sql_query_input.sql_code)
    except Exception as e:
        node.results.errors = str(e)

add_sql_source(external_source_input)

Adds a node that reads data from a SQL source.

This is a convenience alias for add_external_source.

Parameters:

Name	Type	Description	Default
external_source_input	NodeExternalSource	The settings for the external SQL source node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_sql_source(self, external_source_input: input_schema.NodeExternalSource):
    """Adds a node that reads data from a SQL source.

    This is a convenience alias for `add_external_source`.

    Args:
        external_source_input: The settings for the external SQL source node.
    """
    logger.info("Adding sql source")
    self.add_external_source(external_source_input)

add_text_to_rows(node_text_to_rows)

Adds a node that splits cell values into multiple rows.

This is useful for un-nesting data where a single field contains multiple values separated by a delimiter.

Parameters:

Name	Type	Description	Default
node_text_to_rows	NodeTextToRows	The settings object that specifies the column to split and the delimiter to use.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_text_to_rows(self, node_text_to_rows: input_schema.NodeTextToRows) -> "FlowGraph":
    """Adds a node that splits cell values into multiple rows.

    This is useful for un-nesting data where a single field contains multiple
    values separated by a delimiter.

    Args:
        node_text_to_rows: The settings object that specifies the column to split
            and the delimiter to use.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(table: FlowDataEngine) -> FlowDataEngine:
        return table.split(node_text_to_rows.text_to_rows_input)

    self.add_node_step(
        node_id=node_text_to_rows.node_id,
        function=_func,
        node_type="text_to_rows",
        setting_input=node_text_to_rows,
        input_node_ids=[node_text_to_rows.depending_on_id],
    )
    return self

add_train_model(train_settings)

Adds a Train Model node.

Fits a regression model on the worker, stores the serialised artifact in the global catalog (via :class:ArtifactService), and passes the input data through unchanged so downstream nodes can keep transforming.

Parameters:

Name	Type	Description	Default
train_settings	NodeTrainModel	Settings (model name, target/features, model_type, params).	required

Returns:

Name	Type	Description
The	FlowGraph	class:FlowGraph instance for chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_train_model(self, train_settings: input_schema.NodeTrainModel) -> "FlowGraph":
    """Adds a Train Model node.

    Fits a regression model on the worker, stores the serialised artifact
    in the global catalog (via :class:`ArtifactService`), and passes the
    input data through unchanged so downstream nodes can keep transforming.

    Args:
        train_settings: Settings (model name, target/features, model_type, params).

    Returns:
        The :class:`FlowGraph` instance for chaining.
    """

    def _func(data: FlowDataEngine) -> FlowDataEngine:
        # Imports are deferred to runtime: importing flowfile_core.artifacts
        # at module top would trigger Alembic migrations and SQLAlchemy
        # engine setup (~3.5s), slowing every flow_graph import — including
        # CLI startup. Keep these inside _func.
        import shutil

        from flowfile_core.artifacts import get_storage_backend
        from flowfile_core.artifacts.service import ArtifactService
        from flowfile_core.auth.utils import get_local_user_id
        from flowfile_core.flowfile.catalog_helpers import (
            auto_register_flow,
            resolve_source_registration_id,
        )
        from flowfile_core.schemas.artifact_schema import PrepareUploadRequest
        from shared.ml.trainers import get_trainer

        settings = train_settings.train_input
        if not settings.target_column:
            raise ValueError("Train Model requires a 'target_column'.")
        if not settings.feature_columns:
            raise ValueError("Train Model requires at least one 'feature_columns' entry.")
        if settings.publish_to_catalog and not settings.model_name:
            raise ValueError("Train Model: 'model_name' is required when 'publish_to_catalog' is enabled.")

        # Validate model_type and hyperparameters early so the user gets
        # a clear error from core, not a worker-side stack trace.
        trainer = get_trainer(settings.model_type)
        try:
            trainer.params_class(**settings.params)
        except Exception as e:
            raise ValueError(f"Train Model: invalid params for model_type={settings.model_type!r}: {e}") from e

        # Always write the model to a flow-scoped path keyed off this node's id;
        # downstream Apply Model nodes in this flow read from there, no catalog
        # required. The same path is used as the staging path when publishing
        # so we only fit once.
        flow_path = ml_flow_model_path(self.flow_id, train_settings.node_id)
        flow_path.parent.mkdir(parents=True, exist_ok=True)

        prepared = None
        owner_id = train_settings.user_id or get_local_user_id() or 1
        staging_path = flow_path
        storage_backend = get_storage_backend()

        if settings.publish_to_catalog:
            # If the flow has a path on disk but no registration yet,
            # auto-register it (idempotently — same mechanism the open/save
            # routes use). This routes scratch flows under "General >
            # Unnamed Flows" / "Local Flows" so artifacts have a stable
            # lineage without forcing the user to explicitly register first.
            registration_id = self._flow_settings.source_registration_id
            if registration_id is None and self._flow_settings.path:
                auto_register_flow(
                    self._flow_settings.path,
                    self._flow_settings.name or "",
                    owner_id,
                )
                resolve_source_registration_id(self)
                registration_id = self._flow_settings.source_registration_id
            if registration_id is None:
                raise ValueError(
                    "Publishing to catalog requires the flow to be registered. "
                    "Save the flow first, or disable 'Publish to catalog'."
                )

            tags = list({"ml", trainer.task_type, settings.model_type, *settings.catalog_tags})
            with get_db_context() as _ns_db:
                effective_namespace_id = _effective_namespace_id(
                    CatalogService(SQLAlchemyCatalogRepository(_ns_db)), settings
                )
                # A published model is a new catalog artifact — gate the target
                # namespace on the executing principal, mirroring the catalog writer.
                _authorize_catalog_write(
                    _ns_db, train_settings.user_id, existing=None, namespace_id=effective_namespace_id
                )
            prepare_request = PrepareUploadRequest(
                name=settings.model_name,
                source_registration_id=registration_id,
                namespace_id=effective_namespace_id,
                serialization_format=trainer.serialization_format,
                description=settings.catalog_description
                or f"Trained via Flowfile node {train_settings.node_id} ({settings.model_type})",
                tags=tags,
                source_flow_id=self.flow_id,
                source_node_id=train_settings.node_id,
                python_type=f"flowfile.ml.{settings.model_type}",
                python_module="flowfile.ml",
            )
            with get_db_context() as db:
                prepared = ArtifactService(db, storage_backend).prepare_upload(prepare_request, owner_id=owner_id)
            if prepared.method != "file":
                # v1 only supports the shared-filesystem backend; S3 needs a
                # presigned-URL path on the worker which we haven't wired yet.
                with get_db_context() as db:
                    ArtifactService(db, storage_backend).delete_artifact(prepared.artifact_id)
                raise ValueError(
                    "Train Model currently requires the filesystem artifact backend "
                    "(FLOWFILE_ARTIFACT_STORAGE=filesystem). S3 support is not implemented."
                )
            # Train into the catalog staging path; we'll copy to the flow
            # path after success so finalize_upload (which moves the
            # staging file to the permanent location) still works.
            staging_path = Path(prepared.path)

        node = self.get_node(node_id=train_settings.node_id)
        flow_path_written = False
        try:
            fetcher = MLTrainFetcher(
                lf=data.data_frame,
                staging_path=str(staging_path),
                model_type=settings.model_type,
                target_column=settings.target_column,
                feature_columns=settings.feature_columns,
                params=settings.params,
                flow_id=self.flow_id,
                node_id=train_settings.node_id,
                file_ref=node.hash,
                wait_on_completion=False,
            )
            node._fetch_cached_df = fetcher
            result = fetcher.get_result()
            if not isinstance(result, dict) or "sha256" not in result or "size_bytes" not in result:
                raise RuntimeError(f"Worker did not return expected sha256/size_bytes payload, got: {result!r}")

            if prepared is not None:
                # The staging file is also our flow-scoped copy. Atomically
                # replace flow_path (write to .tmp, then os.replace) so a
                # concurrent Apply Model reader can't see a half-written
                # file. Done before finalize_upload (which moves the
                # staging file away).
                flow_tmp = flow_path.with_suffix(flow_path.suffix + ".tmp")
                shutil.copyfile(staging_path, flow_tmp)
                os.replace(flow_tmp, flow_path)
                flow_path_written = True
                with get_db_context() as db:
                    ArtifactService(db, storage_backend).finalize_upload(
                        artifact_id=prepared.artifact_id,
                        storage_key=prepared.storage_key,
                        sha256=result["sha256"],
                        size_bytes=result["size_bytes"],
                    )
        except Exception:
            if prepared is not None:
                # Roll back the pending row on any failure so the user
                # doesn't see ghost artifacts; subsequent re-runs auto-clean
                # pending rows too.
                with get_db_context() as db:
                    try:
                        ArtifactService(db, storage_backend).delete_artifact(prepared.artifact_id)
                    except Exception:
                        logger.exception("Failed to roll back pending artifact %s", prepared.artifact_id)
                # Also roll back the flow_path copy if we wrote it; otherwise
                # the next Apply Model run could quietly use the artifact
                # whose catalog row we just deleted.
                if flow_path_written:
                    try:
                        flow_path.unlink(missing_ok=True)
                    except Exception:
                        logger.exception("Failed to roll back flow_path copy %s", flow_path)
            raise

        if prepared is not None:
            self.flow_logger.info(
                f"Train Model: stored '{settings.model_name}' v{prepared.version} "
                f"(artifact_id={prepared.artifact_id}, size={result['size_bytes']}B); "
                f"flow copy at {flow_path}"
            )
            artifact_name = f"{settings.model_name} v{prepared.version}"
        else:
            self.flow_logger.info(f"Train Model: wrote {result['size_bytes']}B to flow path {flow_path}")
            artifact_name = f"{settings.model_type} (flow only)"

        # Surface the trained model in the node's Artifacts tab + canvas badge.
        # Re-runs replace any prior entry rather than accumulating duplicates.
        self.artifact_context.clear_nodes({train_settings.node_id})
        self.artifact_context.record_published(
            node_id=train_settings.node_id,
            kernel_id="",
            artifacts=[
                {
                    "name": artifact_name,
                    "type_name": f"flowfile.ml.{settings.model_type}",
                    "module": "flowfile.ml",
                    "size_bytes": result["size_bytes"],
                }
            ],
        )
        return data

    def schema_callback():
        input_node: FlowNode = self.get_node(train_settings.node_id).node_inputs.main_inputs[0]
        return input_node.schema

    depending_on_id = train_settings.depending_on_id if hasattr(train_settings, "depending_on_id") else None
    self.add_node_step(
        node_id=train_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="train_model",
        setting_input=train_settings,
        schema_callback=schema_callback,
        input_node_ids=[depending_on_id] if depending_on_id is not None else None,
    )
    return self

add_union(union_settings)

Adds a union node to combine multiple data streams.

Parameters:

Name	Type	Description	Default
union_settings	NodeUnion	The settings for the union operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_union(self, union_settings: input_schema.NodeUnion):
    """Adds a union node to combine multiple data streams.

    Args:
        union_settings: The settings for the union operation.
    """

    def _func(*flowfile_tables: FlowDataEngine):
        dfs: list[pl.LazyFrame] | list[pl.DataFrame] = [flt.data_frame for flt in flowfile_tables]
        return FlowDataEngine(pl.concat(dfs, how="diagonal_relaxed"))

    self.add_node_step(
        node_id=union_settings.node_id,
        function=_func,
        node_type="union",
        setting_input=union_settings,
        input_node_ids=union_settings.depending_on_ids,
    )

add_unique(unique_settings)

Adds a node to find and remove duplicate rows.

Parameters:

Name	Type	Description	Default
unique_settings	NodeUnique	The settings for the unique operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_unique(self, unique_settings: input_schema.NodeUnique):
    """Adds a node to find and remove duplicate rows.

    Args:
        unique_settings: The settings for the unique operation.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.make_unique(unique_settings.unique_input)

    self.add_node_step(
        node_id=unique_settings.node_id,
        function=_func,
        input_columns=[],
        node_type="unique",
        setting_input=unique_settings,
        input_node_ids=[unique_settings.depending_on_id],
    )

add_unpivot(unpivot_settings)

Adds an unpivot node to the graph.

Parameters:

Name	Type	Description	Default
unpivot_settings	NodeUnpivot	The settings for the unpivot operation.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_unpivot(self, unpivot_settings: input_schema.NodeUnpivot):
    """Adds an unpivot node to the graph.

    Args:
        unpivot_settings: The settings for the unpivot operation.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.unpivot(unpivot_settings.unpivot_input)

    self.add_node_step(
        node_id=unpivot_settings.node_id,
        function=_func,
        node_type="unpivot",
        setting_input=unpivot_settings,
        input_node_ids=[unpivot_settings.depending_on_id],
    )

add_user_defined_node(*, custom_node, user_defined_node_settings)

Adds a user-defined custom node to the graph.

When the custom node has a kernel_id set, the process code is sent to the kernel for execution instead of running locally. This enables custom nodes to use external packages installed on the kernel.

Parameters:

Name	Type	Description	Default
custom_node	CustomNodeBase	The custom node instance to add.	required
user_defined_node_settings	UserDefinedNode	The settings for the user-defined node.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def add_user_defined_node(
    self, *, custom_node: CustomNodeBase, user_defined_node_settings: input_schema.UserDefinedNode
):
    """Adds a user-defined custom node to the graph.

    When the custom node has a ``kernel_id`` set, the process code is sent
    to the kernel for execution instead of running locally.  This enables
    custom nodes to use external packages installed on the kernel.

    Args:
        custom_node: The custom node instance to add.
        user_defined_node_settings: The settings for the user-defined node.
    """
    kernel_id = user_defined_node_settings.kernel_id or custom_node.kernel_id
    if (custom_node.environment == "kernel" or custom_node.requires_kernel) and not kernel_id:
        raise KernelRequiredError(custom_node.item)

    registry_entry = user_defined_registry.get(custom_node.item)
    if registry_entry is not None and registry_entry.source_hash:
        user_defined_node_settings.node_source_hash = registry_entry.source_hash

    # Output handles are structural — the node class declares them; the settings
    # copy is a persistence snapshot kept in sync for save/codegen.
    output_names = list(custom_node.output_names or user_defined_node_settings.output_names)
    user_defined_node_settings.output_names = output_names

    if kernel_id:
        _func = self._make_kernel_user_defined_func(
            custom_node=custom_node,
            user_defined_node_settings=user_defined_node_settings,
            kernel_id=kernel_id,
            output_names=output_names,
            registry_entry=registry_entry,
        )
    else:
        _func = self._make_local_user_defined_func(
            custom_node=custom_node,
            user_defined_node_settings=user_defined_node_settings,
            output_names=output_names,
            registry_entry=registry_entry,
        )

    # Wire the hook through add_node_step so user_provided_schema_callback is set
    # BEFORE setting_input triggers reset(): otherwise a 0-input node's eager
    # schema prefetch would run the real function (kernel/worker) in the background.
    schema_callback = None
    if type(custom_node).predict_output_schema is not CustomNodeBase.predict_output_schema:
        schema_callback = self._make_user_defined_schema_callback(
            custom_node=custom_node,
            node_id=user_defined_node_settings.node_id,
            output_names=output_names,
        )
    elif not kernel_id and bool(getattr(custom_node, "requires_data_for_prediction", False)):
        # Hookless data-dependent node: never predict by executing; block until run.
        schema_callback = self._make_blocked_prediction_callback(node_id=user_defined_node_settings.node_id)
    else:
        # Traceability: a stale registry class (or a genuinely hook-less node)
        # lands here and schema prediction degrades to the execution tier.
        logger.info(
            f"custom node {custom_node.item}: no predict_output_schema override on "
            f"{type(custom_node).__module__}.{type(custom_node).__name__}; "
            f"schema prediction uses the execution tier"
        )

    self.add_node_step(
        node_id=user_defined_node_settings.node_id,
        function=_func,
        setting_input=user_defined_node_settings,
        input_node_ids=user_defined_node_settings.depending_on_ids,
        node_type=custom_node.item,
        schema_callback=schema_callback,
    )
    node = self.get_node(user_defined_node_settings.node_id)
    node._executes_on_kernel = bool(kernel_id)
    node._prediction_requires_data = bool(getattr(custom_node, "requires_data_for_prediction", False))
    if custom_node.number_of_inputs == 0:
        self.add_node_to_starting_list(node)
    if custom_node.settings_schema is not None and user_defined_node_settings.settings:
        report = custom_node.settings_schema.populate_values_report(user_defined_node_settings.settings)
        if report.has_drift:
            unknown = report.unknown_sections + report.unknown_components
            node.results.warnings = (
                f"Stored settings no longer match the node's schema; ignored keys: {', '.join(sorted(unknown))}"
            )

add_wait_for(settings)

Adds a Wait For node — passes the left input through and waits on the right.

Two distinct input handles like Join: connect the data path to the left and the dependency node (e.g. Train Model) to the right. The right input's data is discarded; only its completion gates this node.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_wait_for(self, settings: input_schema.NodeWaitFor) -> "FlowGraph":
    """Adds a Wait For node — passes the left input through and waits on the right.

    Two distinct input handles like Join: connect the data path to the left
    and the dependency node (e.g. Train Model) to the right. The right
    input's data is discarded; only its completion gates this node.
    """

    def _func(main: FlowDataEngine, right: FlowDataEngine) -> FlowDataEngine:
        # *right* is intentionally unused — its only job is to make sure
        # the framework waits for the dependency node to finish.
        del right
        return main

    def schema_callback():
        node = self.get_node(settings.node_id)
        if node.node_inputs.main_inputs:
            return node.node_inputs.main_inputs[0].schema
        return []

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        input_columns=[],
        node_type="wait_for",
        setting_input=settings,
        schema_callback=schema_callback,
        input_node_ids=settings.depending_on_ids,
    )
    return self

add_window_functions(settings)

Adds a window-functions node (rolling, cumulative, rank, tile).

Parameters:

Name	Type	Description	Default
settings	NodeWindowFunctions	The settings for the window-functions operation.	required

Returns:

Type	Description
FlowGraph	The FlowGraph instance for method chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

@with_history_capture(HistoryActionType.UPDATE_SETTINGS)
def add_window_functions(self, settings: input_schema.NodeWindowFunctions) -> "FlowGraph":
    """Adds a window-functions node (rolling, cumulative, rank, tile).

    Args:
        settings: The settings for the window-functions operation.

    Returns:
        The `FlowGraph` instance for method chaining.
    """

    def _func(fl: FlowDataEngine) -> FlowDataEngine:
        return fl.do_window_functions(settings.window_input, False)

    self.add_node_step(
        node_id=settings.node_id,
        function=_func,
        node_type="window_functions",
        setting_input=settings,
        input_node_ids=[settings.depending_on_id],
    )

    node = self.get_node(settings.node_id)

    def schema_callback():
        depends_on = node.node_inputs.main_inputs[0]
        input_schema_list = list(depends_on.schema)
        input_types = {s.name: s.data_type for s in depends_on.schema}
        output_schema = list(input_schema_list)
        for w in settings.window_input.window_functions:
            src_type = input_types.get(w.column) if w.column else None
            out_type = w.output_type or transform_schema.get_window_output_type(w.function, src_type)
            if out_type is None:
                out_type = src_type or "Float64"
            output_schema.append(FlowfileColumn.from_input(data_type=out_type, column_name=w.new_column_name))
        return output_schema

    node.schema_callback = schema_callback
    return self

apply_layout(y_spacing=150, x_spacing=200, initial_y=100)

Calculates and applies a layered layout to all nodes in the graph.

This updates their x and y positions for UI rendering.

Parameters:

Name	Type	Description	Default
y_spacing	int	The minimum vertical spacing between two nodes in a layer.	150
x_spacing	int	The horizontal spacing between layers.	200
initial_y	int	The y-position of the topmost node.	100

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def apply_layout(self, y_spacing: int = 150, x_spacing: int = 200, initial_y: int = 100):
    """Calculates and applies a layered layout to all nodes in the graph.

    This updates their x and y positions for UI rendering.

    Args:
        y_spacing: The minimum vertical spacing between two nodes in a layer.
        x_spacing: The horizontal spacing between layers.
        initial_y: The y-position of the topmost node.
    """
    self.flow_logger.info("Applying layered layout...")
    start_time = time()
    try:
        new_positions = calculate_layered_layout(
            self, y_spacing=y_spacing, x_spacing=x_spacing, initial_y=initial_y
        )

        if not new_positions:
            self.flow_logger.warning("Layout calculation returned no positions.")
            return

        updated_count = 0
        for node_id, (pos_x, pos_y) in new_positions.items():
            node = self.get_node(node_id)
            if node and hasattr(node, "setting_input"):
                setting = node.setting_input
                if hasattr(setting, "pos_x") and hasattr(setting, "pos_y"):
                    setting.pos_x = pos_x
                    setting.pos_y = pos_y
                    updated_count += 1
                else:
                    self.flow_logger.warning(
                        f"Node {node_id} setting_input ({type(setting)}) lacks pos_x/pos_y attributes."
                    )
            elif node:
                self.flow_logger.warning(f"Node {node_id} lacks setting_input attribute.")
            # else: node removed between calculation and apply; skip it

        # Reflowed node positions invalidate group boxes — refit them.
        self._recompute_group_bounds()

        end_time = time()
        self.flow_logger.info(
            f"Layout applied to {updated_count}/{len(self.nodes)} nodes in {end_time - start_time:.2f} seconds."
        )

    except Exception as e:
        self.flow_logger.error(f"Layout failed, keeping current positions: {e}")

assign_node_to_named_group(node_id, name, *, color=None)

Assign a node to a group identified by name, creating it if absent (find-or-create).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def assign_node_to_named_group(
    self, node_id: int, name: str, *, color: schemas.GroupColor | None = None
) -> schemas.GroupInformation:
    """Assign a node to a group identified by name, creating it if absent (find-or-create)."""
    existing = next((group for group in self._groups.values() if group.name == name), None)
    if existing is not None:
        return self.add_nodes_to_group(existing.id, [node_id])
    return self.create_group(name, [node_id], color=color)

cancel()

Cancels an ongoing graph execution.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def cancel(self):
    """Cancels an ongoing graph execution."""

    if not self.flow_settings.is_running:
        return
    self.flow_settings.is_canceled = True
    for node in self.nodes:
        node.cancel()

capture_history_if_changed(pre_snapshot, action_type, description, node_id=None)

Capture history only if the flow state actually changed.

Use this for settings updates where the change might be a no-op. Call this AFTER the change is applied.

Parameters:

Name	Type	Description	Default
pre_snapshot	FlowfileData	The FlowfileData captured BEFORE the change.	required
action_type	HistoryActionType	The type of action that was performed.	required
description	str	Human-readable description of the action.	required
node_id	int	Optional ID of the affected node.	None

Returns:

Type	Description
bool	True if a change was detected and snapshot was captured.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def capture_history_if_changed(
    self,
    pre_snapshot: schemas.FlowfileData,
    action_type: HistoryActionType,
    description: str,
    node_id: int = None,
) -> bool:
    """Capture history only if the flow state actually changed.

    Use this for settings updates where the change might be a no-op.
    Call this AFTER the change is applied.

    Args:
        pre_snapshot: The FlowfileData captured BEFORE the change.
        action_type: The type of action that was performed.
        description: Human-readable description of the action.
        node_id: Optional ID of the affected node.

    Returns:
        True if a change was detected and snapshot was captured.
    """
    return self._history_manager.capture_if_changed(self, pre_snapshot, action_type, description, node_id)

capture_history_snapshot(action_type, description, node_id=None)

Capture the current state before a change for undo support.

Parameters:

Name	Type	Description	Default
action_type	HistoryActionType	The type of action being performed.	required
description	str	Human-readable description of the action.	required
node_id	int	Optional ID of the affected node.	None

Returns:

Type	Description
bool	True if snapshot was captured, False if skipped.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def capture_history_snapshot(
    self,
    action_type: HistoryActionType,
    description: str,
    node_id: int = None,
) -> bool:
    """Capture the current state before a change for undo support.

    Args:
        action_type: The type of action being performed.
        description: Human-readable description of the action.
        node_id: Optional ID of the affected node.

    Returns:
        True if snapshot was captured, False if skipped.
    """
    return self._history_manager.capture_snapshot(self, action_type, description, node_id)

check_flow_laziness()

Check whether the flow supports lazy execution for virtual tables.

Finds all catalog-writer nodes in the graph and checks whether their upstream dependencies are fully lazy. Only the nodes that actually feed into a catalog writer matter — unrelated branches (e.g. an Explore Data node on a separate path) are ignored.

Returns a tuple of (is_optimizable, reasons_if_not).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def check_flow_laziness(self) -> tuple[bool, list[str]]:
    """Check whether the flow supports lazy execution for virtual tables.

    Finds all catalog-writer nodes in the graph and checks whether their
    upstream dependencies are fully lazy.  Only the nodes that actually
    feed into a catalog writer matter — unrelated branches (e.g. an
    Explore Data node on a separate path) are ignored.

    Returns a tuple of (is_optimizable, reasons_if_not).
    """
    catalog_writers = [n for n in self.nodes if n.node_type == "catalog_writer"]
    if not catalog_writers:
        # No catalog writer → nothing to optimise; treat as non-lazy
        return False, ["No catalog writer node found in the flow"]
    all_reasons: list[str] = []
    for writer in catalog_writers:
        _, reasons = writer.check_upstream_laziness()
        all_reasons.extend(reasons)
    seen: set[str] = set()
    unique: list[str] = []
    for r in all_reasons:
        if r not in seen:
            seen.add(r)
            unique.append(r)
    return len(unique) == 0, unique

close_flow()

Performs cleanup operations, such as clearing node caches.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def close_flow(self):
    """Performs cleanup operations, such as clearing node caches."""

    for node in self.nodes:
        node.remove_cache()

copy_node(new_node_settings, existing_setting_input, node_type)

Creates a copy of an existing node.

Parameters:

Name	Type	Description	Default
new_node_settings	NodePromise	The promise containing new settings (like ID and position).	required
existing_setting_input	Any	The settings object from the node being copied.	required
node_type	str	The type of the node being copied.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def copy_node(
    self, new_node_settings: input_schema.NodePromise, existing_setting_input: Any, node_type: str
) -> None:
    """Creates a copy of an existing node.

    Args:
        new_node_settings: The promise containing new settings (like ID and position).
        existing_setting_input: The settings object from the node being copied.
        node_type: The type of the node being copied.
    """
    # A custom node whose type isn't installed needs a placeholder template before
    # the promise can be placed (mirrors the flow-restore path).
    if getattr(existing_setting_input, "is_user_defined", False) and node_type not in CUSTOM_NODE_STORE:
        register_missing_node_template(node_type)
    self.add_node_promise(new_node_settings)

    if isinstance(existing_setting_input, input_schema.NodePromise):
        return

    combined_settings = combine_existing_settings_and_new_settings(existing_setting_input, new_node_settings)
    # Subflow port names must stay unique; auto-rename the copy so it doesn't collide with the source.
    if node_type == "flow_output" and isinstance(combined_settings, input_schema.NodeFlowOutput):
        combined_settings.output_name = self._unique_subflow_port_name(
            combined_settings.output_name, node_type, combined_settings.node_id
        )
    elif node_type == "flow_input" and isinstance(combined_settings, input_schema.NodeFlowInput):
        combined_settings.input_name = self._unique_subflow_port_name(
            combined_settings.input_name, node_type, combined_settings.node_id
        )
    try:
        if getattr(existing_setting_input, "is_user_defined", False):
            self._place_user_defined_node(node_type, combined_settings)
        else:
            getattr(self, f"add_{node_type}")(combined_settings)
    except Exception:
        # A failed copy must not leave the pre-added promise dangling in the graph.
        if self.get_node(new_node_settings.node_id) is not None:
            self.delete_node(new_node_settings.node_id)
        raise

create_group(name, node_ids, *, color=None, bounds=None, parent_group_id=None, child_group_ids=None)

Create a visual group. Organizational only.

Members are the given nodes (group_id) and child groups (their parent_group_id). The new group itself nests under parent_group_id. Bounds are computed when not supplied.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def create_group(
    self,
    name: str,
    node_ids: list[int],
    *,
    color: schemas.GroupColor | None = None,
    bounds: schemas.GroupBounds | None = None,
    parent_group_id: int | None = None,
    child_group_ids: list[int] | None = None,
) -> schemas.GroupInformation:
    """Create a visual group. Organizational only.

    Members are the given nodes (group_id) and child groups (their parent_group_id).
    The new group itself nests under parent_group_id. Bounds are computed when not supplied.
    """

    def _do() -> schemas.GroupInformation:
        group_id = self._next_group_id()
        group = schemas.GroupInformation(id=group_id, name=name, color=color, parent_group_id=parent_group_id)
        if bounds is not None:
            group.x_position, group.y_position, group.width, group.height = bounds
        self._groups[group_id] = group
        for node_id in node_ids:
            self._set_node_group(node_id, group_id)
        for cid in child_group_ids or []:
            child = self._groups.get(cid)
            if child is not None and not self._is_ancestor_group(cid, group_id):
                child.parent_group_id = group_id
        if bounds is None:
            self._recompute_group_bounds(group_id)
        return group

    return self._execute_with_history(_do, HistoryActionType.CREATE_GROUP, f"Create group '{name}'")

delete_group(group_id)

Remove a group box (ungroup). Members and sub-groups lift up one level.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def delete_group(self, group_id: int) -> None:
    """Remove a group box (ungroup). Members and sub-groups lift up one level."""
    group = self._groups.get(group_id)
    if group is None:
        return
    new_parent = group.parent_group_id

    def _do() -> None:
        for node_id in self._member_node_ids(group_id):
            self._set_node_group(node_id, new_parent)
        for cid in self._child_group_ids(group_id):
            child = self._groups.get(cid)
            if child is not None:
                child.parent_group_id = new_parent
        self._groups.pop(group_id, None)

    self._execute_with_history(_do, HistoryActionType.DELETE_GROUP, f"Delete group '{group.name}'")

delete_node(node_id)

Deletes a node from the graph and updates all its connections.

Parameters:

Name	Type	Description	Default
node_id	int | str	The ID of the node to delete.	required

Raises:

Type	Description
Exception	If the node with the given ID does not exist.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def delete_node(self, node_id: int | str):
    """Deletes a node from the graph and updates all its connections.

    Args:
        node_id: The ID of the node to delete.

    Raises:
        Exception: If the node with the given ID does not exist.
    """
    logger.info(f"Starting deletion of node with ID: {node_id}")

    node = self._node_db.get(node_id)
    if node:
        logger.info(f"Found node: {node_id}, processing deletion")
        group_id = getattr(node.setting_input, "group_id", None)

        lead_to_steps: list[FlowNode] = node.leads_to_nodes
        logger.debug(f"Node {node_id} leads to {len(lead_to_steps)} other nodes")

        if len(lead_to_steps) > 0:
            for lead_to_step in lead_to_steps:
                logger.debug(f"Deleting input node {node_id} from dependent node {lead_to_step}")
                lead_to_step.delete_input_node(node_id, complete=True)

        if not node.is_start:
            depends_on: list[FlowNode] = node.node_inputs.get_all_inputs()
            logger.debug(f"Node {node_id} depends on {len(depends_on)} other nodes")

            for depend_on in depends_on:
                logger.debug(f"Removing lead_to reference {node_id} from node {depend_on}")
                depend_on.delete_lead_to_node(node_id)

        self._node_db.pop(node_id)
        logger.debug(f"Successfully removed node {node_id} from node_db")
        del node
        logger.info("Node object deleted")
        # Drop a group that just lost its last member (keep it if it still holds sub-groups).
        if (
            group_id is not None
            and group_id in self._groups
            and not self._member_node_ids(group_id)
            and not self._child_group_ids(group_id)
        ):
            self._groups.pop(group_id, None)
    else:
        logger.error(f"Failed to find node with id {node_id}")
        raise Exception(f"Node with id {node_id} does not exist")

generate_code()

Generates code for the flow graph. This method exports the flow graph to a Polars-compatible format.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def generate_code(self):
    """Generates code for the flow graph.
    This method exports the flow graph to a Polars-compatible format.
    """
    from flowfile_core.flowfile.code_generator.code_generator import export_flow_to_polars

    print(export_flow_to_polars(self))

get_frontend_data()

Formats the graph structure into a JSON-like dictionary for a specific legacy frontend.

This method transforms the graph's state into a format compatible with the Drawflow.js library.

Returns:

Type	Description
dict	A dictionary representing the graph in Drawflow format.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_frontend_data(self) -> dict:
    """Formats the graph structure into a JSON-like dictionary for a specific legacy frontend.

    This method transforms the graph's state into a format compatible with the
    Drawflow.js library.

    Returns:
        A dictionary representing the graph in Drawflow format.
    """
    result = {"Home": {"data": {}}}
    flow_info: schemas.FlowInformation = self.get_node_storage()

    for node_id, node_info in flow_info.data.items():
        if node_info.is_setup:
            try:
                pos_x = node_info.data.pos_x
                pos_y = node_info.data.pos_y
                result["Home"]["data"][str(node_id)] = {
                    "id": node_info.id,
                    "name": node_info.type,
                    "data": {},
                    "class": node_info.type,
                    "html": node_info.type,
                    "typenode": "vue",
                    "inputs": {},
                    "outputs": {},
                    "pos_x": pos_x,
                    "pos_y": pos_y,
                }
            except Exception as e:
                logger.error(e)
        if node_info.outputs:
            outputs = {o: 0 for o in node_info.outputs}
            for o in node_info.outputs:
                outputs[o] += 1
            connections = []
            for output_node_id, _n_connections in outputs.items():
                leading_to_node = self.get_node(output_node_id)
                input_types = leading_to_node.get_input_type(node_info.id)
                for input_type in input_types:
                    if input_type == "main":
                        input_frontend_id = "input_1"
                    elif input_type == "right":
                        input_frontend_id = "input_2"
                    elif input_type == "left":
                        input_frontend_id = "input_3"
                    else:
                        input_frontend_id = "input_1"
                    connection = {"node": str(output_node_id), "input": input_frontend_id}
                    connections.append(connection)

            result["Home"]["data"][str(node_id)]["outputs"]["output_1"] = {"connections": connections}
        else:
            result["Home"]["data"][str(node_id)]["outputs"] = {"output_1": {"connections": []}}

        if (
            node_info.left_input_id is not None
            or node_info.right_input_id is not None
            or node_info.input_ids is not None
        ):
            main_inputs = node_info.main_input_ids
            result["Home"]["data"][str(node_id)]["inputs"]["input_1"] = {
                "connections": [{"node": str(main_node_id), "input": "output_1"} for main_node_id in main_inputs]
            }
            if node_info.right_input_id is not None:
                result["Home"]["data"][str(node_id)]["inputs"]["input_2"] = {
                    "connections": [{"node": str(node_info.right_input_id), "input": "output_1"}]
                }
            if node_info.left_input_id is not None:
                result["Home"]["data"][str(node_id)]["inputs"]["input_3"] = {
                    "connections": [{"node": str(node_info.left_input_id), "input": "output_1"}]
                }
    return result

get_history_state()

Get the current state of the history system.

Returns:

Type	Description
HistoryState	HistoryState with information about available undo/redo operations.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_history_state(self) -> HistoryState:
    """Get the current state of the history system.

    Returns:
        HistoryState with information about available undo/redo operations.
    """
    return self._history_manager.get_state()

get_implicit_starter_nodes()

Finds nodes that can act as starting points but are not explicitly defined as such.

Some nodes, like the Polars Code node, can function without an input. This method identifies such nodes if they have no incoming connections.

Returns:

Type	Description
list[FlowNode]	A list of FlowNode objects that are implicit starting nodes.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_implicit_starter_nodes(self) -> list[FlowNode]:
    """Finds nodes that can act as starting points but are not explicitly defined as such.

    Some nodes, like the Polars Code node, can function without an input. This
    method identifies such nodes if they have no incoming connections.

    Returns:
        A list of `FlowNode` objects that are implicit starting nodes.
    """
    starting_node_ids = [node.node_id for node in self._flow_starts]
    implicit_starting_nodes = []
    for node in self.nodes:
        if node.node_template.can_be_start and not node.has_input and node.node_id not in starting_node_ids:
            implicit_starting_nodes.append(node)
    return implicit_starting_nodes

get_node(node_id=None)

Retrieves a node from the graph by its ID.

Parameters:

Name	Type	Description	Default
node_id	int | str	The ID of the node to retrieve. If None, retrieves the last added node.	None

Returns:

Type	Description
FlowNode | None	The FlowNode object, or None if not found.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_node(self, node_id: int | str = None) -> FlowNode | None:
    """Retrieves a node from the graph by its ID.

    Args:
        node_id: The ID of the node to retrieve. If None, retrieves the last added node.

    Returns:
        The FlowNode object, or None if not found.
    """
    if node_id is None:
        node_id = self._node_ids[-1]
    node = self._node_db.get(node_id)
    if node is not None:
        return node

get_node_data(node_id, include_example=True)

Retrieves all data needed to render a node in the UI.

Parameters:

Name	Type	Description	Default
node_id	int	The ID of the node.	required
include_example	bool	Whether to include data samples in the result.	True

Returns:

Type	Description
NodeData	A NodeData object, or None if the node is not found.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_node_data(self, node_id: int, include_example: bool = True) -> NodeData:
    """Retrieves all data needed to render a node in the UI.

    Args:
        node_id: The ID of the node.
        include_example: Whether to include data samples in the result.

    Returns:
        A NodeData object, or None if the node is not found.
    """
    node = self._node_db[node_id]
    return node.get_node_data(flow_id=self.flow_id, include_example=include_example)

get_node_storage()

Serializes the entire graph's state into a storable format.

Returns:

Type	Description
FlowInformation	A FlowInformation object representing the complete graph.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_node_storage(self) -> schemas.FlowInformation:
    """Serializes the entire graph's state into a storable format.

    Returns:
        A FlowInformation object representing the complete graph.
    """
    node_information = {
        node.node_id: node.get_node_information() for node in self.nodes if node.is_setup and node.is_correct
    }

    return schemas.FlowInformation(
        flow_id=self.flow_id,
        flow_name=self.__name__,
        flow_settings=self.flow_settings,
        data=node_information,
        node_starts=[v.node_id for v in self._flow_starts],
        node_connections=self.node_connections,
    )

get_nodes_overview()

Gets a list of dictionary representations for all nodes in the graph.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_nodes_overview(self):
    """Gets a list of dictionary representations for all nodes in the graph."""
    output = []
    for v in self._node_db.values():
        output.append(v.get_repr())
    return output

get_run_info()

Gets a summary of the most recent graph execution.

Returns:

Type	Description
RunInformation	A RunInformation object with details about the last run.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_run_info(self) -> RunInformation:
    """Gets a summary of the most recent graph execution.

    Returns:
        A RunInformation object with details about the last run.
    """
    is_running = self.flow_settings.is_running
    if self.latest_run_info is None:
        return self.create_empty_run_information()

    run_info = self.latest_run_info
    run_info.is_running = is_running
    run_info.execution_mode = self.flow_settings.execution_mode
    if not is_running and run_info.success is None:
        run_info.success = all(nr.success for nr in run_info.node_step_result)
    return run_info

get_vue_flow_input()

Formats the graph's nodes and edges into a schema suitable for the VueFlow frontend.

Returns:

Type	Description
VueFlowInput	A VueFlowInput object.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def get_vue_flow_input(self) -> schemas.VueFlowInput:
    """Formats the graph's nodes and edges into a schema suitable for the VueFlow frontend.

    Returns:
        A VueFlowInput object.
    """
    edges: list[schemas.NodeEdge] = []
    nodes: list[schemas.NodeInput] = []
    for node in self.nodes:
        nodes.append(node.get_node_input())
        edges.extend(node.get_edge_input())
    groups = [
        schemas.FlowfileGroup(**self._groups[group_id].model_dump())
        for group_id in self._groups
        if self._member_node_ids(group_id) or self._child_group_ids(group_id)
    ]
    return schemas.VueFlowInput(node_edges=edges, node_inputs=nodes, groups=groups)

has_unsaved_changes()

Return True if the flow has changed since the last save point.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def has_unsaved_changes(self) -> bool:
    """Return True if the flow has changed since the last save point."""
    return self._history_manager.has_unsaved_changes(self)

mark_as_saved()

Mark the current flow state as the saved baseline (for dirty tracking).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def mark_as_saved(self) -> None:
    """Mark the current flow state as the saved baseline (for dirty tracking)."""
    self._history_manager.mark_saved(self)

print_tree()

Print flow_graph as a visual tree structure, showing the DAG relationships with ASCII art.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def print_tree(self):
    """Print flow_graph as a visual tree structure, showing the DAG relationships with ASCII art."""
    if not self._node_db:
        self.flow_logger.info("Empty flow graph")
        return

    node_info = build_node_info(self.nodes)

    for node_id in node_info:
        calculate_depth(node_id, node_info)

    depth_groups, max_depth = group_nodes_by_depth(node_info)

    for depth in depth_groups:
        depth_groups[depth].sort()

    lines = ["=" * 80, "Flow Graph Visualization", "=" * 80, ""]

    merge_points = define_node_connections(node_info)

    max_label_length = {}
    for depth in range(max_depth + 1):
        if depth in depth_groups:
            max_len = max(len(node_info[nid].label) for nid in depth_groups[depth])
            max_label_length[depth] = max_len

    drawn_nodes = set()
    merge_drawn = set()

    paths_by_merge = {}
    standalone_paths = []

    paths = build_flow_paths(node_info, self._flow_starts, merge_points)

    for path in paths:
        if len(path) > 1 and path[-1] in merge_points and len(merge_points[path[-1]]) > 1:
            merge_id = path[-1]
            if merge_id not in paths_by_merge:
                paths_by_merge[merge_id] = []
            paths_by_merge[merge_id].append(path)
        else:
            standalone_paths.append(path)

    draw_merged_paths(node_info, merge_points, paths_by_merge, merge_drawn, drawn_nodes, lines)

    draw_standalone_paths(drawn_nodes, standalone_paths, lines, node_info)

    add_un_drawn_nodes(drawn_nodes, node_info, lines)

    try:
        execution_plan = compute_execution_plan(
            nodes=self.nodes, flow_starts=self._flow_starts + self.get_implicit_starter_nodes()
        )
        ordered_nodes = execution_plan.all_nodes
        if ordered_nodes:
            for i, node in enumerate(ordered_nodes, 1):
                lines.append(f"  {i:3d}. {node_info[node.node_id].label}")
    except Exception as e:
        lines.append(f"  Could not determine execution order: {e}")

    output = "\n".join(lines)

    print(output)

redo()

Redo the last undone action.

Returns:

Type	Description
UndoRedoResult	UndoRedoResult indicating success or failure.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def redo(self) -> UndoRedoResult:
    """Redo the last undone action.

    Returns:
        UndoRedoResult indicating success or failure.
    """
    return self._history_manager.redo(self)

release_run()

Release the single-run slot claimed by try_claim_run (idempotent).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def release_run(self) -> None:
    """Release the single-run slot claimed by try_claim_run (idempotent)."""
    with self._run_claim_lock:
        self.flow_settings.is_running = False

remove_from_output_cols(columns)

Removes specified columns from the list of expected output columns.

Parameters:

Name	Type	Description	Default
columns	list[str]	A list of column names to remove.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def remove_from_output_cols(self, columns: list[str]):
    """Removes specified columns from the list of expected output columns.

    Args:
        columns: A list of column names to remove.
    """
    cols = set(columns)
    self._output_cols = [c for c in self._output_cols if c not in cols]

remove_nodes_from_group(node_ids)

Remove nodes from whatever group they belong to; prune groups left empty.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def remove_nodes_from_group(self, node_ids: list[int]) -> None:
    """Remove nodes from whatever group they belong to; prune groups left empty."""

    def _do() -> None:
        affected: set[int] = set()
        for node_id in node_ids:
            node = self.get_node(node_id)
            current = getattr(node.setting_input, "group_id", None) if node is not None else None
            if current is not None:
                affected.add(current)
                self._set_node_group(node_id, None)
        for gid in affected:
            if not self._member_node_ids(gid) and not self._child_group_ids(gid):
                self._groups.pop(gid, None)
            else:
                self._recompute_group_bounds(gid)

    self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP_MEMBERSHIP, "Remove nodes from group")

reset()

Forces a deep reset on all nodes in the graph.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def reset(self):
    """Forces a deep reset on all nodes in the graph."""

    for node in self.nodes:
        node.reset(True)

restore_from_snapshot(snapshot)

Clear current state and rebuild from a snapshot.

This method is used internally by undo/redo to restore a previous state.

Parameters:

Name	Type	Description	Default
snapshot	FlowfileData	The FlowfileData snapshot to restore from.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def restore_from_snapshot(self, snapshot: schemas.FlowfileData) -> None:
    """Clear current state and rebuild from a snapshot.

    This method is used internally by undo/redo to restore a previous state.

    Args:
        snapshot: The FlowfileData snapshot to restore from.
    """
    from flowfile_core.flowfile.manage.io_flowfile import (
        _flowfile_data_to_flow_information,
        determine_insertion_order,
    )

    identity = _FlowIdentity.capture(self)
    node_owners = _NodeOwners.capture(self)

    flow_info = _flowfile_data_to_flow_information(snapshot)

    self._node_db.clear()
    self._node_ids.clear()
    self._flow_starts.clear()
    self._groups.clear()
    self._results = None

    self._flow_settings = flow_info.flow_settings
    identity.restore_onto(self)

    ingestion_order = determine_insertion_order(flow_info)

    for node_id in ingestion_order:
        node_info = flow_info.data[node_id]
        if getattr(node_info.setting_input, "is_user_defined", False) and node_info.type not in CUSTOM_NODE_STORE:
            register_missing_node_template(node_info.type)
        node_promise = input_schema.NodePromise(
            flow_id=identity.flow_id,
            node_id=node_info.id,
            pos_x=node_info.x_position or 0,
            pos_y=node_info.y_position or 0,
            node_type=node_info.type,
        )
        if hasattr(node_info.setting_input, "cache_results"):
            node_promise.cache_results = node_info.setting_input.cache_results
        self.add_node_promise(node_promise)

    for node_id in ingestion_order:
        node_info = flow_info.data[node_id]
        if node_info.is_setup and node_info.setting_input is not None:
            if hasattr(node_info.setting_input, "flow_id"):
                node_info.setting_input.flow_id = identity.flow_id

            if hasattr(node_info.setting_input, "user_id"):
                node_info.setting_input.user_id = node_owners.owner_of(node_id)

            if hasattr(node_info.setting_input, "is_user_defined") and node_info.setting_input.is_user_defined:
                # .get() execs the node module lazily; on any failure the node
                # lands in the missing/error path so the flow still opens.
                self._place_user_defined_node(node_info.type, node_info.setting_input)
            else:
                add_method = getattr(self, "add_" + node_info.type, None)
                if add_method:
                    add_method(node_info.setting_input)

    for node_id in ingestion_order:
        node_info = flow_info.data[node_id]
        from_node = self.get_node(node_id)
        if from_node is None:
            continue

        for output_node_id in node_info.outputs or []:
            to_node = self.get_node(output_node_id)
            if to_node is None:
                continue
            if to_node.accepts_dynamic_inputs:
                continue  # keyed edges are restored from input_connections below

            output_node_info = flow_info.data.get(output_node_id)
            if output_node_info is None:
                continue

            is_left_input = (output_node_info.left_input_id == node_id) and (
                to_node.left_input is None or to_node.left_input.node_id != node_id
            )
            is_right_input = (output_node_info.right_input_id == node_id) and (
                to_node.right_input is None or to_node.right_input.node_id != node_id
            )
            is_main_input = node_id in (output_node_info.input_ids or [])

            if is_left_input:
                insert_type = "left"
            elif is_right_input:
                insert_type = "right"
            elif is_main_input:
                insert_type = "main"
            else:
                continue

            to_node.add_node_connection(from_node, insert_type)

    restore_dynamic_input_connections(self, flow_info)

    # Member group_ids were re-applied above via add_<type>(setting_input);
    # repopulate the box registry (name/color/bounds) from the snapshot.
    self.restore_groups(flow_info.groups)

    logger.info(f"Restored flow from snapshot with {len(self._node_db)} nodes")

restore_groups(groups)

Replace the runtime group registry (used by open_flow and restore_from_snapshot).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def restore_groups(self, groups: list[schemas.GroupInformation]) -> None:
    """Replace the runtime group registry (used by open_flow and restore_from_snapshot)."""
    self._groups = {group.id: group for group in groups}
    self._group_id_seq = max(self._groups, default=0)  # next id resumes above the highest restored
    for group in self._groups.values():
        if group.width <= 0 or group.height <= 0:
            self._recompute_group_bounds(group.id)

run_graph()

Executes the entire data flow graph from start to finish.

Independent nodes within the same execution stage are run in parallel using threads. Stages are processed sequentially so that all dependencies are satisfied before a stage begins.

Returns:

Type	Description
RunInformation | None	A RunInformation object summarizing the execution results.

Raises:

Type	Description
Exception	If the flow is already running.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def run_graph(self) -> RunInformation | None:
    """Executes the entire data flow graph from start to finish.

    Independent nodes within the same execution stage are run in parallel
    using threads. Stages are processed sequentially so that all dependencies
    are satisfied before a stage begins.

    Returns:
        A RunInformation object summarizing the execution results.

    Raises:
        Exception: If the flow is already running.
    """
    if not self.try_claim_run():
        raise Exception("Flow is already running")
    try:
        self.flow_settings.is_canceled = False
        self.flow_logger.clear_log_file()
        self.flow_logger.info("Starting to run flowfile flow...")

        self._refresh_catalog_reader_freshness()

        execution_plan = compute_execution_plan(
            nodes=self.nodes, flow_starts=self._flow_starts + self.get_implicit_starter_nodes()
        )

        plan_skip_ids: set[str | int] = {n.node_id for n in execution_plan.skip_nodes}
        self._prepare_rerun_artifacts(plan_skip_ids)

        self.latest_run_info = self.create_initial_run_information(execution_plan.node_count, "full_run")
        skip_node_message(self.flow_logger, execution_plan.skip_nodes)
        execution_order_message(self.flow_logger, execution_plan.stages)

        performance_mode = self.flow_settings.execution_mode == "Performance"
        params: dict[str, ParamValue] = {p.name: p.typed_default() for p in self.flow_settings.parameters}

        failed_node_ids = self._execute_stages(execution_plan, performance_mode, params, plan_skip_ids)
        if not self.flow_settings.is_canceled:
            self._run_post_execution_callbacks(failed_node_ids, plan_skip_ids)

        self.latest_run_info.end_time = datetime.datetime.now()
        self.flow_logger.info("Flow completed!")
        self.end_datetime = datetime.datetime.now()
        self.release_run()
        if self.flow_settings.is_canceled:
            self.flow_logger.info("Flow canceled")
        return self.get_run_info()
    except Exception as e:
        raise e
    finally:
        self.release_run()

save_flow(flow_path)

Saves the current state of the flow graph to a file.

Supports multiple formats based on file extension: - .yaml / .yml: New YAML format - .json: JSON format

Parameters:

Name	Type	Description	Default
flow_path	str	The path where the flow file will be saved.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def save_flow(self, flow_path: str):
    """Saves the current state of the flow graph to a file.

    Supports multiple formats based on file extension:
    - .yaml / .yml: New YAML format
    - .json: JSON format

    Args:
        flow_path: The path where the flow file will be saved.
    """
    logger.info("Saving flow to %s", flow_path)
    path = Path(flow_path)
    os.makedirs(path.parent, exist_ok=True)
    suffix = path.suffix.lower()
    new_flow_name = path.name.replace(suffix, "")
    self._handle_flow_renaming(new_flow_name, path)
    self.flow_settings.modified_on = datetime.datetime.now().timestamp()
    self._validate_registration_ownership(flow_path)
    try:
        if suffix == ".flowfile":
            raise DeprecationWarning(
                "The .flowfile format is deprecated. Please use .yaml or .json formats.\n\n"
                "Or stay on.1 if you still need .flowfile support.\n\n"
            )
        elif suffix in (".yaml", ".yml"):
            flowfile_data = self.get_flowfile_data()
            data = flowfile_data.model_dump(mode="json")
            with open(flow_path, "w", encoding="utf-8") as f:
                yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)
        elif suffix == ".json":
            flowfile_data = self.get_flowfile_data()
            data = flowfile_data.model_dump(mode="json")
            with open(flow_path, "w", encoding="utf-8") as f:
                json.dump(data, f, indent=2, ensure_ascii=False)

        else:
            flowfile_data = self.get_flowfile_data()
            logger.warning(f"Unknown file extension {suffix}. Defaulting to YAML format.")
            data = flowfile_data.model_dump(mode="json")
            with open(flow_path, "w", encoding="utf-8") as f:
                yaml.dump(data, f, default_flow_style=False, sort_keys=False, allow_unicode=True)

    except Exception as e:
        logger.error(f"Error saving flow: {e}")
        raise

    self.flow_settings.path = flow_path
    self._sync_catalog_read_links()
    # Record the current state as the clean baseline for dirty tracking
    self.mark_as_saved()

set_group_bounds(updates)

Persist group box bounds (used together with set_node_positions on drag/resize).

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def set_group_bounds(self, updates: list[schemas.GroupBoundsUpdate]) -> None:
    """Persist group box bounds (used together with set_node_positions on drag/resize)."""
    for update in updates:
        group = self._groups.get(update.group_id)
        if group is not None:
            group.x_position = update.x_position
            group.y_position = update.y_position
            group.width = update.width
            group.height = update.height

set_node_positions(updates)

Persist dragged node positions (absolute canvas coordinates) onto setting_input.

Plain mutator: the caller (update_layout route) captures history once for the whole drag-end batch so node moves and group-bounds changes share one snapshot.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def set_node_positions(self, updates: list[schemas.NodePositionUpdate]) -> None:
    """Persist dragged node positions (absolute canvas coordinates) onto setting_input.

    Plain mutator: the caller (update_layout route) captures history once for the
    whole drag-end batch so node moves and group-bounds changes share one snapshot.
    """
    for update in updates:
        node = self.get_node(update.node_id)
        if node is not None and node.setting_input is not None and hasattr(node.setting_input, "pos_x"):
            node.setting_input.pos_x = update.pos_x
            node.setting_input.pos_y = update.pos_y

trigger_fetch_node(node_id, *, performance_mode=False, reset_cache=True)

Executes a specific node in the graph by its ID.

The defaults are the data-preview contract: a non-performance run, so the node stores its result and can serve the 100-row example grid. Callers that only need the node's query plan (the Explore Data drawer) pass performance_mode=True, which skips that store entirely, and reset_cache=False so exploring doesn't evict a useful cache.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def trigger_fetch_node(
    self,
    node_id: int,
    *,
    performance_mode: bool = False,
    reset_cache: bool = True,
) -> RunInformation | None:
    """Executes a specific node in the graph by its ID.

    The defaults are the data-preview contract: a non-performance run, so the
    node stores its result and can serve the 100-row example grid. Callers
    that only need the node's query plan (the Explore Data drawer) pass
    ``performance_mode=True``, which skips that store entirely, and
    ``reset_cache=False`` so exploring doesn't evict a useful cache.
    """
    if not self.try_claim_run():
        raise Exception("Flow is already running")
    flow_node = self.get_node(node_id)
    self.flow_settings.is_canceled = False
    self.flow_logger.clear_log_file()
    self.latest_run_info = self.create_initial_run_information(1, "fetch_one")
    node_logger = self.flow_logger.get_node_logger(flow_node.node_id)
    node_result = NodeResult(
        node_id=flow_node.node_id,
        node_name=flow_node.name,
        description=flow_node.get_node_information().description,
    )
    logger.info(f"Starting to run: node {flow_node.node_id}, start time: {node_result.start_timestamp}")
    try:
        self.latest_run_info.node_step_result.append(node_result)
        flow_node.execute_node(
            run_location=self.flow_settings.execution_location,
            performance_mode=performance_mode,
            node_logger=node_logger,
            optimize_for_downstream=False,
            reset_cache=reset_cache,
        )
        node_result.error = str(flow_node.results.errors)
        if self.flow_settings.is_canceled:
            node_result.success = None
            node_result.success = None
            node_result.is_running = False
        node_result.success = flow_node.results.errors is None
        node_result.end_timestamp = time()
        node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
        node_result.is_running = False
        self.latest_run_info.nodes_completed += 1
        self.latest_run_info.end_time = datetime.datetime.now()
        self.release_run()
        return self.get_run_info()
    except Exception as e:
        node_result.error = "Node did not run"
        node_result.success = False
        node_result.end_timestamp = time()
        node_result.run_time_ms = int((node_result.end_timestamp - node_result.start_timestamp) * 1000)
        node_result.is_running = False
        node_logger.error(f"Error in node {flow_node.node_id}: {e}")
    finally:
        self.release_run()

try_claim_run()

Atomically claim the flow's single-run slot; False when a run is already in flight.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def try_claim_run(self) -> bool:
    """Atomically claim the flow's single-run slot; False when a run is already in flight."""
    with self._run_claim_lock:
        if self.flow_settings.is_running:
            return False
        self.flow_settings.is_running = True
        return True

undo()

Undo the last action by restoring to the previous state.

Returns:

Type	Description
UndoRedoResult	UndoRedoResult indicating success or failure.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def undo(self) -> UndoRedoResult:
    """Undo the last action by restoring to the previous state.

    Returns:
        UndoRedoResult indicating success or failure.
    """
    return self._history_manager.undo(self)

update_group(group_id, *, name=None, color=None, bounds=None, collapsed=None)

Rename / recolor / move / resize / collapse a group box.

Source code in flowfile_core/flowfile_core/flowfile/flow_graph.py

def update_group(
    self,
    group_id: int,
    *,
    name: str | None = None,
    color: schemas.GroupColor | None = None,
    bounds: schemas.GroupBounds | None = None,
    collapsed: bool | None = None,
) -> schemas.GroupInformation:
    """Rename / recolor / move / resize / collapse a group box."""
    group = self._groups.get(group_id)
    if group is None:
        raise ValueError(f"Group {group_id} does not exist")

    def _do() -> schemas.GroupInformation:
        if name is not None:
            group.name = name
        if color is not None:
            group.color = color
        if bounds is not None:
            group.x_position, group.y_position, group.width, group.height = bounds
        if collapsed is not None:
            group.collapsed = collapsed
        return group

    return self._execute_with_history(_do, HistoryActionType.UPDATE_GROUP, f"Update group '{group.name}'")

FlowNode

The FlowNode represents a single operation in the FlowGraph. Each node corresponds to a specific transformation or action, such as filtering or grouping data.

flowfile_core.flowfile.flow_node.flow_node.FlowNode

Represents a single node in a data flow graph.

This class manages the node's state, its data processing function, and its connections to other nodes within the graph.

Methods:

Name	Description
__call__	Makes the node instance callable, acting as an alias for execute_node.
__init__	Initializes a FlowNode instance.
__repr__	Provides a string representation of the FlowNode instance.
add_lead_to_in_depend_source	Ensures this node is registered in the leads_to_nodes list of its inputs.
add_node_connection	Adds a connection from a source node to this node.
calculate_hash	Calculates a hash based on settings and input node hashes.
cancel	Cancels an ongoing external process if one is running.
check_upstream_laziness	Check whether all upstream dependencies of this node support lazy execution.
clear_table_example	Clear the table example in the results so that it clears the existing results
create_schema_callback_from_function	Wraps a node's function to create a schema callback that extracts the schema.
delete_input_node	Removes a connection from a specific input node.
delete_lead_to_node	Removes a connection to a specific downstream node.
evaluate_nodes	Triggers a state reset for all directly connected downstream nodes.
execute_full_local	Backward-compatible alias for _do_execute_full_local.
execute_local	Backward-compatible alias for _do_execute_local_with_sampling.
execute_node	Execute the node based on its current state and settings.
execute_remote	Backward-compatible alias for _do_execute_remote.
get_all_dependent_node_ids	Yields the IDs of all downstream nodes recursively.
get_all_dependent_nodes	Yields all downstream nodes recursively.
get_column_stats	Computes on-demand stats for one column of this node's cached result.
get_edge_input	Generates NodeEdge objects for all input connections to this node.
get_flow_file_column_schema	Retrieves the schema for a specific column from the output schema.
get_input_type	Gets the type of connection ('main', 'left', 'right') for a given input node ID.
get_node_data	Gathers all necessary data for representing the node in the UI.
get_node_information	Updates and returns the node's information object.
get_node_input	Creates a NodeInput schema object for representing this node in the UI.
get_output	Get the result for a specific output handle.
get_output_data	Gets the full output data sample for this node.
get_predicted_resulting_data	Creates a FlowDataEngine instance based on the predicted schema.
get_predicted_schema	Predicts the output schema of the node without full execution.
get_repr	Gets a detailed dictionary representation of the node's state.
get_resulting_data	Executes the node's function to produce the actual output data.
get_table_example	Generates a TableExample model summarizing the node's output.
invalidate_cache	Force cache invalidation by incrementing the cache epoch.
needs_reset	Checks if the node's hash has changed, indicating an outdated state.
needs_run	Determines if the node needs to be executed.
peek_output_engine	Passively resolves the cached result engine for an output handle.
post_init	Reset every instance attribute to its default state.
prepare_before_run	Resets results and errors before a new execution.
print	Helper method to log messages with node context.
remap_dynamic_inputs	Re-key keyed connections after the node's input slots changed.
remove_cache	Removes cached results for this node.
reset	Resets the node's execution state and schema information.
schema_for_handle	Return the cached schema for a specific output handle.
set_node_information	Populates the node_information attribute with the current state.
store_example_data_generator	Stores a generator function for fetching a sample of the result data.
update_node	Updates the properties of the node.

Attributes:

Name	Type	Description
accepts_dynamic_inputs	bool	True when this node's connections are keyed by target handle (run_flow).
all_inputs	list[FlowNode]	Gets a list of all nodes connected to any input port.
executor	NodeExecutor	Lazy-initialized executor instance.
function	Callable	Gets the core processing function of the node.
has_input	bool	Checks if this node has any input connections.
has_next_step	bool	Checks if this node has any downstream connections.
hash	str	Gets the cached hash for the node, calculating it if it doesn't exist.
is_correct	bool	Checks if the node's input connections satisfy its template requirements.
is_setup	bool	Checks if the node has been properly configured and is ready for execution.
is_start	bool	Determines if the node is a starting node in the flow.
left_input	Optional[FlowNode]	Gets the node connected to the left input port.
main_input	list[FlowNode]	Gets the list of nodes connected to the main input port(s).
name	str	Gets the name of the node.
node_id	str | int	Gets the unique identifier of the node.
number_of_leads_to_nodes	int | None	Counts the number of downstream node connections.
right_input	Optional[FlowNode]	Gets the node connected to the right input port.
schema	list[FlowfileColumn]	Gets the definitive output schema of the node.
schema_callback	SingleExecutionFuture	Gets the schema callback function, creating one if it doesn't exist.
setting_input	Any	Gets the node's specific configuration settings.
singular_input	bool	Checks if the node template specifies exactly one input.
singular_main_input	FlowNode	Gets the input node, assuming it is a single-input type.
state_needs_reset	bool	Checks if the node's state needs to be reset.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

class FlowNode:
    """Represents a single node in a data flow graph.

    This class manages the node's state, its data processing function,
    and its connections to other nodes within the graph.
    """

    parent_uuid: str
    node_type: str
    active: bool
    node_template: node_store.NodeTemplate
    node_default: schemas.NodeDefault
    node_information: schemas.NodeInformation | None

    node_schema: NodeSchemaInformation
    node_inputs: NodeStepInputs
    node_stats: NodeStepStats
    node_settings: NodeStepSettings
    results: NodeResults
    leads_to_nodes: list["FlowNode"]

    _name: str | None
    _function: Callable
    _setting_input: Any
    _executor: NodeExecutor | None
    _execution_state: NodeExecutionState
    _execution_lock: threading.RLock  # guards concurrent get_resulting_data
    _state_needs_reset: bool
    # invoked by run_graph() once all downstream dependents finish; e.g. Kafka commits offsets on success
    _on_flow_complete: Callable[[bool], None] | None

    user_provided_schema_callback: Callable | None
    _schema_callback: SingleExecutionFuture | None
    _named_outputs: dict[str, FlowDataEngine]
    _named_schemas: dict[str, list[FlowfileColumn]]
    _input_output_handles: dict[int, str]

    _hash: str | None
    _cache_epoch: int  # bumped by invalidate_cache() to bust the hash
    _fetch_cached_df: ExternalTaskHandle | None
    _cache_progress: ExternalTaskHandle | None

    _kernel_cancel_context: Any  # (kernel_id, manager, exec_token) of this node's in-flight cell
    _kernel_cancel_event: threading.Event | None
    _subflow_cancel_context: Any  # a running child FlowGraph of a run_flow node

    # a live getter (not a snapshot) so ${...} refs resolve against current parameter edits
    _params_getter: Callable[[], dict[str, ParamValue]] | None
    # warning text when prediction would require executing an un-run kernel node
    _schema_prediction_blocked: str | None
    _prediction_requires_data: bool  # stamped at placement: prediction needs a collect
    _executes_on_kernel: bool  # stamped at placement: function runs on a kernel container

    def __init__(
        self,
        node_id: str | int,
        function: Callable,
        parent_uuid: str,
        setting_input: Any,
        name: str,
        node_type: str,
        input_columns: list[str] = None,
        output_schema: list[FlowfileColumn] = None,
        drop_columns: list[str] = None,
        renew_schema: bool = True,
        pos_x: float = 0,
        pos_y: float = 0,
        schema_callback: Callable = None,
    ):
        """Initializes a FlowNode instance.

        Args:
            node_id: Unique identifier for the node.
            function: The core data processing function for the node.
            parent_uuid: The UUID of the parent flow.
            setting_input: The configuration/settings object for the node.
            name: The name of the node.
            node_type: The type identifier of the node (e.g., 'join', 'filter').
            input_columns: List of column names expected as input.
            output_schema: The schema of the columns to be added.
            drop_columns: List of column names to be dropped.
            renew_schema: Flag to indicate if the schema should be renewed.
            pos_x: The x-coordinate on the canvas.
            pos_y: The y-coordinate on the canvas.
            schema_callback: A custom function to calculate the output schema.
        """
        self.parent_uuid = parent_uuid
        self.post_init()
        self.node_information.id = node_id
        self.node_type = node_type
        self.node_settings.renew_schema = renew_schema
        self.update_node(
            function=function,
            input_columns=input_columns,
            output_schema=output_schema,
            drop_columns=drop_columns,
            setting_input=setting_input,
            name=name,
            pos_x=pos_x,
            pos_y=pos_y,
            schema_callback=schema_callback,
        )

    def post_init(self):
        """Reset every instance attribute to its default state."""
        self.active = True
        self.node_information = schemas.NodeInformation()
        self.node_inputs = NodeStepInputs()
        self.node_stats = NodeStepStats()
        self.node_settings = NodeStepSettings()
        self.node_schema = NodeSchemaInformation()
        self.results = NodeResults()
        self.leads_to_nodes = []

        self._name = None
        self._function = None
        self._setting_input = None
        self._executor = None
        self._execution_state = NodeExecutionState()
        self._execution_lock = threading.RLock()
        self._state_needs_reset = False
        self._on_flow_complete = None

        self.user_provided_schema_callback = None
        self._schema_callback = None
        self._named_outputs = {}  # per-handle output engines, keyed by output handle
        self._named_schemas = {}  # per-handle schemas, populated alongside _named_outputs
        self._input_output_handles = {}  # source node id -> the output handle it connects through

        self._hash = None
        self._cache_epoch = 0
        self._cache_progress = None
        self._fetch_cached_df = None

        self._kernel_cancel_context = None
        self._kernel_cancel_event = None
        self._subflow_cancel_context = None

        self._params_getter = None
        self._schema_prediction_blocked = None
        self._prediction_requires_data = False
        self._executes_on_kernel = False

    @property
    def state_needs_reset(self) -> bool:
        """Checks if the node's state needs to be reset.

        Returns:
            True if a reset is required, False otherwise.
        """
        return self._state_needs_reset

    @state_needs_reset.setter
    def state_needs_reset(self, v: bool):
        """Sets the flag indicating that the node's state needs to be reset.

        Args:
            v: The boolean value to set.
        """
        self._state_needs_reset = v

    @staticmethod
    def _as_default_output(fl: "FlowDataEngine | NamedOutputs") -> "FlowDataEngine":
        """Return the default (first) output for a node-function result.

        Downstream consumers that don't request a specific handle see this one.
        """
        if isinstance(fl, NamedOutputs):
            return fl.default()
        return fl

    def schema_for_handle(self, handle: str) -> list[FlowfileColumn]:
        """Return the cached schema for a specific output handle.

        Falls back to the default ``schema`` property when the handle is unknown
        or the node is single-output, so callers can always rely on this.
        """
        if handle in self._named_schemas:
            return self._named_schemas[handle]
        if handle in self._named_outputs:
            return self._named_outputs[handle].schema
        return self.schema

    def create_schema_callback_from_function(self, f: Callable) -> Callable[[], list[FlowfileColumn]]:
        """Wraps a node's function to create a schema callback that extracts the schema.

        For multi-output functions, every handle's schema is captured in
        ``_named_schemas`` on the single call; the callback itself still
        returns the default handle's schema so the existing contract holds.

        Thread-safe: uses _execution_lock to prevent concurrent execution with get_resulting_data.

        Args:
            f: The node's core function that returns a FlowDataEngine or NamedOutputs.

        Returns:
            A callable that, when executed, returns the default output's schema.
        """

        def schema_callback() -> list[FlowfileColumn]:
            try:
                logger.info("Executing the schema callback function based on the node function")
                with self._execution_lock:
                    result = f()
                    if isinstance(result, NamedOutputs):
                        self._named_schemas = {
                            output_handle(i): engine.schema for i, engine in enumerate(result.engines)
                        }
                        return self._named_schemas.get(DEFAULT_OUTPUT_HANDLE, [])
                    return result.schema
            except Exception as e:
                logger.warning(f"Error with the schema callback: {e}")
                return []

        return schema_callback

    @property
    def schema_callback(self) -> SingleExecutionFuture:
        """Gets the schema callback function, creating one if it doesn't exist.

        The callback is used for predicting the output schema without full execution.

        Returns:
            A SingleExecutionFuture instance wrapping the schema function.
        """
        if self._schema_callback is None:
            if self.user_provided_schema_callback is not None:
                self.schema_callback = self.user_provided_schema_callback
            elif self.is_start:
                self.schema_callback = self.create_schema_callback_from_function(self._function)
        return self._schema_callback

    @schema_callback.setter
    def schema_callback(self, f: Callable):
        """Sets the schema callback function for the node.

        If the node has an enabled output_field_config, the callback is automatically
        wrapped to use the output_field_config schema for prediction.

        Args:
            f: The function to be used for schema calculation.
        """
        if f is None:
            return

        # Wrap callback with output_field_config support if present and enabled
        output_field_config = getattr(self._setting_input, "output_field_config", None)
        if output_field_config and output_field_config.enabled:
            f = create_schema_callback_with_output_config(f, output_field_config)

        def error_callback(e: Exception) -> list:
            logger.warning(e)

            self.node_settings.setup_errors = True
            return []

        self._schema_callback = SingleExecutionFuture(f, error_callback)

    @property
    def executor(self) -> NodeExecutor:
        """Lazy-initialized executor instance.

        Reusing the same executor avoids object creation overhead
        when execute_node is called multiple times.
        """
        if self._executor is None:
            self._executor = NodeExecutor(self)
        return self._executor

    @property
    def is_start(self) -> bool:
        """Determines if the node is a starting node in the flow.

        A starting node requires no inputs.

        Returns:
            True if the node is a start node, False otherwise.
        """
        return not self.has_input and self.node_template.input == 0

    def get_input_type(self, node_id: int) -> list:
        """Gets the type of connection ('main', 'left', 'right') for a given input node ID.

        Args:
            node_id: The ID of the input node.

        Returns:
            A list of connection types for that node ID.
        """
        relation_type = []
        if node_id in [n.node_id for n in self.node_inputs.main_inputs]:
            relation_type.append("main")
        if self.node_inputs.left_input is not None and node_id == self.node_inputs.left_input.node_id:
            relation_type.append("left")
        if self.node_inputs.right_input is not None and node_id == self.node_inputs.right_input.node_id:
            relation_type.append("right")
        return list(set(relation_type))

    def update_node(
        self,
        function: Callable,
        input_columns: list[str] = None,
        output_schema: list[FlowfileColumn] = None,
        drop_columns: list[str] = None,
        name: str = None,
        setting_input: Any = None,
        pos_x: float = 0,
        pos_y: float = 0,
        schema_callback: Callable = None,
    ):
        """Updates the properties of the node.

        This is called during initialization and when settings are changed.

        Args:
            function: The new core data processing function.
            input_columns: The new list of input columns.
            output_schema: The new schema of added columns.
            drop_columns: The new list of dropped columns.
            name: The new name for the node.
            setting_input: The new settings object.
            pos_x: The new x-coordinate.
            pos_y: The new y-coordinate.
            schema_callback: The new custom schema callback function.
        """
        self.user_provided_schema_callback = schema_callback
        self.node_information.y_position = int(pos_y)
        self.node_information.x_position = int(pos_x)
        self.node_information.setting_input = setting_input
        self.name = self.node_type if name is None else name
        self._function = function

        self.node_schema.input_columns = [] if input_columns is None else input_columns
        self.node_schema.output_columns = [] if output_schema is None else output_schema
        self.node_schema.drop_columns = [] if drop_columns is None else drop_columns
        self.node_settings.renew_schema = True
        if hasattr(setting_input, "cache_results"):
            self.node_settings.cache_results = setting_input.cache_results

        self.results.errors = None
        self.add_lead_to_in_depend_source()
        _ = self.hash
        self.node_template = node_store.node_dict.get(self.node_type)
        if self.node_template is None:
            raise Exception(f"Node template {self.node_type} not found")
        self.node_default = node_store.node_defaults.get(self.node_type)
        self.setting_input = setting_input  # wait until the end so that the hash is calculated correctly

    @property
    def name(self) -> str:
        """Gets the name of the node.

        Returns:
            The node's name.
        """
        return self._name

    @name.setter
    def name(self, name: str):
        """Sets the name of the node.

        Args:
            name: The new name.
        """
        self._name = name
        self.__name__ = name

    @property
    def setting_input(self) -> Any:
        """Gets the node's specific configuration settings.

        Returns:
            The settings object.
        """
        return self._setting_input

    @setting_input.setter
    def setting_input(self, setting_input: Any):
        """Sets the node's configuration and triggers a reset if necessary.

        Args:
            setting_input: The new settings object.
        """
        is_manual_input = (
            self.node_type == "manual_input"
            and isinstance(setting_input, input_schema.NodeManualInput)
            and isinstance(self._setting_input, input_schema.NodeManualInput)
        )
        if is_manual_input:
            _ = self.hash
        self._setting_input = setting_input
        if hasattr(setting_input, "cache_results"):
            self.node_settings.cache_results = setting_input.cache_results
        self.set_node_information()
        if is_manual_input:
            if self.hash != self.calculate_hash(setting_input) or not self.node_stats.has_run_with_current_setup:
                self.function = FlowDataEngine(setting_input.raw_data_format)
                self.reset()
                self.get_predicted_schema()
        elif self._setting_input is not None:
            self.reset()

    @property
    def node_id(self) -> str | int:
        """Gets the unique identifier of the node.

        Returns:
            The node's ID.
        """
        return self.node_information.id

    @property
    def left_input(self) -> Optional["FlowNode"]:
        """Gets the node connected to the left input port.

        Returns:
            The left input FlowNode, or None.
        """
        return self.node_inputs.left_input

    @property
    def right_input(self) -> Optional["FlowNode"]:
        """Gets the node connected to the right input port.

        Returns:
            The right input FlowNode, or None.
        """
        return self.node_inputs.right_input

    @property
    def main_input(self) -> list["FlowNode"]:
        """Gets the list of nodes connected to the main input port(s).

        Returns:
            A list of main input FlowNodes.
        """
        return self.node_inputs.main_inputs

    @property
    def accepts_dynamic_inputs(self) -> bool:
        """True when this node's connections are keyed by target handle (run_flow)."""
        # node_template is assigned late in update_node; hash calculation runs before it.
        template = getattr(self, "node_template", None)
        return template is not None and bool(getattr(template, "dynamic_inputs", False))

    @property
    def is_correct(self) -> bool:
        """Checks if the node's input connections satisfy its template requirements.

        Returns:
            True if connections are valid, False otherwise.
        """
        if isinstance(self.setting_input, input_schema.NodePromise):
            return False
        return (
            self.node_template.input == len(self.node_inputs.get_all_inputs())
            or (self.node_template.multi and len(self.node_inputs.get_all_inputs()) > 0)
            or (self.node_template.multi and self.node_template.can_be_start)
            or self.accepts_dynamic_inputs
        )

    def set_node_information(self):
        """Populates the `node_information` attribute with the current state.

        This includes the node's connections, settings, and position.
        """
        node_information = self.node_information
        node_information.left_input_id = self.node_inputs.left_input.node_id if self.left_input else None
        node_information.right_input_id = self.node_inputs.right_input.node_id if self.right_input else None
        node_information.input_ids = (
            [mi.node_id for mi in self.node_inputs.main_inputs] if self.node_inputs.main_inputs is not None else None
        )
        node_information.setting_input = self.setting_input
        node_information.outputs = [n.node_id for n in self.leads_to_nodes]
        # Source-side handle for each downstream connection — the downstream
        # node tracks this in ``_input_output_handles[from_node_id]``.
        node_information.output_handles = [
            n._input_output_handles.get(self.node_id, DEFAULT_OUTPUT_HANDLE) for n in self.leads_to_nodes
        ]
        if self.accepts_dynamic_inputs and self.node_inputs.keyed_inputs:
            source_handles = self.node_inputs.keyed_source_handles or {}
            node_information.input_connections = [
                schemas.FlowfileInputConnection(
                    from_id=source.node_id,
                    input_handle=handle,
                    source_handle=source_handles.get(handle, DEFAULT_OUTPUT_HANDLE),
                )
                for handle, source in self.node_inputs.slot_items()
            ]
        else:
            node_information.input_connections = None
        user_description = self.setting_input.description if hasattr(self.setting_input, "description") else ""
        if user_description:
            node_information.description = user_description
        elif hasattr(self.setting_input, "get_default_description"):
            node_information.description = self.setting_input.get_default_description()
        else:
            node_information.description = ""
        node_information.node_reference = (
            self.setting_input.node_reference if hasattr(self.setting_input, "node_reference") else None
        )
        node_information.is_setup = self.is_setup
        node_information.x_position = self.setting_input.pos_x
        node_information.y_position = self.setting_input.pos_y
        node_information.group_id = getattr(self.setting_input, "group_id", None)
        node_information.type = self.node_type

    def get_node_information(self) -> schemas.NodeInformation:
        """Updates and returns the node's information object.

        Returns:
            The `NodeInformation` object for this node.
        """
        self.set_node_information()
        return self.node_information

    @property
    def function(self) -> Callable:
        """Gets the core processing function of the node.

        Returns:
            The callable function.
        """
        return self._function

    @function.setter
    def function(self, function: Callable):
        """Sets the core processing function of the node.

        Args:
            function: The new callable function.
        """
        self._function = function

    @property
    def all_inputs(self) -> list["FlowNode"]:
        """Gets a list of all nodes connected to any input port.

        Returns:
            A list of all input FlowNodes.
        """
        return self.node_inputs.get_all_inputs()

    def check_upstream_laziness(self) -> tuple[bool, list[str]]:
        """Check whether all upstream dependencies of this node support lazy execution.

        Walks the DAG backwards from this node (excluding itself) and reports
        any eager or conditional nodes that would prevent a lazy/optimized
        execution path.

        Returns:
            A tuple of (is_lazy, reasons).  ``is_lazy`` is True when every
            upstream node has ``laziness == "lazy"``.
        """
        visited: set[FlowNode] = set()
        stack = list(self.all_inputs)
        reasons: list[str] = []

        while stack:
            current = stack.pop()
            if current in visited:
                continue
            visited.add(current)

            if isinstance(current.setting_input, input_schema.NodeCatalogReader):
                if current.setting_input.is_virtual_optimized is False:
                    reasons.append(
                        f"Node '{current.node_template.name}' (id={current.node_id}) reads a "
                        "non-optimized virtual table"
                    )
            else:
                laziness: schemas.LazinessLiteral = current.node_template.laziness
                if laziness == "eager":
                    reasons.append(f"Node '{current.node_template.name}' (id={current.node_id}) is eager")
                elif laziness == "conditional":
                    # TODO: resolve conditional nodes (read_data, polars_code, cloud_storage_reader)
                    # via isinstance checks like catalog_reader, then raise ValueError here instead
                    reasons.append(
                        f"Node '{current.node_template.name}' (id={current.node_id}) is conditional"
                        " — defaulting to non-optimized"
                    )

            stack.extend(current.all_inputs)
        return len(reasons) == 0, reasons

    def calculate_hash(self, setting_input: Any) -> str:
        """Calculates a hash based on settings and input node hashes.

        Args:
            setting_input: The node's settings object to be included in the hash.

        Returns:
            A string hash value.
        """
        if self.accepts_dynamic_inputs:
            # Fold in the target handle and source handle so re-wiring the same
            # upstream node to a different named input invalidates caches.
            source_handles = self.node_inputs.keyed_source_handles or {}
            depends_on_hashes = [
                f"{handle}:{source_handles.get(handle, DEFAULT_OUTPUT_HANDLE)}:{_node.hash}"
                for handle, _node in self.node_inputs.slot_items()
            ]
        else:
            depends_on_hashes = [_node.hash for _node in self.all_inputs]
        node_data_hash = get_hash(setting_input)
        return get_hash(depends_on_hashes + [node_data_hash, self.parent_uuid, self._cache_epoch])

    @property
    def hash(self) -> str:
        """Gets the cached hash for the node, calculating it if it doesn't exist.

        Returns:
            The string hash value.
        """
        if not self._hash:
            self._hash = self.calculate_hash(self.setting_input)
        return self._hash

    def add_node_connection(
        self,
        from_node: "FlowNode",
        insert_type: Literal["main", "left", "right"] = "main",
        output_handle: str = DEFAULT_OUTPUT_HANDLE,
        target_handle: str | None = None,
    ) -> None:
        """Adds a connection from a source node to this node.

        Args:
            from_node: The node to connect from.
            insert_type: The type of input to connect to ('main', 'left', 'right').
            output_handle: The output handle on the source node (e.g. 'output-0', 'output-1').
            target_handle: For dynamic-input nodes only: the target handle the edge
                lands on ('input-0'..'input-N'). Ignored for static nodes.

        Raises:
            Exception: If the insert_type is invalid.
        """
        if self.accepts_dynamic_inputs:
            self._add_keyed_connection(from_node, target_handle or PARAM_INPUT_HANDLE, output_handle)
            return
        from_node.leads_to_nodes.append(self)
        if insert_type == "main":
            if self.node_template.input <= 2 or self.node_inputs.main_inputs is None:
                self.node_inputs.main_inputs = [from_node]
            else:
                self.node_inputs.main_inputs.append(from_node)
        elif insert_type == "right":
            self.node_inputs.right_input = from_node
        elif insert_type == "left":
            self.node_inputs.left_input = from_node
        else:
            raise Exception("Cannot find the connection")
        # Track which output handle of the source node this connection uses
        self._input_output_handles[from_node.node_id] = output_handle
        if self.setting_input.is_setup:
            if hasattr(self.setting_input, "depending_on_id") and insert_type == "main":
                self.setting_input.depending_on_id = from_node.node_id
        self.reset()
        from_node.reset()

    def _add_keyed_connection(self, from_node: "FlowNode", target_handle: str, output_handle: str) -> None:
        """Attach *from_node* to a specific handle of a dynamic-input node.

        Replace-if-occupied (permissive) so load/restore/remap paths never fail;
        the connect API rejects occupied handles before calling this.
        """
        inputs = self.node_inputs
        if inputs.keyed_inputs is None:
            inputs.keyed_inputs = {}
            inputs.keyed_source_handles = {}
        existing = inputs.keyed_inputs.get(target_handle)
        if existing is not None:
            existing.delete_lead_to_node(self.node_id)
            if not any(n is existing for h, n in inputs.keyed_inputs.items() if h != target_handle):
                self._input_output_handles.pop(existing.node_id, None)
        from_node.leads_to_nodes.append(self)
        inputs.keyed_inputs[target_handle] = from_node
        inputs.keyed_source_handles[target_handle] = output_handle
        # Legacy readers key by source node id; ambiguous when one source feeds
        # two handles — keyed_source_handles is authoritative for this node.
        self._input_output_handles[from_node.node_id] = output_handle
        inputs.rebuild_keyed_projection()
        self.reset()
        from_node.reset()

    def _delete_keyed_connection(self, node_id: int, handle: str | None, complete: bool) -> bool:
        """Remove keyed connection(s) fed by *node_id*; all of them when *complete*."""
        inputs = self.node_inputs
        if not inputs.keyed_inputs:
            return False
        if complete:
            handles = [h for h, n in inputs.keyed_inputs.items() if n.node_id == node_id]
        else:
            handles = [handle] if inputs.keyed_connection_exists(handle, node_id) else []
        if not handles:
            return False
        for h in handles:
            inputs.keyed_inputs.pop(h, None)
            if inputs.keyed_source_handles:
                inputs.keyed_source_handles.pop(h, None)
        if not any(n.node_id == node_id for n in inputs.keyed_inputs.values()):
            self._input_output_handles.pop(node_id, None)
        inputs.rebuild_keyed_projection()
        self.reset()
        return True

    def remap_dynamic_inputs(self, mapping: dict[str, str | None]) -> dict[str, list[str]]:
        """Re-key keyed connections after the node's input slots changed.

        Args:
            mapping: old handle -> new handle, or None to drop that connection.
                Handles absent from the mapping keep their key.

        Returns:
            {"moved": [...], "dropped": [...]} describing what happened, so the
            API layer can surface removed connections to the UI.
        """
        inputs = self.node_inputs
        if not inputs.keyed_inputs:
            return {"moved": [], "dropped": []}
        moved: list[str] = []
        dropped: list[str] = []
        new_inputs: dict[str, FlowNode] = {}
        new_source_handles: dict[str, str] = {}
        for handle, node in inputs.keyed_inputs.items():
            new_handle = mapping.get(handle, handle)
            if new_handle is None:
                dropped.append(handle)
                node.delete_lead_to_node(self.node_id)
                continue
            if new_handle != handle:
                moved.append(f"{handle}->{new_handle}")
            new_inputs[new_handle] = node
            new_source_handles[new_handle] = (inputs.keyed_source_handles or {}).get(handle, DEFAULT_OUTPUT_HANDLE)
        inputs.keyed_inputs = new_inputs
        inputs.keyed_source_handles = new_source_handles
        remaining_ids = {n.node_id for n in new_inputs.values()}
        for node_id in list(self._input_output_handles):
            if node_id not in remaining_ids:
                self._input_output_handles.pop(node_id, None)
        inputs.rebuild_keyed_projection()
        self.reset()
        return {"moved": moved, "dropped": dropped}

    def evaluate_nodes(self, deep: bool = False) -> None:
        """Triggers a state reset for all directly connected downstream nodes.

        Args:
            deep: If True, the reset propagates recursively through the entire downstream graph.
        """
        for node in self.leads_to_nodes:
            self.print(f"resetting node: {node.node_id}")
            node.reset(deep)

    def get_flow_file_column_schema(self, col_name: str) -> FlowfileColumn | None:
        """Retrieves the schema for a specific column from the output schema.

        Args:
            col_name: The name of the column.

        Returns:
            The FlowfileColumn object for that column, or None if not found.
        """
        for s in self.schema:
            if s.column_name == col_name:
                return s

    def get_predicted_schema(self, force: bool = False) -> list[FlowfileColumn] | None:
        """Predicts the output schema of the node without full execution.

        It uses the schema_callback or infers from predicted data.

        Args:
            force: If True, forces recalculation even if a predicted schema exists.

        Returns:
            A list of FlowfileColumn objects representing the predicted schema.
        """
        _has_output_field_config = (
            hasattr(self._setting_input, "output_field_config") and self._setting_input.output_field_config is not None
            if self._setting_input
            else False
        )
        logger.info(
            f"get_predicted_schema: node_id={self.node_id}, node_type={self.node_type}, force={force}, "
            f"has_predicted_schema={self.node_schema.predicted_schema is not None}, "
            f"has_schema_callback={self.schema_callback is not None}, "
            f"has_output_field_config={_has_output_field_config}"
        )

        if self.node_schema.predicted_schema and not force:
            logger.debug(f"get_predicted_schema: node_id={self.node_id} - returning cached predicted_schema")
            return self.node_schema.predicted_schema

        if self.schema_callback is not None and (self.node_schema.predicted_schema is None or force):
            self.print("Getting the data from a schema callback")
            logger.info(f"get_predicted_schema: node_id={self.node_id} - invoking schema_callback")
            if force:
                # Force the schema callback to reset, so that it will be executed again
                logger.debug(f"get_predicted_schema: node_id={self.node_id} - forcing schema_callback reset")
                self.schema_callback.reset()

            try:
                schema = self.schema_callback()
                logger.info(
                    f"get_predicted_schema: node_id={self.node_id} - schema_callback returned "
                    f"{len(schema) if schema else 0} columns: {[c.name for c in schema] if schema else []}"
                )
            except Exception as e:
                logger.error(f"get_predicted_schema: node_id={self.node_id} - schema_callback raised exception: {e}")
                schema = None

            if schema is not None and len(schema) > 0:
                self.print("Calculating the schema based on the schema callback")
                self.node_schema.predicted_schema = schema
                logger.info(f"get_predicted_schema: node_id={self.node_id} - set predicted_schema from schema_callback")
                return self.node_schema.predicted_schema
            else:
                logger.warning(
                    f"get_predicted_schema: node_id={self.node_id} - schema_callback returned empty/None schema"
                )
        else:
            logger.debug(f"get_predicted_schema: node_id={self.node_id} - no schema_callback available")

        if self._schema_prediction_blocked is None and (self._prediction_requires_data or self._executes_on_kernel):
            self._schema_prediction_blocked = kernel_block_reason(self, include_self=True)
        if self._schema_prediction_blocked:
            # Prediction would require executing an un-run kernel node: never do
            # that implicitly — surface the warning and skip the exec tier.
            self.results.warnings = self._schema_prediction_blocked
            return self.node_schema.predicted_schema

        logger.debug(f"get_predicted_schema: node_id={self.node_id} - falling back to _predicted_data_getter")
        # Serialize the fallback: without a callback, prediction executes the node's
        # real function (kernel/worker for custom nodes) — concurrent callers must
        # not run it twice into the same working dirs.
        with self._execution_lock:
            if self.node_schema.predicted_schema and not force:
                return self.node_schema.predicted_schema
            predicted_data = self._predicted_data_getter()
            if predicted_data is not None and predicted_data.schema is not None:
                self.print("Calculating the schema based on the predicted resulting data")
                logger.info(
                    f"get_predicted_schema: node_id={self.node_id} - using schema from predicted_data "
                    f"({len(predicted_data.schema)} columns)"
                )
                self.node_schema.predicted_schema = predicted_data.schema
            else:
                logger.warning(
                    f"get_predicted_schema: node_id={self.node_id} - no schema available from any source "
                    f"(predicted_data={'None' if predicted_data is None else 'has_data'}, "
                    f"schema={'None' if predicted_data is None or predicted_data.schema is None else 'has_schema'})"
                )

        return self.node_schema.predicted_schema

    @property
    def is_setup(self) -> bool:
        """Checks if the node has been properly configured and is ready for execution.

        Returns:
            True if the node is set up, False otherwise.
        """
        if not self.node_information.is_setup:
            if self.function.__name__ != "placeholder":
                self.node_information.is_setup = True
                self.setting_input.is_setup = True
        return self.node_information.is_setup

    def print(self, v: Any):
        """Helper method to log messages with node context.

        Args:
            v: The message or value to log.
        """
        logger.info(f"{self.node_type}, node_id: {self.node_id}: {v}")

    def get_output(self, handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine | None:
        """Get the result for a specific output handle.

        For nodes with multiple outputs (e.g. kernel-based custom nodes),
        returns the FlowDataEngine associated with the given handle.
        Falls back to the default ``results.resulting_data`` for single-output nodes.

        Args:
            handle: The output handle identifier (e.g. ``"output-0"``, ``"output-1"``).

        Returns:
            The FlowDataEngine for the requested output, or None.
        """
        self.get_resulting_data()
        if handle in self._named_outputs:
            return self._named_outputs[handle]
        return self.results.resulting_data

    def _resolve_input_result(self, input_node: "FlowNode") -> FlowDataEngine | None:
        """Resolve the correct output from an input node based on connection handle.

        Args:
            input_node: The upstream node to get data from.

        Returns:
            The FlowDataEngine from the appropriate output handle.
        """
        handle = self._input_output_handles.get(input_node.node_id, DEFAULT_OUTPUT_HANDLE)
        return self._resolve_input_result_for_handle(input_node, handle)

    @staticmethod
    def _resolve_input_result_for_handle(input_node: "FlowNode", handle: str) -> FlowDataEngine | None:
        if handle != DEFAULT_OUTPUT_HANDLE:
            # get_output triggers execution first, then routes by handle. Required
            # for the first-call case where _named_outputs is still empty.
            return input_node.get_output(handle)
        return input_node.get_resulting_data()

    def _slot_input_pairs(self) -> list[tuple[Optional["FlowNode"], str]]:
        """Positional inputs for the node function, as (source_node, source_handle).

        Static nodes: one pair per connected input, in canvas handle order
        (input-0, input-1, input-2 -> main, right, left). ``all_inputs`` yields
        main + [left, right], which swaps the last two for a 3-input node — the
        canvas labels the handles top-to-bottom, so the arguments must follow.
        Dynamic-input nodes: index i corresponds to handle ``input-i`` (0 = the
        parameter handle); unconnected slots yield ``(None, DEFAULT_OUTPUT_HANDLE)``
        so the function signature stays positional with gaps preserved.
        """
        if not self.accepts_dynamic_inputs:
            ordered = [
                *(self.node_inputs.main_inputs or []),
                self.node_inputs.right_input,
                self.node_inputs.left_input,
            ]
            return [
                (node, self._input_output_handles.get(node.node_id, DEFAULT_OUTPUT_HANDLE))
                for node in ordered
                if node is not None
            ]
        keyed = self.node_inputs.keyed_inputs or {}
        source_handles = self.node_inputs.keyed_source_handles or {}
        max_index = len(getattr(self.setting_input, "input_slots", None) or [])
        for handle in keyed:
            max_index = max(max_index, input_handle_index(handle))
        pairs: list[tuple[FlowNode | None, str]] = []
        for i in range(max_index + 1):
            handle = input_handle(i)
            pairs.append((keyed.get(handle), source_handles.get(handle, DEFAULT_OUTPUT_HANDLE)))
        return pairs

    @contextmanager
    def _execution_lock_held(self, poll: float = 0.25) -> Generator[None, None, None]:
        """Hold this node's own ``_execution_lock`` while staying cancellable.

        A plain ``with self._execution_lock`` blocks uninterruptibly, so a cancel
        request issued while another thread is slowly materializing this node
        (single-flight contention) would never be observed. Poll the acquire and
        re-check the cancel flag so ``flow.cancel()`` can break a hung/slow collect.
        ``RLock.acquire(timeout=...)`` still returns immediately for the owning
        thread, so same-thread reentrancy (recursion through
        ``_resolve_input_result_for_handle``) is preserved.
        """
        while not self._execution_lock.acquire(timeout=poll):
            if self._execution_state.is_canceled:
                # Deliberately record nothing on results here: we do not hold
                # the lock, and writing results.errors would race the current
                # holder (the success path never clears errors). The executor's
                # state.is_canceled reclassification records the clean cancel.
                raise Exception("Node execution canceled")
        try:
            yield
        finally:
            self._execution_lock.release()

    def get_resulting_data(self) -> FlowDataEngine | None:
        """Executes the node's function to produce the actual output data.

        Handles both regular functions and external data sources.
        Thread-safe and single-flight: the node's own ``_execution_lock`` ensures
        the function runs at most once and the result is memoized, so N downstream
        consumers materialize it once. A node acquires only its OWN lock; upstream
        inputs are read through each upstream's own ``get_resulting_data()``, so
        lock acquisition always follows the DAG (a node -> its parents) and cannot
        form a cross-node cycle.

        Returns:
            A FlowDataEngine instance containing the result, or None on error.

        Raises:
            Exception: Propagates exceptions from the node's function execution.
        """
        if self.is_setup:
            with self._execution_lock_held():
                if self.results.resulting_data is None and self.results.errors is None:
                    self.print("getting resulting data")
                    try:
                        if self._execution_state.is_canceled:
                            raise Exception("Node execution canceled")
                        if isinstance(self.function, FlowDataEngine):
                            fl: FlowDataEngine = self.function
                        elif self.node_type == "external_source":
                            fl: FlowDataEngine = self.function()
                            fl.collect_external()
                            self.node_settings.streamable = False
                        else:
                            self.print("Collecting input data from all inputs")
                            input_data = []
                            for i, (v, src_handle) in enumerate(self._slot_input_pairs()):
                                if v is None:
                                    input_data.append(None)
                                    continue
                                if self._execution_state.is_canceled:
                                    raise Exception("Node execution canceled")
                                self.print(f"Getting resulting data from input {i} (node {v.node_id})")
                                # Read the upstream via its own get_resulting_data(), which
                                # single-flights materialization under the upstream's own
                                # _execution_lock (memoized into results.resulting_data). We do
                                # NOT acquire the upstream's lock here: a node holds only its own
                                # lock, so lock acquisition always follows the DAG (a node -> its
                                # parents) and can never form a cross-node cycle. Taking upstream
                                # locks in per-input slot order used to deadlock a parallel stage
                                # when two sibling nodes consumed the same two upstreams in
                                # opposite left/right order (AB-BA).
                                input_result = self._resolve_input_result_for_handle(v, src_handle)
                                if input_result is not None:
                                    # De-alias: hand the node function a private view. Some node
                                    # functions mutate their input (df.lazy = True in the writers,
                                    # cross_join/fuzzy prep) or return it unchanged (output,
                                    # filter passthrough, ...), and this node's own post-processing
                                    # (set_streamable, output_field_config) mutates whatever the
                                    # function returned — the copy keeps all of that off the
                                    # engine shared with sibling consumers in the same stage.
                                    input_result = input_result.shallow_copy()
                                _df_type = type(input_result.data_frame) if input_result else "None"
                                self.print(f"Input {i} data type: {type(input_result)}, " f"dataframe type: {_df_type}")
                                input_data.append(input_result)
                            self.print(f"All {len(input_data)} inputs collected, calling node function")
                            fl = self._function(*input_data)
                        if isinstance(fl, NamedOutputs):
                            self._named_outputs = fl.by_handle()
                            self._named_schemas = {h: e.schema for h, e in self._named_outputs.items()}
                            for v in self._named_outputs.values():
                                v.set_streamable(self.node_settings.streamable)
                            # Default downstream-without-handle consumers to the first output.
                            # output_field_config (below) only applies to this default; future
                            # multi-output nodes that need per-output config must extend the loop.
                            fl = self._named_outputs[DEFAULT_OUTPUT_HANDLE]
                        else:
                            fl.set_streamable(self.node_settings.streamable)

                        if (
                            hasattr(self._setting_input, "output_field_config")
                            and self._setting_input.output_field_config
                        ):
                            try:
                                fl = apply_output_field_config(fl, self._setting_input.output_field_config)
                            except Exception as e:
                                logger.error(f"Error applying output field config for node {self.node_id}: {e}")
                                raise

                        self.results.resulting_data = fl
                        self.node_schema.result_schema = fl.schema
                    except Exception as e:
                        self.results.resulting_data = FlowDataEngine()
                        self.results.errors = str(e)
                        self.node_stats.has_run_with_current_setup = False
                        self.node_stats.has_completed_last_run = False
                        raise e
                return self.results.resulting_data

    def _predicted_data_getter(self) -> FlowDataEngine | None:
        """Internal helper to get a predicted data result.

        This calls the function with predicted data from input nodes.
        If flow_parameters is set, ${...} references in setting_input are resolved
        temporarily so that nodes like polars_code produce a correct predicted schema.

        Returns:
            A FlowDataEngine instance with predicted data, or an empty one on error.
        """
        restorations = []
        flow_params = self._params_getter() if self._params_getter else {}
        if flow_params:
            try:
                restorations = apply_parameters_in_place(self.setting_input, flow_params)
            except ValueError:
                # Unresolved parameters during lazy eval are non-fatal; just run without substitution
                restorations = []
        try:
            fl = self._function(
                *[
                    (v.get_predicted_resulting_data(src_handle) if v is not None else None)
                    for v, src_handle in self._slot_input_pairs()
                ]
            )
            fl = self._as_default_output(fl)

            # Apply output field configuration if enabled (mirrors get_resulting_data behavior)
            # This ensures schema prediction accounts for output_field_config validation
            if hasattr(self._setting_input, "output_field_config") and self._setting_input.output_field_config:
                if self._setting_input.output_field_config.enabled:
                    fl = apply_output_field_config(fl, self._setting_input.output_field_config)

            return fl
        except ValueError as e:
            if str(e) == "generator already executing":
                logger.info("Generator already executing, waiting for the result")
                sleep(1)
                return self._predicted_data_getter()
            fl = FlowDataEngine()
            return fl

        except Exception as e:
            logger.warning("there was an issue with the function, returning an empty Flowfile")
            logger.warning(e)
        finally:
            if restorations:
                restore_parameters(restorations)

    def get_predicted_resulting_data(self, handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine:
        """Creates a `FlowDataEngine` instance based on the predicted schema.

        This avoids executing the node's full logic. For multi-output nodes the
        ``handle`` argument selects which output's schema to reflect so that a
        downstream node wired to e.g. ``output-1`` sees that partition's schema.

        Args:
            handle: The output handle to reflect. Ignored for single-output nodes.

        Returns:
            A FlowDataEngine instance with a schema but no data.
        """
        # Multi-output: prefer the handle-specific cached schema if we have it.
        if handle != DEFAULT_OUTPUT_HANDLE and (self._named_schemas or self._named_outputs):
            schema = self.schema_for_handle(handle)
            if schema:
                return FlowDataEngine.create_from_schema(schema)

        if self.needs_run(False) and self.schema_callback is not None or self.node_schema.result_schema is not None:
            self.print("Getting data based on the schema")
            # Running the schema callback populates _named_schemas for multi-output
            # nodes; re-check the handle cache afterward before falling back.
            if self.node_schema.result_schema is None:
                _s = self.schema_callback()
                if handle != DEFAULT_OUTPUT_HANDLE and handle in self._named_schemas:
                    _s = self._named_schemas[handle]
                if not _s:
                    # Empty is the callback's "no declared schema" sentinel (e.g. a
                    # predict_output_schema hook opting out) — use the full prediction
                    # ladder, which falls back to execution-based prediction.
                    _s = self.get_predicted_schema()
                    if handle != DEFAULT_OUTPUT_HANDLE:
                        # The exec fallback fills the per-handle caches; prefer them.
                        _s = self.schema_for_handle(handle)
            else:
                _s = self.node_schema.result_schema
            return FlowDataEngine.create_from_schema(_s or [])
        else:
            if isinstance(self.function, FlowDataEngine):
                fl = self.function
            else:
                fl = FlowDataEngine.create_from_schema(self.get_predicted_schema())
            return fl

    def add_lead_to_in_depend_source(self):
        """Ensures this node is registered in the `leads_to_nodes` list of its inputs."""
        for input_node in self.all_inputs:
            if self.node_id not in [n.node_id for n in input_node.leads_to_nodes]:
                input_node.leads_to_nodes.append(self)

    def get_all_dependent_nodes(self) -> Generator["FlowNode", None, None]:
        """Yields all downstream nodes recursively.

        Returns:
            A generator of all dependent FlowNode objects.
        """
        for node in self.leads_to_nodes:
            yield node
            yield from node.get_all_dependent_nodes()

    def get_all_dependent_node_ids(self) -> Generator[int, None, None]:
        """Yields the IDs of all downstream nodes recursively.

        Returns:
            A generator of all dependent node IDs.
        """
        for node in self.leads_to_nodes:
            yield node.node_id
            yield from node.get_all_dependent_node_ids()

    @property
    def schema(self) -> list[FlowfileColumn]:
        """Gets the definitive output schema of the node.

        If not already run, it falls back to the predicted schema.

        Returns:
            A list of FlowfileColumn objects.
        """
        try:
            if self.is_setup and self.results.errors is None:
                if self.node_schema.result_schema is not None and len(self.node_schema.result_schema) > 0:
                    return self.node_schema.result_schema
                elif self.node_type in ("output", "api_response", "flow_output"):
                    if len(self.node_inputs.main_inputs) > 0:
                        self.node_schema.result_schema = self.node_inputs.main_inputs[0].schema
                else:
                    self.node_schema.result_schema = self.get_predicted_schema()
                return self.node_schema.result_schema
            else:
                return []
        except Exception as e:
            logger.error(e)
            return []

    def remove_cache(self):
        """Removes cached results for this node.

        Note: Currently not fully implemented.
        """

        if results_exists(self.hash):
            logger.warning("Not implemented")
            clear_task_from_worker(self.hash)

    def needs_run(
        self,
        performance_mode: bool,
        node_logger: NodeLogger = None,
        execution_location: schemas.ExecutionLocationsLiteral = "remote",
    ) -> bool:
        """Determines if the node needs to be executed.

        The decision is based on its run state, caching settings, and execution mode.

        Args:
            performance_mode: True if the flow is in performance mode.
            node_logger: The logger instance for this node.
            execution_location: The target execution location.

        Returns:
            True if the node should be run, False otherwise.
        """
        if execution_location == "local":
            return False

        flow_logger = logger if node_logger is None else node_logger
        cache_result_exists = results_exists(self.hash)
        if not self.node_stats.has_run_with_current_setup:
            flow_logger.info("Node has not run, needs to run")
            return True
        if self.node_settings.cache_results and cache_result_exists:
            return False
        elif self.node_settings.cache_results and not cache_result_exists:
            return True
        elif not performance_mode and cache_result_exists:
            return False
        else:
            return True

    def __call__(self, *args, **kwargs):
        """Makes the node instance callable, acting as an alias for execute_node."""
        self.execute_node(*args, **kwargs)

    def _do_execute_full_local(self, performance_mode: bool = False) -> None:
        """Executes the node's logic locally, including example data generation.

        Internal method called by NodeExecutor.

        Args:
            performance_mode: If True, skips generating example data.

        Raises:
            Exception: Propagates exceptions from the execution.
        """
        self.clear_table_example()

        def example_data_generator():
            example_data = None

            def get_example_data():
                nonlocal example_data
                if example_data is None:
                    example_data = resulting_data.get_sample(100).to_arrow()
                return example_data

            return get_example_data

        resulting_data = self.get_resulting_data()

        if not performance_mode:
            self.node_stats.has_run_with_current_setup = True
            self.results.example_data_generator = example_data_generator()
            self.node_schema.result_schema = self.results.resulting_data.schema
            self.node_stats.has_completed_last_run = True

    def _do_execute_local_with_sampling(self, performance_mode: bool = False, flow_id: int = None):
        """Executes the node's logic locally with external sampling.

        Internal method called by NodeExecutor.

        Args:
            performance_mode: If True, skips generating example data.
            flow_id: The ID of the parent flow.

        Raises:
            Exception: Propagates exceptions from the execution.
        """
        try:
            resulting_data = self.get_resulting_data()
            if not performance_mode:
                external_sampler = ExternalSampler(
                    lf=resulting_data.data_frame,
                    file_ref=self.hash,
                    wait_on_completion=True,
                    node_id=self.node_id,
                    flow_id=flow_id,
                )
                self.store_example_data_generator(external_sampler)
                if self.results.errors is None and not self.node_stats.is_canceled:
                    self.node_stats.has_run_with_current_setup = True
            self.node_schema.result_schema = resulting_data.schema

        except Exception as e:
            logger.warning(f"Error with step {self.__name__}")
            logger.error(str(e))
            self.results.errors = str(e)
            self.node_stats.has_run_with_current_setup = False
            self.node_stats.has_completed_last_run = False
            raise e

        if self.node_stats.has_run_with_current_setup:
            for step in self.leads_to_nodes:
                if not self.node_settings.streamable:
                    step.node_settings.streamable = self.node_settings.streamable

    _INFER_SCHEMA_RUNGS = (10_000, 100_000)

    @staticmethod
    def _is_type_inference_error(error_description: str | None) -> bool:
        """True when a worker error looks like a CSV schema/type-inference failure worth widening for."""
        if not error_description:
            return False
        lowered = error_description.lower()
        signatures = ("could not parse", "as dtype", "conversion from", "schemaerror", "computeerror", "infer_schema")
        return any(sig in lowered for sig in signatures)

    @classmethod
    def _next_infer_rung(cls, current: int) -> int | None:
        """Next inference length above ``current`` on the escalation ladder, or None if exhausted."""
        return min((rung for rung in cls._INFER_SCHEMA_RUNGS if rung > current), default=None)

    def _eligible_infer_length(self) -> int | None:
        """Configured infer length if this is a CSV read with inference on, else None (ineligible).

        Scoped to file_type == "csv": _escalated_read_frame rebuilds via
        FlowDataEngine.create_from_path, which has no "json" handler (json reads route
        through the worker), so an InputJsonTable (a subclass of InputCsvTable) must not
        be treated as eligible.
        """
        if self.node_type != "read":
            return None
        received_file = getattr(self.setting_input, "received_file", None)
        if getattr(received_file, "file_type", None) != "csv":
            return None
        table_settings = getattr(received_file, "table_settings", None)
        if not isinstance(table_settings, input_schema.InputCsvTable):
            return None
        if not getattr(table_settings, "infer_schema", True):
            return None
        return table_settings.infer_schema_length

    def _escalated_read_frame(self, rung: int) -> FlowDataEngine | None:
        """Rebuild the read frame at a higher infer_schema_length on a copy (saved setting untouched)."""
        if self._eligible_infer_length() is None:
            return None
        escalated_file = self.setting_input.received_file.model_copy(deep=True)
        escalated_file.table_settings.infer_schema_length = rung
        escalated_file.set_absolute_filepath()
        return FlowDataEngine.create_from_path(escalated_file)

    def _do_execute_remote(self, performance_mode: bool = False, node_logger: NodeLogger = None):
        """Executes the node's logic remotely or handles cached results.

        Internal method called by NodeExecutor.

        Args:
            performance_mode: If True, skips generating example data.
            node_logger: The logger for this node execution.

        Raises:
            Exception: If the node_logger is not provided or if execution fails.
        """
        if node_logger is None:
            raise Exception("Node logger is not defined")
        if self.node_settings.cache_results and results_exists(self.hash):
            try:
                self.results.resulting_data = FlowDataEngine(get_external_df_result(self.hash))
                self._cache_progress = None
                return
            except Exception:
                node_logger.warning("Failed to read the cache, rerunning the code")
        if self.node_type in ("output", "api_response", "flow_output", "run_flow"):
            # Stay in-core: sinks have nothing to offload, and run_flow's child
            # graph already executed in-process while building its result.
            self.results.resulting_data = self.get_resulting_data()
            self.node_stats.has_run_with_current_setup = True
            return

        try:
            result_data = self.get_resulting_data()
            # Use 'is not None' instead of truthiness check to avoid triggering __len__()
            # which calls .collect() on the LazyFrame and can cause issues
            if result_data is None:
                self.results.errors = "Error with creating the lazy frame, most likely due to invalid graph"
                raise Exception("get_resulting_data returned None")
        except Exception as e:
            self.results.errors = "Error with creating the lazy frame, most likely due to invalid graph"
            raise e

        if not performance_mode:
            # Transient CSV schema-inference escalation: on a type-parse failure a read node
            # re-stores at a higher infer_schema_length (configured -> 10k -> 100k) without ever
            # mutating the saved setting. Ineligible nodes keep current_infer=None and never escalate.
            store_frame = self.get_resulting_data().data_frame
            current_infer = self._eligible_infer_length()
            file_ref = self.hash

            while True:
                external_df_fetcher = ExternalDfFetcher(
                    lf=store_frame,
                    file_ref=file_ref,
                    wait_on_completion=False,
                    flow_id=node_logger.flow_id,
                    node_id=self.node_id,
                )
                self._fetch_cached_df = external_df_fetcher

                try:
                    lf = external_df_fetcher.get_result()
                    # Row count rides along on the store result — no extra worker round-trip.
                    status = external_df_fetcher.status
                    self.results.resulting_data = FlowDataEngine(
                        lf,
                        number_of_records=status.number_of_records if status is not None else None,
                    )
                    self.store_example_data_generator(external_df_fetcher)
                    self.node_stats.has_run_with_current_setup = True
                    break

                except Exception as e:
                    node_logger.error("Error with external process")
                    if current_infer is not None and self._is_type_inference_error(
                        external_df_fetcher.error_description
                    ):
                        next_rung = self._next_infer_rung(current_infer)
                        escalated = self._escalated_read_frame(next_rung) if next_rung is not None else None
                        if escalated is not None:
                            node_logger.warning(
                                "CSV type inference failed; retrying read with "
                                f"infer_schema_length={next_rung}."
                            )
                            store_frame = escalated.data_frame
                            current_infer = next_rung
                            file_ref = f"{self.hash}_infer{next_rung}"
                            self._fetch_cached_df = None
                            continue
                        # Ladder exhausted on a type conflict: surface the real cause with guidance.
                        guidance = (
                            f"{external_df_fetcher.error_description}\n\n"
                            "Automatic schema-inference escalation reached its maximum "
                            f"({self._INFER_SCHEMA_RUNGS[-1]} rows) but the file still has a type conflict. "
                            "Increase 'Schema Infer Length' further, turn off 'Infer data types' to read "
                            "every column as text, or enable 'Ignore Errors' to null the unparseable values."
                        )
                        self.results.errors = guidance
                        raise Exception(guidance) from e
                    # Never degrade-gracefully on a canceled node: the raise below
                    # feeds the executor's clean-cancel reclassification instead.
                    if external_df_fetcher.error_code == -1 and not self._execution_state.is_canceled:
                        try:
                            self.results.resulting_data = self.get_resulting_data()
                            self.results.warnings = (
                                "Error with external process (unknown error), "
                                "likely the process was killed by the server because of memory constraints, "
                                "continue with the process. "
                                "We cannot display example data..."
                            )
                        except Exception as e:
                            self.results.errors = str(e)
                            raise e
                    elif external_df_fetcher.error_description is None:
                        self.results.errors = str(e)
                        raise e
                    else:
                        self.results.errors = external_df_fetcher.error_description
                        raise Exception(external_df_fetcher.error_description) from e
                    break
                finally:
                    self._fetch_cached_df = None

    # Backward-compatible aliases for renamed methods
    def execute_full_local(self, performance_mode: bool = False) -> None:
        """Backward-compatible alias for _do_execute_full_local."""
        return self._do_execute_full_local(performance_mode)

    def execute_local(self, flow_id: int, performance_mode: bool = False):
        """Backward-compatible alias for _do_execute_local_with_sampling."""
        return self._do_execute_local_with_sampling(performance_mode, flow_id)

    def execute_remote(self, performance_mode: bool = False, node_logger: NodeLogger = None):
        """Backward-compatible alias for _do_execute_remote."""
        return self._do_execute_remote(performance_mode, node_logger)

    def prepare_before_run(self):
        """Resets results and errors before a new execution."""

        self.results.errors = None
        self.results.resulting_data = None
        self.results.example_data = None
        self._named_outputs = {}
        self._named_schemas = {}

    def cancel(self):
        """Cancels an ongoing external process if one is running."""

        if self._fetch_cached_df is not None:
            self._fetch_cached_df.cancel()
        elif self._kernel_cancel_context is not None:
            kernel_id, manager, exec_token = self._kernel_cancel_context
            logger.info("Cancelling kernel execution for kernel '%s'", kernel_id)
            # Signal the cancel event so execute_sync returns promptly
            if self._kernel_cancel_event is not None:
                self._kernel_cancel_event.set()
            try:
                # Addressed to this node's own cell: a node still queued behind
                # another flow on this shared kernel interrupts nothing.
                manager.interrupt_execution_sync(kernel_id, exec_token)
            except Exception:
                logger.exception("Failed to interrupt kernel execution for kernel '%s'", kernel_id)
        elif self._subflow_cancel_context is not None:
            logger.info("Cancelling running subflow for node %s", self.node_id)
            try:
                self._subflow_cancel_context.cancel()
            except Exception:
                logger.exception("Failed to cancel subflow for node %s", self.node_id)
        else:
            logger.info("No external process to cancel; signalling in-process cancellation")
        self.node_stats.is_canceled = True
        self._execution_state.is_canceled = True

    def execute_node(
        self,
        run_location: schemas.ExecutionLocationsLiteral,
        reset_cache: bool = False,
        performance_mode: bool = False,
        retry: bool = True,
        node_logger: NodeLogger | None = None,
        optimize_for_downstream: bool = True,
    ) -> None:
        """Execute the node based on its current state and settings.

        Delegates all execution and skip logic to the NodeExecutor, which is
        the single source of truth for deciding whether a node should run.

        Args:
            run_location: Where to execute ('local' or 'remote')
            reset_cache: Force cache invalidation
            performance_mode: Skip example data generation for speed
            retry: Allow retry on recoverable errors
            node_logger: Logger for this node's execution
            optimize_for_downstream: Cache wide transforms for downstream nodes
        """
        if node_logger is None:
            raise ValueError("node_logger is required")
        if not self.is_setup:
            node_logger.warning(f"Node {self.__name__} is not setup, cannot run")
            return

        self.executor.execute(
            run_location=run_location,
            reset_cache=reset_cache,
            performance_mode=performance_mode,
            retry=retry,
            node_logger=node_logger,
            optimize_for_downstream=optimize_for_downstream,
        )

    def store_example_data_generator(self, external_df_fetcher: ExternalDfFetcher | ExternalSampler):
        """Stores a generator function for fetching a sample of the result data.

        Args:
            external_df_fetcher: The process that generated the sample data.
        """
        if external_df_fetcher.status is not None:
            file_ref = external_df_fetcher.status.file_ref
            self.results.example_data_path = file_ref
            self.results.example_data_generator = get_read_top_n(file_path=file_ref, n=100)
        else:
            logger.error("Could not get the sample data, the external process is not ready")

    def needs_reset(self) -> bool:
        """Checks if the node's hash has changed, indicating an outdated state.

        Returns:
            True if the calculated hash differs from the stored hash.
        """
        return self._hash != self.calculate_hash(self.setting_input)

    def reset(self, deep: bool = False):
        """Resets the node's execution state and schema information.

        This also triggers a reset on all downstream nodes.

        Args:
            deep: If True, forces a reset even if the hash hasn't changed.
        """
        needs_reset = self.needs_reset() or deep
        if needs_reset:
            logger.info(f"{self.node_id}: Node needs reset")
            self.node_stats.has_run_with_current_setup = False
            self.results.reset()
            self.node_schema.result_schema = None
            self.node_schema.predicted_schema = None
            self._schema_prediction_blocked = None
            self._hash = None
            self.node_information.is_setup = None
            self.results.errors = None

            # Reset execution state but preserve source file info for change detection
            self._execution_state.has_run_with_current_setup = False
            self._execution_state.has_completed_last_run = False
            self._execution_state.is_canceled = False
            self._execution_state.result_schema = None
            self._execution_state.predicted_schema = None
            self._execution_state.execution_hash = None
            # Note: source_file_info / source_version_info NOT reset - needed for change detection

            if self.is_correct:
                self._schema_callback = None
                # Eagerly prefetch only for source/start nodes — they have no
                # upstream dependencies, so a background fetch is safe and
                # masks I/O latency. Downstream nodes' callbacks read upstream
                # node state, so eagerly starting them races with the cascade
                # of resets that graph.reset() is currently performing.
                if self.is_start and self.schema_callback:
                    logger.info(f"{self.node_id}: Resetting the schema callback")
                    self.schema_callback.start()
            self.evaluate_nodes()
            _ = self.hash  # Recalculate the hash after reset

    def invalidate_cache(self):
        """Force cache invalidation by incrementing the cache epoch.

        Changes the node's hash so Development mode re-executes instead
        of returning stale results.  Used after external state changes
        (e.g. Kafka consumer group offset reset) that don't alter the
        node's configuration.
        """
        self._cache_epoch += 1
        self._hash = None
        self._execution_state.reset()
        self.node_stats.has_run_with_current_setup = False
        self.node_stats.has_completed_last_run = False

    def delete_lead_to_node(self, node_id: int) -> bool:
        """Removes a connection to a specific downstream node.

        Args:
            node_id: The ID of the downstream node to disconnect.

        Returns:
            True if the connection was found and removed, False otherwise.
        """
        logger.info(f"Deleting lead to node: {node_id}")
        for i, lead_to_node in enumerate(self.leads_to_nodes):
            logger.info(f"Checking lead to node: {lead_to_node.node_id}")
            if lead_to_node.node_id == node_id:
                logger.info(f"Found the node to delete: {node_id}")
                self.leads_to_nodes.pop(i)
                return True
        return False

    def delete_input_node(
        self, node_id: int, connection_type: input_schema.InputConnectionClass = "input-0", complete: bool = False
    ) -> bool:
        """Removes a connection from a specific input node.

        Args:
            node_id: The ID of the input node to disconnect.
            connection_type: The specific input handle (e.g., 'input-0', 'input-1').
            complete: If True, tries to delete from all input types.

        Returns:
            True if a connection was found and removed, False otherwise.
        """
        if self.accepts_dynamic_inputs:
            return self._delete_keyed_connection(node_id, connection_type, complete)
        deleted: bool = False
        if connection_type == "input-0" or complete:
            for i, node in enumerate(self.node_inputs.main_inputs or []):
                if node.node_id == node_id:
                    self.node_inputs.main_inputs.pop(i)
                    deleted = True
                    if not complete:
                        continue
        if connection_type == "input-1" or complete:
            if self.node_inputs.right_input is not None and self.node_inputs.right_input.node_id == node_id:
                self.node_inputs.right_input = None
                deleted = True
        if connection_type == "input-2" or complete:
            if self.node_inputs.left_input is not None and self.node_inputs.left_input.node_id == node_id:
                self.node_inputs.left_input = None
                deleted = True
        if not deleted and connection_type not in ("input-0", "input-1", "input-2"):
            logger.warning("Could not find the connection to delete...")
        if deleted:
            self._input_output_handles.pop(node_id, None)
            self.reset()
        return deleted

    def __repr__(self) -> str:
        """Provides a string representation of the FlowNode instance.

        Returns:
            A string showing the node's ID and type.
        """
        return f"Node id: {self.node_id} ({self.node_type})"

    def _get_readable_schema(self) -> list[dict] | None:
        """Helper to get a simplified, dictionary representation of the output schema.

        Returns:
            A list of dictionaries, each with 'column_name' and 'data_type'.
        """
        if self.is_setup:
            output = []
            for s in self.schema:
                output.append(dict(column_name=s.column_name, data_type=s.data_type))
            return output

    def get_repr(self) -> dict:
        """Gets a detailed dictionary representation of the node's state.

        Returns:
            A dictionary containing key information about the node.
        """
        return dict(
            FlowNode=dict(
                node_id=self.node_id,
                step_name=self.__name__,
                output_columns=self.node_schema.output_columns,
                output_schema=self._get_readable_schema(),
            )
        )

    @property
    def number_of_leads_to_nodes(self) -> int | None:
        """Counts the number of downstream node connections.

        Returns:
            The number of nodes this node leads to.
        """
        if self.is_setup:
            return len(self.leads_to_nodes)

    @property
    def has_next_step(self) -> bool:
        """Checks if this node has any downstream connections.

        Returns:
            True if it has at least one downstream node.
        """
        return len(self.leads_to_nodes) > 0

    @property
    def has_input(self) -> bool:
        """Checks if this node has any input connections.

        Returns:
            True if it has at least one input node.
        """
        return len(self.all_inputs) > 0

    @property
    def singular_input(self) -> bool:
        """Checks if the node template specifies exactly one input.

        Returns:
            True if the node is a single-input type.
        """
        return self.node_template.input == 1

    @property
    def singular_main_input(self) -> "FlowNode":
        """Gets the input node, assuming it is a single-input type.

        Returns:
            The single input FlowNode, or None.
        """
        if self.singular_input:
            return self.all_inputs[0]

    def clear_table_example(self) -> None:
        """
        Clear the table example in the results so that it clears the existing results
        Returns:
            None
        """

        self.results.example_data = None
        self.results.example_data_generator = None
        self.results.example_data_path = None

    @staticmethod
    def _preview_record_count(
        engine: FlowDataEngine | None, sample_len: int | None, sample_size: int = 100
    ) -> int | None:
        """Best-effort exact row count for the preview header — never computed.

        Uses only counts that are already free (a stored count, an eager frame's
        height) plus the sample-shortfall rule: a sample strictly shorter than
        the requested sample size is the whole table. Must never call
        ``get_number_of_records()`` — on a lazy engine that collects the whole
        upstream plan just to render a header.
        """
        if engine is not None:
            count = engine.known_record_count()
            if count is not None:
                return count
        if sample_len is not None and sample_len < sample_size:
            return sample_len
        return None

    def peek_output_engine(self, output_handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine | None:
        """Passively resolves the cached result engine for an output handle.

        Never routes through ``get_output()``/``get_resulting_data()``, which
        can re-execute the node. Identity checks only: FlowDataEngine truthiness
        goes through ``__len__``, which can trigger a full collect.
        """
        engine = self._named_outputs.get(output_handle)
        if engine is None:
            engine = self.results.resulting_data
        return engine

    def get_column_stats(
        self,
        column_name: str,
        output_handle: str = DEFAULT_OUTPUT_HANDLE,
        offload_to_worker: bool = False,
    ) -> FileColumn:
        """Computes on-demand stats for one column of this node's cached result.

        The stats land on the result engine's ``FlowfileColumn`` (the single
        source of truth), so this run's later previews carry them too; the
        return value is that column's ordinary ``FileColumn`` representation.
        ``offload_to_worker`` ships the aggregate to the worker — used for
        locally-run flows, whose cached engine is the full upstream plan.
        Raises ``ColumnStatsUnavailable`` when there is no cached result to
        aggregate over without executing or re-pulling data, and
        ``pl.exceptions.ColumnNotFoundError`` for an unknown column.
        """
        if self.node_template.node_group == "output":
            # An output node previews its upstream input; mirror get_table_example.
            if not self.main_input:
                raise ColumnStatsUnavailable("Output node has no input connected.")
            return self.main_input[0].get_column_stats(column_name, offload_to_worker=offload_to_worker)
        engine = self.peek_output_engine(output_handle)
        if engine is None:
            raise ColumnStatsUnavailable("Node has no cached result. Run the flow first.")
        if engine.external_source is not None:
            raise ColumnStatsUnavailable("Result is an external source; stats would re-pull it.")
        if engine.is_future and not engine.is_collected:
            raise ColumnStatsUnavailable("Result is still being computed.")
        stats = compute_column_stats(engine, column_name, offload_to_worker=offload_to_worker)
        # compute rebinds the engine's schema list (copy-on-write); follow it on
        # this node so previews via self.schema carry the stats too.
        if engine is self.results.resulting_data:
            self.node_schema.result_schema = engine.schema
        return stats

    def get_table_example(
        self, include_data: bool = False, output_handle: str = DEFAULT_OUTPUT_HANDLE
    ) -> TableExample | None:
        """Generates a `TableExample` model summarizing the node's output.

        This can optionally include a sample of the data. For multi-output
        nodes, ``output_handle`` selects which named output to preview.

        Args:
            include_data: If True, includes a data sample in the result.
            output_handle: The output handle to preview (e.g. ``"output-0"``).
                For single-output nodes the default is the only choice.

        Returns:
            A `TableExample` object, or None if the node is not set up.
        """
        self.print("Getting a table example")
        if self.is_setup and include_data and self.node_stats.has_completed_last_run:
            if self.node_template.node_group == "output":
                self.print("getting the table example")
                return self.main_input[0].get_table_example(include_data)

            logger.info("getting the table example since the node has run")
            # For multi-output nodes, pull the sample from the requested named
            # output instead of the default cached example_data_generator.
            if self._named_outputs and output_handle in self._named_outputs:
                engine = self._named_outputs[output_handle]
                preview_df = engine.data_frame.head(100)
                if isinstance(preview_df, pl.LazyFrame):
                    preview_df = preview_df.collect()
                data = preview_df.to_dicts() if preview_df is not None else []
                schema = [FileColumn.model_validate(c.get_column_repr()) for c in engine.schema]
                return TableExample(
                    node_id=self.node_id,
                    name=str(self.node_id),
                    number_of_records=self._preview_record_count(engine, len(data)),
                    number_of_columns=len(schema),
                    table_schema=schema,
                    columns=[c.name for c in schema],
                    data=data,
                    has_example_data=True,
                    has_run_with_current_setup=self.node_stats.has_run_with_current_setup,
                )

            example_data_getter = self.results.example_data_generator
            if example_data_getter is not None:
                data = example_data_getter().to_pylist()
                if data is None:
                    data = []
            else:
                data = []
            schema = [FileColumn.model_validate(c.get_column_repr()) for c in self.schema]
            has_example_data = self.results.example_data_generator is not None

            return TableExample(
                node_id=self.node_id,
                name=str(self.node_id),
                number_of_records=self._preview_record_count(
                    self.results.resulting_data, len(data) if has_example_data else None
                ),
                number_of_columns=len(schema),
                table_schema=schema,
                columns=[c.name for c in schema],
                data=data,
                has_example_data=has_example_data,
                has_run_with_current_setup=self.node_stats.has_run_with_current_setup,
            )
        else:
            logger.warning("getting the table example but the node has not run")
            try:
                schema = [FileColumn.model_validate(c.get_column_repr()) for c in self.schema]
            except Exception as e:
                logger.warning(e)
                schema = []
            columns = [s.name for s in schema]
            return TableExample(
                node_id=self.node_id,
                name=str(self.node_id),
                number_of_records=None,
                number_of_columns=len(columns),
                table_schema=schema,
                columns=columns,
                data=[],
            )

    def get_node_data(
        self,
        flow_id: int,
        include_example: bool = False,
        include_output: bool = True,
        include_inputs: bool = True,
    ) -> NodeData:
        """Gathers all necessary data for representing the node in the UI.

        Args:
            flow_id: The ID of the parent flow.
            include_example: If True, includes data samples.
            include_output: If True, computes this node's own output preview
                (``main_output``). The settings panel only needs the input
                schemas, so callers that just open settings pass False to skip
                the potentially expensive output-schema prediction (e.g. a pivot
                must materialize data to determine its output columns).
            include_inputs: If True, resolves each connected input's schema
                (``main_input``/``left_input``/``right_input``). False is the
                settings-open fast path: it skips upstream schema prediction
                entirely (which can execute un-run custom nodes on a kernel or
                worker) and therefore also the main_input-guarded setting
                generators/updators (join/cross_join/fuzzy_match). Only the
                custom-node drawer should use it for now.

        Returns:
            A `NodeData` object.
        """
        node = NodeData(
            flow_id=flow_id,
            node_id=self.node_id,
            has_run=self.node_stats.has_run_with_current_setup,
            setting_input=self.setting_input,
            flow_type=self.node_type,
        )
        if include_inputs:
            if self.accepts_dynamic_inputs:
                # The settings panel's main_input is the parameter-data connection
                # (handle input-0) specifically — not whichever slot happens to be
                # connected first.
                param_node = (self.node_inputs.keyed_inputs or {}).get(PARAM_INPUT_HANDLE)
                if param_node is not None:
                    node.main_input = param_node.get_table_example()
            elif self.main_input:
                node.main_input = self.main_input[0].get_table_example()
            if self.left_input:
                node.left_input = self.left_input.get_table_example()
            if self.right_input:
                node.right_input = self.right_input.get_table_example()
            # The get_table_example calls above cascade the inputs' predictions,
            # setting the kernel-gate flag; walk the whole upstream chain so the
            # warning reaches every downstream node.
            warning = first_upstream_prediction_warning(self)
            if warning and not (self.node_schema.result_schema or self.node_schema.predicted_schema):
                # This node may resolve its own columns independently (declared in
                # the Schema Validator). Compute its prediction — cheap here since
                # the blocked upstream means it either resolves from the
                # declaration or hits the gate, never materializing through it.
                try:
                    self.get_predicted_schema()
                except Exception:
                    pass
                if self.node_schema.result_schema or self.node_schema.predicted_schema:
                    warning = None
            node.prediction_warning = warning
        if self.is_setup and include_output:
            node.main_output = self.get_table_example(include_example)
        node = setting_generator.get_setting_generator(self.node_type)(node)

        node = setting_updator.get_setting_updator(self.node_type)(node)
        # Save the updated settings back to the node so they persist across calls
        if node.setting_input is not None and not isinstance(node.setting_input, input_schema.NodePromise):
            self.setting_input = node.setting_input
        return node

    def get_output_data(self) -> TableExample:
        """Gets the full output data sample for this node.

        Returns:
            A `TableExample` object with data.
        """
        return self.get_table_example(True)

    def get_node_input(self) -> schemas.NodeInput:
        """Creates a `NodeInput` schema object for representing this node in the UI.

        Returns:
            A `NodeInput` object.
        """
        output_names = getattr(self.setting_input, "output_names", None)
        node_reference = getattr(self.setting_input, "node_reference", None)
        template_fields = {**self.node_template.__dict__}
        if self.accepts_dynamic_inputs:
            # Per-instance handles: pass names verbatim (may be [] -> zero outputs,
            # or a single labeled output) so the frontend derives counts from them.
            template_fields["output_names"] = output_names
        elif output_names and len(output_names) > 1:
            template_fields["output_names"] = output_names
        # else: keep the template's own names — an unconfigured node has no
        # settings snapshot yet, and clobbering here would drop the tooltips.
        return schemas.NodeInput(
            pos_y=self.setting_input.pos_y,
            pos_x=self.setting_input.pos_x,
            group_id=getattr(self.setting_input, "group_id", None),
            id=self.node_id,
            node_reference=node_reference,
            input_names=getattr(self.setting_input, "input_names", None),
            **template_fields,
        )

    def get_edge_input(self) -> list[schemas.NodeEdge]:
        """Generates `NodeEdge` objects for all input connections to this node.

        Returns:
            A list of `NodeEdge` objects.
        """
        edges = []
        if self.accepts_dynamic_inputs:
            source_handles = self.node_inputs.keyed_source_handles or {}
            return [
                schemas.NodeEdge(
                    id=f"{source.node_id}-{self.node_id}-{handle}",
                    source=source.node_id,
                    target=self.node_id,
                    sourceHandle=source_handles.get(handle, DEFAULT_OUTPUT_HANDLE),
                    targetHandle=handle,
                )
                for handle, source in self.node_inputs.slot_items()
            ]
        if self.node_inputs.main_inputs is not None:
            for i, main_input in enumerate(self.node_inputs.main_inputs):
                source_handle = self._input_output_handles.get(main_input.node_id, DEFAULT_OUTPUT_HANDLE)
                edges.append(
                    schemas.NodeEdge(
                        id=f"{main_input.node_id}-{self.node_id}-{i}",
                        source=main_input.node_id,
                        target=self.node_id,
                        sourceHandle=source_handle,
                        targetHandle="input-0",
                    )
                )
        if self.node_inputs.left_input is not None:
            left_handle = self._input_output_handles.get(self.node_inputs.left_input.node_id, DEFAULT_OUTPUT_HANDLE)
            edges.append(
                schemas.NodeEdge(
                    id=f"{self.node_inputs.left_input.node_id}-{self.node_id}-right",
                    source=self.node_inputs.left_input.node_id,
                    target=self.node_id,
                    sourceHandle=left_handle,
                    targetHandle="input-2",
                )
            )
        if self.node_inputs.right_input is not None:
            right_handle = self._input_output_handles.get(self.node_inputs.right_input.node_id, DEFAULT_OUTPUT_HANDLE)
            edges.append(
                schemas.NodeEdge(
                    id=f"{self.node_inputs.right_input.node_id}-{self.node_id}-left",
                    source=self.node_inputs.right_input.node_id,
                    target=self.node_id,
                    sourceHandle=right_handle,
                    targetHandle="input-1",
                )
            )
        return edges

accepts_dynamic_inputs property

True when this node's connections are keyed by target handle (run_flow).

all_inputs property

Gets a list of all nodes connected to any input port.

Returns:

Type	Description
list[FlowNode]	A list of all input FlowNodes.

executor property

Lazy-initialized executor instance.

Reusing the same executor avoids object creation overhead when execute_node is called multiple times.

function property writable

Gets the core processing function of the node.

Returns:

Type	Description
Callable	The callable function.

has_input property

Checks if this node has any input connections.

Returns:

Type	Description
bool	True if it has at least one input node.

has_next_step property

Checks if this node has any downstream connections.

Returns:

Type	Description
bool	True if it has at least one downstream node.

hash property

Gets the cached hash for the node, calculating it if it doesn't exist.

Returns:

Type	Description
str	The string hash value.

is_correct property

Checks if the node's input connections satisfy its template requirements.

Returns:

Type	Description
bool	True if connections are valid, False otherwise.

is_setup property

Checks if the node has been properly configured and is ready for execution.

Returns:

Type	Description
bool	True if the node is set up, False otherwise.

is_start property

Determines if the node is a starting node in the flow.

A starting node requires no inputs.

Returns:

Type	Description
bool	True if the node is a start node, False otherwise.

left_input property

Gets the node connected to the left input port.

Returns:

Type	Description
Optional[FlowNode]	The left input FlowNode, or None.

main_input property

Gets the list of nodes connected to the main input port(s).

Returns:

Type	Description
list[FlowNode]	A list of main input FlowNodes.

name property writable

Gets the name of the node.

Returns:

Type	Description
str	The node's name.

node_id property

Gets the unique identifier of the node.

Returns:

Type	Description
str | int	The node's ID.

number_of_leads_to_nodes property

Counts the number of downstream node connections.

Returns:

Type	Description
int | None	The number of nodes this node leads to.

right_input property

Gets the node connected to the right input port.

Returns:

Type	Description
Optional[FlowNode]	The right input FlowNode, or None.

schema property

Gets the definitive output schema of the node.

If not already run, it falls back to the predicted schema.

Returns:

Type	Description
list[FlowfileColumn]	A list of FlowfileColumn objects.

schema_callback property writable

Gets the schema callback function, creating one if it doesn't exist.

The callback is used for predicting the output schema without full execution.

Returns:

Type	Description
SingleExecutionFuture	A SingleExecutionFuture instance wrapping the schema function.

setting_input property writable

Gets the node's specific configuration settings.

Returns:

Type	Description
Any	The settings object.

singular_input property

Checks if the node template specifies exactly one input.

Returns:

Type	Description
bool	True if the node is a single-input type.

singular_main_input property

Gets the input node, assuming it is a single-input type.

Returns:

Type	Description
FlowNode	The single input FlowNode, or None.

state_needs_reset property writable

Checks if the node's state needs to be reset.

Returns:

Type	Description
bool	True if a reset is required, False otherwise.

__call__(*args, **kwargs)

Makes the node instance callable, acting as an alias for execute_node.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def __call__(self, *args, **kwargs):
    """Makes the node instance callable, acting as an alias for execute_node."""
    self.execute_node(*args, **kwargs)

__init__(node_id, function, parent_uuid, setting_input, name, node_type, input_columns=None, output_schema=None, drop_columns=None, renew_schema=True, pos_x=0, pos_y=0, schema_callback=None)

Initializes a FlowNode instance.

Parameters:

Name	Type	Description	Default
node_id	str | int	Unique identifier for the node.	required
function	Callable	The core data processing function for the node.	required
parent_uuid	str	The UUID of the parent flow.	required
setting_input	Any	The configuration/settings object for the node.	required
name	str	The name of the node.	required
node_type	str	The type identifier of the node (e.g., 'join', 'filter').	required
input_columns	list[str]	List of column names expected as input.	None
output_schema	list[FlowfileColumn]	The schema of the columns to be added.	None
drop_columns	list[str]	List of column names to be dropped.	None
renew_schema	bool	Flag to indicate if the schema should be renewed.	True
pos_x	float	The x-coordinate on the canvas.	0
pos_y	float	The y-coordinate on the canvas.	0
schema_callback	Callable	A custom function to calculate the output schema.	None

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def __init__(
    self,
    node_id: str | int,
    function: Callable,
    parent_uuid: str,
    setting_input: Any,
    name: str,
    node_type: str,
    input_columns: list[str] = None,
    output_schema: list[FlowfileColumn] = None,
    drop_columns: list[str] = None,
    renew_schema: bool = True,
    pos_x: float = 0,
    pos_y: float = 0,
    schema_callback: Callable = None,
):
    """Initializes a FlowNode instance.

    Args:
        node_id: Unique identifier for the node.
        function: The core data processing function for the node.
        parent_uuid: The UUID of the parent flow.
        setting_input: The configuration/settings object for the node.
        name: The name of the node.
        node_type: The type identifier of the node (e.g., 'join', 'filter').
        input_columns: List of column names expected as input.
        output_schema: The schema of the columns to be added.
        drop_columns: List of column names to be dropped.
        renew_schema: Flag to indicate if the schema should be renewed.
        pos_x: The x-coordinate on the canvas.
        pos_y: The y-coordinate on the canvas.
        schema_callback: A custom function to calculate the output schema.
    """
    self.parent_uuid = parent_uuid
    self.post_init()
    self.node_information.id = node_id
    self.node_type = node_type
    self.node_settings.renew_schema = renew_schema
    self.update_node(
        function=function,
        input_columns=input_columns,
        output_schema=output_schema,
        drop_columns=drop_columns,
        setting_input=setting_input,
        name=name,
        pos_x=pos_x,
        pos_y=pos_y,
        schema_callback=schema_callback,
    )

__repr__()

Provides a string representation of the FlowNode instance.

Returns:

Type	Description
str	A string showing the node's ID and type.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def __repr__(self) -> str:
    """Provides a string representation of the FlowNode instance.

    Returns:
        A string showing the node's ID and type.
    """
    return f"Node id: {self.node_id} ({self.node_type})"

add_lead_to_in_depend_source()

Ensures this node is registered in the leads_to_nodes list of its inputs.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def add_lead_to_in_depend_source(self):
    """Ensures this node is registered in the `leads_to_nodes` list of its inputs."""
    for input_node in self.all_inputs:
        if self.node_id not in [n.node_id for n in input_node.leads_to_nodes]:
            input_node.leads_to_nodes.append(self)

add_node_connection(from_node, insert_type='main', output_handle=DEFAULT_OUTPUT_HANDLE, target_handle=None)

Adds a connection from a source node to this node.

Parameters:

Name	Type	Description	Default
from_node	FlowNode	The node to connect from.	required
insert_type	Literal['main', 'left', 'right']	The type of input to connect to ('main', 'left', 'right').	'main'
output_handle	str	The output handle on the source node (e.g. 'output-0', 'output-1').	DEFAULT_OUTPUT_HANDLE
target_handle	str | None	For dynamic-input nodes only: the target handle the edge lands on ('input-0'..'input-N'). Ignored for static nodes.	None

Raises:

Type	Description
Exception	If the insert_type is invalid.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def add_node_connection(
    self,
    from_node: "FlowNode",
    insert_type: Literal["main", "left", "right"] = "main",
    output_handle: str = DEFAULT_OUTPUT_HANDLE,
    target_handle: str | None = None,
) -> None:
    """Adds a connection from a source node to this node.

    Args:
        from_node: The node to connect from.
        insert_type: The type of input to connect to ('main', 'left', 'right').
        output_handle: The output handle on the source node (e.g. 'output-0', 'output-1').
        target_handle: For dynamic-input nodes only: the target handle the edge
            lands on ('input-0'..'input-N'). Ignored for static nodes.

    Raises:
        Exception: If the insert_type is invalid.
    """
    if self.accepts_dynamic_inputs:
        self._add_keyed_connection(from_node, target_handle or PARAM_INPUT_HANDLE, output_handle)
        return
    from_node.leads_to_nodes.append(self)
    if insert_type == "main":
        if self.node_template.input <= 2 or self.node_inputs.main_inputs is None:
            self.node_inputs.main_inputs = [from_node]
        else:
            self.node_inputs.main_inputs.append(from_node)
    elif insert_type == "right":
        self.node_inputs.right_input = from_node
    elif insert_type == "left":
        self.node_inputs.left_input = from_node
    else:
        raise Exception("Cannot find the connection")
    # Track which output handle of the source node this connection uses
    self._input_output_handles[from_node.node_id] = output_handle
    if self.setting_input.is_setup:
        if hasattr(self.setting_input, "depending_on_id") and insert_type == "main":
            self.setting_input.depending_on_id = from_node.node_id
    self.reset()
    from_node.reset()

calculate_hash(setting_input)

Calculates a hash based on settings and input node hashes.

Parameters:

Name	Type	Description	Default
setting_input	Any	The node's settings object to be included in the hash.	required

Returns:

Type	Description
str	A string hash value.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def calculate_hash(self, setting_input: Any) -> str:
    """Calculates a hash based on settings and input node hashes.

    Args:
        setting_input: The node's settings object to be included in the hash.

    Returns:
        A string hash value.
    """
    if self.accepts_dynamic_inputs:
        # Fold in the target handle and source handle so re-wiring the same
        # upstream node to a different named input invalidates caches.
        source_handles = self.node_inputs.keyed_source_handles or {}
        depends_on_hashes = [
            f"{handle}:{source_handles.get(handle, DEFAULT_OUTPUT_HANDLE)}:{_node.hash}"
            for handle, _node in self.node_inputs.slot_items()
        ]
    else:
        depends_on_hashes = [_node.hash for _node in self.all_inputs]
    node_data_hash = get_hash(setting_input)
    return get_hash(depends_on_hashes + [node_data_hash, self.parent_uuid, self._cache_epoch])

cancel()

Cancels an ongoing external process if one is running.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def cancel(self):
    """Cancels an ongoing external process if one is running."""

    if self._fetch_cached_df is not None:
        self._fetch_cached_df.cancel()
    elif self._kernel_cancel_context is not None:
        kernel_id, manager, exec_token = self._kernel_cancel_context
        logger.info("Cancelling kernel execution for kernel '%s'", kernel_id)
        # Signal the cancel event so execute_sync returns promptly
        if self._kernel_cancel_event is not None:
            self._kernel_cancel_event.set()
        try:
            # Addressed to this node's own cell: a node still queued behind
            # another flow on this shared kernel interrupts nothing.
            manager.interrupt_execution_sync(kernel_id, exec_token)
        except Exception:
            logger.exception("Failed to interrupt kernel execution for kernel '%s'", kernel_id)
    elif self._subflow_cancel_context is not None:
        logger.info("Cancelling running subflow for node %s", self.node_id)
        try:
            self._subflow_cancel_context.cancel()
        except Exception:
            logger.exception("Failed to cancel subflow for node %s", self.node_id)
    else:
        logger.info("No external process to cancel; signalling in-process cancellation")
    self.node_stats.is_canceled = True
    self._execution_state.is_canceled = True

check_upstream_laziness()

Check whether all upstream dependencies of this node support lazy execution.

Walks the DAG backwards from this node (excluding itself) and reports any eager or conditional nodes that would prevent a lazy/optimized execution path.

Returns:

Type	Description
bool	A tuple of (is_lazy, reasons). is_lazy is True when every
list[str]	upstream node has laziness == "lazy".

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def check_upstream_laziness(self) -> tuple[bool, list[str]]:
    """Check whether all upstream dependencies of this node support lazy execution.

    Walks the DAG backwards from this node (excluding itself) and reports
    any eager or conditional nodes that would prevent a lazy/optimized
    execution path.

    Returns:
        A tuple of (is_lazy, reasons).  ``is_lazy`` is True when every
        upstream node has ``laziness == "lazy"``.
    """
    visited: set[FlowNode] = set()
    stack = list(self.all_inputs)
    reasons: list[str] = []

    while stack:
        current = stack.pop()
        if current in visited:
            continue
        visited.add(current)

        if isinstance(current.setting_input, input_schema.NodeCatalogReader):
            if current.setting_input.is_virtual_optimized is False:
                reasons.append(
                    f"Node '{current.node_template.name}' (id={current.node_id}) reads a "
                    "non-optimized virtual table"
                )
        else:
            laziness: schemas.LazinessLiteral = current.node_template.laziness
            if laziness == "eager":
                reasons.append(f"Node '{current.node_template.name}' (id={current.node_id}) is eager")
            elif laziness == "conditional":
                # TODO: resolve conditional nodes (read_data, polars_code, cloud_storage_reader)
                # via isinstance checks like catalog_reader, then raise ValueError here instead
                reasons.append(
                    f"Node '{current.node_template.name}' (id={current.node_id}) is conditional"
                    " — defaulting to non-optimized"
                )

        stack.extend(current.all_inputs)
    return len(reasons) == 0, reasons

clear_table_example()

Clear the table example in the results so that it clears the existing results Returns: None

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def clear_table_example(self) -> None:
    """
    Clear the table example in the results so that it clears the existing results
    Returns:
        None
    """

    self.results.example_data = None
    self.results.example_data_generator = None
    self.results.example_data_path = None

create_schema_callback_from_function(f)

Wraps a node's function to create a schema callback that extracts the schema.

For multi-output functions, every handle's schema is captured in _named_schemas on the single call; the callback itself still returns the default handle's schema so the existing contract holds.

Thread-safe: uses _execution_lock to prevent concurrent execution with get_resulting_data.

Parameters:

Name	Type	Description	Default
f	Callable	The node's core function that returns a FlowDataEngine or NamedOutputs.	required

Returns:

Type	Description
Callable[[], list[FlowfileColumn]]	A callable that, when executed, returns the default output's schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def create_schema_callback_from_function(self, f: Callable) -> Callable[[], list[FlowfileColumn]]:
    """Wraps a node's function to create a schema callback that extracts the schema.

    For multi-output functions, every handle's schema is captured in
    ``_named_schemas`` on the single call; the callback itself still
    returns the default handle's schema so the existing contract holds.

    Thread-safe: uses _execution_lock to prevent concurrent execution with get_resulting_data.

    Args:
        f: The node's core function that returns a FlowDataEngine or NamedOutputs.

    Returns:
        A callable that, when executed, returns the default output's schema.
    """

    def schema_callback() -> list[FlowfileColumn]:
        try:
            logger.info("Executing the schema callback function based on the node function")
            with self._execution_lock:
                result = f()
                if isinstance(result, NamedOutputs):
                    self._named_schemas = {
                        output_handle(i): engine.schema for i, engine in enumerate(result.engines)
                    }
                    return self._named_schemas.get(DEFAULT_OUTPUT_HANDLE, [])
                return result.schema
        except Exception as e:
            logger.warning(f"Error with the schema callback: {e}")
            return []

    return schema_callback

delete_input_node(node_id, connection_type='input-0', complete=False)

Removes a connection from a specific input node.

Parameters:

Name	Type	Description	Default
node_id	int	The ID of the input node to disconnect.	required
connection_type	InputConnectionClass	The specific input handle (e.g., 'input-0', 'input-1').	'input-0'
complete	bool	If True, tries to delete from all input types.	False

Returns:

Type	Description
bool	True if a connection was found and removed, False otherwise.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def delete_input_node(
    self, node_id: int, connection_type: input_schema.InputConnectionClass = "input-0", complete: bool = False
) -> bool:
    """Removes a connection from a specific input node.

    Args:
        node_id: The ID of the input node to disconnect.
        connection_type: The specific input handle (e.g., 'input-0', 'input-1').
        complete: If True, tries to delete from all input types.

    Returns:
        True if a connection was found and removed, False otherwise.
    """
    if self.accepts_dynamic_inputs:
        return self._delete_keyed_connection(node_id, connection_type, complete)
    deleted: bool = False
    if connection_type == "input-0" or complete:
        for i, node in enumerate(self.node_inputs.main_inputs or []):
            if node.node_id == node_id:
                self.node_inputs.main_inputs.pop(i)
                deleted = True
                if not complete:
                    continue
    if connection_type == "input-1" or complete:
        if self.node_inputs.right_input is not None and self.node_inputs.right_input.node_id == node_id:
            self.node_inputs.right_input = None
            deleted = True
    if connection_type == "input-2" or complete:
        if self.node_inputs.left_input is not None and self.node_inputs.left_input.node_id == node_id:
            self.node_inputs.left_input = None
            deleted = True
    if not deleted and connection_type not in ("input-0", "input-1", "input-2"):
        logger.warning("Could not find the connection to delete...")
    if deleted:
        self._input_output_handles.pop(node_id, None)
        self.reset()
    return deleted

delete_lead_to_node(node_id)

Removes a connection to a specific downstream node.

Parameters:

Name	Type	Description	Default
node_id	int	The ID of the downstream node to disconnect.	required

Returns:

Type	Description
bool	True if the connection was found and removed, False otherwise.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def delete_lead_to_node(self, node_id: int) -> bool:
    """Removes a connection to a specific downstream node.

    Args:
        node_id: The ID of the downstream node to disconnect.

    Returns:
        True if the connection was found and removed, False otherwise.
    """
    logger.info(f"Deleting lead to node: {node_id}")
    for i, lead_to_node in enumerate(self.leads_to_nodes):
        logger.info(f"Checking lead to node: {lead_to_node.node_id}")
        if lead_to_node.node_id == node_id:
            logger.info(f"Found the node to delete: {node_id}")
            self.leads_to_nodes.pop(i)
            return True
    return False

evaluate_nodes(deep=False)

Triggers a state reset for all directly connected downstream nodes.

Parameters:

Name	Type	Description	Default
deep	bool	If True, the reset propagates recursively through the entire downstream graph.	False

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def evaluate_nodes(self, deep: bool = False) -> None:
    """Triggers a state reset for all directly connected downstream nodes.

    Args:
        deep: If True, the reset propagates recursively through the entire downstream graph.
    """
    for node in self.leads_to_nodes:
        self.print(f"resetting node: {node.node_id}")
        node.reset(deep)

execute_full_local(performance_mode=False)

Backward-compatible alias for _do_execute_full_local.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def execute_full_local(self, performance_mode: bool = False) -> None:
    """Backward-compatible alias for _do_execute_full_local."""
    return self._do_execute_full_local(performance_mode)

execute_local(flow_id, performance_mode=False)

Backward-compatible alias for _do_execute_local_with_sampling.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def execute_local(self, flow_id: int, performance_mode: bool = False):
    """Backward-compatible alias for _do_execute_local_with_sampling."""
    return self._do_execute_local_with_sampling(performance_mode, flow_id)

execute_node(run_location, reset_cache=False, performance_mode=False, retry=True, node_logger=None, optimize_for_downstream=True)

Execute the node based on its current state and settings.

Delegates all execution and skip logic to the NodeExecutor, which is the single source of truth for deciding whether a node should run.

Parameters:

Name	Type	Description	Default
run_location	ExecutionLocationsLiteral	Where to execute ('local' or 'remote')	required
reset_cache	bool	Force cache invalidation	False
performance_mode	bool	Skip example data generation for speed	False
retry	bool	Allow retry on recoverable errors	True
node_logger	NodeLogger | None	Logger for this node's execution	None
optimize_for_downstream	bool	Cache wide transforms for downstream nodes	True

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def execute_node(
    self,
    run_location: schemas.ExecutionLocationsLiteral,
    reset_cache: bool = False,
    performance_mode: bool = False,
    retry: bool = True,
    node_logger: NodeLogger | None = None,
    optimize_for_downstream: bool = True,
) -> None:
    """Execute the node based on its current state and settings.

    Delegates all execution and skip logic to the NodeExecutor, which is
    the single source of truth for deciding whether a node should run.

    Args:
        run_location: Where to execute ('local' or 'remote')
        reset_cache: Force cache invalidation
        performance_mode: Skip example data generation for speed
        retry: Allow retry on recoverable errors
        node_logger: Logger for this node's execution
        optimize_for_downstream: Cache wide transforms for downstream nodes
    """
    if node_logger is None:
        raise ValueError("node_logger is required")
    if not self.is_setup:
        node_logger.warning(f"Node {self.__name__} is not setup, cannot run")
        return

    self.executor.execute(
        run_location=run_location,
        reset_cache=reset_cache,
        performance_mode=performance_mode,
        retry=retry,
        node_logger=node_logger,
        optimize_for_downstream=optimize_for_downstream,
    )

execute_remote(performance_mode=False, node_logger=None)

Backward-compatible alias for _do_execute_remote.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def execute_remote(self, performance_mode: bool = False, node_logger: NodeLogger = None):
    """Backward-compatible alias for _do_execute_remote."""
    return self._do_execute_remote(performance_mode, node_logger)

get_all_dependent_node_ids()

Yields the IDs of all downstream nodes recursively.

Returns:

Type	Description
None	A generator of all dependent node IDs.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_all_dependent_node_ids(self) -> Generator[int, None, None]:
    """Yields the IDs of all downstream nodes recursively.

    Returns:
        A generator of all dependent node IDs.
    """
    for node in self.leads_to_nodes:
        yield node.node_id
        yield from node.get_all_dependent_node_ids()

get_all_dependent_nodes()

Yields all downstream nodes recursively.

Returns:

Type	Description
None	A generator of all dependent FlowNode objects.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_all_dependent_nodes(self) -> Generator["FlowNode", None, None]:
    """Yields all downstream nodes recursively.

    Returns:
        A generator of all dependent FlowNode objects.
    """
    for node in self.leads_to_nodes:
        yield node
        yield from node.get_all_dependent_nodes()

get_column_stats(column_name, output_handle=DEFAULT_OUTPUT_HANDLE, offload_to_worker=False)

Computes on-demand stats for one column of this node's cached result.

The stats land on the result engine's FlowfileColumn (the single source of truth), so this run's later previews carry them too; the return value is that column's ordinary FileColumn representation. offload_to_worker ships the aggregate to the worker — used for locally-run flows, whose cached engine is the full upstream plan. Raises ColumnStatsUnavailable when there is no cached result to aggregate over without executing or re-pulling data, and pl.exceptions.ColumnNotFoundError for an unknown column.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_column_stats(
    self,
    column_name: str,
    output_handle: str = DEFAULT_OUTPUT_HANDLE,
    offload_to_worker: bool = False,
) -> FileColumn:
    """Computes on-demand stats for one column of this node's cached result.

    The stats land on the result engine's ``FlowfileColumn`` (the single
    source of truth), so this run's later previews carry them too; the
    return value is that column's ordinary ``FileColumn`` representation.
    ``offload_to_worker`` ships the aggregate to the worker — used for
    locally-run flows, whose cached engine is the full upstream plan.
    Raises ``ColumnStatsUnavailable`` when there is no cached result to
    aggregate over without executing or re-pulling data, and
    ``pl.exceptions.ColumnNotFoundError`` for an unknown column.
    """
    if self.node_template.node_group == "output":
        # An output node previews its upstream input; mirror get_table_example.
        if not self.main_input:
            raise ColumnStatsUnavailable("Output node has no input connected.")
        return self.main_input[0].get_column_stats(column_name, offload_to_worker=offload_to_worker)
    engine = self.peek_output_engine(output_handle)
    if engine is None:
        raise ColumnStatsUnavailable("Node has no cached result. Run the flow first.")
    if engine.external_source is not None:
        raise ColumnStatsUnavailable("Result is an external source; stats would re-pull it.")
    if engine.is_future and not engine.is_collected:
        raise ColumnStatsUnavailable("Result is still being computed.")
    stats = compute_column_stats(engine, column_name, offload_to_worker=offload_to_worker)
    # compute rebinds the engine's schema list (copy-on-write); follow it on
    # this node so previews via self.schema carry the stats too.
    if engine is self.results.resulting_data:
        self.node_schema.result_schema = engine.schema
    return stats

get_edge_input()

Generates NodeEdge objects for all input connections to this node.

Returns:

Type	Description
list[NodeEdge]	A list of NodeEdge objects.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_edge_input(self) -> list[schemas.NodeEdge]:
    """Generates `NodeEdge` objects for all input connections to this node.

    Returns:
        A list of `NodeEdge` objects.
    """
    edges = []
    if self.accepts_dynamic_inputs:
        source_handles = self.node_inputs.keyed_source_handles or {}
        return [
            schemas.NodeEdge(
                id=f"{source.node_id}-{self.node_id}-{handle}",
                source=source.node_id,
                target=self.node_id,
                sourceHandle=source_handles.get(handle, DEFAULT_OUTPUT_HANDLE),
                targetHandle=handle,
            )
            for handle, source in self.node_inputs.slot_items()
        ]
    if self.node_inputs.main_inputs is not None:
        for i, main_input in enumerate(self.node_inputs.main_inputs):
            source_handle = self._input_output_handles.get(main_input.node_id, DEFAULT_OUTPUT_HANDLE)
            edges.append(
                schemas.NodeEdge(
                    id=f"{main_input.node_id}-{self.node_id}-{i}",
                    source=main_input.node_id,
                    target=self.node_id,
                    sourceHandle=source_handle,
                    targetHandle="input-0",
                )
            )
    if self.node_inputs.left_input is not None:
        left_handle = self._input_output_handles.get(self.node_inputs.left_input.node_id, DEFAULT_OUTPUT_HANDLE)
        edges.append(
            schemas.NodeEdge(
                id=f"{self.node_inputs.left_input.node_id}-{self.node_id}-right",
                source=self.node_inputs.left_input.node_id,
                target=self.node_id,
                sourceHandle=left_handle,
                targetHandle="input-2",
            )
        )
    if self.node_inputs.right_input is not None:
        right_handle = self._input_output_handles.get(self.node_inputs.right_input.node_id, DEFAULT_OUTPUT_HANDLE)
        edges.append(
            schemas.NodeEdge(
                id=f"{self.node_inputs.right_input.node_id}-{self.node_id}-left",
                source=self.node_inputs.right_input.node_id,
                target=self.node_id,
                sourceHandle=right_handle,
                targetHandle="input-1",
            )
        )
    return edges

get_flow_file_column_schema(col_name)

Retrieves the schema for a specific column from the output schema.

Parameters:

Name	Type	Description	Default
col_name	str	The name of the column.	required

Returns:

Type	Description
FlowfileColumn | None	The FlowfileColumn object for that column, or None if not found.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_flow_file_column_schema(self, col_name: str) -> FlowfileColumn | None:
    """Retrieves the schema for a specific column from the output schema.

    Args:
        col_name: The name of the column.

    Returns:
        The FlowfileColumn object for that column, or None if not found.
    """
    for s in self.schema:
        if s.column_name == col_name:
            return s

get_input_type(node_id)

Gets the type of connection ('main', 'left', 'right') for a given input node ID.

Parameters:

Name	Type	Description	Default
node_id	int	The ID of the input node.	required

Returns:

Type	Description
list	A list of connection types for that node ID.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_input_type(self, node_id: int) -> list:
    """Gets the type of connection ('main', 'left', 'right') for a given input node ID.

    Args:
        node_id: The ID of the input node.

    Returns:
        A list of connection types for that node ID.
    """
    relation_type = []
    if node_id in [n.node_id for n in self.node_inputs.main_inputs]:
        relation_type.append("main")
    if self.node_inputs.left_input is not None and node_id == self.node_inputs.left_input.node_id:
        relation_type.append("left")
    if self.node_inputs.right_input is not None and node_id == self.node_inputs.right_input.node_id:
        relation_type.append("right")
    return list(set(relation_type))

get_node_data(flow_id, include_example=False, include_output=True, include_inputs=True)

Gathers all necessary data for representing the node in the UI.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the parent flow.	required
include_example	bool	If True, includes data samples.	False
include_output	bool	If True, computes this node's own output preview (main_output). The settings panel only needs the input schemas, so callers that just open settings pass False to skip the potentially expensive output-schema prediction (e.g. a pivot must materialize data to determine its output columns).	True
include_inputs	bool	If True, resolves each connected input's schema (main_input/left_input/right_input). False is the settings-open fast path: it skips upstream schema prediction entirely (which can execute un-run custom nodes on a kernel or worker) and therefore also the main_input-guarded setting generators/updators (join/cross_join/fuzzy_match). Only the custom-node drawer should use it for now.	True

Returns:

Type	Description
NodeData	A NodeData object.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_node_data(
    self,
    flow_id: int,
    include_example: bool = False,
    include_output: bool = True,
    include_inputs: bool = True,
) -> NodeData:
    """Gathers all necessary data for representing the node in the UI.

    Args:
        flow_id: The ID of the parent flow.
        include_example: If True, includes data samples.
        include_output: If True, computes this node's own output preview
            (``main_output``). The settings panel only needs the input
            schemas, so callers that just open settings pass False to skip
            the potentially expensive output-schema prediction (e.g. a pivot
            must materialize data to determine its output columns).
        include_inputs: If True, resolves each connected input's schema
            (``main_input``/``left_input``/``right_input``). False is the
            settings-open fast path: it skips upstream schema prediction
            entirely (which can execute un-run custom nodes on a kernel or
            worker) and therefore also the main_input-guarded setting
            generators/updators (join/cross_join/fuzzy_match). Only the
            custom-node drawer should use it for now.

    Returns:
        A `NodeData` object.
    """
    node = NodeData(
        flow_id=flow_id,
        node_id=self.node_id,
        has_run=self.node_stats.has_run_with_current_setup,
        setting_input=self.setting_input,
        flow_type=self.node_type,
    )
    if include_inputs:
        if self.accepts_dynamic_inputs:
            # The settings panel's main_input is the parameter-data connection
            # (handle input-0) specifically — not whichever slot happens to be
            # connected first.
            param_node = (self.node_inputs.keyed_inputs or {}).get(PARAM_INPUT_HANDLE)
            if param_node is not None:
                node.main_input = param_node.get_table_example()
        elif self.main_input:
            node.main_input = self.main_input[0].get_table_example()
        if self.left_input:
            node.left_input = self.left_input.get_table_example()
        if self.right_input:
            node.right_input = self.right_input.get_table_example()
        # The get_table_example calls above cascade the inputs' predictions,
        # setting the kernel-gate flag; walk the whole upstream chain so the
        # warning reaches every downstream node.
        warning = first_upstream_prediction_warning(self)
        if warning and not (self.node_schema.result_schema or self.node_schema.predicted_schema):
            # This node may resolve its own columns independently (declared in
            # the Schema Validator). Compute its prediction — cheap here since
            # the blocked upstream means it either resolves from the
            # declaration or hits the gate, never materializing through it.
            try:
                self.get_predicted_schema()
            except Exception:
                pass
            if self.node_schema.result_schema or self.node_schema.predicted_schema:
                warning = None
        node.prediction_warning = warning
    if self.is_setup and include_output:
        node.main_output = self.get_table_example(include_example)
    node = setting_generator.get_setting_generator(self.node_type)(node)

    node = setting_updator.get_setting_updator(self.node_type)(node)
    # Save the updated settings back to the node so they persist across calls
    if node.setting_input is not None and not isinstance(node.setting_input, input_schema.NodePromise):
        self.setting_input = node.setting_input
    return node

get_node_information()

Updates and returns the node's information object.

Returns:

Type	Description
NodeInformation	The NodeInformation object for this node.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_node_information(self) -> schemas.NodeInformation:
    """Updates and returns the node's information object.

    Returns:
        The `NodeInformation` object for this node.
    """
    self.set_node_information()
    return self.node_information

get_node_input()

Creates a NodeInput schema object for representing this node in the UI.

Returns:

Type	Description
NodeInput	A NodeInput object.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_node_input(self) -> schemas.NodeInput:
    """Creates a `NodeInput` schema object for representing this node in the UI.

    Returns:
        A `NodeInput` object.
    """
    output_names = getattr(self.setting_input, "output_names", None)
    node_reference = getattr(self.setting_input, "node_reference", None)
    template_fields = {**self.node_template.__dict__}
    if self.accepts_dynamic_inputs:
        # Per-instance handles: pass names verbatim (may be [] -> zero outputs,
        # or a single labeled output) so the frontend derives counts from them.
        template_fields["output_names"] = output_names
    elif output_names and len(output_names) > 1:
        template_fields["output_names"] = output_names
    # else: keep the template's own names — an unconfigured node has no
    # settings snapshot yet, and clobbering here would drop the tooltips.
    return schemas.NodeInput(
        pos_y=self.setting_input.pos_y,
        pos_x=self.setting_input.pos_x,
        group_id=getattr(self.setting_input, "group_id", None),
        id=self.node_id,
        node_reference=node_reference,
        input_names=getattr(self.setting_input, "input_names", None),
        **template_fields,
    )

get_output(handle=DEFAULT_OUTPUT_HANDLE)

Get the result for a specific output handle.

For nodes with multiple outputs (e.g. kernel-based custom nodes), returns the FlowDataEngine associated with the given handle. Falls back to the default results.resulting_data for single-output nodes.

Parameters:

Name	Type	Description	Default
handle	str	The output handle identifier (e.g. "output-0", "output-1").	DEFAULT_OUTPUT_HANDLE

Returns:

Type	Description
FlowDataEngine | None	The FlowDataEngine for the requested output, or None.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_output(self, handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine | None:
    """Get the result for a specific output handle.

    For nodes with multiple outputs (e.g. kernel-based custom nodes),
    returns the FlowDataEngine associated with the given handle.
    Falls back to the default ``results.resulting_data`` for single-output nodes.

    Args:
        handle: The output handle identifier (e.g. ``"output-0"``, ``"output-1"``).

    Returns:
        The FlowDataEngine for the requested output, or None.
    """
    self.get_resulting_data()
    if handle in self._named_outputs:
        return self._named_outputs[handle]
    return self.results.resulting_data

get_output_data()

Gets the full output data sample for this node.

Returns:

Type	Description
TableExample	A TableExample object with data.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_output_data(self) -> TableExample:
    """Gets the full output data sample for this node.

    Returns:
        A `TableExample` object with data.
    """
    return self.get_table_example(True)

get_predicted_resulting_data(handle=DEFAULT_OUTPUT_HANDLE)

Creates a FlowDataEngine instance based on the predicted schema.

This avoids executing the node's full logic. For multi-output nodes the handle argument selects which output's schema to reflect so that a downstream node wired to e.g. output-1 sees that partition's schema.

Parameters:

Name	Type	Description	Default
handle	str	The output handle to reflect. Ignored for single-output nodes.	DEFAULT_OUTPUT_HANDLE

Returns:

Type	Description
FlowDataEngine	A FlowDataEngine instance with a schema but no data.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_predicted_resulting_data(self, handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine:
    """Creates a `FlowDataEngine` instance based on the predicted schema.

    This avoids executing the node's full logic. For multi-output nodes the
    ``handle`` argument selects which output's schema to reflect so that a
    downstream node wired to e.g. ``output-1`` sees that partition's schema.

    Args:
        handle: The output handle to reflect. Ignored for single-output nodes.

    Returns:
        A FlowDataEngine instance with a schema but no data.
    """
    # Multi-output: prefer the handle-specific cached schema if we have it.
    if handle != DEFAULT_OUTPUT_HANDLE and (self._named_schemas or self._named_outputs):
        schema = self.schema_for_handle(handle)
        if schema:
            return FlowDataEngine.create_from_schema(schema)

    if self.needs_run(False) and self.schema_callback is not None or self.node_schema.result_schema is not None:
        self.print("Getting data based on the schema")
        # Running the schema callback populates _named_schemas for multi-output
        # nodes; re-check the handle cache afterward before falling back.
        if self.node_schema.result_schema is None:
            _s = self.schema_callback()
            if handle != DEFAULT_OUTPUT_HANDLE and handle in self._named_schemas:
                _s = self._named_schemas[handle]
            if not _s:
                # Empty is the callback's "no declared schema" sentinel (e.g. a
                # predict_output_schema hook opting out) — use the full prediction
                # ladder, which falls back to execution-based prediction.
                _s = self.get_predicted_schema()
                if handle != DEFAULT_OUTPUT_HANDLE:
                    # The exec fallback fills the per-handle caches; prefer them.
                    _s = self.schema_for_handle(handle)
        else:
            _s = self.node_schema.result_schema
        return FlowDataEngine.create_from_schema(_s or [])
    else:
        if isinstance(self.function, FlowDataEngine):
            fl = self.function
        else:
            fl = FlowDataEngine.create_from_schema(self.get_predicted_schema())
        return fl

get_predicted_schema(force=False)

Predicts the output schema of the node without full execution.

It uses the schema_callback or infers from predicted data.

Parameters:

Name	Type	Description	Default
force	bool	If True, forces recalculation even if a predicted schema exists.	False

Returns:

Type	Description
list[FlowfileColumn] | None	A list of FlowfileColumn objects representing the predicted schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_predicted_schema(self, force: bool = False) -> list[FlowfileColumn] | None:
    """Predicts the output schema of the node without full execution.

    It uses the schema_callback or infers from predicted data.

    Args:
        force: If True, forces recalculation even if a predicted schema exists.

    Returns:
        A list of FlowfileColumn objects representing the predicted schema.
    """
    _has_output_field_config = (
        hasattr(self._setting_input, "output_field_config") and self._setting_input.output_field_config is not None
        if self._setting_input
        else False
    )
    logger.info(
        f"get_predicted_schema: node_id={self.node_id}, node_type={self.node_type}, force={force}, "
        f"has_predicted_schema={self.node_schema.predicted_schema is not None}, "
        f"has_schema_callback={self.schema_callback is not None}, "
        f"has_output_field_config={_has_output_field_config}"
    )

    if self.node_schema.predicted_schema and not force:
        logger.debug(f"get_predicted_schema: node_id={self.node_id} - returning cached predicted_schema")
        return self.node_schema.predicted_schema

    if self.schema_callback is not None and (self.node_schema.predicted_schema is None or force):
        self.print("Getting the data from a schema callback")
        logger.info(f"get_predicted_schema: node_id={self.node_id} - invoking schema_callback")
        if force:
            # Force the schema callback to reset, so that it will be executed again
            logger.debug(f"get_predicted_schema: node_id={self.node_id} - forcing schema_callback reset")
            self.schema_callback.reset()

        try:
            schema = self.schema_callback()
            logger.info(
                f"get_predicted_schema: node_id={self.node_id} - schema_callback returned "
                f"{len(schema) if schema else 0} columns: {[c.name for c in schema] if schema else []}"
            )
        except Exception as e:
            logger.error(f"get_predicted_schema: node_id={self.node_id} - schema_callback raised exception: {e}")
            schema = None

        if schema is not None and len(schema) > 0:
            self.print("Calculating the schema based on the schema callback")
            self.node_schema.predicted_schema = schema
            logger.info(f"get_predicted_schema: node_id={self.node_id} - set predicted_schema from schema_callback")
            return self.node_schema.predicted_schema
        else:
            logger.warning(
                f"get_predicted_schema: node_id={self.node_id} - schema_callback returned empty/None schema"
            )
    else:
        logger.debug(f"get_predicted_schema: node_id={self.node_id} - no schema_callback available")

    if self._schema_prediction_blocked is None and (self._prediction_requires_data or self._executes_on_kernel):
        self._schema_prediction_blocked = kernel_block_reason(self, include_self=True)
    if self._schema_prediction_blocked:
        # Prediction would require executing an un-run kernel node: never do
        # that implicitly — surface the warning and skip the exec tier.
        self.results.warnings = self._schema_prediction_blocked
        return self.node_schema.predicted_schema

    logger.debug(f"get_predicted_schema: node_id={self.node_id} - falling back to _predicted_data_getter")
    # Serialize the fallback: without a callback, prediction executes the node's
    # real function (kernel/worker for custom nodes) — concurrent callers must
    # not run it twice into the same working dirs.
    with self._execution_lock:
        if self.node_schema.predicted_schema and not force:
            return self.node_schema.predicted_schema
        predicted_data = self._predicted_data_getter()
        if predicted_data is not None and predicted_data.schema is not None:
            self.print("Calculating the schema based on the predicted resulting data")
            logger.info(
                f"get_predicted_schema: node_id={self.node_id} - using schema from predicted_data "
                f"({len(predicted_data.schema)} columns)"
            )
            self.node_schema.predicted_schema = predicted_data.schema
        else:
            logger.warning(
                f"get_predicted_schema: node_id={self.node_id} - no schema available from any source "
                f"(predicted_data={'None' if predicted_data is None else 'has_data'}, "
                f"schema={'None' if predicted_data is None or predicted_data.schema is None else 'has_schema'})"
            )

    return self.node_schema.predicted_schema

get_repr()

Gets a detailed dictionary representation of the node's state.

Returns:

Type	Description
dict	A dictionary containing key information about the node.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_repr(self) -> dict:
    """Gets a detailed dictionary representation of the node's state.

    Returns:
        A dictionary containing key information about the node.
    """
    return dict(
        FlowNode=dict(
            node_id=self.node_id,
            step_name=self.__name__,
            output_columns=self.node_schema.output_columns,
            output_schema=self._get_readable_schema(),
        )
    )

get_resulting_data()

Executes the node's function to produce the actual output data.

Handles both regular functions and external data sources. Thread-safe and single-flight: the node's own _execution_lock ensures the function runs at most once and the result is memoized, so N downstream consumers materialize it once. A node acquires only its OWN lock; upstream inputs are read through each upstream's own get_resulting_data(), so lock acquisition always follows the DAG (a node -> its parents) and cannot form a cross-node cycle.

Returns:

Type	Description
FlowDataEngine | None	A FlowDataEngine instance containing the result, or None on error.

Raises:

Type	Description
Exception	Propagates exceptions from the node's function execution.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_resulting_data(self) -> FlowDataEngine | None:
    """Executes the node's function to produce the actual output data.

    Handles both regular functions and external data sources.
    Thread-safe and single-flight: the node's own ``_execution_lock`` ensures
    the function runs at most once and the result is memoized, so N downstream
    consumers materialize it once. A node acquires only its OWN lock; upstream
    inputs are read through each upstream's own ``get_resulting_data()``, so
    lock acquisition always follows the DAG (a node -> its parents) and cannot
    form a cross-node cycle.

    Returns:
        A FlowDataEngine instance containing the result, or None on error.

    Raises:
        Exception: Propagates exceptions from the node's function execution.
    """
    if self.is_setup:
        with self._execution_lock_held():
            if self.results.resulting_data is None and self.results.errors is None:
                self.print("getting resulting data")
                try:
                    if self._execution_state.is_canceled:
                        raise Exception("Node execution canceled")
                    if isinstance(self.function, FlowDataEngine):
                        fl: FlowDataEngine = self.function
                    elif self.node_type == "external_source":
                        fl: FlowDataEngine = self.function()
                        fl.collect_external()
                        self.node_settings.streamable = False
                    else:
                        self.print("Collecting input data from all inputs")
                        input_data = []
                        for i, (v, src_handle) in enumerate(self._slot_input_pairs()):
                            if v is None:
                                input_data.append(None)
                                continue
                            if self._execution_state.is_canceled:
                                raise Exception("Node execution canceled")
                            self.print(f"Getting resulting data from input {i} (node {v.node_id})")
                            # Read the upstream via its own get_resulting_data(), which
                            # single-flights materialization under the upstream's own
                            # _execution_lock (memoized into results.resulting_data). We do
                            # NOT acquire the upstream's lock here: a node holds only its own
                            # lock, so lock acquisition always follows the DAG (a node -> its
                            # parents) and can never form a cross-node cycle. Taking upstream
                            # locks in per-input slot order used to deadlock a parallel stage
                            # when two sibling nodes consumed the same two upstreams in
                            # opposite left/right order (AB-BA).
                            input_result = self._resolve_input_result_for_handle(v, src_handle)
                            if input_result is not None:
                                # De-alias: hand the node function a private view. Some node
                                # functions mutate their input (df.lazy = True in the writers,
                                # cross_join/fuzzy prep) or return it unchanged (output,
                                # filter passthrough, ...), and this node's own post-processing
                                # (set_streamable, output_field_config) mutates whatever the
                                # function returned — the copy keeps all of that off the
                                # engine shared with sibling consumers in the same stage.
                                input_result = input_result.shallow_copy()
                            _df_type = type(input_result.data_frame) if input_result else "None"
                            self.print(f"Input {i} data type: {type(input_result)}, " f"dataframe type: {_df_type}")
                            input_data.append(input_result)
                        self.print(f"All {len(input_data)} inputs collected, calling node function")
                        fl = self._function(*input_data)
                    if isinstance(fl, NamedOutputs):
                        self._named_outputs = fl.by_handle()
                        self._named_schemas = {h: e.schema for h, e in self._named_outputs.items()}
                        for v in self._named_outputs.values():
                            v.set_streamable(self.node_settings.streamable)
                        # Default downstream-without-handle consumers to the first output.
                        # output_field_config (below) only applies to this default; future
                        # multi-output nodes that need per-output config must extend the loop.
                        fl = self._named_outputs[DEFAULT_OUTPUT_HANDLE]
                    else:
                        fl.set_streamable(self.node_settings.streamable)

                    if (
                        hasattr(self._setting_input, "output_field_config")
                        and self._setting_input.output_field_config
                    ):
                        try:
                            fl = apply_output_field_config(fl, self._setting_input.output_field_config)
                        except Exception as e:
                            logger.error(f"Error applying output field config for node {self.node_id}: {e}")
                            raise

                    self.results.resulting_data = fl
                    self.node_schema.result_schema = fl.schema
                except Exception as e:
                    self.results.resulting_data = FlowDataEngine()
                    self.results.errors = str(e)
                    self.node_stats.has_run_with_current_setup = False
                    self.node_stats.has_completed_last_run = False
                    raise e
            return self.results.resulting_data

get_table_example(include_data=False, output_handle=DEFAULT_OUTPUT_HANDLE)

Generates a TableExample model summarizing the node's output.

This can optionally include a sample of the data. For multi-output nodes, output_handle selects which named output to preview.

Parameters:

Name	Type	Description	Default
include_data	bool	If True, includes a data sample in the result.	False
output_handle	str	The output handle to preview (e.g. "output-0"). For single-output nodes the default is the only choice.	DEFAULT_OUTPUT_HANDLE

Returns:

Type	Description
TableExample | None	A TableExample object, or None if the node is not set up.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def get_table_example(
    self, include_data: bool = False, output_handle: str = DEFAULT_OUTPUT_HANDLE
) -> TableExample | None:
    """Generates a `TableExample` model summarizing the node's output.

    This can optionally include a sample of the data. For multi-output
    nodes, ``output_handle`` selects which named output to preview.

    Args:
        include_data: If True, includes a data sample in the result.
        output_handle: The output handle to preview (e.g. ``"output-0"``).
            For single-output nodes the default is the only choice.

    Returns:
        A `TableExample` object, or None if the node is not set up.
    """
    self.print("Getting a table example")
    if self.is_setup and include_data and self.node_stats.has_completed_last_run:
        if self.node_template.node_group == "output":
            self.print("getting the table example")
            return self.main_input[0].get_table_example(include_data)

        logger.info("getting the table example since the node has run")
        # For multi-output nodes, pull the sample from the requested named
        # output instead of the default cached example_data_generator.
        if self._named_outputs and output_handle in self._named_outputs:
            engine = self._named_outputs[output_handle]
            preview_df = engine.data_frame.head(100)
            if isinstance(preview_df, pl.LazyFrame):
                preview_df = preview_df.collect()
            data = preview_df.to_dicts() if preview_df is not None else []
            schema = [FileColumn.model_validate(c.get_column_repr()) for c in engine.schema]
            return TableExample(
                node_id=self.node_id,
                name=str(self.node_id),
                number_of_records=self._preview_record_count(engine, len(data)),
                number_of_columns=len(schema),
                table_schema=schema,
                columns=[c.name for c in schema],
                data=data,
                has_example_data=True,
                has_run_with_current_setup=self.node_stats.has_run_with_current_setup,
            )

        example_data_getter = self.results.example_data_generator
        if example_data_getter is not None:
            data = example_data_getter().to_pylist()
            if data is None:
                data = []
        else:
            data = []
        schema = [FileColumn.model_validate(c.get_column_repr()) for c in self.schema]
        has_example_data = self.results.example_data_generator is not None

        return TableExample(
            node_id=self.node_id,
            name=str(self.node_id),
            number_of_records=self._preview_record_count(
                self.results.resulting_data, len(data) if has_example_data else None
            ),
            number_of_columns=len(schema),
            table_schema=schema,
            columns=[c.name for c in schema],
            data=data,
            has_example_data=has_example_data,
            has_run_with_current_setup=self.node_stats.has_run_with_current_setup,
        )
    else:
        logger.warning("getting the table example but the node has not run")
        try:
            schema = [FileColumn.model_validate(c.get_column_repr()) for c in self.schema]
        except Exception as e:
            logger.warning(e)
            schema = []
        columns = [s.name for s in schema]
        return TableExample(
            node_id=self.node_id,
            name=str(self.node_id),
            number_of_records=None,
            number_of_columns=len(columns),
            table_schema=schema,
            columns=columns,
            data=[],
        )

invalidate_cache()

Force cache invalidation by incrementing the cache epoch.

Changes the node's hash so Development mode re-executes instead of returning stale results. Used after external state changes (e.g. Kafka consumer group offset reset) that don't alter the node's configuration.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def invalidate_cache(self):
    """Force cache invalidation by incrementing the cache epoch.

    Changes the node's hash so Development mode re-executes instead
    of returning stale results.  Used after external state changes
    (e.g. Kafka consumer group offset reset) that don't alter the
    node's configuration.
    """
    self._cache_epoch += 1
    self._hash = None
    self._execution_state.reset()
    self.node_stats.has_run_with_current_setup = False
    self.node_stats.has_completed_last_run = False

needs_reset()

Checks if the node's hash has changed, indicating an outdated state.

Returns:

Type	Description
bool	True if the calculated hash differs from the stored hash.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def needs_reset(self) -> bool:
    """Checks if the node's hash has changed, indicating an outdated state.

    Returns:
        True if the calculated hash differs from the stored hash.
    """
    return self._hash != self.calculate_hash(self.setting_input)

needs_run(performance_mode, node_logger=None, execution_location='remote')

Determines if the node needs to be executed.

The decision is based on its run state, caching settings, and execution mode.

Parameters:

Name	Type	Description	Default
performance_mode	bool	True if the flow is in performance mode.	required
node_logger	NodeLogger	The logger instance for this node.	None
execution_location	ExecutionLocationsLiteral	The target execution location.	'remote'

Returns:

Type	Description
bool	True if the node should be run, False otherwise.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def needs_run(
    self,
    performance_mode: bool,
    node_logger: NodeLogger = None,
    execution_location: schemas.ExecutionLocationsLiteral = "remote",
) -> bool:
    """Determines if the node needs to be executed.

    The decision is based on its run state, caching settings, and execution mode.

    Args:
        performance_mode: True if the flow is in performance mode.
        node_logger: The logger instance for this node.
        execution_location: The target execution location.

    Returns:
        True if the node should be run, False otherwise.
    """
    if execution_location == "local":
        return False

    flow_logger = logger if node_logger is None else node_logger
    cache_result_exists = results_exists(self.hash)
    if not self.node_stats.has_run_with_current_setup:
        flow_logger.info("Node has not run, needs to run")
        return True
    if self.node_settings.cache_results and cache_result_exists:
        return False
    elif self.node_settings.cache_results and not cache_result_exists:
        return True
    elif not performance_mode and cache_result_exists:
        return False
    else:
        return True

peek_output_engine(output_handle=DEFAULT_OUTPUT_HANDLE)

Passively resolves the cached result engine for an output handle.

Never routes through get_output()/get_resulting_data(), which can re-execute the node. Identity checks only: FlowDataEngine truthiness goes through __len__, which can trigger a full collect.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def peek_output_engine(self, output_handle: str = DEFAULT_OUTPUT_HANDLE) -> FlowDataEngine | None:
    """Passively resolves the cached result engine for an output handle.

    Never routes through ``get_output()``/``get_resulting_data()``, which
    can re-execute the node. Identity checks only: FlowDataEngine truthiness
    goes through ``__len__``, which can trigger a full collect.
    """
    engine = self._named_outputs.get(output_handle)
    if engine is None:
        engine = self.results.resulting_data
    return engine

post_init()

Reset every instance attribute to its default state.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def post_init(self):
    """Reset every instance attribute to its default state."""
    self.active = True
    self.node_information = schemas.NodeInformation()
    self.node_inputs = NodeStepInputs()
    self.node_stats = NodeStepStats()
    self.node_settings = NodeStepSettings()
    self.node_schema = NodeSchemaInformation()
    self.results = NodeResults()
    self.leads_to_nodes = []

    self._name = None
    self._function = None
    self._setting_input = None
    self._executor = None
    self._execution_state = NodeExecutionState()
    self._execution_lock = threading.RLock()
    self._state_needs_reset = False
    self._on_flow_complete = None

    self.user_provided_schema_callback = None
    self._schema_callback = None
    self._named_outputs = {}  # per-handle output engines, keyed by output handle
    self._named_schemas = {}  # per-handle schemas, populated alongside _named_outputs
    self._input_output_handles = {}  # source node id -> the output handle it connects through

    self._hash = None
    self._cache_epoch = 0
    self._cache_progress = None
    self._fetch_cached_df = None

    self._kernel_cancel_context = None
    self._kernel_cancel_event = None
    self._subflow_cancel_context = None

    self._params_getter = None
    self._schema_prediction_blocked = None
    self._prediction_requires_data = False
    self._executes_on_kernel = False

prepare_before_run()

Resets results and errors before a new execution.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def prepare_before_run(self):
    """Resets results and errors before a new execution."""

    self.results.errors = None
    self.results.resulting_data = None
    self.results.example_data = None
    self._named_outputs = {}
    self._named_schemas = {}

print(v)

Helper method to log messages with node context.

Parameters:

Name	Type	Description	Default
v	Any	The message or value to log.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def print(self, v: Any):
    """Helper method to log messages with node context.

    Args:
        v: The message or value to log.
    """
    logger.info(f"{self.node_type}, node_id: {self.node_id}: {v}")

remap_dynamic_inputs(mapping)

Re-key keyed connections after the node's input slots changed.

Parameters:

Name	Type	Description	Default
mapping	dict[str, str | None]	old handle -> new handle, or None to drop that connection. Handles absent from the mapping keep their key.	required

Returns:

Type	Description
dict[str, list[str]]	{"moved": [...], "dropped": [...]} describing what happened, so the
dict[str, list[str]]	API layer can surface removed connections to the UI.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def remap_dynamic_inputs(self, mapping: dict[str, str | None]) -> dict[str, list[str]]:
    """Re-key keyed connections after the node's input slots changed.

    Args:
        mapping: old handle -> new handle, or None to drop that connection.
            Handles absent from the mapping keep their key.

    Returns:
        {"moved": [...], "dropped": [...]} describing what happened, so the
        API layer can surface removed connections to the UI.
    """
    inputs = self.node_inputs
    if not inputs.keyed_inputs:
        return {"moved": [], "dropped": []}
    moved: list[str] = []
    dropped: list[str] = []
    new_inputs: dict[str, FlowNode] = {}
    new_source_handles: dict[str, str] = {}
    for handle, node in inputs.keyed_inputs.items():
        new_handle = mapping.get(handle, handle)
        if new_handle is None:
            dropped.append(handle)
            node.delete_lead_to_node(self.node_id)
            continue
        if new_handle != handle:
            moved.append(f"{handle}->{new_handle}")
        new_inputs[new_handle] = node
        new_source_handles[new_handle] = (inputs.keyed_source_handles or {}).get(handle, DEFAULT_OUTPUT_HANDLE)
    inputs.keyed_inputs = new_inputs
    inputs.keyed_source_handles = new_source_handles
    remaining_ids = {n.node_id for n in new_inputs.values()}
    for node_id in list(self._input_output_handles):
        if node_id not in remaining_ids:
            self._input_output_handles.pop(node_id, None)
    inputs.rebuild_keyed_projection()
    self.reset()
    return {"moved": moved, "dropped": dropped}

remove_cache()

Removes cached results for this node.

Note: Currently not fully implemented.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def remove_cache(self):
    """Removes cached results for this node.

    Note: Currently not fully implemented.
    """

    if results_exists(self.hash):
        logger.warning("Not implemented")
        clear_task_from_worker(self.hash)

reset(deep=False)

Resets the node's execution state and schema information.

This also triggers a reset on all downstream nodes.

Parameters:

Name	Type	Description	Default
deep	bool	If True, forces a reset even if the hash hasn't changed.	False

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def reset(self, deep: bool = False):
    """Resets the node's execution state and schema information.

    This also triggers a reset on all downstream nodes.

    Args:
        deep: If True, forces a reset even if the hash hasn't changed.
    """
    needs_reset = self.needs_reset() or deep
    if needs_reset:
        logger.info(f"{self.node_id}: Node needs reset")
        self.node_stats.has_run_with_current_setup = False
        self.results.reset()
        self.node_schema.result_schema = None
        self.node_schema.predicted_schema = None
        self._schema_prediction_blocked = None
        self._hash = None
        self.node_information.is_setup = None
        self.results.errors = None

        # Reset execution state but preserve source file info for change detection
        self._execution_state.has_run_with_current_setup = False
        self._execution_state.has_completed_last_run = False
        self._execution_state.is_canceled = False
        self._execution_state.result_schema = None
        self._execution_state.predicted_schema = None
        self._execution_state.execution_hash = None
        # Note: source_file_info / source_version_info NOT reset - needed for change detection

        if self.is_correct:
            self._schema_callback = None
            # Eagerly prefetch only for source/start nodes — they have no
            # upstream dependencies, so a background fetch is safe and
            # masks I/O latency. Downstream nodes' callbacks read upstream
            # node state, so eagerly starting them races with the cascade
            # of resets that graph.reset() is currently performing.
            if self.is_start and self.schema_callback:
                logger.info(f"{self.node_id}: Resetting the schema callback")
                self.schema_callback.start()
        self.evaluate_nodes()
        _ = self.hash  # Recalculate the hash after reset

schema_for_handle(handle)

Return the cached schema for a specific output handle.

Falls back to the default schema property when the handle is unknown or the node is single-output, so callers can always rely on this.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def schema_for_handle(self, handle: str) -> list[FlowfileColumn]:
    """Return the cached schema for a specific output handle.

    Falls back to the default ``schema`` property when the handle is unknown
    or the node is single-output, so callers can always rely on this.
    """
    if handle in self._named_schemas:
        return self._named_schemas[handle]
    if handle in self._named_outputs:
        return self._named_outputs[handle].schema
    return self.schema

set_node_information()

Populates the node_information attribute with the current state.

This includes the node's connections, settings, and position.

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def set_node_information(self):
    """Populates the `node_information` attribute with the current state.

    This includes the node's connections, settings, and position.
    """
    node_information = self.node_information
    node_information.left_input_id = self.node_inputs.left_input.node_id if self.left_input else None
    node_information.right_input_id = self.node_inputs.right_input.node_id if self.right_input else None
    node_information.input_ids = (
        [mi.node_id for mi in self.node_inputs.main_inputs] if self.node_inputs.main_inputs is not None else None
    )
    node_information.setting_input = self.setting_input
    node_information.outputs = [n.node_id for n in self.leads_to_nodes]
    # Source-side handle for each downstream connection — the downstream
    # node tracks this in ``_input_output_handles[from_node_id]``.
    node_information.output_handles = [
        n._input_output_handles.get(self.node_id, DEFAULT_OUTPUT_HANDLE) for n in self.leads_to_nodes
    ]
    if self.accepts_dynamic_inputs and self.node_inputs.keyed_inputs:
        source_handles = self.node_inputs.keyed_source_handles or {}
        node_information.input_connections = [
            schemas.FlowfileInputConnection(
                from_id=source.node_id,
                input_handle=handle,
                source_handle=source_handles.get(handle, DEFAULT_OUTPUT_HANDLE),
            )
            for handle, source in self.node_inputs.slot_items()
        ]
    else:
        node_information.input_connections = None
    user_description = self.setting_input.description if hasattr(self.setting_input, "description") else ""
    if user_description:
        node_information.description = user_description
    elif hasattr(self.setting_input, "get_default_description"):
        node_information.description = self.setting_input.get_default_description()
    else:
        node_information.description = ""
    node_information.node_reference = (
        self.setting_input.node_reference if hasattr(self.setting_input, "node_reference") else None
    )
    node_information.is_setup = self.is_setup
    node_information.x_position = self.setting_input.pos_x
    node_information.y_position = self.setting_input.pos_y
    node_information.group_id = getattr(self.setting_input, "group_id", None)
    node_information.type = self.node_type

store_example_data_generator(external_df_fetcher)

Stores a generator function for fetching a sample of the result data.

Parameters:

Name	Type	Description	Default
external_df_fetcher	ExternalDfFetcher | ExternalSampler	The process that generated the sample data.	required

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def store_example_data_generator(self, external_df_fetcher: ExternalDfFetcher | ExternalSampler):
    """Stores a generator function for fetching a sample of the result data.

    Args:
        external_df_fetcher: The process that generated the sample data.
    """
    if external_df_fetcher.status is not None:
        file_ref = external_df_fetcher.status.file_ref
        self.results.example_data_path = file_ref
        self.results.example_data_generator = get_read_top_n(file_path=file_ref, n=100)
    else:
        logger.error("Could not get the sample data, the external process is not ready")

update_node(function, input_columns=None, output_schema=None, drop_columns=None, name=None, setting_input=None, pos_x=0, pos_y=0, schema_callback=None)

Updates the properties of the node.

This is called during initialization and when settings are changed.

Parameters:

Name	Type	Description	Default
function	Callable	The new core data processing function.	required
input_columns	list[str]	The new list of input columns.	None
output_schema	list[FlowfileColumn]	The new schema of added columns.	None
drop_columns	list[str]	The new list of dropped columns.	None
name	str	The new name for the node.	None
setting_input	Any	The new settings object.	None
pos_x	float	The new x-coordinate.	0
pos_y	float	The new y-coordinate.	0
schema_callback	Callable	The new custom schema callback function.	None

Source code in flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py

def update_node(
    self,
    function: Callable,
    input_columns: list[str] = None,
    output_schema: list[FlowfileColumn] = None,
    drop_columns: list[str] = None,
    name: str = None,
    setting_input: Any = None,
    pos_x: float = 0,
    pos_y: float = 0,
    schema_callback: Callable = None,
):
    """Updates the properties of the node.

    This is called during initialization and when settings are changed.

    Args:
        function: The new core data processing function.
        input_columns: The new list of input columns.
        output_schema: The new schema of added columns.
        drop_columns: The new list of dropped columns.
        name: The new name for the node.
        setting_input: The new settings object.
        pos_x: The new x-coordinate.
        pos_y: The new y-coordinate.
        schema_callback: The new custom schema callback function.
    """
    self.user_provided_schema_callback = schema_callback
    self.node_information.y_position = int(pos_y)
    self.node_information.x_position = int(pos_x)
    self.node_information.setting_input = setting_input
    self.name = self.node_type if name is None else name
    self._function = function

    self.node_schema.input_columns = [] if input_columns is None else input_columns
    self.node_schema.output_columns = [] if output_schema is None else output_schema
    self.node_schema.drop_columns = [] if drop_columns is None else drop_columns
    self.node_settings.renew_schema = True
    if hasattr(setting_input, "cache_results"):
        self.node_settings.cache_results = setting_input.cache_results

    self.results.errors = None
    self.add_lead_to_in_depend_source()
    _ = self.hash
    self.node_template = node_store.node_dict.get(self.node_type)
    if self.node_template is None:
        raise Exception(f"Node template {self.node_type} not found")
    self.node_default = node_store.node_defaults.get(self.node_type)
    self.setting_input = setting_input  # wait until the end so that the hash is calculated correctly

The FlowDataEngine

The FlowDataEngine is the primary engine of the library, providing a rich API for data manipulation, I/O, and transformation. Its methods are grouped below by functionality.

flowfile_core.flowfile.flow_data_engine.flow_data_engine.FlowDataEngine dataclass

The core data handling engine for Flowfile.

This class acts as a high-level wrapper around a Polars DataFrame or LazyFrame, providing a unified API for data ingestion, transformation, and output. It manages data state (lazy vs. eager), schema information, and execution logic.

Attributes:

Name	Type	Description
_data_frame	DataFrame | LazyFrame	The underlying Polars DataFrame or LazyFrame.
columns	list[Any]	A list of column names in the current data frame.
name	str	An optional name for the data engine instance.
number_of_records	int	The number of records. Can be -1 for lazy frames.
errors	list	A list of errors encountered during operations.
_schema	list[FlowfileColumn] | None	A cached list of FlowfileColumn objects representing the schema.

Methods:

Name	Description
__call__	Makes the class instance callable, returning itself.
__get_sample__	Internal method to get a sample of the data.
__getitem__	Accesses a specific column or item from the DataFrame.
__init__	Initializes the FlowDataEngine from various data sources.
__len__	Returns the number of records in the table.
__repr__	Returns a string representation of the FlowDataEngine.
add_new_values	Adds a new column with the provided values.
add_record_id	Adds a record ID (row number) column to the DataFrame.
align_to_schema	Aligns the DataFrame to an expected schema.
apply_dynamic_rename	Renames a subset of columns according to a single rule.
apply_flowfile_formula	Applies a formula to create a new column or transform an existing one.
apply_sql_formula	Applies an SQL-style formula using pl.sql_expr.
assert_equal	Asserts that this DataFrame is equal to another.
cache	Caches the current DataFrame to disk and updates the internal reference.
calculate_schema	Calculates and returns the schema.
change_column_types	Changes the data type of one or more columns.
collect	Collects the data and returns it as a Polars DataFrame.
collect_external	Materializes data from a tracked external source.
concat	Concatenates this DataFrame with one or more other DataFrames.
count	Gets the total number of records.
create_from_external_source	Creates a FlowDataEngine from an external data source.
create_from_path	Creates a FlowDataEngine from a local file path.
create_from_path_worker	Creates a FlowDataEngine from a path in a worker process.
create_from_schema	Creates an empty FlowDataEngine from a schema definition.
create_from_sql	Creates a FlowDataEngine by executing a SQL query.
create_random	Creates a FlowDataEngine with randomly generated data.
do_cross_join	Performs a cross join with another DataFrame.
do_filter	Filters rows based on a predicate expression.
do_group_by	Performs a group-by operation on the DataFrame.
do_pivot	Converts the DataFrame from a long to a wide format, aggregating values.
do_select	Performs a complex column selection, renaming, and reordering operation.
do_sort	Sorts the DataFrame by one or more columns.
do_window_functions	Applies window functions (rolling, cumulative, rank, tile) to the data.
drop_columns	Drops specified columns from the DataFrame.
filter_split	Partition rows by predicate into pass and fail streams.
from_cloud_storage_obj	Creates a FlowDataEngine from an object in cloud storage.
generate_enumerator	Generates a FlowDataEngine with a single column containing a sequence of integers.
get_estimated_file_size	Estimates the file size in bytes if the data originated from a local file.
get_number_of_records	Gets the total number of records in the DataFrame.
get_number_of_records_in_process	Get the number of records in the DataFrame in the local process.
get_output_sample	Gets a sample of the data as a list of dictionaries.
get_record_count	Returns a new FlowDataEngine with a single column 'number_of_records'
get_sample	Gets a sample of rows from the DataFrame.
get_schema_column	Retrieves the schema information for a single column by its name.
get_select_inputs	Gets SelectInput specifications for all columns in the current schema.
get_subset	Gets the first n_rows from the DataFrame.
initialize_empty_fl	Initializes an empty LazyFrame.
iter_batches	Iterates over the DataFrame in batches.
join	Performs a standard SQL-style join with another DataFrame.
known_record_count	Returns the exact record count only when it is already known for free.
make_unique	Gets the unique rows from the DataFrame.
output	Writes the DataFrame to a local output file.
random_sample	Takes a uniform random sample of rows without materialising the frame.
random_split	Randomly partition rows into N labeled groups (in-process).
random_split_external	Worker-offloaded variant of :meth:random_split.
reorganize_order	Reorganizes columns into a specified order.
resolve_dynamic_rename_map	Compute the {old_name: new_name} map for a dynamic-rename operation.
save	Saves the DataFrame to a file in a separate thread.
select_columns	Selects a subset of columns from the DataFrame.
set_streamable	Sets whether DataFrame operations should be streamable.
shallow_copy	Cheap de-aliasing wrapper around the same (immutable) Polars frame.
solve_graph	Solves a graph problem represented by 'from' and 'to' columns.
split	Splits a column's text values into multiple rows based on a delimiter.
start_fuzzy_join	Starts a fuzzy join operation in a background process.
to_arrow	Converts the DataFrame to a PyArrow Table.
to_cloud_storage_obj	Writes the DataFrame to an object in cloud storage.
to_database_obj	Writes the DataFrame to a SQL database in-process (local execution path).
to_dict	Converts the DataFrame to a Python dictionary of columns.
to_pylist	Converts the DataFrame to a list of Python dictionaries.
to_raw_data	Converts the DataFrame to a RawData schema object.
unpivot	Converts the DataFrame from a wide to a long format.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@dataclass
class FlowDataEngine:
    """The core data handling engine for Flowfile.

    This class acts as a high-level wrapper around a Polars DataFrame or
    LazyFrame, providing a unified API for data ingestion, transformation,
    and output. It manages data state (lazy vs. eager), schema information,
    and execution logic.

    Attributes:
        _data_frame: The underlying Polars DataFrame or LazyFrame.
        columns: A list of column names in the current data frame.
        name: An optional name for the data engine instance.
        number_of_records: The number of records. Can be -1 for lazy frames.
        errors: A list of errors encountered during operations.
        _schema: A cached list of `FlowfileColumn` objects representing the schema.
    """

    # Core attributes
    _data_frame: pl.DataFrame | pl.LazyFrame
    columns: list[Any]

    # Metadata attributes
    name: str = None
    number_of_records: int = None
    errors: list = None
    _schema: list[FlowfileColumn] | None = None

    # Configuration attributes
    _optimize_memory: bool = False
    _lazy: bool = None
    _streamable: bool = True
    _calculate_schema_stats: bool = False

    # Cache and optimization attributes
    __col_name_idx_map: dict = None
    __data_map: dict = None
    __optimized_columns: list = None
    __sample__: str = None
    __number_of_fields: int = None
    _col_idx: dict[str, int] = None

    # Source tracking
    _org_path: str | None = None
    _external_source: ExternalDataSource | None = None

    # State tracking
    sorted_by: int = None
    is_future: bool = False
    is_collected: bool = True
    ind_schema_calculated: bool = False

    # Callbacks
    _future: Future = None
    _number_of_records_callback: Callable = None
    _data_callback: Callable = None

    def __init__(
        self,
        raw_data: (
            list[dict] | list[Any] | dict[str, Any] | ParquetFile | pl.DataFrame | pl.LazyFrame | input_schema.RawData
        ) = None,
        path_ref: str = None,
        name: str = None,
        optimize_memory: bool = True,
        schema: list[FlowfileColumn] | list[str] | pl.Schema = None,
        number_of_records: int = None,
        calculate_schema_stats: bool = False,
        streamable: bool = True,
        number_of_records_callback: Callable = None,
        data_callback: Callable = None,
    ):
        """Initializes the FlowDataEngine from various data sources.

        Args:
            raw_data: The input data. Can be a list of dicts, a Polars DataFrame/LazyFrame,
                or a `RawData` schema object.
            path_ref: A string path to a Parquet file.
            name: An optional name for the data engine instance.
            optimize_memory: If True, prefers lazy operations to conserve memory.
            schema: An optional schema definition. Can be a list of `FlowfileColumn` objects,
                a list of column names, or a Polars `Schema`.
            number_of_records: The number of records, if known.
            calculate_schema_stats: If True, computes detailed statistics for each column.
            streamable: If True, allows for streaming operations when possible.
            number_of_records_callback: A callback function to retrieve the number of records.
            data_callback: A callback function to retrieve the data.
        """
        self._initialize_attributes(number_of_records_callback, data_callback, streamable)

        if raw_data is not None:
            self._handle_raw_data(raw_data, number_of_records, optimize_memory)
        elif path_ref:
            self._handle_path_ref(path_ref, optimize_memory)
        else:
            self.initialize_empty_fl()
        self._finalize_initialization(name, optimize_memory, schema, calculate_schema_stats)

    def _initialize_attributes(self, number_of_records_callback, data_callback, streamable):
        """(Internal) Sets the initial default attributes for a new instance.

        This helper is called first during initialization to ensure all state-tracking
        and configuration attributes have a clean default value before data is processed.
        """
        self._external_source = None
        self._number_of_records_callback = number_of_records_callback
        self._data_callback = data_callback
        self.ind_schema_calculated = False
        self._streamable = streamable
        self._org_path = None
        self._lazy = False
        self.errors = []
        self._calculate_schema_stats = False
        self.is_collected = True
        self.is_future = False

    def _handle_raw_data(self, raw_data, number_of_records, optimize_memory):
        """(Internal) Dispatches raw data to the appropriate handler based on its type.

        This acts as a router during initialization, inspecting the type of `raw_data`
        and calling the corresponding specialized `_handle_*` method to process it.
        """
        if isinstance(raw_data, input_schema.RawData):
            self._handle_raw_data_format(raw_data)
        elif isinstance(raw_data, pl.DataFrame):
            self._handle_polars_dataframe(raw_data, number_of_records)
        elif isinstance(raw_data, pl.LazyFrame):
            self._handle_polars_lazy_frame(raw_data, number_of_records, optimize_memory)
        elif isinstance(raw_data, list | dict):
            self._handle_python_data(raw_data)

    def _handle_polars_dataframe(self, df: pl.DataFrame, number_of_records: int | None):
        """(Internal) Initializes the engine from an eager Polars DataFrame."""
        self.data_frame = df
        self.number_of_records = number_of_records or df.select(pl.len())[0, 0]

    def _handle_polars_lazy_frame(self, lf: pl.LazyFrame, number_of_records: int | None, optimize_memory: bool):
        """(Internal) Initializes the engine from a Polars LazyFrame."""
        self.data_frame = lf
        self._lazy = True
        if number_of_records is not None:
            self.number_of_records = number_of_records
        elif optimize_memory:
            self.number_of_records = -1
        else:
            self.number_of_records = lf.select(pl.len()).collect()[0, 0]

    def _handle_python_data(self, data: list | dict):
        """(Internal) Dispatches Python collections to the correct handler."""
        if isinstance(data, dict):
            self._handle_dict_input(data)
        else:
            self._handle_list_input(data)

    def _handle_dict_input(self, data: dict):
        """(Internal) Initializes the engine from a Python dictionary."""
        if len(data) == 0:
            self.initialize_empty_fl()
        lengths = [len(v) if isinstance(v, list | tuple) else 1 for v in data.values()]

        if len(set(lengths)) == 1 and lengths[0] > 1:
            self.number_of_records = lengths[0]
            self.data_frame = pl.DataFrame(data)
        else:
            self.number_of_records = 1
            self.data_frame = pl.DataFrame([data])
        self.lazy = True

    def _handle_raw_data_format(self, raw_data: input_schema.RawData):
        """(Internal) Initializes the engine from a `RawData` schema object.

        This method uses the schema provided in the `RawData` object to correctly
        infer data types when creating the Polars DataFrame.

        Args:
            raw_data: An instance of `RawData` containing the data and schema.
        """
        flowfile_schema = list(FlowfileColumn.create_from_minimal_field_info(c) for c in raw_data.columns)
        polars_schema = pl.Schema(
            [
                (flowfile_column.column_name, flowfile_column.get_polars_type().pl_datatype)
                for flowfile_column in flowfile_schema
            ]
        )
        try:
            df = pl.DataFrame(raw_data.data, polars_schema, strict=False)
        except TypeError as e:
            logger.warning(f"Could not parse the data with the schema:\n{e}")
            df = pl.DataFrame(raw_data.data)
        self.number_of_records = len(df)
        self.data_frame = df.lazy()
        self.lazy = True

    def _handle_list_input(self, data: list):
        """(Internal) Initializes the engine from a list of records."""
        number_of_records = len(data)
        if number_of_records > 0:
            processed_data = self._process_list_data(data)
            self.number_of_records = number_of_records
            self.data_frame = pl.DataFrame(processed_data)
            self.lazy = True
        else:
            self.initialize_empty_fl()
            self.number_of_records = 0

    @staticmethod
    def _process_list_data(data: list) -> list[dict]:
        """(Internal) Normalizes list data into a list of dictionaries.

        Ensures that a list of objects or non-dict items is converted into a
        uniform list of dictionaries suitable for Polars DataFrame creation.
        """
        if not (isinstance(data[0], dict) or hasattr(data[0], "__dict__")):
            try:
                return pl.DataFrame(data).to_dicts()
            except TypeError:
                raise Exception("Value must be able to be converted to dictionary") from None
            except Exception as e:
                raise Exception(f"Value must be able to be converted to dictionary: {e}") from e

        if not isinstance(data[0], dict):
            data = [row.__dict__ for row in data]

        return ensure_similarity_dicts(data)

    def to_cloud_storage_obj(self, settings: cloud_storage_schemas.CloudStorageWriteSettingsInternal):
        """Writes the DataFrame to an object in cloud storage.

        This method supports writing to various cloud storage providers like AWS S3,
        Azure Data Lake Storage, and Google Cloud Storage.

        Args:
            settings: A `CloudStorageWriteSettingsInternal` object containing connection
                details, file format, and write options.

        Raises:
            ValueError: If the specified file format is not supported for writing.
            NotImplementedError: If the 'append' write mode is used with an unsupported format.
            Exception: If the write operation to cloud storage fails for any reason.
        """
        connection = settings.connection
        write_settings = settings.write_settings
        logger.info(f"Writing to {connection.storage_type} storage: {write_settings.resource_path}")

        storage_options = CloudStorageReader.get_storage_options(connection)
        credential_provider = CloudStorageReader.get_credential_provider(connection)
        use_pyarrow = CloudStorageReader.use_pyarrow_for_gcs(connection)

        write_to_cloud(
            df=self.data_frame,
            resource_path=write_settings.resource_path,
            storage_options=storage_options,
            file_format=write_settings.file_format,
            write_mode=write_settings.write_mode,
            compression=write_settings.parquet_compression,
            separator=write_settings.csv_delimiter,
            partition_by=write_settings.partition_by,
            credential_provider=credential_provider,
            use_pyarrow=use_pyarrow,
            logger=logger,
        )

    def to_database_obj(self, *, database_type: str, uri: str, table_name: str, if_exists: str) -> None:
        """Writes the DataFrame to a SQL database in-process (local execution path)."""
        logger.info(f"Writing to {database_type} table {table_name}")
        write_dataframe_to_database(
            self.collect(),
            database_type=database_type,
            uri=uri,
            table_name=table_name,
            if_exists=if_exists,
        )

    @classmethod
    def from_cloud_storage_obj(cls, settings: cloud_storage_schemas.CloudStorageReadSettingsInternal) -> FlowDataEngine:
        """Creates a FlowDataEngine from an object in cloud storage.

        This method supports reading from various cloud storage providers like AWS S3,
        Azure Data Lake Storage, and Google Cloud Storage, with support for
        various authentication methods.

        Args:
            settings: A `CloudStorageReadSettingsInternal` object containing connection
                details, file format, and read options.

        Returns:
            A new `FlowDataEngine` instance containing the data from cloud storage.

        Raises:
            ValueError: If the storage type or file format is not supported.
            NotImplementedError: If a requested file format like "delta" or "iceberg"
                is not yet implemented.
            Exception: If reading from cloud storage fails.
        """
        connection = settings.connection
        read_settings = settings.read_settings

        logger.info(f"Reading from {connection.storage_type} storage: {read_settings.resource_path}")
        storage_options = CloudStorageReader.get_storage_options(connection)
        credential_provider = CloudStorageReader.get_credential_provider(connection)
        use_pyarrow = CloudStorageReader.use_pyarrow_for_gcs(connection)
        if read_settings.file_format == "parquet":
            return cls._read_parquet_from_cloud(
                read_settings.resource_path,
                storage_options,
                credential_provider,
                read_settings.scan_mode == "directory",
                use_pyarrow=use_pyarrow,
            )
        elif read_settings.file_format == "delta":
            return cls._read_delta_from_cloud(
                read_settings.resource_path,
                storage_options,
                credential_provider,
                read_settings,
                use_pyarrow=use_pyarrow,
            )
        elif read_settings.file_format == "csv":
            return cls._read_csv_from_cloud(
                read_settings.resource_path,
                storage_options,
                credential_provider,
                read_settings,
                use_pyarrow=use_pyarrow,
            )
        elif read_settings.file_format == "json":
            return cls._read_json_from_cloud(
                read_settings.resource_path,
                storage_options,
                credential_provider,
                read_settings.scan_mode == "directory",
                use_pyarrow=use_pyarrow,
            )
        elif read_settings.file_format == "iceberg":
            return cls._read_iceberg_from_cloud(
                read_settings.resource_path, storage_options, credential_provider, read_settings
            )

        elif read_settings.file_format in ["delta", "iceberg"]:
            # These would require additional libraries
            raise NotImplementedError(f"File format {read_settings.file_format} not yet implemented")
        else:
            raise ValueError(f"Unsupported file format: {read_settings.file_format}")

    @classmethod
    def _read_directory_via_gcsfs(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        file_format: str,
        read_settings: cloud_storage_schemas.CloudStorageReadSettings | None = None,
    ) -> FlowDataEngine:
        """Read multiple files from a GCS directory using gcsfs glob + open."""
        import gcsfs

        fs = gcsfs.GCSFileSystem(**storage_options)
        path = resource_path.replace("gs://", "").rstrip("/")
        files = fs.glob(f"{path}/*.{file_format}")
        if not files:
            raise ValueError(f"No {file_format} files found in {resource_path}")

        dfs = []
        for f in files:
            if file_format == "parquet":
                dfs.append(pl.read_parquet(fs.open(f)))
            elif file_format == "csv":
                dfs.append(
                    pl.read_csv(
                        fs.open(f),
                        has_header=read_settings.csv_has_header if read_settings else True,
                        separator=read_settings.csv_delimiter if read_settings else ",",
                        encoding=read_settings.csv_encoding if read_settings else "utf8",
                    )
                )
            elif file_format == "json":
                dfs.append(pl.read_ndjson(fs.open(f)))

        df = pl.concat(dfs)
        return cls(df.lazy(), number_of_records=len(df), optimize_memory=True, streamable=True)

    @staticmethod
    def _get_schema_from_first_file_in_dir(
        source: str,
        storage_options: dict[str, Any],
        file_format: Literal["csv", "parquet", "json", "delta"],
        use_pyarrow: bool = False,
    ) -> list[FlowfileColumn] | None:
        """Infers the schema by scanning the first file in a cloud directory."""
        from pyarrow import parquet as pq

        try:
            first_file_ref = get_first_file_from_cloud_dir(source, storage_options=storage_options)

            if use_pyarrow:
                import gcsfs

                fs = gcsfs.GCSFileSystem(**storage_options)
                return convert_stats_to_column_info(
                    FlowDataEngine._create_schema_stats_from_pl_schema(
                        pl.from_arrow(pq.read_schema(first_file_ref, filesystem=fs).empty_table()).collect_schema()
                    )
                )
            else:
                read_func = getattr(pl, "scan_" + file_format)

                return convert_stats_to_column_info(
                    FlowDataEngine._create_schema_stats_from_pl_schema(
                        read_func(first_file_ref, storage_options=storage_options).collect_schema()
                    )
                )
        except Exception as e:
            logger.warning(f"Could not read schema from first file in directory, using default schema: {e}")

    @classmethod
    def _read_iceberg_from_cloud(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        credential_provider: Callable | None,
        read_settings: cloud_storage_schemas.CloudStorageReadSettings,
    ) -> FlowDataEngine:
        """Reads Iceberg table(s) from cloud storage."""
        raise NotImplementedError("Failed to read Iceberg table from cloud storage: Not yet implemented")

    @classmethod
    def _read_parquet_from_cloud(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        credential_provider: Callable | None,
        is_directory: bool,
        use_pyarrow: bool = False,
    ) -> FlowDataEngine:
        """Reads Parquet file(s) from cloud storage."""
        try:
            if is_directory:
                resource_path = ensure_path_has_wildcard_pattern(resource_path=resource_path, file_format="parquet")
            scan_kwargs = {"source": resource_path}
            if storage_options:
                scan_kwargs["storage_options"] = storage_options

            if credential_provider:
                scan_kwargs["credential_provider"] = credential_provider
            if storage_options and is_directory:
                schema = cls._get_schema_from_first_file_in_dir(
                    resource_path, storage_options, "parquet", use_pyarrow=use_pyarrow
                )
            else:
                schema = None
            if use_pyarrow:
                lf = get_lazy_frame_from_gcs_pyarrow_dataset(
                    resource_path=resource_path, storage_options=storage_options, is_directory=is_directory
                )
            else:
                lf = pl.scan_parquet(**scan_kwargs)
            return cls(
                lf,
                number_of_records=CLOUD_PLACEHOLDER_RECORD_COUNT,
                optimize_memory=True,
                streamable=True,
                schema=schema,
            )

        except Exception as e:
            logger.error(f"Failed to read Parquet from {resource_path}: {str(e)}")
            raise Exception(f"Failed to read Parquet from cloud storage: {str(e)}") from e

    @classmethod
    def _read_delta_from_cloud(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        credential_provider: Callable | None,
        read_settings: cloud_storage_schemas.CloudStorageReadSettings,
        use_pyarrow: bool = False,
    ) -> FlowDataEngine:
        """Reads a Delta Lake table from cloud storage."""
        try:
            logger.info("Reading Delta file from cloud storage...")
            logger.info(f"read_settings: {read_settings}")
            if use_pyarrow:
                lf = scan_delta_from_gcs(resource_path, storage_options, delta_version=read_settings.delta_version)
            else:
                scan_kwargs = {"source": normalize_delta_path(resource_path)}
                if read_settings.delta_version:
                    scan_kwargs["version"] = read_settings.delta_version
                if storage_options:
                    scan_kwargs["storage_options"] = storage_options
                if credential_provider:
                    scan_kwargs["credential_provider"] = credential_provider
                lf = pl.scan_delta(**scan_kwargs)

            return cls(
                lf,
                number_of_records=CLOUD_PLACEHOLDER_RECORD_COUNT,
                optimize_memory=True,
                streamable=True,
            )
        except Exception as e:
            logger.error(f"Failed to read Delta file from {resource_path}: {str(e)}")
            raise Exception(f"Failed to read Delta file from cloud storage: {str(e)}") from e

    @classmethod
    def _read_csv_from_cloud(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        credential_provider: Callable | None,
        read_settings: cloud_storage_schemas.CloudStorageReadSettings,
        use_pyarrow: bool = False,
    ) -> FlowDataEngine:
        """Reads CSV file(s) from cloud storage."""
        try:
            if use_pyarrow and read_settings.scan_mode == "directory":
                return cls._read_directory_via_gcsfs(resource_path, storage_options, "csv", read_settings)

            scan_kwargs = {
                "source": resource_path,
                "has_header": read_settings.csv_has_header,
                "separator": read_settings.csv_delimiter,
                "encoding": read_settings.csv_encoding,
            }
            if storage_options:
                scan_kwargs["storage_options"] = storage_options
            if credential_provider:
                scan_kwargs["credential_provider"] = credential_provider

            if read_settings.scan_mode == "directory":
                resource_path = ensure_path_has_wildcard_pattern(resource_path=resource_path, file_format="csv")
                scan_kwargs["source"] = resource_path
            if storage_options and read_settings.scan_mode == "directory":
                schema = cls._get_schema_from_first_file_in_dir(resource_path, storage_options, "csv")
            else:
                schema = None

            if use_pyarrow:
                df = pl.read_csv(**scan_kwargs)
                lf = df.lazy()
            else:
                lf = pl.scan_csv(**scan_kwargs)

            return cls(
                lf,
                number_of_records=CLOUD_PLACEHOLDER_RECORD_COUNT,
                optimize_memory=True,
                streamable=True,
                schema=schema,
            )

        except Exception as e:
            logger.error(f"Failed to read CSV from {resource_path}: {str(e)}")
            raise Exception(f"Failed to read CSV from cloud storage: {str(e)}") from e

    @classmethod
    def _read_json_from_cloud(
        cls,
        resource_path: str,
        storage_options: dict[str, Any],
        credential_provider: Callable | None,
        is_directory: bool,
        use_pyarrow: bool = False,
    ) -> FlowDataEngine:
        """Reads JSON file(s) from cloud storage."""
        try:
            if use_pyarrow and is_directory:
                return cls._read_directory_via_gcsfs(resource_path, storage_options, "json")

            if is_directory:
                resource_path = ensure_path_has_wildcard_pattern(resource_path, "json")
            scan_kwargs = {"source": resource_path}

            if storage_options:
                scan_kwargs["storage_options"] = storage_options
            if credential_provider:
                scan_kwargs["credential_provider"] = credential_provider

            if use_pyarrow:
                # For GCS via gcsfs: use gcsfs.open for single-file JSON
                import gcsfs

                fs = gcsfs.GCSFileSystem(**storage_options)
                path = resource_path.replace("gs://", "")
                with fs.open(path) as f:
                    df = pl.read_ndjson(f)
                return cls(df.lazy(), number_of_records=len(df), optimize_memory=True, streamable=True)

            lf = pl.scan_ndjson(**scan_kwargs)

            return cls(
                lf,
                number_of_records=-1,
                optimize_memory=True,
                streamable=True,
            )

        except Exception as e:
            logger.error(f"Failed to read JSON from {resource_path}: {str(e)}")
            raise Exception(f"Failed to read JSON from cloud storage: {str(e)}") from e

    def _handle_path_ref(self, path_ref: str, optimize_memory: bool):
        """Handles file path reference input."""
        try:
            pf = ParquetFile(path_ref)
        except Exception as e:
            logger.error(e)
            raise Exception("Provided ref is not a parquet file") from e

        self.number_of_records = pf.metadata.num_rows
        if optimize_memory:
            self._lazy = True
            self.data_frame = pl.scan_parquet(path_ref)
        else:
            self.data_frame = pl.read_parquet(path_ref)

    def _finalize_initialization(
        self, name: str, optimize_memory: bool, schema: Any | None, calculate_schema_stats: bool
    ):
        """Finalizes initialization by setting remaining attributes."""
        _ = calculate_schema_stats
        self.name = name
        self._optimize_memory = optimize_memory
        if assert_if_flowfile_schema(schema):
            self._schema = schema
            self.columns = [c.column_name for c in self._schema]
        else:
            pl_schema = self.data_frame.collect_schema()
            self._schema = self._handle_schema(schema, pl_schema)
            self.columns = [c.column_name for c in self._schema] if self._schema else pl_schema.names()

    def __getitem__(self, item):
        """Accesses a specific column or item from the DataFrame."""
        return self.data_frame.select([item])

    @property
    def data_frame(self) -> pl.LazyFrame | pl.DataFrame | None:
        """The underlying Polars DataFrame or LazyFrame.

        This property provides access to the Polars object that backs the
        FlowDataEngine. It handles lazy-loading from external sources if necessary.

        Returns:
            The active Polars `DataFrame` or `LazyFrame`.
        """
        if self._data_frame is not None and not self.is_future:
            return self._data_frame
        elif self.is_future:
            return self._data_frame
        elif self._external_source is not None and self.lazy:
            return self._data_frame
        elif self._external_source is not None and not self.lazy:
            if self._external_source.get_pl_df() is None:
                data_frame = list(self._external_source.get_iter())
                if len(data_frame) > 0:
                    self.data_frame = pl.DataFrame(data_frame)
            else:
                self.data_frame = self._external_source.get_pl_df()
            self.calculate_schema()
            return self._data_frame

    @data_frame.setter
    def data_frame(self, df: pl.LazyFrame | pl.DataFrame):
        """Sets the underlying Polars DataFrame or LazyFrame."""
        if self.lazy and isinstance(df, pl.DataFrame):
            raise Exception("Cannot set a non-lazy dataframe to a lazy flowfile")
        self._data_frame = df
        self._schema = None

    @staticmethod
    def _create_schema_stats_from_pl_schema(pl_schema: pl.Schema) -> list[dict]:
        """Converts a Polars Schema into a list of schema statistics dictionaries."""
        return [dict(column_name=k, pl_datatype=v, col_index=i) for i, (k, v) in enumerate(pl_schema.items())]

    def _add_schema_from_schema_stats(self, schema_stats: list[dict]):
        """Populates the schema from a list of schema statistics dictionaries."""
        self._schema = convert_stats_to_column_info(schema_stats)

    @property
    def schema(self) -> list[FlowfileColumn]:
        """The schema of the DataFrame as a list of `FlowfileColumn` objects.

        This property lazily calculates the schema if it hasn't been determined yet.

        Returns:
            A list of `FlowfileColumn` objects describing the schema.
        """
        if self.number_of_fields == 0:
            return []
        if self._schema is None or (self._calculate_schema_stats and not self.ind_schema_calculated):
            if self._calculate_schema_stats and not self.ind_schema_calculated:
                schema_stats = self._calculate_schema()
                self.ind_schema_calculated = True
            else:
                schema_stats = self._create_schema_stats_from_pl_schema(self.data_frame.collect_schema())
            self._add_schema_from_schema_stats(schema_stats)
        return self._schema

    @property
    def number_of_fields(self) -> int:
        """The number of columns (fields) in the DataFrame.

        Returns:
            The integer count of columns.
        """
        if self.__number_of_fields is None:
            self.__number_of_fields = len(self.columns)
        return self.__number_of_fields

    def collect(self, n_records: int = None) -> pl.DataFrame:
        """Collects the data and returns it as a Polars DataFrame.

        This method triggers the execution of the lazy query plan (if applicable)
        and returns the result. It supports streaming to optimize memory usage
        for large datasets.

        Args:
            n_records: The maximum number of records to collect. If None, all
                records are collected.

        Returns:
            A Polars `DataFrame` containing the collected data.
        """
        if n_records is None:
            logger.info(f'Fetching all data for Table object "{id(self)}". Settings: streaming={self._streamable}')
        else:
            logger.info(
                f'Fetching {n_records} record(s) for Table object "{id(self)}". '
                f"Settings: streaming={self._streamable}"
            )

        if not self.lazy:
            return self.data_frame

        try:
            return self._collect_data(n_records)
        except Exception as e:
            self.errors = [e]
            return self._handle_collection_error(n_records)

    def _collect_data(self, n_records: int = None) -> pl.DataFrame:
        """Internal method to handle data collection logic."""
        if n_records is None:
            self.collect_external()
            if self._streamable:
                try:
                    logger.info("Collecting data in streaming mode")
                    return self.data_frame.collect(engine="streaming")
                except PanicException:
                    self._streamable = False

            logger.info("Collecting data in non-streaming mode")
            return self.data_frame.collect()

        if self.external_source is not None:
            return self._collect_from_external_source(n_records)

        if self._streamable:
            return self.data_frame.head(n_records).collect(engine="streaming")
        return self.data_frame.head(n_records).collect()

    def _collect_from_external_source(self, n_records: int) -> pl.DataFrame:
        """Handles collection from an external source."""
        if self.external_source.get_pl_df() is not None:
            all_data = self.external_source.get_pl_df().head(n_records)
            self.data_frame = all_data
        else:
            all_data = self.external_source.get_sample(n_records)
            self.data_frame = pl.LazyFrame(all_data)
        return self.data_frame

    def _handle_collection_error(self, n_records: int) -> pl.DataFrame:
        """Handles errors during collection by attempting partial collection."""
        n_records = 100000000 if n_records is None else n_records
        ok_cols, error_cols = self._identify_valid_columns(n_records)

        if len(ok_cols) > 0:
            return self._create_partial_dataframe(ok_cols, error_cols, n_records)
        return self._create_empty_dataframe(n_records)

    def _identify_valid_columns(self, n_records: int) -> tuple[list[str], list[tuple[str, Any]]]:
        """Identifies which columns can be collected successfully."""
        ok_cols = []
        error_cols = []
        for c in self.columns:
            try:
                _ = self.data_frame.select(c).head(n_records).collect()
                ok_cols.append(c)
            except Exception:
                error_cols.append((c, self.data_frame.schema[c]))
        return ok_cols, error_cols

    def _create_partial_dataframe(
        self, ok_cols: list[str], error_cols: list[tuple[str, Any]], n_records: int
    ) -> pl.DataFrame:
        """Creates a DataFrame with partial data for columns that could be collected."""
        df = self.data_frame.select(ok_cols)
        df = df.with_columns([pl.lit(None).alias(column_name).cast(data_type) for column_name, data_type in error_cols])
        return df.select(self.columns).head(n_records).collect()

    def _create_empty_dataframe(self, n_records: int) -> pl.DataFrame:
        """Creates an empty DataFrame with the correct schema."""
        if self.number_of_records > 0:
            return pl.DataFrame(
                {
                    column_name: pl.Series(
                        name=column_name, values=[None] * min(self.number_of_records, n_records)
                    ).cast(data_type)
                    for column_name, data_type in self.data_frame.schema.items()
                }
            )
        return pl.DataFrame(schema=self.data_frame.schema)

    def do_group_by(
        self, group_by_input: transform_schemas.GroupByInput, calculate_schema_stats: bool = True
    ) -> FlowDataEngine:
        """Performs a group-by operation on the DataFrame.

        Args:
            group_by_input: A `GroupByInput` object defining the grouping columns
                and aggregations.
            calculate_schema_stats: If True, calculates schema statistics for the
                resulting DataFrame.

        Returns:
            A new `FlowDataEngine` instance with the grouped and aggregated data.
        """
        aggregations = [c for c in group_by_input.agg_cols if c.agg != "groupby"]
        group_columns = [c for c in group_by_input.agg_cols if c.agg == "groupby"]

        if len(group_columns) == 0:
            return FlowDataEngine(
                self.data_frame.select(ac.agg_func(ac.old_name).alias(ac.new_name) for ac in aggregations),
                calculate_schema_stats=calculate_schema_stats,
            )

        df = self.data_frame.rename({c.old_name: c.new_name for c in group_columns})
        group_by_columns = [n_c.new_name for n_c in group_columns]

        if len(aggregations) == 0:
            return FlowDataEngine(
                df.select(group_by_columns).unique(),
                calculate_schema_stats=calculate_schema_stats,
            )

        grouped_df = df.group_by(*group_by_columns)
        agg_exprs = [ac.agg_func(ac.old_name).alias(ac.new_name) for ac in aggregations]
        result_df = grouped_df.agg(agg_exprs)

        return FlowDataEngine(
            result_df,
            calculate_schema_stats=calculate_schema_stats,
        )

    def do_window_functions(
        self, settings: transform_schemas.WindowFunctionsInput, calculate_schema_stats: bool = False
    ) -> FlowDataEngine:
        """Applies window functions (rolling, cumulative, rank, tile) to the data.

        When ``settings.order_by`` is provided, rows are sorted first so that
        rolling and tile operations have a deterministic order; the sort is
        preserved in the output. Partitioning (``partition_by``) is applied via
        ``.over(...)`` so operations reset for each group.
        """
        if not settings.window_functions:
            return self

        df = self.data_frame
        if settings.order_by:
            descending = [s.descending for s in settings.order_by]
            df = df.sort([s.column for s in settings.order_by], descending=descending)

        exprs = [
            _build_window_expr(w, settings.partition_by) for w in settings.window_functions
        ]
        df = df.with_columns(exprs)
        return FlowDataEngine(df, calculate_schema_stats=calculate_schema_stats)

    def do_sort(self, sorts: list[transform_schemas.SortByInput]) -> FlowDataEngine:
        """Sorts the DataFrame by one or more columns.

        Args:
            sorts: A list of `SortByInput` objects, each specifying a column
                and sort direction ('asc' or 'desc').

        Returns:
            A new `FlowDataEngine` instance with the sorted data.
        """
        if not sorts:
            return self

        descending = [s.descending for s in sorts]
        df = self.data_frame.sort([sort_by.column for sort_by in sorts], descending=descending)
        return FlowDataEngine(df, number_of_records=self.number_of_records, schema=self.schema)

    def change_column_types(
        self, transforms: list[transform_schemas.SelectInput], calculate_schema: bool = False
    ) -> FlowDataEngine:
        """Changes the data type of one or more columns.

        Args:
            transforms: A list of `SelectInput` objects, where each object specifies
                the column and its new `polars_type`.
            calculate_schema: If True, recalculates the schema after the type change.

        Returns:
            A new `FlowDataEngine` instance with the updated column types.
        """
        dtypes = [dtype.base_type() for dtype in self.data_frame.collect_schema().dtypes()]
        idx_mapping = list(
            (transform.old_name, self.cols_idx.get(transform.old_name), get_polars_type(transform.polars_type))
            for transform in transforms
            if transform.data_type is not None
        )

        actual_transforms = [c for c in idx_mapping if c[2] != dtypes[c[1]]]
        transformations = [
            utils.define_pl_col_transformation(col_name=transform[0], col_type=transform[2])
            for transform in actual_transforms
        ]

        df = self.data_frame.with_columns(transformations)
        return FlowDataEngine(
            df,
            number_of_records=self.number_of_records,
            calculate_schema_stats=calculate_schema,
            streamable=self._streamable,
        )

    def save(self, path: str, data_type: str = "parquet") -> Future:
        """Saves the DataFrame to a file in a separate thread.

        Args:
            path: The file path to save to.
            data_type: The format to save in (e.g., 'parquet', 'csv').

        Returns:
            A `loky.Future` object representing the asynchronous save operation.
        """
        estimated_size = deepcopy(self.get_estimated_file_size() * 4)
        df = deepcopy(self.data_frame)
        return write_threaded(_df=df, path=path, data_type=data_type, estimated_size=estimated_size)

    def to_pylist(self) -> list[dict]:
        """Converts the DataFrame to a list of Python dictionaries.

        Returns:
            A list where each item is a dictionary representing a row.
        """
        if self.lazy:
            return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_dicts()
        return self.data_frame.to_dicts()

    def to_arrow(self) -> PaTable:
        """Converts the DataFrame to a PyArrow Table.

        This method triggers a `.collect()` call if the data is lazy,
        then converts the resulting eager DataFrame into a `pyarrow.Table`.

        Returns:
            A `pyarrow.Table` instance representing the data.
        """
        if self.lazy:
            return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_arrow()
        else:
            return self.data_frame.to_arrow()

    def to_raw_data(self) -> input_schema.RawData:
        """Converts the DataFrame to a `RawData` schema object.

        Returns:
            An `input_schema.RawData` object containing the schema and data.
        """
        columns = [c.get_minimal_field_info() for c in self.schema]
        data = list(self.to_dict().values())
        return input_schema.RawData(columns=columns, data=data)

    def to_dict(self) -> dict[str, list]:
        """Converts the DataFrame to a Python dictionary of columns.

        Each key in the dictionary is a column name, and the corresponding value
        is a list of the data in that column.

        Returns:
            A dictionary mapping column names to lists of their values.
        """
        if self.lazy:
            return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_dict(as_series=False)
        else:
            return self.data_frame.to_dict(as_series=False)

    @classmethod
    def create_from_external_source(cls, external_source: ExternalDataSource) -> FlowDataEngine:
        """Creates a FlowDataEngine from an external data source.

        Args:
            external_source: An object that conforms to the `ExternalDataSource`
                interface.

        Returns:
            A new `FlowDataEngine` instance.
        """
        if external_source.schema is not None:
            ff = cls.create_from_schema(external_source.schema)
        elif external_source.initial_data_getter is not None:
            ff = cls(raw_data=external_source.initial_data_getter())
        else:
            ff = cls()
        ff._external_source = external_source
        return ff

    @classmethod
    def create_from_sql(cls, sql: str, conn: Any) -> FlowDataEngine:
        """Creates a FlowDataEngine by executing a SQL query.

        Args:
            sql: The SQL query string to execute.
            conn: A database connection object or connection URI string.

        Returns:
            A new `FlowDataEngine` instance with the query result.
        """
        return cls(pl.read_sql(sql, conn))

    @classmethod
    def create_from_schema(cls, schema: list[FlowfileColumn]) -> FlowDataEngine:
        """Creates an empty FlowDataEngine from a schema definition.

        Args:
            schema: A list of `FlowfileColumn` objects defining the schema.

        Returns:
            A new, empty `FlowDataEngine` instance with the specified schema.
        """
        pl_schema = []
        for i, flow_file_column in enumerate(schema):
            pl_schema.append((flow_file_column.name, cast_str_to_polars_type(flow_file_column.data_type)))
            schema[i].col_index = i
        df = pl.LazyFrame(schema=pl_schema)
        return cls(df, schema=schema, calculate_schema_stats=False, number_of_records=0)

    @classmethod
    def create_from_path(cls, received_table: input_schema.ReceivedTable) -> FlowDataEngine:
        """Creates a FlowDataEngine from a local file path.

        Supports various file types like CSV, Parquet, and Excel.

        Args:
            received_table: A `ReceivedTableBase` object containing the file path
                and format details.

        Returns:
            A new `FlowDataEngine` instance with data from the file.
        """
        received_table.set_absolute_filepath()
        file_type_handlers = {
            "csv": create_funcs.create_from_path_csv,
            "parquet": create_funcs.create_from_path_parquet,
            "excel": create_funcs.create_from_path_excel,
            "ipc": create_funcs.create_from_path_ipc,
            "ndjson": create_funcs.create_from_path_ndjson,
            "avro": create_funcs.create_from_path_avro,
        }

        handler = file_type_handlers.get(received_table.file_type)
        if not handler:
            raise Exception(f"Cannot create from {received_table.file_type}")

        flow_file = cls(handler(received_table))
        flow_file._org_path = received_table.abs_file_path
        return flow_file

    @classmethod
    def create_random(cls, number_of_records: int = 1000) -> FlowDataEngine:
        """Creates a FlowDataEngine with randomly generated data.

        Useful for testing and examples.

        Args:
            number_of_records: The number of random records to generate.

        Returns:
            A new `FlowDataEngine` instance with fake data.
        """
        return cls(create_fake_data(number_of_records))

    @classmethod
    def generate_enumerator(cls, length: int = 1000, output_name: str = "output_column") -> FlowDataEngine:
        """Generates a FlowDataEngine with a single column containing a sequence of integers.

        Args:
            length: The number of integers to generate in the sequence.
            output_name: The name of the output column.

        Returns:
            A new `FlowDataEngine` instance.
        """
        if length > 10_000_000:
            length = 10_000_000
        return cls(pl.LazyFrame().select((pl.int_range(0, length, dtype=pl.UInt32)).alias(output_name)))

    def _handle_schema(
        self, schema: list[FlowfileColumn] | list[str] | pl.Schema | None, pl_schema: pl.Schema
    ) -> list[FlowfileColumn] | None:
        """Handles schema processing and validation during initialization."""
        if schema is None and pl_schema is not None:
            return convert_stats_to_column_info(self._create_schema_stats_from_pl_schema(pl_schema))
        elif schema is None and pl_schema is None:
            return None
        elif assert_if_flowfile_schema(schema) and pl_schema is None:
            return schema
        elif pl_schema is not None and schema is not None:
            if schema.__len__() != pl_schema.__len__():
                raise Exception(
                    f"Schema does not match the data got {schema.__len__()} columns expected {pl_schema.__len__()}"
                )
            if isinstance(schema, pl.Schema):
                return self._handle_polars_schema(schema, pl_schema)
            elif isinstance(schema, list) and len(schema) == 0:
                return []
            elif isinstance(schema[0], str):
                return self._handle_string_schema(schema, pl_schema)
            return schema

    def _handle_polars_schema(self, schema: pl.Schema, pl_schema: pl.Schema) -> list[FlowfileColumn]:
        """Handles Polars schema conversion."""
        flow_file_columns = [
            FlowfileColumn.create_from_polars_dtype(column_name=col_name, data_type=dtype)
            for col_name, dtype in zip(schema.names(), schema.dtypes(), strict=False)
        ]

        select_arg = [
            pl.col(o).alias(n).cast(schema_dtype)
            for o, n, schema_dtype in zip(pl_schema.names(), schema.names(), schema.dtypes(), strict=False)
        ]

        self.data_frame = self.data_frame.select(select_arg)
        return flow_file_columns

    def _handle_string_schema(self, schema: list[str], pl_schema: pl.Schema) -> list[FlowfileColumn]:
        """Handles string-based schema conversion."""
        flow_file_columns = [
            FlowfileColumn.create_from_polars_dtype(column_name=col_name, data_type=dtype)
            for col_name, dtype in zip(schema, pl_schema.dtypes(), strict=False)
        ]

        self.data_frame = self.data_frame.rename({o: n for o, n in zip(pl_schema.names(), schema, strict=False)})

        return flow_file_columns

    def split(self, split_input: transform_schemas.TextToRowsInput) -> FlowDataEngine:
        """Splits a column's text values into multiple rows based on a delimiter.

        This operation is often referred to as "exploding" the DataFrame, as it
        increases the number of rows.

        Args:
            split_input: A `TextToRowsInput` object specifying the column to split,
                the delimiter, and the output column name.

        Returns:
            A new `FlowDataEngine` instance with the exploded rows.
        """
        output_column_name = (
            split_input.output_column_name if split_input.output_column_name else split_input.column_to_split
        )

        split_value = (
            split_input.split_fixed_value if split_input.split_by_fixed_value else pl.col(split_input.split_by_column)
        )

        df = self.data_frame.with_columns(
            pl.col(split_input.column_to_split).str.split(by=split_value).alias(output_column_name)
        ).explode(output_column_name)

        return FlowDataEngine(df)

    def unpivot(self, unpivot_input: transform_schemas.UnpivotInput) -> FlowDataEngine:
        """Converts the DataFrame from a wide to a long format.

        This is the inverse of a pivot operation, taking columns and transforming
        them into `variable` and `value` rows.

        Args:
            unpivot_input: An `UnpivotInput` object specifying which columns to
                unpivot and which to keep as index columns.

        Returns:
            A new, unpivoted `FlowDataEngine` instance.
        """
        lf = self.data_frame

        if unpivot_input.data_type_selector_expr is not None:
            result = lf.unpivot(on=unpivot_input.data_type_selector_expr(), index=unpivot_input.index_columns)
        elif unpivot_input.value_columns is not None:
            result = lf.unpivot(on=unpivot_input.value_columns, index=unpivot_input.index_columns)
        else:
            result = lf.unpivot()

        return FlowDataEngine(result)

    def do_pivot(self, pivot_input: transform_schemas.PivotInput, node_logger: NodeLogger = None) -> FlowDataEngine:
        """Converts the DataFrame from a long to a wide format, aggregating values.

        Args:
            pivot_input: A `PivotInput` object defining the index, pivot, and value
                columns, along with the aggregation logic.
            node_logger: An optional logger for reporting warnings, e.g., if the
                pivot column has too many unique values.

        Returns:
            A new, pivoted `FlowDataEngine` instance.
        """
        max_unique_vals = 200
        new_cols_unique = fetch_unique_values(
            self.data_frame.select(pivot_input.pivot_column)
            .unique()
            .sort(pivot_input.pivot_column)
            .limit(max_unique_vals)
            .cast(pl.String)
        )
        if len(new_cols_unique) >= max_unique_vals:
            if node_logger:
                node_logger.warning(
                    "Pivot column has too many unique values. Please consider using a different column."
                    f" Max unique values: {max_unique_vals}"
                )

        if len(pivot_input.index_columns) == 0:
            no_index_cols = True
            pivot_input.index_columns = ["__temp__"]
            ff = self.apply_flowfile_formula("1", col_name="__temp__")
        else:
            no_index_cols = False
            ff = self

        index_columns = pivot_input.get_index_columns()
        grouped_ff = ff.do_group_by(pivot_input.get_group_by_input(), False)
        pivot_column = pivot_input.get_pivot_column()

        input_df = grouped_ff.data_frame.with_columns(pivot_column.cast(pl.String).alias(pivot_input.pivot_column))
        number_of_aggregations = len(pivot_input.aggregations)
        # Aggregations where missing combinations should be filled with 0 to match
        # native polars pivot behavior (polars >= 1.32)
        _zero_fill_aggs = {"sum", "count", "len"}
        df = (
            input_df.select(*index_columns, pivot_column, pivot_input.get_values_expr())
            .group_by(*index_columns)
            .agg(
                [
                    (pl.col("vals").filter(pivot_column == new_col_value)).first().alias(new_col_value)
                    for new_col_value in new_cols_unique
                ]
            )
            .select(
                *index_columns,
                *[
                    (
                        pl.col(new_col).struct.field(agg).fill_null(0)
                        if agg in _zero_fill_aggs
                        else pl.col(new_col).struct.field(agg)
                    ).alias(f'{new_col + "_" + agg if number_of_aggregations > 1 else new_col}')
                    for new_col in new_cols_unique
                    for agg in pivot_input.aggregations
                ],
            )
        )

        if no_index_cols:
            df = df.drop("__temp__")
            pivot_input.index_columns = []

        return FlowDataEngine(df, calculate_schema_stats=False)

    def do_filter(self, predicate: str) -> FlowDataEngine:
        """Filters rows based on a predicate expression.

        Args:
            predicate: A string containing a Polars expression that evaluates to
                a boolean value.

        Returns:
            A new `FlowDataEngine` instance containing only the rows that match
            the predicate.
        """
        try:
            f = to_expr(predicate)
        except Exception as e:
            logger.warning(f"Error in filter expression: {e}")
            f = to_expr("False")
        df = self.data_frame.filter(f)
        _ = df.collect_schema()  # Collecting schema to ensure the filter is valid

        return FlowDataEngine(df, streamable=self._streamable)

    def filter_split(self, predicate: str) -> NamedOutputs:
        """Partition rows by ``predicate`` into ``pass`` and ``fail`` streams.

        Rows where the predicate evaluates to null are dropped from both
        streams — matching the behaviour of two manually-wired filter nodes
        with opposing predicates.
        """
        from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

        try:
            f = to_expr(predicate)
        except Exception as e:
            logger.warning(f"Error in filter expression: {e}")
            f = to_expr("False")
        pass_df = self.data_frame.filter(f)
        fail_df = self.data_frame.filter(~f)
        _ = pass_df.collect_schema()
        return NamedOutputs(
            {
                "pass": FlowDataEngine(pass_df, streamable=self._streamable),
                "fail": FlowDataEngine(fail_df, streamable=self._streamable),
            }
        )

    def add_record_id(self, record_id_settings: transform_schemas.RecordIdInput) -> FlowDataEngine:
        """Adds a record ID (row number) column to the DataFrame.

        Can generate a simple sequential ID or a grouped ID that resets for
        each group.

        Args:
            record_id_settings: A `RecordIdInput` object specifying the output
                column name, offset, and optional grouping columns.

        Returns:
            A new `FlowDataEngine` instance with the added record ID column.
        """
        if record_id_settings.group_by and len(record_id_settings.group_by_columns) > 0:
            return self._add_grouped_record_id(record_id_settings)
        return self._add_simple_record_id(record_id_settings)

    def _add_grouped_record_id(self, record_id_settings: transform_schemas.RecordIdInput) -> FlowDataEngine:
        """Adds a record ID column with grouping."""
        select_cols = [pl.col(record_id_settings.output_column_name)] + [pl.col(c) for c in self.columns]

        df = (
            self.data_frame.with_columns(pl.lit(1).alias(record_id_settings.output_column_name))
            .with_columns(
                (
                    pl.cum_count(record_id_settings.output_column_name).over(record_id_settings.group_by_columns)
                    + record_id_settings.offset
                    - 1
                ).alias(record_id_settings.output_column_name)
            )
            .select(select_cols)
        )

        output_schema = [FlowfileColumn.from_input(record_id_settings.output_column_name, "UInt64")]
        output_schema.extend(self.schema)

        return FlowDataEngine(df, schema=output_schema)

    def _add_simple_record_id(self, record_id_settings: transform_schemas.RecordIdInput) -> FlowDataEngine:
        """Adds a simple sequential record ID column."""
        df = self.data_frame.with_row_index(record_id_settings.output_column_name, record_id_settings.offset)

        output_schema = [FlowfileColumn.from_input(record_id_settings.output_column_name, "UInt64")]
        output_schema.extend(self.schema)

        return FlowDataEngine(df, schema=output_schema)

    def get_schema_column(self, col_name: str) -> FlowfileColumn:
        """Retrieves the schema information for a single column by its name.

        Args:
            col_name: The name of the column to retrieve.

        Returns:
            A `FlowfileColumn` object for the specified column, or `None` if not found.
        """
        for s in self.schema:
            if s.name == col_name:
                return s

    def get_estimated_file_size(self) -> int:
        """Estimates the file size in bytes if the data originated from a local file.

        This relies on the original path being tracked during file ingestion.

        Returns:
            The file size in bytes, or 0 if the original path is unknown.
        """
        if self._org_path is not None and not is_url(self._org_path):
            return os.path.getsize(self._org_path)
        return 0

    def __repr__(self) -> str:
        """Returns a string representation of the FlowDataEngine."""
        return f"flow data engine\n{self.data_frame.__repr__()}"

    def __call__(self) -> FlowDataEngine:
        """Makes the class instance callable, returning itself."""
        return self

    def __len__(self) -> int:
        """Returns the number of records in the table."""
        return self.number_of_records if self.number_of_records >= 0 else self.get_number_of_records()

    def cache(self) -> FlowDataEngine:
        """Caches the current DataFrame to disk and updates the internal reference.

        This triggers a background process to write the current LazyFrame's result
        to a temporary file. Subsequent operations on this `FlowDataEngine` instance
        will read from the cached file, which can speed up downstream computations.

        Returns:
            The same `FlowDataEngine` instance, now backed by the cached data.
        """
        edf = ExternalDfFetcher(
            lf=self.data_frame, file_ref=str(id(self)), wait_on_completion=False, flow_id=-1, node_id=-1
        )
        logger.info("Caching data in background")
        result = edf.get_result()
        if isinstance(result, pl.LazyFrame):
            logger.info("Data cached")
            del self._data_frame
            self.data_frame = result
            logger.info("Data loaded from cache")
        return self

    def collect_external(self):
        """Materializes data from a tracked external source.

        If the `FlowDataEngine` was created from an `ExternalDataSource`, this
        method will trigger the data retrieval, update the internal `_data_frame`
        to a `LazyFrame` of the collected data, and reset the schema to be
        re-evaluated.
        """
        if self._external_source is not None:
            logger.info("Collecting external source")
            if self.external_source.get_pl_df() is not None:
                self.data_frame = self.external_source.get_pl_df().lazy()
            else:
                self.data_frame = pl.LazyFrame(list(self.external_source.get_iter()))
            self._schema = None

    def get_output_sample(self, n_rows: int = 10) -> list[dict]:
        """Gets a sample of the data as a list of dictionaries.

        This is typically used to display a preview of the data in a UI.

        Args:
            n_rows: The number of rows to sample.

        Returns:
            A list of dictionaries, where each dictionary represents a row.
        """
        if self.number_of_records > n_rows or self.number_of_records < 0:
            df = self.collect(n_rows)
        else:
            df = self.collect()
        return df.to_dicts()

    def __get_sample__(self, n_rows: int = 100, streamable: bool = True) -> FlowDataEngine:
        """Internal method to get a sample of the data."""
        if not self.lazy:
            df = self.data_frame.lazy()
        else:
            df = self.data_frame

        if streamable:
            try:
                df = df.head(n_rows).collect()
            except Exception as e:
                logger.warning(f"Error in getting sample: {e}")
                df = df.head(n_rows).collect(engine="auto")
        else:
            df = self.collect()
        return FlowDataEngine(df, number_of_records=len(df), schema=self.schema)

    def get_sample(
        self,
        n_rows: int = 100,
        random: bool = False,
        shuffle: bool = False,
        seed: int = None,
        execution_location: ExecutionLocationsLiteral | None = None,
    ) -> FlowDataEngine:
        """Gets a sample of rows from the DataFrame.

        Args:
            n_rows: The number of rows to sample.
            random: If True, performs random sampling. If False, takes the first n_rows.
            shuffle: If True (and `random` is True), shuffles the data before sampling.
            seed: A random seed for reproducibility.
            execution_location: Location which is used to calculate the size of the dataframe
        Returns:
            A new `FlowDataEngine` instance containing the sampled data.
        """
        logging.info(f"Getting sample of {n_rows} rows")
        if random:
            if self.lazy and self.external_source is not None:
                self.collect_external()

            if self.lazy and shuffle:
                sample_df = self.data_frame.collect(engine="streaming" if self._streamable else "auto").sample(
                    n_rows, seed=seed, shuffle=shuffle
                )
            elif shuffle:
                sample_df = self.data_frame.sample(n_rows, seed=seed, shuffle=shuffle)
            else:
                if execution_location is None:
                    execution_location = get_global_execution_location()
                n_rows = min(
                    n_rows, self.get_number_of_records(calculate_in_worker_process=execution_location == "remote")
                )

                every_n_records = ceil(self.number_of_records / n_rows)
                sample_df = self.data_frame.gather_every(every_n_records)
        else:
            if self.external_source:
                self.collect(n_rows)
            sample_df = self.data_frame.head(n_rows)

        return FlowDataEngine(sample_df, schema=self.schema)

    def random_sample(
        self,
        n: int | None = None,
        fraction: float | None = None,
        seed: int | None = None,
    ) -> FlowDataEngine:
        """Takes a uniform random sample of rows without materialising the frame.

        Polars exposes ``sample`` only on eager DataFrames, so the lazy
        equivalent is built from a shuffled row rank: each row draws a distinct
        rank from a random permutation of ``0..len``, and keeping the ranks
        below a threshold keeps a uniform subset. Nothing is collected and the
        row count is never queried, so the result stays a plan that ships to
        the worker like any other lazy transform — unlike :meth:`random_split`,
        which has to materialise because its outputs must share one permutation.

        Sampling more rows than the frame holds yields the whole frame, and the
        original row order is preserved.

        Args:
            n: Number of rows to keep. Mutually exclusive with `fraction`.
            fraction: Share of rows to keep, between 0 and 1. Mutually exclusive with `n`.
            seed: Seed for a reproducible sample; None draws a fresh permutation
                on every execution.

        Returns:
            A new `FlowDataEngine` instance containing the sampled rows.
        """
        if (n is None) == (fraction is None):
            raise ValueError("Provide exactly one of n or fraction")
        df = self.data_frame if self.lazy else self.data_frame.lazy()
        threshold = (pl.len() * fraction).round().cast(pl.Int64) if fraction is not None else max(0, n)
        sampled = df.filter(pl.int_range(0, pl.len()).shuffle(seed=seed) < threshold)
        return FlowDataEngine(sampled, schema=self.schema, streamable=self._streamable)

    def random_split(
        self,
        splits: list[tuple[str, float]],
        seed: int | None = None,
    ) -> NamedOutputs:
        """Randomly partition rows into N labeled groups (in-process).

        Used by ``add_random_split`` when ``execution_location == "local"``
        (WASM / no-worker). For remote mode the worker-offloaded variant
        :meth:`random_split_external` is used instead.

        The shuffled frame is materialized once so that each output shares the
        same shuffle — otherwise every handle's ``.collect()`` would re-run the
        full shuffle+sort independently (O(N·n log n) instead of O(n log n)).

        Args:
            splits: Ordered (name, percentage) pairs; percentages must sum to
                100 (validated upstream in ``NodeRandomSplit``).
            seed: Random seed; if None, one is generated per call.

        Returns:
            ``NamedOutputs`` mapping each split name to a fresh ``FlowDataEngine``.
        """
        from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

        if seed is None:
            seed = random.randint(0, 2**31 - 1)
        shuffled = (
            self.data_frame.with_columns(pl.int_range(0, pl.len()).shuffle(seed=seed).alias("__split_rank__"))
            .sort("__split_rank__")
            .drop("__split_rank__")
            .collect()
        )
        total = shuffled.height
        out: dict[str, FlowDataEngine] = {}
        offset = 0
        for i, (name, percentage) in enumerate(splits):
            length = total - offset if i == len(splits) - 1 else int(round(total * percentage / 100.0))
            out[name] = FlowDataEngine(shuffled.slice(offset, max(0, length)).lazy())
            offset += length
        return NamedOutputs(out)

    def random_split_external(
        self,
        splits: list[tuple[str, float]],
        seed: int | None = None,
        flow_id: int = -1,
        node_id: int | str = -1,
    ) -> NamedOutputs:
        """Worker-offloaded variant of :meth:`random_split`.

        The shuffled frame is materialised once on ``flowfile_worker`` (never
        in this process). Each returned split is a lazy ``slice`` over the
        cached parquet, so downstream ``.collect()`` on a handle reads only
        that split's rows from disk.

        Used by ``add_random_split`` when ``execution_location != "local"``;
        the in-process path is :meth:`random_split`.
        """
        import uuid

        from flowfile_core.flowfile.flow_data_engine.subprocess_operations import (
            ExternalDfFetcher,
        )
        from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

        if seed is None:
            seed = random.randint(0, 2**31 - 1)

        shuffled_lazy = (
            self.data_frame.with_columns(pl.int_range(0, pl.len()).shuffle(seed=seed).alias("__split_rank__"))
            .sort("__split_rank__")
            .drop("__split_rank__")
        )

        # Stable, unique file_ref — avoids id()-reuse collisions with cache().
        file_ref = f"random_split_{flow_id}_{node_id}_{seed}_{uuid.uuid4().hex}"
        edf = ExternalDfFetcher(
            lf=shuffled_lazy,
            file_ref=file_ref,
            wait_on_completion=True,
            flow_id=flow_id,
            node_id=node_id,
        )
        cached_lf = edf.get_result()
        if not isinstance(cached_lf, pl.LazyFrame):
            raise RuntimeError(f"random_split_external: worker did not return a LazyFrame (got {type(cached_lf)!r})")

        # Cheap — reads parquet footer metadata, not row data.
        total = cached_lf.select(pl.len()).collect()[0, 0]

        out: dict[str, FlowDataEngine] = {}
        offset = 0
        for i, (name, percentage) in enumerate(splits):
            length = total - offset if i == len(splits) - 1 else int(round(total * percentage / 100.0))
            length = max(0, length)
            out[name] = FlowDataEngine(
                cached_lf.slice(offset, length),
                number_of_records=length,
                schema=self.schema,
            )
            offset += length
        return NamedOutputs(out)

    def get_subset(self, n_rows: int = 100) -> FlowDataEngine:
        """Gets the first `n_rows` from the DataFrame.

        Args:
            n_rows: The number of rows to include in the subset.

        Returns:
            A new `FlowDataEngine` instance containing the subset of data.
        """
        if not self.lazy:
            return FlowDataEngine(self.data_frame.head(n_rows), calculate_schema_stats=True)
        else:
            return FlowDataEngine(self.data_frame.head(n_rows), calculate_schema_stats=True)

    def iter_batches(
        self, batch_size: int = 1000, columns: list | tuple | str = None
    ) -> Generator[FlowDataEngine, None, None]:
        """Iterates over the DataFrame in batches.

        Args:
            batch_size: The size of each batch.
            columns: A list of column names to include in the batches. If None,
                all columns are included.

        Yields:
            A `FlowDataEngine` instance for each batch.
        """
        if columns:
            self.data_frame = self.data_frame.select(columns)
        self.lazy = False
        batches = self.data_frame.iter_slices(batch_size)
        for batch in batches:
            yield FlowDataEngine(batch)

    def start_fuzzy_join(
        self,
        fuzzy_match_input: transform_schemas.FuzzyMatchInput,
        other: FlowDataEngine,
        file_ref: str,
        flow_id: int = -1,
        node_id: int | str = -1,
    ) -> ExternalFuzzyMatchFetcher:
        """Starts a fuzzy join operation in a background process.

        This method prepares the data and initiates the fuzzy matching in a
        separate process, returning a tracker object immediately.

        Args:
            fuzzy_match_input: A `FuzzyMatchInput` object with the matching parameters.
            other: The right `FlowDataEngine` to join with.
            file_ref: A reference string for temporary files.
            flow_id: The flow ID for tracking.
            node_id: The node ID for tracking.

        Returns:
            An `ExternalFuzzyMatchFetcher` object that can be used to track the
            progress and retrieve the result of the fuzzy join.
        """
        fuzzy_match_input_manager = transform_schemas.FuzzyMatchInputManager(fuzzy_match_input)
        left_df, right_df = prepare_for_fuzzy_match(
            left=self, right=other, fuzzy_match_input_manager=fuzzy_match_input_manager
        )

        return ExternalFuzzyMatchFetcher(
            left_df,
            right_df,
            fuzzy_maps=fuzzy_match_input_manager.fuzzy_maps,
            file_ref=file_ref + "_fm",
            wait_on_completion=False,
            flow_id=flow_id,
            node_id=node_id,
        )

    def fuzzy_join_external(
        self,
        fuzzy_match_input: transform_schemas.FuzzyMatchInput,
        other: FlowDataEngine,
        file_ref: str = None,
        flow_id: int = -1,
        node_id: int = -1,
    ):
        if file_ref is None:
            file_ref = str(id(self)) + "_" + str(id(other))
        fuzzy_match_input_manager = transform_schemas.FuzzyMatchInputManager(fuzzy_match_input)

        left_df, right_df = prepare_for_fuzzy_match(
            left=self, right=other, fuzzy_match_input_manager=fuzzy_match_input_manager
        )
        external_tracker = ExternalFuzzyMatchFetcher(
            left_df,
            right_df,
            fuzzy_maps=fuzzy_match_input_manager.fuzzy_maps,
            file_ref=file_ref + "_fm",
            wait_on_completion=False,
            flow_id=flow_id,
            node_id=node_id,
        )
        return FlowDataEngine(external_tracker.get_result())

    def fuzzy_join(
        self,
        fuzzy_match_input: transform_schemas.FuzzyMatchInput,
        other: FlowDataEngine,
        node_logger: NodeLogger = None,
    ) -> FlowDataEngine:
        fuzzy_match_input_manager = transform_schemas.FuzzyMatchInputManager(fuzzy_match_input)
        left_df, right_df = prepare_for_fuzzy_match(
            left=self, right=other, fuzzy_match_input_manager=fuzzy_match_input_manager
        )
        fuzzy_mappings = [FuzzyMapping(**fm.__dict__) for fm in fuzzy_match_input_manager.fuzzy_maps]
        return FlowDataEngine(
            fuzzy_match_dfs(
                left_df, right_df, fuzzy_maps=fuzzy_mappings, logger=node_logger.logger if node_logger else logger
            ).lazy()
        )

    def do_cross_join(
        self,
        cross_join_input: transform_schemas.CrossJoinInput,
        auto_generate_selection: bool,
        verify_integrity: bool,
        other: FlowDataEngine,
    ) -> FlowDataEngine:
        """Performs a cross join with another DataFrame.

        A cross join produces the Cartesian product of the two DataFrames.

        Args:
            cross_join_input: A `CrossJoinInput` object specifying column selections.
            auto_generate_selection: If True, automatically renames columns to avoid conflicts.
            verify_integrity: If True, checks if the resulting join would be too large.
            other: The right `FlowDataEngine` to join with.

        Returns:
            A new `FlowDataEngine` with the result of the cross join.

        Raises:
            Exception: If `verify_integrity` is True and the join would result in
                an excessively large number of records.
        """
        self.lazy = True
        other.lazy = True
        cross_join_input_manager = transform_schemas.CrossJoinInputManager(cross_join_input)
        _ensure_all_columns_have_select(
            left_cols=self.columns, right_cols=other.columns, manager=cross_join_input_manager
        )
        verify_join_select_integrity(
            cross_join_input_manager.input, left_columns=self.columns, right_columns=other.columns
        )
        right_select = [
            v.old_name
            for v in cross_join_input_manager.right_select.renames
            if (v.keep or v.join_key) and v.is_available
        ]
        left_select = [
            v.old_name
            for v in cross_join_input_manager.left_select.renames
            if (v.keep or v.join_key) and v.is_available
        ]
        cross_join_input_manager.auto_rename(rename_mode="suffix")
        left = self.data_frame.select(left_select).rename(cross_join_input_manager.left_select.rename_table)
        right = other.data_frame.select(right_select).rename(cross_join_input_manager.right_select.rename_table)

        joined_df = left.join(right, how="cross")

        cols_to_delete_after = [
            col.new_name
            for col in cross_join_input_manager.left_select.renames + cross_join_input_manager.right_select.renames
            if col.join_key and not col.keep and col.is_available
        ]

        fl = FlowDataEngine(joined_df.drop(cols_to_delete_after), calculate_schema_stats=False, streamable=False)
        return fl

    def join(
        self,
        join_input: transform_schemas.JoinInput,
        auto_generate_selection: bool,
        verify_integrity: bool,
        other: FlowDataEngine,
    ) -> FlowDataEngine:
        """Performs a standard SQL-style join with another DataFrame."""
        join_manager = transform_schemas.JoinInputManager(join_input)
        _ensure_all_columns_have_select(left_cols=self.columns, right_cols=other.columns, manager=join_manager)
        join_manager.set_join_keys()
        for jk in join_manager.join_mapping:
            if jk.left_col not in {c.old_name for c in join_manager.left_select.renames}:
                join_manager.left_select.append(transform_schemas.SelectInput(jk.left_col, keep=False))
            if jk.right_col not in {c.old_name for c in join_manager.right_select.renames}:
                join_manager.right_select.append(transform_schemas.SelectInput(jk.right_col, keep=False))
        verify_join_select_integrity(join_manager.input, left_columns=self.columns, right_columns=other.columns)
        join_map_problems = get_join_map_problems(
            join_manager.input, left_columns=self.schema, right_columns=other.schema
        )
        if join_map_problems:
            raise Exception("Join is not valid: " + "; ".join(join_map_problems))

        if join_manager.how in ("semi", "anti"):
            # Semi/anti joins push the full left input downstream unchanged (all columns,
            # original order, no rename or drop); the right frame only supplies the join
            # keys for matching. Stale entries in left_select are therefore irrelevant here.
            left_on = [jm.left_col for jm in join_manager.join_mapping]
            right_on = [jm.right_col for jm in join_manager.join_mapping]
            right = other.data_frame.select(list(dict.fromkeys(right_on)))
            joined_df = self.data_frame.join(other=right, left_on=left_on, right_on=right_on, how=join_manager.how)
            # -1 = unknown (not 0): a 0 here reads as a real "empty result" count.
            return FlowDataEngine(joined_df, calculate_schema_stats=False, number_of_records=-1, streamable=False)

        if auto_generate_selection:
            join_manager.auto_rename()

        left = self.data_frame.select(join_manager.left_manager.get_select_cols()).rename(
            join_manager.left_manager.get_rename_table()
        )
        right = other.data_frame.select(join_manager.right_manager.get_select_cols()).rename(
            join_manager.right_manager.get_rename_table()
        )

        left, right, reverse_join_key_mapping = _handle_duplication_join_keys(left, right, join_manager)
        left, right = rename_df_table_for_join(left, right, join_manager.get_join_key_renames())
        if join_manager.how == "right":
            joined_df = right.join(
                other=left,
                left_on=join_manager.right_join_keys,
                right_on=join_manager.left_join_keys,
                how="left",
                suffix="",
            ).rename(reverse_join_key_mapping)
        else:
            joined_df = left.join(
                other=right,
                left_on=join_manager.left_join_keys,
                right_on=join_manager.right_join_keys,
                how=join_manager.how,
                suffix="",
            ).rename(reverse_join_key_mapping)

        left_cols_to_delete_after = [
            get_col_name_to_delete(col, "left")
            for col in join_manager.input.left_select.renames
            if not col.keep and col.is_available and col.join_key
        ]

        right_cols_to_delete_after = [
            get_col_name_to_delete(col, "right")
            for col in join_manager.input.right_select.renames
            if not col.keep
            and col.is_available
            and col.join_key
            and join_manager.how in ("left", "right", "inner", "cross", "outer")
        ]

        if len(right_cols_to_delete_after + left_cols_to_delete_after) > 0:
            joined_df = joined_df.drop(left_cols_to_delete_after + right_cols_to_delete_after)

        undo_join_key_remapping = get_undo_rename_mapping_join(join_manager)
        joined_df = joined_df.rename(undo_join_key_remapping)

        # -1 = unknown (not 0): a 0 here reads as a real "empty result" count.
        return FlowDataEngine(joined_df, calculate_schema_stats=False, number_of_records=-1, streamable=False)

    def solve_graph(self, graph_solver_input: transform_schemas.GraphSolverInput) -> FlowDataEngine:
        """Solves a graph problem represented by 'from' and 'to' columns.

        This is used for operations like finding connected components in a graph.

        Args:
            graph_solver_input: A `GraphSolverInput` object defining the source,
                destination, and output column names.

        Returns:
            A new `FlowDataEngine` instance with the solved graph data.
        """
        lf = self.data_frame.with_columns(
            graph_solver(graph_solver_input.col_from, graph_solver_input.col_to).alias(
                graph_solver_input.output_column_name
            )
        )
        return FlowDataEngine(lf)

    def add_new_values(self, values: Iterable, col_name: str = None) -> FlowDataEngine:
        """Adds a new column with the provided values.

        Args:
            values: An iterable (e.g., list, tuple) of values to add as a new column.
            col_name: The name for the new column. Defaults to 'new_values'.

        Returns:
            A new `FlowDataEngine` instance with the added column.
        """
        if col_name is None:
            col_name = "new_values"
        return FlowDataEngine(self.data_frame.with_columns(pl.Series(values).alias(col_name)))

    def get_record_count(self) -> FlowDataEngine:
        """Returns a new FlowDataEngine with a single column 'number_of_records'
        containing the total number of records.

        Returns:
            A new `FlowDataEngine` instance.
        """
        return FlowDataEngine(self.data_frame.select(pl.len().alias("number_of_records")))

    def assert_equal(self, other: FlowDataEngine, ordered: bool = True, strict_schema: bool = False):
        """Asserts that this DataFrame is equal to another.

        Useful for testing.

        Args:
            other: The other `FlowDataEngine` to compare with.
            ordered: If True, the row order must be identical.
            strict_schema: If True, the data types of the schemas must be identical.

        Raises:
            Exception: If the DataFrames are not equal based on the specified criteria.
        """
        org_laziness = self.lazy, other.lazy
        self.lazy = False
        other.lazy = False
        self.number_of_records = -1
        other.number_of_records = -1
        other = other.select_columns(self.columns)

        if self.get_number_of_records_in_process() != other.get_number_of_records_in_process():
            raise Exception("Number of records is not equal")

        if self.columns != other.columns:
            raise Exception("Schema is not equal")

        if strict_schema:
            assert self.data_frame.schema == other.data_frame.schema, "Data types do not match"

        if ordered:
            self_lf = self.data_frame.sort(by=self.columns)
            other_lf = other.data_frame.sort(by=other.columns)
        else:
            self_lf = self.data_frame
            other_lf = other.data_frame

        self.lazy, other.lazy = org_laziness
        assert self_lf.equals(other_lf), "Data is not equal"

    def initialize_empty_fl(self):
        """Initializes an empty LazyFrame."""
        self.data_frame = pl.LazyFrame()
        self.number_of_records = 0
        self._lazy = True

    def _calculate_number_of_records_in_worker(self) -> int:
        """Calculates the number of records in a worker process."""
        number_of_records = ExternalDfFetcher(
            lf=self.data_frame,
            operation_type="calculate_number_of_records",
            flow_id=-1,
            node_id=-1,
            wait_on_completion=True,
        ).result
        return number_of_records

    def get_number_of_records_in_process(self, force_calculate: bool = False):
        """
        Get the number of records in the DataFrame in the local process.

        args:
            force_calculate: If True, forces recalculation even if a value is cached.

        Returns:
            The total number of records.
        """
        return self.get_number_of_records(force_calculate=force_calculate)

    def known_record_count(self) -> int | None:
        """Returns the exact record count only when it is already known for free.

        Sources, in order: a previously stored ``number_of_records`` (e.g. the
        count the worker sent along with a remote run result), or the height of
        an eager frame. Returns None otherwise — deliberately never falls back
        to ``get_number_of_records()``, which on a lazy frame collects the whole
        plan to count it. Stored placeholders are not counts: the cloud readers
        stamp ``CLOUD_PLACEHOLDER_RECORD_COUNT`` and external-source engines
        carry a schema-time 0, so both report unknown here.
        """
        if self._external_source is not None:
            return None
        if self.number_of_records is not None and self.number_of_records >= 0:
            if self.number_of_records == CLOUD_PLACEHOLDER_RECORD_COUNT:
                return None
            return self.number_of_records
        if not self.lazy:
            return self.data_frame.height
        return None

    def get_number_of_records(
        self, warn: bool = False, force_calculate: bool = False, calculate_in_worker_process: bool = False
    ) -> int:
        """Gets the total number of records in the DataFrame.

        For lazy frames, this may trigger a full data scan, which can be expensive.

        Args:
            warn: If True, logs a warning if a potentially expensive calculation is triggered.
            force_calculate: If True, forces recalculation even if a value is cached.
            calculate_in_worker_process: If True, offloads the calculation to a worker process.

        Returns:
            The total number of records.

        Raises:
            ValueError: If the number of records could not be determined.
        """
        if self.is_future and not self.is_collected:
            return -1
        if self.number_of_records is None or self.number_of_records < 0 or force_calculate:
            if self._number_of_records_callback is not None:
                self._number_of_records_callback(self)

            if self.lazy:
                if calculate_in_worker_process:
                    try:
                        self.number_of_records = self._calculate_number_of_records_in_worker()
                        return self.number_of_records
                    except Exception as e:
                        logger.error(f"Error: {e}")
                if warn:
                    logger.warning("Calculating the number of records this can be expensive on a lazy frame")
                try:
                    self.number_of_records = self.data_frame.select(pl.len()).collect(
                        engine="streaming" if self._streamable else "auto"
                    )[0, 0]
                except Exception:
                    raise ValueError("Could not get number of records") from None
            else:
                self.number_of_records = self.data_frame.__len__()
        return self.number_of_records

    @property
    def has_errors(self) -> bool:
        """Checks if there are any errors."""
        return len(self.errors) > 0

    @property
    def lazy(self) -> bool:
        """Indicates if the DataFrame is in lazy mode."""
        return self._lazy

    @lazy.setter
    def lazy(self, exec_lazy: bool = False):
        """Sets the laziness of the DataFrame.

        Args:
            exec_lazy: If True, converts the DataFrame to a LazyFrame. If False,
                collects the data and converts it to an eager DataFrame.
        """
        if exec_lazy != self._lazy:
            if exec_lazy:
                self.data_frame = self.data_frame.lazy()
            else:
                self._lazy = exec_lazy
                if self.external_source is not None:
                    df = self.collect()
                    self.data_frame = df
                else:
                    self.data_frame = self.data_frame.collect(engine="streaming" if self._streamable else "auto")
            self._lazy = exec_lazy

    @property
    def external_source(self) -> ExternalDataSource:
        """The external data source, if any."""
        return self._external_source

    @property
    def cols_idx(self) -> dict[str, int]:
        """A dictionary mapping column names to their integer index."""
        if self._col_idx is None:
            self._col_idx = {c: i for i, c in enumerate(self.columns)}
        return self._col_idx

    @property
    def __name__(self) -> str:
        """The name of the table."""
        return self.name

    def get_select_inputs(self) -> transform_schemas.SelectInputs:
        """Gets `SelectInput` specifications for all columns in the current schema.

        Returns:
            A `SelectInputs` object that can be used to configure selection or
            transformation operations.
        """
        return transform_schemas.SelectInputs(
            [transform_schemas.SelectInput(old_name=c.name, data_type=c.data_type) for c in self.schema]
        )

    def select_columns(self, list_select: list[str] | tuple[str] | str) -> FlowDataEngine:
        """Selects a subset of columns from the DataFrame.

        Args:
            list_select: A list, tuple, or single string of column names to select.

        Returns:
            A new `FlowDataEngine` instance containing only the selected columns.
        """
        if isinstance(list_select, str):
            list_select = [list_select]

        idx_to_keep = [self.cols_idx.get(c) for c in list_select]
        selects = [ls for ls, id_to_keep in zip(list_select, idx_to_keep, strict=False) if id_to_keep is not None]
        new_schema = [self.schema[i] for i in idx_to_keep if i is not None]

        return FlowDataEngine(
            self.data_frame.select(selects),
            number_of_records=self.number_of_records,
            schema=new_schema,
            streamable=self._streamable,
        )

    def drop_columns(self, columns: list[str]) -> FlowDataEngine:
        """Drops specified columns from the DataFrame.

        Args:
            columns: A list of column names to drop.

        Returns:
            A new `FlowDataEngine` instance without the dropped columns.
        """
        cols_for_select = tuple(set(self.columns) - set(columns))
        idx_to_keep = [self.cols_idx.get(c) for c in cols_for_select]
        new_schema = [self.schema[i] for i in idx_to_keep]

        return FlowDataEngine(
            self.data_frame.select(cols_for_select), number_of_records=self.number_of_records, schema=new_schema
        )

    def align_to_schema(self, expected_schema: list[FlowfileColumn]) -> FlowDataEngine:
        """Aligns the DataFrame to an expected schema.

        Adds any missing columns as typed nulls and reorders all columns to
        match the order defined in *expected_schema*.  Extra columns present
        in the data but absent from the expected schema are appended at the
        end so no data is silently dropped.

        Args:
            expected_schema: The desired column list, in order.

        Returns:
            A new ``FlowDataEngine`` whose columns match *expected_schema*.
        """
        actual_names = set(self.columns)
        expected_names = [c.column_name for c in expected_schema]

        missing_exprs = []
        for col in expected_schema:
            if col.column_name not in actual_names:
                pl_type = cast_str_to_polars_type(col.data_type)
                missing_exprs.append(pl.lit(None).cast(pl_type).alias(col.column_name))

        df = self.data_frame
        if missing_exprs:
            df = df.with_columns(missing_exprs)

        expected_set = set(expected_names)
        extra_names = [c for c in self.columns if c not in expected_set]
        final_order = expected_names + extra_names

        return FlowDataEngine(
            df.select(final_order),
            number_of_records=self.number_of_records,
            streamable=self._streamable,
        )

    def reorganize_order(self, column_order: list[str]) -> FlowDataEngine:
        """Reorganizes columns into a specified order.

        Args:
            column_order: A list of column names in the desired order.

        Returns:
            A new `FlowDataEngine` instance with the columns reordered.
        """
        df = self.data_frame.select(column_order)
        schema = sorted(self.schema, key=lambda x: column_order.index(x.column_name))
        return FlowDataEngine(df, schema=schema, number_of_records=self.number_of_records)

    def apply_flowfile_formula(self, func: str, col_name: str, output_data_type: pl.DataType = None) -> FlowDataEngine:
        """Applies a formula to create a new column or transform an existing one.

        Args:
            func: A string containing a Polars expression formula.
            col_name: The name of the new or transformed column.
            output_data_type: The desired Polars data type for the output column.

        Returns:
            A new `FlowDataEngine` instance with the applied formula.
        """
        parsed_func = to_expr(func)
        if output_data_type is not None:
            df2 = self.data_frame.with_columns(parsed_func.cast(output_data_type).alias(col_name))
        else:
            df2 = self.data_frame.with_columns(parsed_func.alias(col_name))

        return FlowDataEngine(df2, number_of_records=self.number_of_records)

    @staticmethod
    def _select_rename_targets(
        columns: list[tuple[str, str]],
        settings: transform_schemas.DynamicRenameInput,
    ) -> list[str]:
        """Return the ordered list of column names the rename rule applies to.

        Applies the `selection_mode` filter (`"all"`, `"list"`, or `"data_type"`) to
        the incoming schema, preserving the original column order. Unknown column
        names in `settings.selected_columns` are silently dropped so stale UI state
        does not break execution. An unknown or unset `selection_mode` returns `[]`.

        Args:
            columns: Incoming schema as `(column_name, data_type_group)` tuples, in order.
            settings: The dynamic rename configuration.

        Returns:
            The column names the rename rule should be applied to, in schema order.
        """
        mode = settings.selection_mode
        if mode == "all":
            return [name for name, _ in columns]
        if mode == "list":
            available = {name for name, _ in columns}
            return [c for c in settings.selected_columns if c in available]
        if mode == "data_type":
            wanted = settings.selected_data_type
            if wanted is None:
                return []
            return [name for name, group in columns if group == wanted]
        return []

    @staticmethod
    def _compute_renamed_names(
        targets: list[str],
        settings: transform_schemas.DynamicRenameInput,
        first_row_values: dict[str, Any] | None = None,
    ) -> list[str]:
        """Return new names aligned 1:1 with `targets` for the configured rename mode.

        For `"prefix"` / `"suffix"` modes the transformation is applied string-wise.
        For `"formula"` mode the user's flowfile-formula expression is evaluated once
        against a one-column DataFrame (`column_name`) whose rows are `targets`, so
        the formula can reference the original name via the `column_name` field. A
        scalar/literal result (length 1) is broadcast to match `targets`; any other
        cardinality mismatch is an error. An empty/whitespace formula is treated as
        a no-op (returns `targets` unchanged). Non-string formula results are
        coerced to `str`.

        For `"first_row"` mode the new names come from `first_row_values` (a dict
        keyed by original column name). When called without `first_row_values`
        (schema-only preview) the function returns `targets` unchanged.

        Args:
            targets: Column names the rename rule applies to.
            settings: The dynamic rename configuration.
            first_row_values: First-row values keyed by original column name, used
                only in `"first_row"` mode.

        Returns:
            New names aligned 1:1 with `targets`.

        Raises:
            ValueError: If a formula yields a null, changes cardinality in a way
                that cannot be broadcast (e.g. an aggregation collapsing N rows to
                a different N), or — in `"first_row"` mode — a first-row value is
                null or empty.
        """
        if not targets:
            return []
        mode = settings.rename_mode
        if mode == "prefix":
            return [f"{settings.prefix}{n}" for n in targets]
        if mode == "suffix":
            return [f"{n}{settings.suffix}" for n in targets]
        if mode == "first_row":
            if first_row_values is None:
                return list(targets)
            new = [first_row_values.get(name) for name in targets]
            for original, v in zip(targets, new, strict=True):
                if v is None or (isinstance(v, str) and v.strip() == ""):
                    raise ValueError(f"Dynamic rename (first_row) got a null/empty value for column '{original}'.")
            return [str(v) for v in new]
        if mode == "formula":
            if not settings.formula.strip():
                return list(targets)
            expr = to_expr(settings.formula)
            tmp = pl.DataFrame({"column_name": targets})
            results = tmp.select(expr.alias("__ff_rename__"))["__ff_rename__"].to_list()
            # A scalar/literal formula (e.g. `"x"`) returns a single row; broadcast
            # it to match `targets`. Any other length mismatch means the formula
            # changed cardinality (e.g. an aggregation), which is not a valid rename.
            if len(results) == 1 and len(targets) > 1:
                results = results * len(targets)
            elif len(results) != len(targets):
                raise ValueError(
                    "Dynamic rename formula must produce one value per column "
                    f"(got {len(results)} for {len(targets)} target column(s))."
                )
            for original, new in zip(targets, results, strict=True):
                if new is None:
                    raise ValueError(f"Dynamic rename formula returned null for column '{original}'.")
            return [str(n) for n in results]
        return list(targets)

    @staticmethod
    def _assert_rename_has_no_duplicates(
        rename_map: dict[str, str],
        all_columns: list[tuple[str, str]],
    ) -> None:
        """Raise if the rename map would yield duplicate final column names.

        Checks two kinds of collision: (1) two renames producing the same new name,
        and (2) a rename producing a name that already exists on an untouched
        (non-renamed) column. The set of "untouched" names is derived from
        `all_columns` minus `rename_map.keys()`.

        Args:
            rename_map: The proposed `{old_name: new_name}` map (no-ops already removed).
            all_columns: The full incoming schema as `(column_name, data_type_group)` tuples.

        Raises:
            ValueError: If applying `rename_map` would produce duplicate column names.
                The error message lists all offending new names in sorted order.
        """
        untouched = {name for name, _ in all_columns} - rename_map.keys()
        duplicates: set[str] = set()
        seen: set[str] = set()
        for new in rename_map.values():
            if new in seen or new in untouched:
                duplicates.add(new)
            seen.add(new)
        if duplicates:
            raise ValueError("Dynamic rename produces duplicate column name(s): " + ", ".join(sorted(duplicates)))

    @staticmethod
    def resolve_dynamic_rename_map(
        columns: list[tuple[str, str]],
        settings: transform_schemas.DynamicRenameInput,
        first_row_values: dict[str, Any] | None = None,
    ) -> dict[str, str]:
        """Compute the `{old_name: new_name}` map for a dynamic-rename operation.

        Pure function — takes the incoming schema as `(name, data_type_group)` tuples
        (where `data_type_group` is `FlowfileColumn.data_type_group`, e.g. `"Numeric"`,
        `"String"`, `"Date"`, …) and the user's settings, and returns the rename map.
        Raises `ValueError` if the rule would produce duplicate column names.

        Args:
            columns: Incoming schema as `(column_name, data_type_group)` tuples, in order.
            settings: The dynamic rename configuration.
            first_row_values: First-row values keyed by original column name. Required
                for `"first_row"` mode to produce a real rename map; when omitted in
                `"first_row"` mode the result is an empty map (schema-only preview).

        Returns:
            A dict mapping original column name to new column name. No-op renames are
            omitted, so the result is safe to pass directly to `pl.DataFrame.rename`.
        """
        targets = FlowDataEngine._select_rename_targets(columns, settings)
        new_names = FlowDataEngine._compute_renamed_names(targets, settings, first_row_values=first_row_values)
        rename_map = {old: new for old, new in zip(targets, new_names, strict=True) if old != new}
        FlowDataEngine._assert_rename_has_no_duplicates(rename_map, columns)
        return rename_map

    def _peek_first_row_as_dict(self) -> dict[str, Any]:
        """Return the first row of the underlying frame keyed by column name.

        Runs on the external worker via `ExternalDfFetcher` to keep the heavy
        compute out of the core process (same pattern as `fetch_unique_values`
        used by `do_pivot`). Falls back to an in-core collect only if the
        external fetcher is unavailable. Raises `ValueError` if the frame is
        empty.
        """
        df = self.data_frame
        lf = df.lazy() if isinstance(df, pl.DataFrame) else df
        head_lf = lf.head(1)

        table = None
        try:
            external = ExternalDfFetcher(lf=head_lf, flow_id=1, node_id=-1, wait_on_completion=True)
            if external.status is not None and external.status.status == "Completed":
                table = arrow_read(external.status.file_ref)
        except Exception as e:  # noqa: BLE001 - worker availability is best-effort
            logger.debug(f"ExternalDfFetcher unavailable for first_row peek ({e}); using in-core fallback")

        if table is not None:
            if table.num_rows == 0:
                raise ValueError("Dynamic rename (first_row) requires at least one row in the input; got 0.")
            return {name: table.column(name)[0].as_py() for name in table.column_names}

        head = head_lf.collect()
        if head.height == 0:
            raise ValueError("Dynamic rename (first_row) requires at least one row in the input; got 0.")
        return dict(zip(head.columns, head.row(0), strict=True))

    def apply_dynamic_rename(self, settings: transform_schemas.DynamicRenameInput) -> FlowDataEngine:
        """Renames a subset of columns according to a single rule.

        Supports prefix, suffix, flowfile-formula, and first-row rename modes, with
        column selection by name list, by data type, or across all columns. In
        `"first_row"` mode the first row is always dropped from the output after
        its values are promoted to column headers.

        Args:
            settings: The dynamic rename configuration.

        Returns:
            A new `FlowDataEngine` with the renamed columns (or this instance's DataFrame
            unchanged if the rule resolves to no renames).
        """
        columns = [(c.column_name, c.data_type_group) for c in self.schema]
        first_row_values = None
        if settings.rename_mode == "first_row":
            first_row_values = self._peek_first_row_as_dict()
        rename_map = self.resolve_dynamic_rename_map(columns, settings, first_row_values=first_row_values)
        new_df = self.data_frame.rename(rename_map) if rename_map else self.data_frame
        if settings.rename_mode == "first_row":
            new_df = new_df.slice(1)
            new_records = max(0, self.number_of_records - 1)
            return FlowDataEngine(new_df, number_of_records=new_records)
        if not rename_map:
            return FlowDataEngine(
                self.data_frame,
                number_of_records=self.number_of_records,
                schema=self.schema,
            )
        return FlowDataEngine(new_df, number_of_records=self.number_of_records)

    def apply_sql_formula(self, func: str, col_name: str, output_data_type: pl.DataType = None) -> FlowDataEngine:
        """Applies an SQL-style formula using `pl.sql_expr`.

        Args:
            func: A string containing an SQL expression.
            col_name: The name of the new or transformed column.
            output_data_type: The desired Polars data type for the output column.

        Returns:
            A new `FlowDataEngine` instance with the applied formula.
        """
        expr = to_expr(func)
        if output_data_type not in (None, transform_schemas.AUTO_DATA_TYPE):
            df = self.data_frame.with_columns(expr.cast(output_data_type).alias(col_name))
        else:
            df = self.data_frame.with_columns(expr.alias(col_name))

        return FlowDataEngine(df, number_of_records=self.number_of_records)

    def output(
        self, output_fs: input_schema.OutputSettings, flow_id: int, node_id: int | str, execute_remote: bool = False
    ) -> FlowDataEngine:
        """Writes the DataFrame to a local output file.

        For remote-worker writes the caller (``add_output._func``) uses
        ``ExternalOutputWriter`` directly so the fetcher can be exposed on
        the node for cancellation; this method only handles the local path.

        Args:
            output_fs: An `OutputSettings` object with details about the output file.
            flow_id: The flow ID for tracking.
            node_id: The node ID for tracking.
            execute_remote: Retained for signature compatibility; ignored.

        Returns:
            The same `FlowDataEngine` instance for chaining.
        """
        logger.info("Starting to write results locally")
        utils.local_write_output(
            self.data_frame,
            data_type=output_fs.file_type,
            path=output_fs.abs_file_path,
            write_mode=output_fs.write_mode,
            sheet_name=output_fs.sheet_name,
            delimiter=output_fs.delimiter,
            compression=output_fs.compression,
            flow_id=flow_id,
            node_id=node_id,
        )
        logger.info("Finished writing output")
        return self

    def make_unique(self, unique_input: transform_schemas.UniqueInput = None) -> FlowDataEngine:
        """Gets the unique rows from the DataFrame.

        Args:
            unique_input: A `UniqueInput` object specifying a subset of columns
                to consider for uniqueness and a strategy for keeping rows.

        Returns:
            A new `FlowDataEngine` instance with unique rows.
        """
        if unique_input is None or unique_input.columns is None:
            return FlowDataEngine(self.data_frame.unique())
        return FlowDataEngine(self.data_frame.unique(unique_input.columns, keep=unique_input.strategy))

    def concat(self, other: Iterable[FlowDataEngine] | FlowDataEngine) -> FlowDataEngine:
        """Concatenates this DataFrame with one or more other DataFrames.

        Args:
            other: A single `FlowDataEngine` or an iterable of them.

        Returns:
            A new `FlowDataEngine` containing the concatenated data.
        """
        if isinstance(other, FlowDataEngine):
            other = [other]

        dfs: list[pl.LazyFrame] | list[pl.DataFrame] = [self.data_frame] + [flt.data_frame for flt in other]
        return FlowDataEngine(pl.concat(dfs, how="diagonal_relaxed"))

    def do_select(self, select_inputs: transform_schemas.SelectInputs, keep_missing: bool = True) -> FlowDataEngine:
        """Performs a complex column selection, renaming, and reordering operation.

        Args:
            select_inputs: A `SelectInputs` object defining the desired transformations.
            keep_missing: If True, columns not specified in `select_inputs` are kept.
                If False, they are dropped.

        Returns:
            A new `FlowDataEngine` with the transformed selection.
        """
        new_schema = deepcopy(self.schema)
        renames = [r for r in select_inputs.renames if r.is_available]
        if not keep_missing:
            drop_cols = set(self.data_frame.collect_schema().names()) - set(r.old_name for r in renames).union(
                set(r.old_name for r in renames if not r.keep)
            )
            keep_cols = []
        else:
            keep_cols = list(set(self.data_frame.collect_schema().names()) - set(r.old_name for r in renames))
            drop_cols = set(r.old_name for r in renames if not r.keep)

        if len(drop_cols) > 0:
            new_schema = [s for s in new_schema if s.name not in drop_cols]
        new_schema_mapping = {v.name: v for v in new_schema}

        available_renames = []
        for rename in renames:
            if (rename.new_name != rename.old_name or rename.new_name not in new_schema_mapping) and rename.keep:
                schema_entry = new_schema_mapping.get(rename.old_name)
                if schema_entry is not None:
                    available_renames.append(rename)
                    schema_entry.column_name = rename.new_name

        rename_dict = {r.old_name: r.new_name for r in available_renames}
        fl = self.select_columns(
            list_select=[col_to_keep.old_name for col_to_keep in renames if col_to_keep.keep] + keep_cols
        )
        fl = fl.change_column_types(transforms=[r for r in renames if r.keep])
        ndf = fl.data_frame.rename(rename_dict)
        renames.sort(key=lambda r: 0 if r.position is None else r.position)
        sorted_cols = utils.match_order(
            ndf.collect_schema().names(), [r.new_name for r in renames] + self.data_frame.collect_schema().names()
        )
        output_file = FlowDataEngine(ndf, number_of_records=self.number_of_records)
        return output_file.reorganize_order(sorted_cols)

    def set_streamable(self, streamable: bool = False):
        """Sets whether DataFrame operations should be streamable."""
        self._streamable = streamable

    def shallow_copy(self) -> FlowDataEngine:
        """Cheap de-aliasing wrapper around the same (immutable) Polars frame.

        Shares the frame and the cached schema, but owns its own mutable flags
        (_lazy, _streamable, number_of_records, _schema), so a consumer handed
        this copy can never mutate an engine shared with sibling consumers.
        Collect-free: forwarding number_of_records and the cached schema skips
        both pl.len() and collect_schema() in __init__ (the schema fallback only
        fires when _schema is unset, and is metadata-only). Deliberately does
        not carry external_source: memoized results are materialized before
        they are shared, so the plain frame is the whole result.
        """
        return FlowDataEngine(
            self.data_frame,
            name=self.name,
            optimize_memory=self._optimize_memory,
            schema=self._schema,
            number_of_records=self.number_of_records,
            streamable=self._streamable,
            number_of_records_callback=self._number_of_records_callback,
            data_callback=self._data_callback,
        )

    def _calculate_schema(self) -> list[dict]:
        """Calculates schema statistics."""
        if self.external_source is not None:
            self.collect_external()
        v = utils.calculate_schema(self.data_frame)
        return v

    def calculate_schema(self):
        """Calculates and returns the schema."""
        self._calculate_schema_stats = True
        return self.schema

    def count(self) -> int:
        """Gets the total number of records."""
        return self.get_number_of_records()

    @classmethod
    def create_from_path_worker(cls, received_table: input_schema.ReceivedTable, flow_id: int, node_id: int | str):
        """Creates a FlowDataEngine from a path in a worker process."""
        received_table.set_absolute_filepath()

        external_fetcher = ExternalCreateFetcher(
            received_table=received_table, file_type=received_table.file_type, flow_id=flow_id, node_id=node_id
        )
        return cls(external_fetcher.get_result())

__name__ property

The name of the table.

cols_idx property

A dictionary mapping column names to their integer index.

data_frame property writable

The underlying Polars DataFrame or LazyFrame.

This property provides access to the Polars object that backs the FlowDataEngine. It handles lazy-loading from external sources if necessary.

Returns:

Type	Description
LazyFrame | DataFrame | None	The active Polars DataFrame or LazyFrame.

external_source property

The external data source, if any.

has_errors property

Checks if there are any errors.

lazy property writable

Indicates if the DataFrame is in lazy mode.

number_of_fields property

The number of columns (fields) in the DataFrame.

Returns:

Type	Description
int	The integer count of columns.

schema property

The schema of the DataFrame as a list of FlowfileColumn objects.

This property lazily calculates the schema if it hasn't been determined yet.

Returns:

Type	Description
list[FlowfileColumn]	A list of FlowfileColumn objects describing the schema.

__call__()

Makes the class instance callable, returning itself.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __call__(self) -> FlowDataEngine:
    """Makes the class instance callable, returning itself."""
    return self

__get_sample__(n_rows=100, streamable=True)

Internal method to get a sample of the data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __get_sample__(self, n_rows: int = 100, streamable: bool = True) -> FlowDataEngine:
    """Internal method to get a sample of the data."""
    if not self.lazy:
        df = self.data_frame.lazy()
    else:
        df = self.data_frame

    if streamable:
        try:
            df = df.head(n_rows).collect()
        except Exception as e:
            logger.warning(f"Error in getting sample: {e}")
            df = df.head(n_rows).collect(engine="auto")
    else:
        df = self.collect()
    return FlowDataEngine(df, number_of_records=len(df), schema=self.schema)

__getitem__(item)

Accesses a specific column or item from the DataFrame.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __getitem__(self, item):
    """Accesses a specific column or item from the DataFrame."""
    return self.data_frame.select([item])

__init__(raw_data=None, path_ref=None, name=None, optimize_memory=True, schema=None, number_of_records=None, calculate_schema_stats=False, streamable=True, number_of_records_callback=None, data_callback=None)

Initializes the FlowDataEngine from various data sources.

Parameters:

Name	Type	Description	Default
raw_data	list[dict] | list[Any] | dict[str, Any] | ParquetFile | DataFrame | LazyFrame | RawData	The input data. Can be a list of dicts, a Polars DataFrame/LazyFrame, or a RawData schema object.	None
path_ref	str	A string path to a Parquet file.	None
name	str	An optional name for the data engine instance.	None
optimize_memory	bool	If True, prefers lazy operations to conserve memory.	True
schema	list[FlowfileColumn] | list[str] | Schema	An optional schema definition. Can be a list of FlowfileColumn objects, a list of column names, or a Polars Schema.	None
number_of_records	int	The number of records, if known.	None
calculate_schema_stats	bool	If True, computes detailed statistics for each column.	False
streamable	bool	If True, allows for streaming operations when possible.	True
number_of_records_callback	Callable	A callback function to retrieve the number of records.	None
data_callback	Callable	A callback function to retrieve the data.	None

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __init__(
    self,
    raw_data: (
        list[dict] | list[Any] | dict[str, Any] | ParquetFile | pl.DataFrame | pl.LazyFrame | input_schema.RawData
    ) = None,
    path_ref: str = None,
    name: str = None,
    optimize_memory: bool = True,
    schema: list[FlowfileColumn] | list[str] | pl.Schema = None,
    number_of_records: int = None,
    calculate_schema_stats: bool = False,
    streamable: bool = True,
    number_of_records_callback: Callable = None,
    data_callback: Callable = None,
):
    """Initializes the FlowDataEngine from various data sources.

    Args:
        raw_data: The input data. Can be a list of dicts, a Polars DataFrame/LazyFrame,
            or a `RawData` schema object.
        path_ref: A string path to a Parquet file.
        name: An optional name for the data engine instance.
        optimize_memory: If True, prefers lazy operations to conserve memory.
        schema: An optional schema definition. Can be a list of `FlowfileColumn` objects,
            a list of column names, or a Polars `Schema`.
        number_of_records: The number of records, if known.
        calculate_schema_stats: If True, computes detailed statistics for each column.
        streamable: If True, allows for streaming operations when possible.
        number_of_records_callback: A callback function to retrieve the number of records.
        data_callback: A callback function to retrieve the data.
    """
    self._initialize_attributes(number_of_records_callback, data_callback, streamable)

    if raw_data is not None:
        self._handle_raw_data(raw_data, number_of_records, optimize_memory)
    elif path_ref:
        self._handle_path_ref(path_ref, optimize_memory)
    else:
        self.initialize_empty_fl()
    self._finalize_initialization(name, optimize_memory, schema, calculate_schema_stats)

__len__()

Returns the number of records in the table.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __len__(self) -> int:
    """Returns the number of records in the table."""
    return self.number_of_records if self.number_of_records >= 0 else self.get_number_of_records()

__repr__()

Returns a string representation of the FlowDataEngine.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def __repr__(self) -> str:
    """Returns a string representation of the FlowDataEngine."""
    return f"flow data engine\n{self.data_frame.__repr__()}"

add_new_values(values, col_name=None)

Adds a new column with the provided values.

Parameters:

Name	Type	Description	Default
values	Iterable	An iterable (e.g., list, tuple) of values to add as a new column.	required
col_name	str	The name for the new column. Defaults to 'new_values'.	None

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the added column.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def add_new_values(self, values: Iterable, col_name: str = None) -> FlowDataEngine:
    """Adds a new column with the provided values.

    Args:
        values: An iterable (e.g., list, tuple) of values to add as a new column.
        col_name: The name for the new column. Defaults to 'new_values'.

    Returns:
        A new `FlowDataEngine` instance with the added column.
    """
    if col_name is None:
        col_name = "new_values"
    return FlowDataEngine(self.data_frame.with_columns(pl.Series(values).alias(col_name)))

add_record_id(record_id_settings)

Adds a record ID (row number) column to the DataFrame.

Can generate a simple sequential ID or a grouped ID that resets for each group.

Parameters:

Name	Type	Description	Default
record_id_settings	RecordIdInput	A RecordIdInput object specifying the output column name, offset, and optional grouping columns.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the added record ID column.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def add_record_id(self, record_id_settings: transform_schemas.RecordIdInput) -> FlowDataEngine:
    """Adds a record ID (row number) column to the DataFrame.

    Can generate a simple sequential ID or a grouped ID that resets for
    each group.

    Args:
        record_id_settings: A `RecordIdInput` object specifying the output
            column name, offset, and optional grouping columns.

    Returns:
        A new `FlowDataEngine` instance with the added record ID column.
    """
    if record_id_settings.group_by and len(record_id_settings.group_by_columns) > 0:
        return self._add_grouped_record_id(record_id_settings)
    return self._add_simple_record_id(record_id_settings)

align_to_schema(expected_schema)

Aligns the DataFrame to an expected schema.

Adds any missing columns as typed nulls and reorders all columns to match the order defined in expected_schema. Extra columns present in the data but absent from the expected schema are appended at the end so no data is silently dropped.

Parameters:

Name	Type	Description	Default
expected_schema	list[FlowfileColumn]	The desired column list, in order.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine whose columns match expected_schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def align_to_schema(self, expected_schema: list[FlowfileColumn]) -> FlowDataEngine:
    """Aligns the DataFrame to an expected schema.

    Adds any missing columns as typed nulls and reorders all columns to
    match the order defined in *expected_schema*.  Extra columns present
    in the data but absent from the expected schema are appended at the
    end so no data is silently dropped.

    Args:
        expected_schema: The desired column list, in order.

    Returns:
        A new ``FlowDataEngine`` whose columns match *expected_schema*.
    """
    actual_names = set(self.columns)
    expected_names = [c.column_name for c in expected_schema]

    missing_exprs = []
    for col in expected_schema:
        if col.column_name not in actual_names:
            pl_type = cast_str_to_polars_type(col.data_type)
            missing_exprs.append(pl.lit(None).cast(pl_type).alias(col.column_name))

    df = self.data_frame
    if missing_exprs:
        df = df.with_columns(missing_exprs)

    expected_set = set(expected_names)
    extra_names = [c for c in self.columns if c not in expected_set]
    final_order = expected_names + extra_names

    return FlowDataEngine(
        df.select(final_order),
        number_of_records=self.number_of_records,
        streamable=self._streamable,
    )

apply_dynamic_rename(settings)

Renames a subset of columns according to a single rule.

Supports prefix, suffix, flowfile-formula, and first-row rename modes, with column selection by name list, by data type, or across all columns. In "first_row" mode the first row is always dropped from the output after its values are promoted to column headers.

Parameters:

Name	Type	Description	Default
settings	DynamicRenameInput	The dynamic rename configuration.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine with the renamed columns (or this instance's DataFrame
FlowDataEngine	unchanged if the rule resolves to no renames).

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def apply_dynamic_rename(self, settings: transform_schemas.DynamicRenameInput) -> FlowDataEngine:
    """Renames a subset of columns according to a single rule.

    Supports prefix, suffix, flowfile-formula, and first-row rename modes, with
    column selection by name list, by data type, or across all columns. In
    `"first_row"` mode the first row is always dropped from the output after
    its values are promoted to column headers.

    Args:
        settings: The dynamic rename configuration.

    Returns:
        A new `FlowDataEngine` with the renamed columns (or this instance's DataFrame
        unchanged if the rule resolves to no renames).
    """
    columns = [(c.column_name, c.data_type_group) for c in self.schema]
    first_row_values = None
    if settings.rename_mode == "first_row":
        first_row_values = self._peek_first_row_as_dict()
    rename_map = self.resolve_dynamic_rename_map(columns, settings, first_row_values=first_row_values)
    new_df = self.data_frame.rename(rename_map) if rename_map else self.data_frame
    if settings.rename_mode == "first_row":
        new_df = new_df.slice(1)
        new_records = max(0, self.number_of_records - 1)
        return FlowDataEngine(new_df, number_of_records=new_records)
    if not rename_map:
        return FlowDataEngine(
            self.data_frame,
            number_of_records=self.number_of_records,
            schema=self.schema,
        )
    return FlowDataEngine(new_df, number_of_records=self.number_of_records)

apply_flowfile_formula(func, col_name, output_data_type=None)

Applies a formula to create a new column or transform an existing one.

Parameters:

Name	Type	Description	Default
func	str	A string containing a Polars expression formula.	required
col_name	str	The name of the new or transformed column.	required
output_data_type	DataType	The desired Polars data type for the output column.	None

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the applied formula.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def apply_flowfile_formula(self, func: str, col_name: str, output_data_type: pl.DataType = None) -> FlowDataEngine:
    """Applies a formula to create a new column or transform an existing one.

    Args:
        func: A string containing a Polars expression formula.
        col_name: The name of the new or transformed column.
        output_data_type: The desired Polars data type for the output column.

    Returns:
        A new `FlowDataEngine` instance with the applied formula.
    """
    parsed_func = to_expr(func)
    if output_data_type is not None:
        df2 = self.data_frame.with_columns(parsed_func.cast(output_data_type).alias(col_name))
    else:
        df2 = self.data_frame.with_columns(parsed_func.alias(col_name))

    return FlowDataEngine(df2, number_of_records=self.number_of_records)

apply_sql_formula(func, col_name, output_data_type=None)

Applies an SQL-style formula using pl.sql_expr.

Parameters:

Name	Type	Description	Default
func	str	A string containing an SQL expression.	required
col_name	str	The name of the new or transformed column.	required
output_data_type	DataType	The desired Polars data type for the output column.	None

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the applied formula.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def apply_sql_formula(self, func: str, col_name: str, output_data_type: pl.DataType = None) -> FlowDataEngine:
    """Applies an SQL-style formula using `pl.sql_expr`.

    Args:
        func: A string containing an SQL expression.
        col_name: The name of the new or transformed column.
        output_data_type: The desired Polars data type for the output column.

    Returns:
        A new `FlowDataEngine` instance with the applied formula.
    """
    expr = to_expr(func)
    if output_data_type not in (None, transform_schemas.AUTO_DATA_TYPE):
        df = self.data_frame.with_columns(expr.cast(output_data_type).alias(col_name))
    else:
        df = self.data_frame.with_columns(expr.alias(col_name))

    return FlowDataEngine(df, number_of_records=self.number_of_records)

assert_equal(other, ordered=True, strict_schema=False)

Asserts that this DataFrame is equal to another.

Useful for testing.

Parameters:

Name	Type	Description	Default
other	FlowDataEngine	The other FlowDataEngine to compare with.	required
ordered	bool	If True, the row order must be identical.	True
strict_schema	bool	If True, the data types of the schemas must be identical.	False

Raises:

Type	Description
Exception	If the DataFrames are not equal based on the specified criteria.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def assert_equal(self, other: FlowDataEngine, ordered: bool = True, strict_schema: bool = False):
    """Asserts that this DataFrame is equal to another.

    Useful for testing.

    Args:
        other: The other `FlowDataEngine` to compare with.
        ordered: If True, the row order must be identical.
        strict_schema: If True, the data types of the schemas must be identical.

    Raises:
        Exception: If the DataFrames are not equal based on the specified criteria.
    """
    org_laziness = self.lazy, other.lazy
    self.lazy = False
    other.lazy = False
    self.number_of_records = -1
    other.number_of_records = -1
    other = other.select_columns(self.columns)

    if self.get_number_of_records_in_process() != other.get_number_of_records_in_process():
        raise Exception("Number of records is not equal")

    if self.columns != other.columns:
        raise Exception("Schema is not equal")

    if strict_schema:
        assert self.data_frame.schema == other.data_frame.schema, "Data types do not match"

    if ordered:
        self_lf = self.data_frame.sort(by=self.columns)
        other_lf = other.data_frame.sort(by=other.columns)
    else:
        self_lf = self.data_frame
        other_lf = other.data_frame

    self.lazy, other.lazy = org_laziness
    assert self_lf.equals(other_lf), "Data is not equal"

cache()

Caches the current DataFrame to disk and updates the internal reference.

This triggers a background process to write the current LazyFrame's result to a temporary file. Subsequent operations on this FlowDataEngine instance will read from the cached file, which can speed up downstream computations.

Returns:

Type	Description
FlowDataEngine	The same FlowDataEngine instance, now backed by the cached data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def cache(self) -> FlowDataEngine:
    """Caches the current DataFrame to disk and updates the internal reference.

    This triggers a background process to write the current LazyFrame's result
    to a temporary file. Subsequent operations on this `FlowDataEngine` instance
    will read from the cached file, which can speed up downstream computations.

    Returns:
        The same `FlowDataEngine` instance, now backed by the cached data.
    """
    edf = ExternalDfFetcher(
        lf=self.data_frame, file_ref=str(id(self)), wait_on_completion=False, flow_id=-1, node_id=-1
    )
    logger.info("Caching data in background")
    result = edf.get_result()
    if isinstance(result, pl.LazyFrame):
        logger.info("Data cached")
        del self._data_frame
        self.data_frame = result
        logger.info("Data loaded from cache")
    return self

calculate_schema()

Calculates and returns the schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def calculate_schema(self):
    """Calculates and returns the schema."""
    self._calculate_schema_stats = True
    return self.schema

change_column_types(transforms, calculate_schema=False)

Changes the data type of one or more columns.

Parameters:

Name	Type	Description	Default
transforms	list[SelectInput]	A list of SelectInput objects, where each object specifies the column and its new polars_type.	required
calculate_schema	bool	If True, recalculates the schema after the type change.	False

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the updated column types.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def change_column_types(
    self, transforms: list[transform_schemas.SelectInput], calculate_schema: bool = False
) -> FlowDataEngine:
    """Changes the data type of one or more columns.

    Args:
        transforms: A list of `SelectInput` objects, where each object specifies
            the column and its new `polars_type`.
        calculate_schema: If True, recalculates the schema after the type change.

    Returns:
        A new `FlowDataEngine` instance with the updated column types.
    """
    dtypes = [dtype.base_type() for dtype in self.data_frame.collect_schema().dtypes()]
    idx_mapping = list(
        (transform.old_name, self.cols_idx.get(transform.old_name), get_polars_type(transform.polars_type))
        for transform in transforms
        if transform.data_type is not None
    )

    actual_transforms = [c for c in idx_mapping if c[2] != dtypes[c[1]]]
    transformations = [
        utils.define_pl_col_transformation(col_name=transform[0], col_type=transform[2])
        for transform in actual_transforms
    ]

    df = self.data_frame.with_columns(transformations)
    return FlowDataEngine(
        df,
        number_of_records=self.number_of_records,
        calculate_schema_stats=calculate_schema,
        streamable=self._streamable,
    )

collect(n_records=None)

Collects the data and returns it as a Polars DataFrame.

This method triggers the execution of the lazy query plan (if applicable) and returns the result. It supports streaming to optimize memory usage for large datasets.

Parameters:

Name	Type	Description	Default
n_records	int	The maximum number of records to collect. If None, all records are collected.	None

Returns:

Type	Description
DataFrame	A Polars DataFrame containing the collected data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def collect(self, n_records: int = None) -> pl.DataFrame:
    """Collects the data and returns it as a Polars DataFrame.

    This method triggers the execution of the lazy query plan (if applicable)
    and returns the result. It supports streaming to optimize memory usage
    for large datasets.

    Args:
        n_records: The maximum number of records to collect. If None, all
            records are collected.

    Returns:
        A Polars `DataFrame` containing the collected data.
    """
    if n_records is None:
        logger.info(f'Fetching all data for Table object "{id(self)}". Settings: streaming={self._streamable}')
    else:
        logger.info(
            f'Fetching {n_records} record(s) for Table object "{id(self)}". '
            f"Settings: streaming={self._streamable}"
        )

    if not self.lazy:
        return self.data_frame

    try:
        return self._collect_data(n_records)
    except Exception as e:
        self.errors = [e]
        return self._handle_collection_error(n_records)

collect_external()

Materializes data from a tracked external source.

If the FlowDataEngine was created from an ExternalDataSource, this method will trigger the data retrieval, update the internal _data_frame to a LazyFrame of the collected data, and reset the schema to be re-evaluated.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def collect_external(self):
    """Materializes data from a tracked external source.

    If the `FlowDataEngine` was created from an `ExternalDataSource`, this
    method will trigger the data retrieval, update the internal `_data_frame`
    to a `LazyFrame` of the collected data, and reset the schema to be
    re-evaluated.
    """
    if self._external_source is not None:
        logger.info("Collecting external source")
        if self.external_source.get_pl_df() is not None:
            self.data_frame = self.external_source.get_pl_df().lazy()
        else:
            self.data_frame = pl.LazyFrame(list(self.external_source.get_iter()))
        self._schema = None

concat(other)

Concatenates this DataFrame with one or more other DataFrames.

Parameters:

Name	Type	Description	Default
other	Iterable[FlowDataEngine] | FlowDataEngine	A single FlowDataEngine or an iterable of them.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine containing the concatenated data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def concat(self, other: Iterable[FlowDataEngine] | FlowDataEngine) -> FlowDataEngine:
    """Concatenates this DataFrame with one or more other DataFrames.

    Args:
        other: A single `FlowDataEngine` or an iterable of them.

    Returns:
        A new `FlowDataEngine` containing the concatenated data.
    """
    if isinstance(other, FlowDataEngine):
        other = [other]

    dfs: list[pl.LazyFrame] | list[pl.DataFrame] = [self.data_frame] + [flt.data_frame for flt in other]
    return FlowDataEngine(pl.concat(dfs, how="diagonal_relaxed"))

count()

Gets the total number of records.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def count(self) -> int:
    """Gets the total number of records."""
    return self.get_number_of_records()

create_from_external_source(external_source) classmethod

Creates a FlowDataEngine from an external data source.

Parameters:

Name	Type	Description	Default
external_source	ExternalDataSource	An object that conforms to the ExternalDataSource interface.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_from_external_source(cls, external_source: ExternalDataSource) -> FlowDataEngine:
    """Creates a FlowDataEngine from an external data source.

    Args:
        external_source: An object that conforms to the `ExternalDataSource`
            interface.

    Returns:
        A new `FlowDataEngine` instance.
    """
    if external_source.schema is not None:
        ff = cls.create_from_schema(external_source.schema)
    elif external_source.initial_data_getter is not None:
        ff = cls(raw_data=external_source.initial_data_getter())
    else:
        ff = cls()
    ff._external_source = external_source
    return ff

create_from_path(received_table) classmethod

Creates a FlowDataEngine from a local file path.

Supports various file types like CSV, Parquet, and Excel.

Parameters:

Name	Type	Description	Default
received_table	ReceivedTable	A ReceivedTableBase object containing the file path and format details.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with data from the file.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_from_path(cls, received_table: input_schema.ReceivedTable) -> FlowDataEngine:
    """Creates a FlowDataEngine from a local file path.

    Supports various file types like CSV, Parquet, and Excel.

    Args:
        received_table: A `ReceivedTableBase` object containing the file path
            and format details.

    Returns:
        A new `FlowDataEngine` instance with data from the file.
    """
    received_table.set_absolute_filepath()
    file_type_handlers = {
        "csv": create_funcs.create_from_path_csv,
        "parquet": create_funcs.create_from_path_parquet,
        "excel": create_funcs.create_from_path_excel,
        "ipc": create_funcs.create_from_path_ipc,
        "ndjson": create_funcs.create_from_path_ndjson,
        "avro": create_funcs.create_from_path_avro,
    }

    handler = file_type_handlers.get(received_table.file_type)
    if not handler:
        raise Exception(f"Cannot create from {received_table.file_type}")

    flow_file = cls(handler(received_table))
    flow_file._org_path = received_table.abs_file_path
    return flow_file

create_from_path_worker(received_table, flow_id, node_id) classmethod

Creates a FlowDataEngine from a path in a worker process.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_from_path_worker(cls, received_table: input_schema.ReceivedTable, flow_id: int, node_id: int | str):
    """Creates a FlowDataEngine from a path in a worker process."""
    received_table.set_absolute_filepath()

    external_fetcher = ExternalCreateFetcher(
        received_table=received_table, file_type=received_table.file_type, flow_id=flow_id, node_id=node_id
    )
    return cls(external_fetcher.get_result())

create_from_schema(schema) classmethod

Creates an empty FlowDataEngine from a schema definition.

Parameters:

Name	Type	Description	Default
schema	list[FlowfileColumn]	A list of FlowfileColumn objects defining the schema.	required

Returns:

Type	Description
FlowDataEngine	A new, empty FlowDataEngine instance with the specified schema.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_from_schema(cls, schema: list[FlowfileColumn]) -> FlowDataEngine:
    """Creates an empty FlowDataEngine from a schema definition.

    Args:
        schema: A list of `FlowfileColumn` objects defining the schema.

    Returns:
        A new, empty `FlowDataEngine` instance with the specified schema.
    """
    pl_schema = []
    for i, flow_file_column in enumerate(schema):
        pl_schema.append((flow_file_column.name, cast_str_to_polars_type(flow_file_column.data_type)))
        schema[i].col_index = i
    df = pl.LazyFrame(schema=pl_schema)
    return cls(df, schema=schema, calculate_schema_stats=False, number_of_records=0)

create_from_sql(sql, conn) classmethod

Creates a FlowDataEngine by executing a SQL query.

Parameters:

Name	Type	Description	Default
sql	str	The SQL query string to execute.	required
conn	Any	A database connection object or connection URI string.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the query result.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_from_sql(cls, sql: str, conn: Any) -> FlowDataEngine:
    """Creates a FlowDataEngine by executing a SQL query.

    Args:
        sql: The SQL query string to execute.
        conn: A database connection object or connection URI string.

    Returns:
        A new `FlowDataEngine` instance with the query result.
    """
    return cls(pl.read_sql(sql, conn))

create_random(number_of_records=1000) classmethod

Creates a FlowDataEngine with randomly generated data.

Useful for testing and examples.

Parameters:

Name	Type	Description	Default
number_of_records	int	The number of random records to generate.	1000

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with fake data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def create_random(cls, number_of_records: int = 1000) -> FlowDataEngine:
    """Creates a FlowDataEngine with randomly generated data.

    Useful for testing and examples.

    Args:
        number_of_records: The number of random records to generate.

    Returns:
        A new `FlowDataEngine` instance with fake data.
    """
    return cls(create_fake_data(number_of_records))

do_cross_join(cross_join_input, auto_generate_selection, verify_integrity, other)

Performs a cross join with another DataFrame.

A cross join produces the Cartesian product of the two DataFrames.

Parameters:

Name	Type	Description	Default
cross_join_input	CrossJoinInput	A CrossJoinInput object specifying column selections.	required
auto_generate_selection	bool	If True, automatically renames columns to avoid conflicts.	required
verify_integrity	bool	If True, checks if the resulting join would be too large.	required
other	FlowDataEngine	The right FlowDataEngine to join with.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine with the result of the cross join.

Raises:

Type	Description
Exception	If verify_integrity is True and the join would result in an excessively large number of records.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_cross_join(
    self,
    cross_join_input: transform_schemas.CrossJoinInput,
    auto_generate_selection: bool,
    verify_integrity: bool,
    other: FlowDataEngine,
) -> FlowDataEngine:
    """Performs a cross join with another DataFrame.

    A cross join produces the Cartesian product of the two DataFrames.

    Args:
        cross_join_input: A `CrossJoinInput` object specifying column selections.
        auto_generate_selection: If True, automatically renames columns to avoid conflicts.
        verify_integrity: If True, checks if the resulting join would be too large.
        other: The right `FlowDataEngine` to join with.

    Returns:
        A new `FlowDataEngine` with the result of the cross join.

    Raises:
        Exception: If `verify_integrity` is True and the join would result in
            an excessively large number of records.
    """
    self.lazy = True
    other.lazy = True
    cross_join_input_manager = transform_schemas.CrossJoinInputManager(cross_join_input)
    _ensure_all_columns_have_select(
        left_cols=self.columns, right_cols=other.columns, manager=cross_join_input_manager
    )
    verify_join_select_integrity(
        cross_join_input_manager.input, left_columns=self.columns, right_columns=other.columns
    )
    right_select = [
        v.old_name
        for v in cross_join_input_manager.right_select.renames
        if (v.keep or v.join_key) and v.is_available
    ]
    left_select = [
        v.old_name
        for v in cross_join_input_manager.left_select.renames
        if (v.keep or v.join_key) and v.is_available
    ]
    cross_join_input_manager.auto_rename(rename_mode="suffix")
    left = self.data_frame.select(left_select).rename(cross_join_input_manager.left_select.rename_table)
    right = other.data_frame.select(right_select).rename(cross_join_input_manager.right_select.rename_table)

    joined_df = left.join(right, how="cross")

    cols_to_delete_after = [
        col.new_name
        for col in cross_join_input_manager.left_select.renames + cross_join_input_manager.right_select.renames
        if col.join_key and not col.keep and col.is_available
    ]

    fl = FlowDataEngine(joined_df.drop(cols_to_delete_after), calculate_schema_stats=False, streamable=False)
    return fl

do_filter(predicate)

Filters rows based on a predicate expression.

Parameters:

Name	Type	Description	Default
predicate	str	A string containing a Polars expression that evaluates to a boolean value.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance containing only the rows that match
FlowDataEngine	the predicate.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_filter(self, predicate: str) -> FlowDataEngine:
    """Filters rows based on a predicate expression.

    Args:
        predicate: A string containing a Polars expression that evaluates to
            a boolean value.

    Returns:
        A new `FlowDataEngine` instance containing only the rows that match
        the predicate.
    """
    try:
        f = to_expr(predicate)
    except Exception as e:
        logger.warning(f"Error in filter expression: {e}")
        f = to_expr("False")
    df = self.data_frame.filter(f)
    _ = df.collect_schema()  # Collecting schema to ensure the filter is valid

    return FlowDataEngine(df, streamable=self._streamable)

do_group_by(group_by_input, calculate_schema_stats=True)

Performs a group-by operation on the DataFrame.

Parameters:

Name	Type	Description	Default
group_by_input	GroupByInput	A GroupByInput object defining the grouping columns and aggregations.	required
calculate_schema_stats	bool	If True, calculates schema statistics for the resulting DataFrame.	True

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the grouped and aggregated data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_group_by(
    self, group_by_input: transform_schemas.GroupByInput, calculate_schema_stats: bool = True
) -> FlowDataEngine:
    """Performs a group-by operation on the DataFrame.

    Args:
        group_by_input: A `GroupByInput` object defining the grouping columns
            and aggregations.
        calculate_schema_stats: If True, calculates schema statistics for the
            resulting DataFrame.

    Returns:
        A new `FlowDataEngine` instance with the grouped and aggregated data.
    """
    aggregations = [c for c in group_by_input.agg_cols if c.agg != "groupby"]
    group_columns = [c for c in group_by_input.agg_cols if c.agg == "groupby"]

    if len(group_columns) == 0:
        return FlowDataEngine(
            self.data_frame.select(ac.agg_func(ac.old_name).alias(ac.new_name) for ac in aggregations),
            calculate_schema_stats=calculate_schema_stats,
        )

    df = self.data_frame.rename({c.old_name: c.new_name for c in group_columns})
    group_by_columns = [n_c.new_name for n_c in group_columns]

    if len(aggregations) == 0:
        return FlowDataEngine(
            df.select(group_by_columns).unique(),
            calculate_schema_stats=calculate_schema_stats,
        )

    grouped_df = df.group_by(*group_by_columns)
    agg_exprs = [ac.agg_func(ac.old_name).alias(ac.new_name) for ac in aggregations]
    result_df = grouped_df.agg(agg_exprs)

    return FlowDataEngine(
        result_df,
        calculate_schema_stats=calculate_schema_stats,
    )

do_pivot(pivot_input, node_logger=None)

Converts the DataFrame from a long to a wide format, aggregating values.

Parameters:

Name	Type	Description	Default
pivot_input	PivotInput	A PivotInput object defining the index, pivot, and value columns, along with the aggregation logic.	required
node_logger	NodeLogger	An optional logger for reporting warnings, e.g., if the pivot column has too many unique values.	None

Returns:

Type	Description
FlowDataEngine	A new, pivoted FlowDataEngine instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_pivot(self, pivot_input: transform_schemas.PivotInput, node_logger: NodeLogger = None) -> FlowDataEngine:
    """Converts the DataFrame from a long to a wide format, aggregating values.

    Args:
        pivot_input: A `PivotInput` object defining the index, pivot, and value
            columns, along with the aggregation logic.
        node_logger: An optional logger for reporting warnings, e.g., if the
            pivot column has too many unique values.

    Returns:
        A new, pivoted `FlowDataEngine` instance.
    """
    max_unique_vals = 200
    new_cols_unique = fetch_unique_values(
        self.data_frame.select(pivot_input.pivot_column)
        .unique()
        .sort(pivot_input.pivot_column)
        .limit(max_unique_vals)
        .cast(pl.String)
    )
    if len(new_cols_unique) >= max_unique_vals:
        if node_logger:
            node_logger.warning(
                "Pivot column has too many unique values. Please consider using a different column."
                f" Max unique values: {max_unique_vals}"
            )

    if len(pivot_input.index_columns) == 0:
        no_index_cols = True
        pivot_input.index_columns = ["__temp__"]
        ff = self.apply_flowfile_formula("1", col_name="__temp__")
    else:
        no_index_cols = False
        ff = self

    index_columns = pivot_input.get_index_columns()
    grouped_ff = ff.do_group_by(pivot_input.get_group_by_input(), False)
    pivot_column = pivot_input.get_pivot_column()

    input_df = grouped_ff.data_frame.with_columns(pivot_column.cast(pl.String).alias(pivot_input.pivot_column))
    number_of_aggregations = len(pivot_input.aggregations)
    # Aggregations where missing combinations should be filled with 0 to match
    # native polars pivot behavior (polars >= 1.32)
    _zero_fill_aggs = {"sum", "count", "len"}
    df = (
        input_df.select(*index_columns, pivot_column, pivot_input.get_values_expr())
        .group_by(*index_columns)
        .agg(
            [
                (pl.col("vals").filter(pivot_column == new_col_value)).first().alias(new_col_value)
                for new_col_value in new_cols_unique
            ]
        )
        .select(
            *index_columns,
            *[
                (
                    pl.col(new_col).struct.field(agg).fill_null(0)
                    if agg in _zero_fill_aggs
                    else pl.col(new_col).struct.field(agg)
                ).alias(f'{new_col + "_" + agg if number_of_aggregations > 1 else new_col}')
                for new_col in new_cols_unique
                for agg in pivot_input.aggregations
            ],
        )
    )

    if no_index_cols:
        df = df.drop("__temp__")
        pivot_input.index_columns = []

    return FlowDataEngine(df, calculate_schema_stats=False)

do_select(select_inputs, keep_missing=True)

Performs a complex column selection, renaming, and reordering operation.

Parameters:

Name	Type	Description	Default
select_inputs	SelectInputs	A SelectInputs object defining the desired transformations.	required
keep_missing	bool	If True, columns not specified in select_inputs are kept. If False, they are dropped.	True

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine with the transformed selection.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_select(self, select_inputs: transform_schemas.SelectInputs, keep_missing: bool = True) -> FlowDataEngine:
    """Performs a complex column selection, renaming, and reordering operation.

    Args:
        select_inputs: A `SelectInputs` object defining the desired transformations.
        keep_missing: If True, columns not specified in `select_inputs` are kept.
            If False, they are dropped.

    Returns:
        A new `FlowDataEngine` with the transformed selection.
    """
    new_schema = deepcopy(self.schema)
    renames = [r for r in select_inputs.renames if r.is_available]
    if not keep_missing:
        drop_cols = set(self.data_frame.collect_schema().names()) - set(r.old_name for r in renames).union(
            set(r.old_name for r in renames if not r.keep)
        )
        keep_cols = []
    else:
        keep_cols = list(set(self.data_frame.collect_schema().names()) - set(r.old_name for r in renames))
        drop_cols = set(r.old_name for r in renames if not r.keep)

    if len(drop_cols) > 0:
        new_schema = [s for s in new_schema if s.name not in drop_cols]
    new_schema_mapping = {v.name: v for v in new_schema}

    available_renames = []
    for rename in renames:
        if (rename.new_name != rename.old_name or rename.new_name not in new_schema_mapping) and rename.keep:
            schema_entry = new_schema_mapping.get(rename.old_name)
            if schema_entry is not None:
                available_renames.append(rename)
                schema_entry.column_name = rename.new_name

    rename_dict = {r.old_name: r.new_name for r in available_renames}
    fl = self.select_columns(
        list_select=[col_to_keep.old_name for col_to_keep in renames if col_to_keep.keep] + keep_cols
    )
    fl = fl.change_column_types(transforms=[r for r in renames if r.keep])
    ndf = fl.data_frame.rename(rename_dict)
    renames.sort(key=lambda r: 0 if r.position is None else r.position)
    sorted_cols = utils.match_order(
        ndf.collect_schema().names(), [r.new_name for r in renames] + self.data_frame.collect_schema().names()
    )
    output_file = FlowDataEngine(ndf, number_of_records=self.number_of_records)
    return output_file.reorganize_order(sorted_cols)

do_sort(sorts)

Sorts the DataFrame by one or more columns.

Parameters:

Name	Type	Description	Default
sorts	list[SortByInput]	A list of SortByInput objects, each specifying a column and sort direction ('asc' or 'desc').	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the sorted data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_sort(self, sorts: list[transform_schemas.SortByInput]) -> FlowDataEngine:
    """Sorts the DataFrame by one or more columns.

    Args:
        sorts: A list of `SortByInput` objects, each specifying a column
            and sort direction ('asc' or 'desc').

    Returns:
        A new `FlowDataEngine` instance with the sorted data.
    """
    if not sorts:
        return self

    descending = [s.descending for s in sorts]
    df = self.data_frame.sort([sort_by.column for sort_by in sorts], descending=descending)
    return FlowDataEngine(df, number_of_records=self.number_of_records, schema=self.schema)

do_window_functions(settings, calculate_schema_stats=False)

Applies window functions (rolling, cumulative, rank, tile) to the data.

When settings.order_by is provided, rows are sorted first so that rolling and tile operations have a deterministic order; the sort is preserved in the output. Partitioning (partition_by) is applied via .over(...) so operations reset for each group.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def do_window_functions(
    self, settings: transform_schemas.WindowFunctionsInput, calculate_schema_stats: bool = False
) -> FlowDataEngine:
    """Applies window functions (rolling, cumulative, rank, tile) to the data.

    When ``settings.order_by`` is provided, rows are sorted first so that
    rolling and tile operations have a deterministic order; the sort is
    preserved in the output. Partitioning (``partition_by``) is applied via
    ``.over(...)`` so operations reset for each group.
    """
    if not settings.window_functions:
        return self

    df = self.data_frame
    if settings.order_by:
        descending = [s.descending for s in settings.order_by]
        df = df.sort([s.column for s in settings.order_by], descending=descending)

    exprs = [
        _build_window_expr(w, settings.partition_by) for w in settings.window_functions
    ]
    df = df.with_columns(exprs)
    return FlowDataEngine(df, calculate_schema_stats=calculate_schema_stats)

drop_columns(columns)

Drops specified columns from the DataFrame.

Parameters:

Name	Type	Description	Default
columns	list[str]	A list of column names to drop.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance without the dropped columns.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def drop_columns(self, columns: list[str]) -> FlowDataEngine:
    """Drops specified columns from the DataFrame.

    Args:
        columns: A list of column names to drop.

    Returns:
        A new `FlowDataEngine` instance without the dropped columns.
    """
    cols_for_select = tuple(set(self.columns) - set(columns))
    idx_to_keep = [self.cols_idx.get(c) for c in cols_for_select]
    new_schema = [self.schema[i] for i in idx_to_keep]

    return FlowDataEngine(
        self.data_frame.select(cols_for_select), number_of_records=self.number_of_records, schema=new_schema
    )

filter_split(predicate)

Partition rows by predicate into pass and fail streams.

Rows where the predicate evaluates to null are dropped from both streams — matching the behaviour of two manually-wired filter nodes with opposing predicates.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def filter_split(self, predicate: str) -> NamedOutputs:
    """Partition rows by ``predicate`` into ``pass`` and ``fail`` streams.

    Rows where the predicate evaluates to null are dropped from both
    streams — matching the behaviour of two manually-wired filter nodes
    with opposing predicates.
    """
    from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

    try:
        f = to_expr(predicate)
    except Exception as e:
        logger.warning(f"Error in filter expression: {e}")
        f = to_expr("False")
    pass_df = self.data_frame.filter(f)
    fail_df = self.data_frame.filter(~f)
    _ = pass_df.collect_schema()
    return NamedOutputs(
        {
            "pass": FlowDataEngine(pass_df, streamable=self._streamable),
            "fail": FlowDataEngine(fail_df, streamable=self._streamable),
        }
    )

from_cloud_storage_obj(settings) classmethod

Creates a FlowDataEngine from an object in cloud storage.

This method supports reading from various cloud storage providers like AWS S3, Azure Data Lake Storage, and Google Cloud Storage, with support for various authentication methods.

Parameters:

Name	Type	Description	Default
settings	CloudStorageReadSettingsInternal	A CloudStorageReadSettingsInternal object containing connection details, file format, and read options.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance containing the data from cloud storage.

Raises:

Type	Description
ValueError	If the storage type or file format is not supported.
NotImplementedError	If a requested file format like "delta" or "iceberg" is not yet implemented.
Exception	If reading from cloud storage fails.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def from_cloud_storage_obj(cls, settings: cloud_storage_schemas.CloudStorageReadSettingsInternal) -> FlowDataEngine:
    """Creates a FlowDataEngine from an object in cloud storage.

    This method supports reading from various cloud storage providers like AWS S3,
    Azure Data Lake Storage, and Google Cloud Storage, with support for
    various authentication methods.

    Args:
        settings: A `CloudStorageReadSettingsInternal` object containing connection
            details, file format, and read options.

    Returns:
        A new `FlowDataEngine` instance containing the data from cloud storage.

    Raises:
        ValueError: If the storage type or file format is not supported.
        NotImplementedError: If a requested file format like "delta" or "iceberg"
            is not yet implemented.
        Exception: If reading from cloud storage fails.
    """
    connection = settings.connection
    read_settings = settings.read_settings

    logger.info(f"Reading from {connection.storage_type} storage: {read_settings.resource_path}")
    storage_options = CloudStorageReader.get_storage_options(connection)
    credential_provider = CloudStorageReader.get_credential_provider(connection)
    use_pyarrow = CloudStorageReader.use_pyarrow_for_gcs(connection)
    if read_settings.file_format == "parquet":
        return cls._read_parquet_from_cloud(
            read_settings.resource_path,
            storage_options,
            credential_provider,
            read_settings.scan_mode == "directory",
            use_pyarrow=use_pyarrow,
        )
    elif read_settings.file_format == "delta":
        return cls._read_delta_from_cloud(
            read_settings.resource_path,
            storage_options,
            credential_provider,
            read_settings,
            use_pyarrow=use_pyarrow,
        )
    elif read_settings.file_format == "csv":
        return cls._read_csv_from_cloud(
            read_settings.resource_path,
            storage_options,
            credential_provider,
            read_settings,
            use_pyarrow=use_pyarrow,
        )
    elif read_settings.file_format == "json":
        return cls._read_json_from_cloud(
            read_settings.resource_path,
            storage_options,
            credential_provider,
            read_settings.scan_mode == "directory",
            use_pyarrow=use_pyarrow,
        )
    elif read_settings.file_format == "iceberg":
        return cls._read_iceberg_from_cloud(
            read_settings.resource_path, storage_options, credential_provider, read_settings
        )

    elif read_settings.file_format in ["delta", "iceberg"]:
        # These would require additional libraries
        raise NotImplementedError(f"File format {read_settings.file_format} not yet implemented")
    else:
        raise ValueError(f"Unsupported file format: {read_settings.file_format}")

generate_enumerator(length=1000, output_name='output_column') classmethod

Generates a FlowDataEngine with a single column containing a sequence of integers.

Parameters:

Name	Type	Description	Default
length	int	The number of integers to generate in the sequence.	1000
output_name	str	The name of the output column.	'output_column'

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@classmethod
def generate_enumerator(cls, length: int = 1000, output_name: str = "output_column") -> FlowDataEngine:
    """Generates a FlowDataEngine with a single column containing a sequence of integers.

    Args:
        length: The number of integers to generate in the sequence.
        output_name: The name of the output column.

    Returns:
        A new `FlowDataEngine` instance.
    """
    if length > 10_000_000:
        length = 10_000_000
    return cls(pl.LazyFrame().select((pl.int_range(0, length, dtype=pl.UInt32)).alias(output_name)))

get_estimated_file_size()

Estimates the file size in bytes if the data originated from a local file.

This relies on the original path being tracked during file ingestion.

Returns:

Type	Description
int	The file size in bytes, or 0 if the original path is unknown.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_estimated_file_size(self) -> int:
    """Estimates the file size in bytes if the data originated from a local file.

    This relies on the original path being tracked during file ingestion.

    Returns:
        The file size in bytes, or 0 if the original path is unknown.
    """
    if self._org_path is not None and not is_url(self._org_path):
        return os.path.getsize(self._org_path)
    return 0

get_number_of_records(warn=False, force_calculate=False, calculate_in_worker_process=False)

Gets the total number of records in the DataFrame.

For lazy frames, this may trigger a full data scan, which can be expensive.

Parameters:

Name	Type	Description	Default
warn	bool	If True, logs a warning if a potentially expensive calculation is triggered.	False
force_calculate	bool	If True, forces recalculation even if a value is cached.	False
calculate_in_worker_process	bool	If True, offloads the calculation to a worker process.	False

Returns:

Type	Description
int	The total number of records.

Raises:

Type	Description
ValueError	If the number of records could not be determined.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_number_of_records(
    self, warn: bool = False, force_calculate: bool = False, calculate_in_worker_process: bool = False
) -> int:
    """Gets the total number of records in the DataFrame.

    For lazy frames, this may trigger a full data scan, which can be expensive.

    Args:
        warn: If True, logs a warning if a potentially expensive calculation is triggered.
        force_calculate: If True, forces recalculation even if a value is cached.
        calculate_in_worker_process: If True, offloads the calculation to a worker process.

    Returns:
        The total number of records.

    Raises:
        ValueError: If the number of records could not be determined.
    """
    if self.is_future and not self.is_collected:
        return -1
    if self.number_of_records is None or self.number_of_records < 0 or force_calculate:
        if self._number_of_records_callback is not None:
            self._number_of_records_callback(self)

        if self.lazy:
            if calculate_in_worker_process:
                try:
                    self.number_of_records = self._calculate_number_of_records_in_worker()
                    return self.number_of_records
                except Exception as e:
                    logger.error(f"Error: {e}")
            if warn:
                logger.warning("Calculating the number of records this can be expensive on a lazy frame")
            try:
                self.number_of_records = self.data_frame.select(pl.len()).collect(
                    engine="streaming" if self._streamable else "auto"
                )[0, 0]
            except Exception:
                raise ValueError("Could not get number of records") from None
        else:
            self.number_of_records = self.data_frame.__len__()
    return self.number_of_records

get_number_of_records_in_process(force_calculate=False)

Get the number of records in the DataFrame in the local process.

Parameters:

Name	Type	Description	Default
force_calculate	bool	If True, forces recalculation even if a value is cached.	False

Returns:

Type	Description
	The total number of records.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_number_of_records_in_process(self, force_calculate: bool = False):
    """
    Get the number of records in the DataFrame in the local process.

    args:
        force_calculate: If True, forces recalculation even if a value is cached.

    Returns:
        The total number of records.
    """
    return self.get_number_of_records(force_calculate=force_calculate)

get_output_sample(n_rows=10)

Gets a sample of the data as a list of dictionaries.

This is typically used to display a preview of the data in a UI.

Parameters:

Name	Type	Description	Default
n_rows	int	The number of rows to sample.	10

Returns:

Type	Description
list[dict]	A list of dictionaries, where each dictionary represents a row.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_output_sample(self, n_rows: int = 10) -> list[dict]:
    """Gets a sample of the data as a list of dictionaries.

    This is typically used to display a preview of the data in a UI.

    Args:
        n_rows: The number of rows to sample.

    Returns:
        A list of dictionaries, where each dictionary represents a row.
    """
    if self.number_of_records > n_rows or self.number_of_records < 0:
        df = self.collect(n_rows)
    else:
        df = self.collect()
    return df.to_dicts()

get_record_count()

Returns a new FlowDataEngine with a single column 'number_of_records' containing the total number of records.

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_record_count(self) -> FlowDataEngine:
    """Returns a new FlowDataEngine with a single column 'number_of_records'
    containing the total number of records.

    Returns:
        A new `FlowDataEngine` instance.
    """
    return FlowDataEngine(self.data_frame.select(pl.len().alias("number_of_records")))

get_sample(n_rows=100, random=False, shuffle=False, seed=None, execution_location=None)

Gets a sample of rows from the DataFrame.

Parameters:

Name	Type	Description	Default
n_rows	int	The number of rows to sample.	100
random	bool	If True, performs random sampling. If False, takes the first n_rows.	False
shuffle	bool	If True (and random is True), shuffles the data before sampling.	False
seed	int	A random seed for reproducibility.	None
execution_location	ExecutionLocationsLiteral | None	Location which is used to calculate the size of the dataframe	None

Returns: A new FlowDataEngine instance containing the sampled data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_sample(
    self,
    n_rows: int = 100,
    random: bool = False,
    shuffle: bool = False,
    seed: int = None,
    execution_location: ExecutionLocationsLiteral | None = None,
) -> FlowDataEngine:
    """Gets a sample of rows from the DataFrame.

    Args:
        n_rows: The number of rows to sample.
        random: If True, performs random sampling. If False, takes the first n_rows.
        shuffle: If True (and `random` is True), shuffles the data before sampling.
        seed: A random seed for reproducibility.
        execution_location: Location which is used to calculate the size of the dataframe
    Returns:
        A new `FlowDataEngine` instance containing the sampled data.
    """
    logging.info(f"Getting sample of {n_rows} rows")
    if random:
        if self.lazy and self.external_source is not None:
            self.collect_external()

        if self.lazy and shuffle:
            sample_df = self.data_frame.collect(engine="streaming" if self._streamable else "auto").sample(
                n_rows, seed=seed, shuffle=shuffle
            )
        elif shuffle:
            sample_df = self.data_frame.sample(n_rows, seed=seed, shuffle=shuffle)
        else:
            if execution_location is None:
                execution_location = get_global_execution_location()
            n_rows = min(
                n_rows, self.get_number_of_records(calculate_in_worker_process=execution_location == "remote")
            )

            every_n_records = ceil(self.number_of_records / n_rows)
            sample_df = self.data_frame.gather_every(every_n_records)
    else:
        if self.external_source:
            self.collect(n_rows)
        sample_df = self.data_frame.head(n_rows)

    return FlowDataEngine(sample_df, schema=self.schema)

get_schema_column(col_name)

Retrieves the schema information for a single column by its name.

Parameters:

Name	Type	Description	Default
col_name	str	The name of the column to retrieve.	required

Returns:

Type	Description
FlowfileColumn	A FlowfileColumn object for the specified column, or None if not found.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_schema_column(self, col_name: str) -> FlowfileColumn:
    """Retrieves the schema information for a single column by its name.

    Args:
        col_name: The name of the column to retrieve.

    Returns:
        A `FlowfileColumn` object for the specified column, or `None` if not found.
    """
    for s in self.schema:
        if s.name == col_name:
            return s

get_select_inputs()

Gets SelectInput specifications for all columns in the current schema.

Returns:

Type	Description
SelectInputs	A SelectInputs object that can be used to configure selection or
SelectInputs	transformation operations.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_select_inputs(self) -> transform_schemas.SelectInputs:
    """Gets `SelectInput` specifications for all columns in the current schema.

    Returns:
        A `SelectInputs` object that can be used to configure selection or
        transformation operations.
    """
    return transform_schemas.SelectInputs(
        [transform_schemas.SelectInput(old_name=c.name, data_type=c.data_type) for c in self.schema]
    )

get_subset(n_rows=100)

Gets the first n_rows from the DataFrame.

Parameters:

Name	Type	Description	Default
n_rows	int	The number of rows to include in the subset.	100

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance containing the subset of data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def get_subset(self, n_rows: int = 100) -> FlowDataEngine:
    """Gets the first `n_rows` from the DataFrame.

    Args:
        n_rows: The number of rows to include in the subset.

    Returns:
        A new `FlowDataEngine` instance containing the subset of data.
    """
    if not self.lazy:
        return FlowDataEngine(self.data_frame.head(n_rows), calculate_schema_stats=True)
    else:
        return FlowDataEngine(self.data_frame.head(n_rows), calculate_schema_stats=True)

initialize_empty_fl()

Initializes an empty LazyFrame.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def initialize_empty_fl(self):
    """Initializes an empty LazyFrame."""
    self.data_frame = pl.LazyFrame()
    self.number_of_records = 0
    self._lazy = True

iter_batches(batch_size=1000, columns=None)

Iterates over the DataFrame in batches.

Parameters:

Name	Type	Description	Default
batch_size	int	The size of each batch.	1000
columns	list | tuple | str	A list of column names to include in the batches. If None, all columns are included.	None

Yields:

Type	Description
FlowDataEngine	A FlowDataEngine instance for each batch.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def iter_batches(
    self, batch_size: int = 1000, columns: list | tuple | str = None
) -> Generator[FlowDataEngine, None, None]:
    """Iterates over the DataFrame in batches.

    Args:
        batch_size: The size of each batch.
        columns: A list of column names to include in the batches. If None,
            all columns are included.

    Yields:
        A `FlowDataEngine` instance for each batch.
    """
    if columns:
        self.data_frame = self.data_frame.select(columns)
    self.lazy = False
    batches = self.data_frame.iter_slices(batch_size)
    for batch in batches:
        yield FlowDataEngine(batch)

join(join_input, auto_generate_selection, verify_integrity, other)

Performs a standard SQL-style join with another DataFrame.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def join(
    self,
    join_input: transform_schemas.JoinInput,
    auto_generate_selection: bool,
    verify_integrity: bool,
    other: FlowDataEngine,
) -> FlowDataEngine:
    """Performs a standard SQL-style join with another DataFrame."""
    join_manager = transform_schemas.JoinInputManager(join_input)
    _ensure_all_columns_have_select(left_cols=self.columns, right_cols=other.columns, manager=join_manager)
    join_manager.set_join_keys()
    for jk in join_manager.join_mapping:
        if jk.left_col not in {c.old_name for c in join_manager.left_select.renames}:
            join_manager.left_select.append(transform_schemas.SelectInput(jk.left_col, keep=False))
        if jk.right_col not in {c.old_name for c in join_manager.right_select.renames}:
            join_manager.right_select.append(transform_schemas.SelectInput(jk.right_col, keep=False))
    verify_join_select_integrity(join_manager.input, left_columns=self.columns, right_columns=other.columns)
    join_map_problems = get_join_map_problems(
        join_manager.input, left_columns=self.schema, right_columns=other.schema
    )
    if join_map_problems:
        raise Exception("Join is not valid: " + "; ".join(join_map_problems))

    if join_manager.how in ("semi", "anti"):
        # Semi/anti joins push the full left input downstream unchanged (all columns,
        # original order, no rename or drop); the right frame only supplies the join
        # keys for matching. Stale entries in left_select are therefore irrelevant here.
        left_on = [jm.left_col for jm in join_manager.join_mapping]
        right_on = [jm.right_col for jm in join_manager.join_mapping]
        right = other.data_frame.select(list(dict.fromkeys(right_on)))
        joined_df = self.data_frame.join(other=right, left_on=left_on, right_on=right_on, how=join_manager.how)
        # -1 = unknown (not 0): a 0 here reads as a real "empty result" count.
        return FlowDataEngine(joined_df, calculate_schema_stats=False, number_of_records=-1, streamable=False)

    if auto_generate_selection:
        join_manager.auto_rename()

    left = self.data_frame.select(join_manager.left_manager.get_select_cols()).rename(
        join_manager.left_manager.get_rename_table()
    )
    right = other.data_frame.select(join_manager.right_manager.get_select_cols()).rename(
        join_manager.right_manager.get_rename_table()
    )

    left, right, reverse_join_key_mapping = _handle_duplication_join_keys(left, right, join_manager)
    left, right = rename_df_table_for_join(left, right, join_manager.get_join_key_renames())
    if join_manager.how == "right":
        joined_df = right.join(
            other=left,
            left_on=join_manager.right_join_keys,
            right_on=join_manager.left_join_keys,
            how="left",
            suffix="",
        ).rename(reverse_join_key_mapping)
    else:
        joined_df = left.join(
            other=right,
            left_on=join_manager.left_join_keys,
            right_on=join_manager.right_join_keys,
            how=join_manager.how,
            suffix="",
        ).rename(reverse_join_key_mapping)

    left_cols_to_delete_after = [
        get_col_name_to_delete(col, "left")
        for col in join_manager.input.left_select.renames
        if not col.keep and col.is_available and col.join_key
    ]

    right_cols_to_delete_after = [
        get_col_name_to_delete(col, "right")
        for col in join_manager.input.right_select.renames
        if not col.keep
        and col.is_available
        and col.join_key
        and join_manager.how in ("left", "right", "inner", "cross", "outer")
    ]

    if len(right_cols_to_delete_after + left_cols_to_delete_after) > 0:
        joined_df = joined_df.drop(left_cols_to_delete_after + right_cols_to_delete_after)

    undo_join_key_remapping = get_undo_rename_mapping_join(join_manager)
    joined_df = joined_df.rename(undo_join_key_remapping)

    # -1 = unknown (not 0): a 0 here reads as a real "empty result" count.
    return FlowDataEngine(joined_df, calculate_schema_stats=False, number_of_records=-1, streamable=False)

known_record_count()

Returns the exact record count only when it is already known for free.

Sources, in order: a previously stored number_of_records (e.g. the count the worker sent along with a remote run result), or the height of an eager frame. Returns None otherwise — deliberately never falls back to get_number_of_records(), which on a lazy frame collects the whole plan to count it. Stored placeholders are not counts: the cloud readers stamp CLOUD_PLACEHOLDER_RECORD_COUNT and external-source engines carry a schema-time 0, so both report unknown here.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def known_record_count(self) -> int | None:
    """Returns the exact record count only when it is already known for free.

    Sources, in order: a previously stored ``number_of_records`` (e.g. the
    count the worker sent along with a remote run result), or the height of
    an eager frame. Returns None otherwise — deliberately never falls back
    to ``get_number_of_records()``, which on a lazy frame collects the whole
    plan to count it. Stored placeholders are not counts: the cloud readers
    stamp ``CLOUD_PLACEHOLDER_RECORD_COUNT`` and external-source engines
    carry a schema-time 0, so both report unknown here.
    """
    if self._external_source is not None:
        return None
    if self.number_of_records is not None and self.number_of_records >= 0:
        if self.number_of_records == CLOUD_PLACEHOLDER_RECORD_COUNT:
            return None
        return self.number_of_records
    if not self.lazy:
        return self.data_frame.height
    return None

make_unique(unique_input=None)

Gets the unique rows from the DataFrame.

Parameters:

Name	Type	Description	Default
unique_input	UniqueInput	A UniqueInput object specifying a subset of columns to consider for uniqueness and a strategy for keeping rows.	None

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with unique rows.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def make_unique(self, unique_input: transform_schemas.UniqueInput = None) -> FlowDataEngine:
    """Gets the unique rows from the DataFrame.

    Args:
        unique_input: A `UniqueInput` object specifying a subset of columns
            to consider for uniqueness and a strategy for keeping rows.

    Returns:
        A new `FlowDataEngine` instance with unique rows.
    """
    if unique_input is None or unique_input.columns is None:
        return FlowDataEngine(self.data_frame.unique())
    return FlowDataEngine(self.data_frame.unique(unique_input.columns, keep=unique_input.strategy))

output(output_fs, flow_id, node_id, execute_remote=False)

Writes the DataFrame to a local output file.

For remote-worker writes the caller (add_output._func) uses ExternalOutputWriter directly so the fetcher can be exposed on the node for cancellation; this method only handles the local path.

Parameters:

Name	Type	Description	Default
output_fs	OutputSettings	An OutputSettings object with details about the output file.	required
flow_id	int	The flow ID for tracking.	required
node_id	int | str	The node ID for tracking.	required
execute_remote	bool	Retained for signature compatibility; ignored.	False

Returns:

Type	Description
FlowDataEngine	The same FlowDataEngine instance for chaining.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def output(
    self, output_fs: input_schema.OutputSettings, flow_id: int, node_id: int | str, execute_remote: bool = False
) -> FlowDataEngine:
    """Writes the DataFrame to a local output file.

    For remote-worker writes the caller (``add_output._func``) uses
    ``ExternalOutputWriter`` directly so the fetcher can be exposed on
    the node for cancellation; this method only handles the local path.

    Args:
        output_fs: An `OutputSettings` object with details about the output file.
        flow_id: The flow ID for tracking.
        node_id: The node ID for tracking.
        execute_remote: Retained for signature compatibility; ignored.

    Returns:
        The same `FlowDataEngine` instance for chaining.
    """
    logger.info("Starting to write results locally")
    utils.local_write_output(
        self.data_frame,
        data_type=output_fs.file_type,
        path=output_fs.abs_file_path,
        write_mode=output_fs.write_mode,
        sheet_name=output_fs.sheet_name,
        delimiter=output_fs.delimiter,
        compression=output_fs.compression,
        flow_id=flow_id,
        node_id=node_id,
    )
    logger.info("Finished writing output")
    return self

random_sample(n=None, fraction=None, seed=None)

Takes a uniform random sample of rows without materialising the frame.

Polars exposes sample only on eager DataFrames, so the lazy equivalent is built from a shuffled row rank: each row draws a distinct rank from a random permutation of 0..len, and keeping the ranks below a threshold keeps a uniform subset. Nothing is collected and the row count is never queried, so the result stays a plan that ships to the worker like any other lazy transform — unlike :meth:random_split, which has to materialise because its outputs must share one permutation.

Sampling more rows than the frame holds yields the whole frame, and the original row order is preserved.

Parameters:

Name	Type	Description	Default
n	int | None	Number of rows to keep. Mutually exclusive with fraction.	None
fraction	float | None	Share of rows to keep, between 0 and 1. Mutually exclusive with n.	None
seed	int | None	Seed for a reproducible sample; None draws a fresh permutation on every execution.	None

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance containing the sampled rows.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def random_sample(
    self,
    n: int | None = None,
    fraction: float | None = None,
    seed: int | None = None,
) -> FlowDataEngine:
    """Takes a uniform random sample of rows without materialising the frame.

    Polars exposes ``sample`` only on eager DataFrames, so the lazy
    equivalent is built from a shuffled row rank: each row draws a distinct
    rank from a random permutation of ``0..len``, and keeping the ranks
    below a threshold keeps a uniform subset. Nothing is collected and the
    row count is never queried, so the result stays a plan that ships to
    the worker like any other lazy transform — unlike :meth:`random_split`,
    which has to materialise because its outputs must share one permutation.

    Sampling more rows than the frame holds yields the whole frame, and the
    original row order is preserved.

    Args:
        n: Number of rows to keep. Mutually exclusive with `fraction`.
        fraction: Share of rows to keep, between 0 and 1. Mutually exclusive with `n`.
        seed: Seed for a reproducible sample; None draws a fresh permutation
            on every execution.

    Returns:
        A new `FlowDataEngine` instance containing the sampled rows.
    """
    if (n is None) == (fraction is None):
        raise ValueError("Provide exactly one of n or fraction")
    df = self.data_frame if self.lazy else self.data_frame.lazy()
    threshold = (pl.len() * fraction).round().cast(pl.Int64) if fraction is not None else max(0, n)
    sampled = df.filter(pl.int_range(0, pl.len()).shuffle(seed=seed) < threshold)
    return FlowDataEngine(sampled, schema=self.schema, streamable=self._streamable)

random_split(splits, seed=None)

Randomly partition rows into N labeled groups (in-process).

Used by add_random_split when execution_location == "local" (WASM / no-worker). For remote mode the worker-offloaded variant :meth:random_split_external is used instead.

The shuffled frame is materialized once so that each output shares the same shuffle — otherwise every handle's .collect() would re-run the full shuffle+sort independently (O(N·n log n) instead of O(n log n)).

Parameters:

Name	Type	Description	Default
splits	list[tuple[str, float]]	Ordered (name, percentage) pairs; percentages must sum to 100 (validated upstream in NodeRandomSplit).	required
seed	int | None	Random seed; if None, one is generated per call.	None

Returns:

Type	Description
NamedOutputs	NamedOutputs mapping each split name to a fresh FlowDataEngine.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def random_split(
    self,
    splits: list[tuple[str, float]],
    seed: int | None = None,
) -> NamedOutputs:
    """Randomly partition rows into N labeled groups (in-process).

    Used by ``add_random_split`` when ``execution_location == "local"``
    (WASM / no-worker). For remote mode the worker-offloaded variant
    :meth:`random_split_external` is used instead.

    The shuffled frame is materialized once so that each output shares the
    same shuffle — otherwise every handle's ``.collect()`` would re-run the
    full shuffle+sort independently (O(N·n log n) instead of O(n log n)).

    Args:
        splits: Ordered (name, percentage) pairs; percentages must sum to
            100 (validated upstream in ``NodeRandomSplit``).
        seed: Random seed; if None, one is generated per call.

    Returns:
        ``NamedOutputs`` mapping each split name to a fresh ``FlowDataEngine``.
    """
    from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

    if seed is None:
        seed = random.randint(0, 2**31 - 1)
    shuffled = (
        self.data_frame.with_columns(pl.int_range(0, pl.len()).shuffle(seed=seed).alias("__split_rank__"))
        .sort("__split_rank__")
        .drop("__split_rank__")
        .collect()
    )
    total = shuffled.height
    out: dict[str, FlowDataEngine] = {}
    offset = 0
    for i, (name, percentage) in enumerate(splits):
        length = total - offset if i == len(splits) - 1 else int(round(total * percentage / 100.0))
        out[name] = FlowDataEngine(shuffled.slice(offset, max(0, length)).lazy())
        offset += length
    return NamedOutputs(out)

random_split_external(splits, seed=None, flow_id=-1, node_id=-1)

Worker-offloaded variant of :meth:random_split.

The shuffled frame is materialised once on flowfile_worker (never in this process). Each returned split is a lazy slice over the cached parquet, so downstream .collect() on a handle reads only that split's rows from disk.

Used by add_random_split when execution_location != "local"; the in-process path is :meth:random_split.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def random_split_external(
    self,
    splits: list[tuple[str, float]],
    seed: int | None = None,
    flow_id: int = -1,
    node_id: int | str = -1,
) -> NamedOutputs:
    """Worker-offloaded variant of :meth:`random_split`.

    The shuffled frame is materialised once on ``flowfile_worker`` (never
    in this process). Each returned split is a lazy ``slice`` over the
    cached parquet, so downstream ``.collect()`` on a handle reads only
    that split's rows from disk.

    Used by ``add_random_split`` when ``execution_location != "local"``;
    the in-process path is :meth:`random_split`.
    """
    import uuid

    from flowfile_core.flowfile.flow_data_engine.subprocess_operations import (
        ExternalDfFetcher,
    )
    from flowfile_core.flowfile.flow_node.multi_output import NamedOutputs

    if seed is None:
        seed = random.randint(0, 2**31 - 1)

    shuffled_lazy = (
        self.data_frame.with_columns(pl.int_range(0, pl.len()).shuffle(seed=seed).alias("__split_rank__"))
        .sort("__split_rank__")
        .drop("__split_rank__")
    )

    # Stable, unique file_ref — avoids id()-reuse collisions with cache().
    file_ref = f"random_split_{flow_id}_{node_id}_{seed}_{uuid.uuid4().hex}"
    edf = ExternalDfFetcher(
        lf=shuffled_lazy,
        file_ref=file_ref,
        wait_on_completion=True,
        flow_id=flow_id,
        node_id=node_id,
    )
    cached_lf = edf.get_result()
    if not isinstance(cached_lf, pl.LazyFrame):
        raise RuntimeError(f"random_split_external: worker did not return a LazyFrame (got {type(cached_lf)!r})")

    # Cheap — reads parquet footer metadata, not row data.
    total = cached_lf.select(pl.len()).collect()[0, 0]

    out: dict[str, FlowDataEngine] = {}
    offset = 0
    for i, (name, percentage) in enumerate(splits):
        length = total - offset if i == len(splits) - 1 else int(round(total * percentage / 100.0))
        length = max(0, length)
        out[name] = FlowDataEngine(
            cached_lf.slice(offset, length),
            number_of_records=length,
            schema=self.schema,
        )
        offset += length
    return NamedOutputs(out)

reorganize_order(column_order)

Reorganizes columns into a specified order.

Parameters:

Name	Type	Description	Default
column_order	list[str]	A list of column names in the desired order.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the columns reordered.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def reorganize_order(self, column_order: list[str]) -> FlowDataEngine:
    """Reorganizes columns into a specified order.

    Args:
        column_order: A list of column names in the desired order.

    Returns:
        A new `FlowDataEngine` instance with the columns reordered.
    """
    df = self.data_frame.select(column_order)
    schema = sorted(self.schema, key=lambda x: column_order.index(x.column_name))
    return FlowDataEngine(df, schema=schema, number_of_records=self.number_of_records)

resolve_dynamic_rename_map(columns, settings, first_row_values=None) staticmethod

Compute the {old_name: new_name} map for a dynamic-rename operation.

Pure function — takes the incoming schema as (name, data_type_group) tuples (where data_type_group is FlowfileColumn.data_type_group, e.g. "Numeric", "String", "Date", …) and the user's settings, and returns the rename map. Raises ValueError if the rule would produce duplicate column names.

Parameters:

Name	Type	Description	Default
columns	list[tuple[str, str]]	Incoming schema as (column_name, data_type_group) tuples, in order.	required
settings	DynamicRenameInput	The dynamic rename configuration.	required
first_row_values	dict[str, Any] | None	First-row values keyed by original column name. Required for "first_row" mode to produce a real rename map; when omitted in "first_row" mode the result is an empty map (schema-only preview).	None

Returns:

Type	Description
dict[str, str]	A dict mapping original column name to new column name. No-op renames are
dict[str, str]	omitted, so the result is safe to pass directly to pl.DataFrame.rename.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

@staticmethod
def resolve_dynamic_rename_map(
    columns: list[tuple[str, str]],
    settings: transform_schemas.DynamicRenameInput,
    first_row_values: dict[str, Any] | None = None,
) -> dict[str, str]:
    """Compute the `{old_name: new_name}` map for a dynamic-rename operation.

    Pure function — takes the incoming schema as `(name, data_type_group)` tuples
    (where `data_type_group` is `FlowfileColumn.data_type_group`, e.g. `"Numeric"`,
    `"String"`, `"Date"`, …) and the user's settings, and returns the rename map.
    Raises `ValueError` if the rule would produce duplicate column names.

    Args:
        columns: Incoming schema as `(column_name, data_type_group)` tuples, in order.
        settings: The dynamic rename configuration.
        first_row_values: First-row values keyed by original column name. Required
            for `"first_row"` mode to produce a real rename map; when omitted in
            `"first_row"` mode the result is an empty map (schema-only preview).

    Returns:
        A dict mapping original column name to new column name. No-op renames are
        omitted, so the result is safe to pass directly to `pl.DataFrame.rename`.
    """
    targets = FlowDataEngine._select_rename_targets(columns, settings)
    new_names = FlowDataEngine._compute_renamed_names(targets, settings, first_row_values=first_row_values)
    rename_map = {old: new for old, new in zip(targets, new_names, strict=True) if old != new}
    FlowDataEngine._assert_rename_has_no_duplicates(rename_map, columns)
    return rename_map

save(path, data_type='parquet')

Saves the DataFrame to a file in a separate thread.

Parameters:

Name	Type	Description	Default
path	str	The file path to save to.	required
data_type	str	The format to save in (e.g., 'parquet', 'csv').	'parquet'

Returns:

Type	Description
Future	A loky.Future object representing the asynchronous save operation.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def save(self, path: str, data_type: str = "parquet") -> Future:
    """Saves the DataFrame to a file in a separate thread.

    Args:
        path: The file path to save to.
        data_type: The format to save in (e.g., 'parquet', 'csv').

    Returns:
        A `loky.Future` object representing the asynchronous save operation.
    """
    estimated_size = deepcopy(self.get_estimated_file_size() * 4)
    df = deepcopy(self.data_frame)
    return write_threaded(_df=df, path=path, data_type=data_type, estimated_size=estimated_size)

select_columns(list_select)

Selects a subset of columns from the DataFrame.

Parameters:

Name	Type	Description	Default
list_select	list[str] | tuple[str] | str	A list, tuple, or single string of column names to select.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance containing only the selected columns.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def select_columns(self, list_select: list[str] | tuple[str] | str) -> FlowDataEngine:
    """Selects a subset of columns from the DataFrame.

    Args:
        list_select: A list, tuple, or single string of column names to select.

    Returns:
        A new `FlowDataEngine` instance containing only the selected columns.
    """
    if isinstance(list_select, str):
        list_select = [list_select]

    idx_to_keep = [self.cols_idx.get(c) for c in list_select]
    selects = [ls for ls, id_to_keep in zip(list_select, idx_to_keep, strict=False) if id_to_keep is not None]
    new_schema = [self.schema[i] for i in idx_to_keep if i is not None]

    return FlowDataEngine(
        self.data_frame.select(selects),
        number_of_records=self.number_of_records,
        schema=new_schema,
        streamable=self._streamable,
    )

set_streamable(streamable=False)

Sets whether DataFrame operations should be streamable.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def set_streamable(self, streamable: bool = False):
    """Sets whether DataFrame operations should be streamable."""
    self._streamable = streamable

shallow_copy()

Cheap de-aliasing wrapper around the same (immutable) Polars frame.

Shares the frame and the cached schema, but owns its own mutable flags (_lazy, _streamable, number_of_records, _schema), so a consumer handed this copy can never mutate an engine shared with sibling consumers. Collect-free: forwarding number_of_records and the cached schema skips both pl.len() and collect_schema() in init (the schema fallback only fires when _schema is unset, and is metadata-only). Deliberately does not carry external_source: memoized results are materialized before they are shared, so the plain frame is the whole result.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def shallow_copy(self) -> FlowDataEngine:
    """Cheap de-aliasing wrapper around the same (immutable) Polars frame.

    Shares the frame and the cached schema, but owns its own mutable flags
    (_lazy, _streamable, number_of_records, _schema), so a consumer handed
    this copy can never mutate an engine shared with sibling consumers.
    Collect-free: forwarding number_of_records and the cached schema skips
    both pl.len() and collect_schema() in __init__ (the schema fallback only
    fires when _schema is unset, and is metadata-only). Deliberately does
    not carry external_source: memoized results are materialized before
    they are shared, so the plain frame is the whole result.
    """
    return FlowDataEngine(
        self.data_frame,
        name=self.name,
        optimize_memory=self._optimize_memory,
        schema=self._schema,
        number_of_records=self.number_of_records,
        streamable=self._streamable,
        number_of_records_callback=self._number_of_records_callback,
        data_callback=self._data_callback,
    )

solve_graph(graph_solver_input)

Solves a graph problem represented by 'from' and 'to' columns.

This is used for operations like finding connected components in a graph.

Parameters:

Name	Type	Description	Default
graph_solver_input	GraphSolverInput	A GraphSolverInput object defining the source, destination, and output column names.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the solved graph data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def solve_graph(self, graph_solver_input: transform_schemas.GraphSolverInput) -> FlowDataEngine:
    """Solves a graph problem represented by 'from' and 'to' columns.

    This is used for operations like finding connected components in a graph.

    Args:
        graph_solver_input: A `GraphSolverInput` object defining the source,
            destination, and output column names.

    Returns:
        A new `FlowDataEngine` instance with the solved graph data.
    """
    lf = self.data_frame.with_columns(
        graph_solver(graph_solver_input.col_from, graph_solver_input.col_to).alias(
            graph_solver_input.output_column_name
        )
    )
    return FlowDataEngine(lf)

split(split_input)

Splits a column's text values into multiple rows based on a delimiter.

This operation is often referred to as "exploding" the DataFrame, as it increases the number of rows.

Parameters:

Name	Type	Description	Default
split_input	TextToRowsInput	A TextToRowsInput object specifying the column to split, the delimiter, and the output column name.	required

Returns:

Type	Description
FlowDataEngine	A new FlowDataEngine instance with the exploded rows.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def split(self, split_input: transform_schemas.TextToRowsInput) -> FlowDataEngine:
    """Splits a column's text values into multiple rows based on a delimiter.

    This operation is often referred to as "exploding" the DataFrame, as it
    increases the number of rows.

    Args:
        split_input: A `TextToRowsInput` object specifying the column to split,
            the delimiter, and the output column name.

    Returns:
        A new `FlowDataEngine` instance with the exploded rows.
    """
    output_column_name = (
        split_input.output_column_name if split_input.output_column_name else split_input.column_to_split
    )

    split_value = (
        split_input.split_fixed_value if split_input.split_by_fixed_value else pl.col(split_input.split_by_column)
    )

    df = self.data_frame.with_columns(
        pl.col(split_input.column_to_split).str.split(by=split_value).alias(output_column_name)
    ).explode(output_column_name)

    return FlowDataEngine(df)

start_fuzzy_join(fuzzy_match_input, other, file_ref, flow_id=-1, node_id=-1)

Starts a fuzzy join operation in a background process.

This method prepares the data and initiates the fuzzy matching in a separate process, returning a tracker object immediately.

Parameters:

Name	Type	Description	Default
fuzzy_match_input	FuzzyMatchInput	A FuzzyMatchInput object with the matching parameters.	required
other	FlowDataEngine	The right FlowDataEngine to join with.	required
file_ref	str	A reference string for temporary files.	required
flow_id	int	The flow ID for tracking.	-1
node_id	int | str	The node ID for tracking.	-1

Returns:

Type	Description
ExternalFuzzyMatchFetcher	An ExternalFuzzyMatchFetcher object that can be used to track the
ExternalFuzzyMatchFetcher	progress and retrieve the result of the fuzzy join.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def start_fuzzy_join(
    self,
    fuzzy_match_input: transform_schemas.FuzzyMatchInput,
    other: FlowDataEngine,
    file_ref: str,
    flow_id: int = -1,
    node_id: int | str = -1,
) -> ExternalFuzzyMatchFetcher:
    """Starts a fuzzy join operation in a background process.

    This method prepares the data and initiates the fuzzy matching in a
    separate process, returning a tracker object immediately.

    Args:
        fuzzy_match_input: A `FuzzyMatchInput` object with the matching parameters.
        other: The right `FlowDataEngine` to join with.
        file_ref: A reference string for temporary files.
        flow_id: The flow ID for tracking.
        node_id: The node ID for tracking.

    Returns:
        An `ExternalFuzzyMatchFetcher` object that can be used to track the
        progress and retrieve the result of the fuzzy join.
    """
    fuzzy_match_input_manager = transform_schemas.FuzzyMatchInputManager(fuzzy_match_input)
    left_df, right_df = prepare_for_fuzzy_match(
        left=self, right=other, fuzzy_match_input_manager=fuzzy_match_input_manager
    )

    return ExternalFuzzyMatchFetcher(
        left_df,
        right_df,
        fuzzy_maps=fuzzy_match_input_manager.fuzzy_maps,
        file_ref=file_ref + "_fm",
        wait_on_completion=False,
        flow_id=flow_id,
        node_id=node_id,
    )

to_arrow()

Converts the DataFrame to a PyArrow Table.

This method triggers a .collect() call if the data is lazy, then converts the resulting eager DataFrame into a pyarrow.Table.

Returns:

Type	Description
Table	A pyarrow.Table instance representing the data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_arrow(self) -> PaTable:
    """Converts the DataFrame to a PyArrow Table.

    This method triggers a `.collect()` call if the data is lazy,
    then converts the resulting eager DataFrame into a `pyarrow.Table`.

    Returns:
        A `pyarrow.Table` instance representing the data.
    """
    if self.lazy:
        return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_arrow()
    else:
        return self.data_frame.to_arrow()

to_cloud_storage_obj(settings)

Writes the DataFrame to an object in cloud storage.

This method supports writing to various cloud storage providers like AWS S3, Azure Data Lake Storage, and Google Cloud Storage.

Parameters:

Name	Type	Description	Default
settings	CloudStorageWriteSettingsInternal	A CloudStorageWriteSettingsInternal object containing connection details, file format, and write options.	required

Raises:

Type	Description
ValueError	If the specified file format is not supported for writing.
NotImplementedError	If the 'append' write mode is used with an unsupported format.
Exception	If the write operation to cloud storage fails for any reason.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_cloud_storage_obj(self, settings: cloud_storage_schemas.CloudStorageWriteSettingsInternal):
    """Writes the DataFrame to an object in cloud storage.

    This method supports writing to various cloud storage providers like AWS S3,
    Azure Data Lake Storage, and Google Cloud Storage.

    Args:
        settings: A `CloudStorageWriteSettingsInternal` object containing connection
            details, file format, and write options.

    Raises:
        ValueError: If the specified file format is not supported for writing.
        NotImplementedError: If the 'append' write mode is used with an unsupported format.
        Exception: If the write operation to cloud storage fails for any reason.
    """
    connection = settings.connection
    write_settings = settings.write_settings
    logger.info(f"Writing to {connection.storage_type} storage: {write_settings.resource_path}")

    storage_options = CloudStorageReader.get_storage_options(connection)
    credential_provider = CloudStorageReader.get_credential_provider(connection)
    use_pyarrow = CloudStorageReader.use_pyarrow_for_gcs(connection)

    write_to_cloud(
        df=self.data_frame,
        resource_path=write_settings.resource_path,
        storage_options=storage_options,
        file_format=write_settings.file_format,
        write_mode=write_settings.write_mode,
        compression=write_settings.parquet_compression,
        separator=write_settings.csv_delimiter,
        partition_by=write_settings.partition_by,
        credential_provider=credential_provider,
        use_pyarrow=use_pyarrow,
        logger=logger,
    )

to_database_obj(*, database_type, uri, table_name, if_exists)

Writes the DataFrame to a SQL database in-process (local execution path).

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_database_obj(self, *, database_type: str, uri: str, table_name: str, if_exists: str) -> None:
    """Writes the DataFrame to a SQL database in-process (local execution path)."""
    logger.info(f"Writing to {database_type} table {table_name}")
    write_dataframe_to_database(
        self.collect(),
        database_type=database_type,
        uri=uri,
        table_name=table_name,
        if_exists=if_exists,
    )

to_dict()

Converts the DataFrame to a Python dictionary of columns.

Each key in the dictionary is a column name, and the corresponding value is a list of the data in that column.

Returns:

Type	Description
dict[str, list]	A dictionary mapping column names to lists of their values.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_dict(self) -> dict[str, list]:
    """Converts the DataFrame to a Python dictionary of columns.

    Each key in the dictionary is a column name, and the corresponding value
    is a list of the data in that column.

    Returns:
        A dictionary mapping column names to lists of their values.
    """
    if self.lazy:
        return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_dict(as_series=False)
    else:
        return self.data_frame.to_dict(as_series=False)

to_pylist()

Converts the DataFrame to a list of Python dictionaries.

Returns:

Type	Description
list[dict]	A list where each item is a dictionary representing a row.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_pylist(self) -> list[dict]:
    """Converts the DataFrame to a list of Python dictionaries.

    Returns:
        A list where each item is a dictionary representing a row.
    """
    if self.lazy:
        return self.data_frame.collect(engine="streaming" if self._streamable else "auto").to_dicts()
    return self.data_frame.to_dicts()

to_raw_data()

Converts the DataFrame to a RawData schema object.

Returns:

Type	Description
RawData	An input_schema.RawData object containing the schema and data.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def to_raw_data(self) -> input_schema.RawData:
    """Converts the DataFrame to a `RawData` schema object.

    Returns:
        An `input_schema.RawData` object containing the schema and data.
    """
    columns = [c.get_minimal_field_info() for c in self.schema]
    data = list(self.to_dict().values())
    return input_schema.RawData(columns=columns, data=data)

unpivot(unpivot_input)

Converts the DataFrame from a wide to a long format.

This is the inverse of a pivot operation, taking columns and transforming them into variable and value rows.

Parameters:

Name	Type	Description	Default
unpivot_input	UnpivotInput	An UnpivotInput object specifying which columns to unpivot and which to keep as index columns.	required

Returns:

Type	Description
FlowDataEngine	A new, unpivoted FlowDataEngine instance.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py

def unpivot(self, unpivot_input: transform_schemas.UnpivotInput) -> FlowDataEngine:
    """Converts the DataFrame from a wide to a long format.

    This is the inverse of a pivot operation, taking columns and transforming
    them into `variable` and `value` rows.

    Args:
        unpivot_input: An `UnpivotInput` object specifying which columns to
            unpivot and which to keep as index columns.

    Returns:
        A new, unpivoted `FlowDataEngine` instance.
    """
    lf = self.data_frame

    if unpivot_input.data_type_selector_expr is not None:
        result = lf.unpivot(on=unpivot_input.data_type_selector_expr(), index=unpivot_input.index_columns)
    elif unpivot_input.value_columns is not None:
        result = lf.unpivot(on=unpivot_input.value_columns, index=unpivot_input.index_columns)
    else:
        result = lf.unpivot()

    return FlowDataEngine(result)

FlowfileColumn

The FlowfileColumn holds the schema and metadata for a single column managed by the FlowDataEngine.

flowfile_core.flowfile.flow_data_engine.flow_file_column.main.FlowfileColumn dataclass

Methods:

Name	Description
__repr__	Provides a concise, developer-friendly representation of the object.
__str__	Provides a detailed, readable summary of the column's metadata.
apply_statistics	Applies exactly-computed statistics onto this column.

Attributes:

Name	Type	Description
stats_applied	bool	True once exact stats were written via apply_statistics.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_file_column/main.py

@dataclass
class FlowfileColumn:
    column_name: str
    data_type: str
    size: int
    max_value: str
    min_value: str
    col_index: int
    number_of_empty_values: int
    number_of_unique_values: int
    example_values: str
    data_type_group: ReadableDataTypeGroup
    __sql_type: Any | None
    __is_unique: bool | None
    __nullable: bool | None
    __has_values: bool | None
    average_value: str | None
    __perc_unique: float | None
    __stats_applied: bool

    def __init__(self, polars_type: PlType):
        self.data_type = convert_pl_type_to_string(polars_type.pl_datatype)
        self.size = polars_type.count - polars_type.null_count
        self.max_value = polars_type.max
        self.min_value = polars_type.min
        self.number_of_unique_values = polars_type.n_unique
        self.number_of_empty_values = polars_type.null_count
        self.example_values = polars_type.examples
        self.column_name = polars_type.column_name
        self.average_value = polars_type.mean
        self.col_index = polars_type.col_index
        self.__has_values = None
        self.__nullable = None
        self.__is_unique = None
        self.__sql_type = None
        self.__perc_unique = None
        self.__stats_applied = False
        self.data_type_group = self.get_readable_datatype_group()

    def __repr__(self):
        """
        Provides a concise, developer-friendly representation of the object.
        Ideal for debugging and console inspection.
        """
        return (
            f"FlowfileColumn(name='{self.column_name}', "
            f"type={self.data_type}, "
            f"size={self.size}, "
            f"nulls={self.number_of_empty_values})"
        )

    def __str__(self):
        """
        Provides a detailed, readable summary of the column's metadata.
        It conditionally omits any attribute that is None, ensuring a clean output.
        """
        # --- Header (Always Shown) ---
        header = f"<FlowfileColumn: '{self.column_name}'>"
        lines = []

        # --- Core Attributes (Conditionally Shown) ---
        if self.data_type is not None:
            lines.append(f"  Type: {self.data_type}")
        if self.size is not None:
            lines.append(f"  Non-Nulls: {self.size}")

        if self.size is not None and self.number_of_empty_values is not None:
            total_entries = self.size + self.number_of_empty_values
            if total_entries > 0:
                null_perc = (self.number_of_empty_values / total_entries) * 100
                null_info = f"{self.number_of_empty_values} ({null_perc:.1f}%)"
            else:
                null_info = "0 (0.0%)"
            lines.append(f"  Nulls: {null_info}")

        if self.number_of_unique_values is not None:
            lines.append(f"  Unique: {self.number_of_unique_values}")

        # --- Conditional Stats Section ---
        stats = []
        if self.min_value is not None:
            stats.append(f"    Min: {self.min_value}")
        if self.max_value is not None:
            stats.append(f"    Max: {self.max_value}")
        if self.average_value is not None:
            stats.append(f"    Mean: {self.average_value}")

        if stats:
            lines.append("  Stats:")
            lines.extend(stats)

        # --- Conditional Examples Section ---
        if self.example_values:
            example_str = str(self.example_values)
            if len(example_str) > 70:
                example_str = example_str[:67] + "..."
            lines.append(f"  Examples: {example_str}")

        return f"{header}\n" + "\n".join(lines)

    @classmethod
    def create_from_polars_type(cls, polars_type: PlType, **kwargs) -> "FlowfileColumn":
        for k, v in kwargs.items():
            if hasattr(polars_type, k):
                setattr(polars_type, k, v)
        return cls(polars_type)

    @classmethod
    def from_input(cls, column_name: str, data_type: str, **kwargs) -> "FlowfileColumn":
        pl_type = cast_str_to_polars_type(data_type)
        if pl_type is not None:
            data_type = pl_type
        return cls(PlType(column_name=column_name, pl_datatype=data_type, **kwargs))

    @classmethod
    def create_from_polars_dtype(cls, column_name: str, data_type: pl.DataType, **kwargs):
        return cls(PlType(column_name=column_name, pl_datatype=data_type, **kwargs))

    def get_minimal_field_info(self) -> input_schema.MinimalFieldInfo:
        return input_schema.MinimalFieldInfo(name=self.column_name, data_type=self.data_type)

    @classmethod
    def create_from_minimal_field_info(cls, minimal_field_info: input_schema.MinimalFieldInfo) -> "FlowfileColumn":
        return cls.from_input(column_name=minimal_field_info.name, data_type=minimal_field_info.data_type)

    @property
    def is_unique(self) -> bool:
        if self.__is_unique is None:
            if self.has_values:
                self.__is_unique = self.number_of_unique_values == self.number_of_filled_values
            else:
                self.__is_unique = False
        return self.__is_unique

    @property
    def perc_unique(self) -> float:
        if self.__perc_unique is None:
            self.__perc_unique = self.number_of_unique_values / self.number_of_filled_values
        return self.__perc_unique

    @property
    def has_values(self) -> bool:
        if not self.__has_values:
            self.__has_values = self.number_of_unique_values > 0
        return self.__has_values

    @property
    def number_of_filled_values(self):
        return self.size

    @property
    def nullable(self):
        if self.__nullable is None:
            self.__nullable = self.number_of_empty_values > 0
        return self.__nullable

    @property
    def name(self):
        return self.column_name

    def apply_statistics(
        self,
        total_rows: int,
        null_count: int,
        n_unique: int | None = None,
        min_value: Any | None = None,
        max_value: Any | None = None,
        average_value: Any | None = None,
    ) -> None:
        """Applies exactly-computed statistics onto this column.

        Replaces the schema-time sentinels with real values (from the on-demand
        column_stats pass) and resets the lazily-derived caches (is_unique,
        perc_unique, …) so they re-derive from the new counts. Every stat field
        is overwritten: a value not in this batch (count-only retry, an
        unsupported dtype) becomes unknown again instead of surviving from an
        earlier state. Values are stringified and bounded so a long-text
        min/max can't blow up payloads.
        """
        self.size = total_rows - null_count
        self.number_of_empty_values = null_count
        self.number_of_unique_values = -1 if n_unique is None else n_unique
        self.min_value = None if min_value is None else str(min_value)[:MAX_STAT_VALUE_LENGTH]
        self.max_value = None if max_value is None else str(max_value)[:MAX_STAT_VALUE_LENGTH]
        if isinstance(average_value, float):
            average_value = round(average_value, 4)
        self.average_value = None if average_value is None else str(average_value)[:MAX_STAT_VALUE_LENGTH]
        self.__stats_applied = True
        self.__is_unique = None
        self.__perc_unique = None
        self.__has_values = None
        self.__nullable = None

    @property
    def stats_applied(self) -> bool:
        """True once exact stats were written via apply_statistics."""
        return self.__stats_applied

    @staticmethod
    def _known_count(value: int | None) -> int | None:
        return None if value is None or value < 0 else value

    def get_column_repr(self):
        # Sentinel stat values (-1 counts, "" bounds — never computed) go out as
        # None so the wire model doesn't dress "unknown" up as data. Once exact
        # stats were applied, an empty string is a real bound ("" is a legal min
        # of a String column) and ships as-is.
        null_count = self._known_count(self.number_of_empty_values)
        empty_is_sentinel = not self.__stats_applied

        def bound_repr(value):
            if value is None or (value == "" and empty_is_sentinel):
                return None
            return str(value)

        return dict(
            name=self.name,
            size=None if null_count is None else self.size,
            data_type=str(self.data_type),
            data_type_group=self.data_type_group,
            has_values=self.has_values,
            is_unique=self.is_unique,
            max_value=bound_repr(self.max_value),
            min_value=bound_repr(self.min_value),
            number_of_unique_values=self._known_count(self.number_of_unique_values),
            number_of_filled_values=None if null_count is None else self.number_of_filled_values,
            number_of_empty_values=null_count,
            average_value=None if self.average_value in (None, "") else str(self.average_value),
        )

    def generic_datatype(self) -> DataTypeGroup:
        if self.data_type in ("Utf8", "VARCHAR", "CHAR", "NVARCHAR", "String"):
            return "str"
        elif self.data_type in (
            "fixed_decimal",
            "decimal",
            "float",
            "integer",
            "boolean",
            "double",
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "Binary",
            "Boolean",
            "Uint8",
            "Uint16",
            "Uint32",
            "Uint64",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
        ):
            return "numeric"
        elif self.data_type in ("datetime", "date", "Date", "Datetime", "Time"):
            return "date"
        else:
            return "str"

    def get_readable_datatype_group(self) -> ReadableDataTypeGroup:
        # Parameterized dtypes stringify with their inner type, e.g. "List(Int64)"
        # or "Datetime(time_unit='us', ...)" — match on the base token.
        base = self.data_type.split("(", 1)[0]
        if base in ("Utf8", "VARCHAR", "CHAR", "NVARCHAR", "String"):
            return "String"
        elif base in ("boolean", "Boolean"):
            return "Boolean"
        elif base in ("binary", "Binary"):
            return "Binary"
        elif base in ("list", "struct", "array", "List", "Struct", "Array"):
            return "Complex"
        elif base in (
            "fixed_decimal",
            "decimal",
            "float",
            "integer",
            "double",
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "Uint8",
            "Uint16",
            "Uint32",
            "Uint64",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
        ):
            return "Numeric"
        elif base in ("datetime", "date", "Date", "Datetime", "Time", "time", "Duration", "duration"):
            return "Date"
        else:
            return "Other"

    def get_polars_type(self) -> PlType:
        pl_datatype = cast_str_to_polars_type(self.data_type)
        pl_type = PlType(pl_datatype=pl_datatype, **self.__dict__)
        return pl_type

    def update_type_from_polars_type(self, pl_type: PlType):
        self.data_type = str(pl_type.pl_datatype.base_type())

stats_applied property

True once exact stats were written via apply_statistics.

__repr__()

Provides a concise, developer-friendly representation of the object. Ideal for debugging and console inspection.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_file_column/main.py

def __repr__(self):
    """
    Provides a concise, developer-friendly representation of the object.
    Ideal for debugging and console inspection.
    """
    return (
        f"FlowfileColumn(name='{self.column_name}', "
        f"type={self.data_type}, "
        f"size={self.size}, "
        f"nulls={self.number_of_empty_values})"
    )

__str__()

Provides a detailed, readable summary of the column's metadata. It conditionally omits any attribute that is None, ensuring a clean output.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_file_column/main.py

def __str__(self):
    """
    Provides a detailed, readable summary of the column's metadata.
    It conditionally omits any attribute that is None, ensuring a clean output.
    """
    # --- Header (Always Shown) ---
    header = f"<FlowfileColumn: '{self.column_name}'>"
    lines = []

    # --- Core Attributes (Conditionally Shown) ---
    if self.data_type is not None:
        lines.append(f"  Type: {self.data_type}")
    if self.size is not None:
        lines.append(f"  Non-Nulls: {self.size}")

    if self.size is not None and self.number_of_empty_values is not None:
        total_entries = self.size + self.number_of_empty_values
        if total_entries > 0:
            null_perc = (self.number_of_empty_values / total_entries) * 100
            null_info = f"{self.number_of_empty_values} ({null_perc:.1f}%)"
        else:
            null_info = "0 (0.0%)"
        lines.append(f"  Nulls: {null_info}")

    if self.number_of_unique_values is not None:
        lines.append(f"  Unique: {self.number_of_unique_values}")

    # --- Conditional Stats Section ---
    stats = []
    if self.min_value is not None:
        stats.append(f"    Min: {self.min_value}")
    if self.max_value is not None:
        stats.append(f"    Max: {self.max_value}")
    if self.average_value is not None:
        stats.append(f"    Mean: {self.average_value}")

    if stats:
        lines.append("  Stats:")
        lines.extend(stats)

    # --- Conditional Examples Section ---
    if self.example_values:
        example_str = str(self.example_values)
        if len(example_str) > 70:
            example_str = example_str[:67] + "..."
        lines.append(f"  Examples: {example_str}")

    return f"{header}\n" + "\n".join(lines)

apply_statistics(total_rows, null_count, n_unique=None, min_value=None, max_value=None, average_value=None)

Applies exactly-computed statistics onto this column.

Replaces the schema-time sentinels with real values (from the on-demand column_stats pass) and resets the lazily-derived caches (is_unique, perc_unique, …) so they re-derive from the new counts. Every stat field is overwritten: a value not in this batch (count-only retry, an unsupported dtype) becomes unknown again instead of surviving from an earlier state. Values are stringified and bounded so a long-text min/max can't blow up payloads.

Source code in flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_file_column/main.py

def apply_statistics(
    self,
    total_rows: int,
    null_count: int,
    n_unique: int | None = None,
    min_value: Any | None = None,
    max_value: Any | None = None,
    average_value: Any | None = None,
) -> None:
    """Applies exactly-computed statistics onto this column.

    Replaces the schema-time sentinels with real values (from the on-demand
    column_stats pass) and resets the lazily-derived caches (is_unique,
    perc_unique, …) so they re-derive from the new counts. Every stat field
    is overwritten: a value not in this batch (count-only retry, an
    unsupported dtype) becomes unknown again instead of surviving from an
    earlier state. Values are stringified and bounded so a long-text
    min/max can't blow up payloads.
    """
    self.size = total_rows - null_count
    self.number_of_empty_values = null_count
    self.number_of_unique_values = -1 if n_unique is None else n_unique
    self.min_value = None if min_value is None else str(min_value)[:MAX_STAT_VALUE_LENGTH]
    self.max_value = None if max_value is None else str(max_value)[:MAX_STAT_VALUE_LENGTH]
    if isinstance(average_value, float):
        average_value = round(average_value, 4)
    self.average_value = None if average_value is None else str(average_value)[:MAX_STAT_VALUE_LENGTH]
    self.__stats_applied = True
    self.__is_unique = None
    self.__perc_unique = None
    self.__has_values = None
    self.__nullable = None

---

Data Modeling (Schemas)

This section documents the Pydantic models that define the structure of settings and data.

schemas

flowfile_core.schemas.schemas

Classes:

Name	Description
CreateGroupRequest	Body for POST /editor/create_group/. Bounds are optional; computed from members if omitted.
FlowGraphConfig	Configuration model for a flow graph's basic properties.
FlowInformation	Represents the complete state of a flow, including settings, nodes, and connections.
FlowSettings	Extends FlowGraphConfig with additional operational settings for a flow.
FlowSettingsResponse	FlowSettings plus runtime-only fields for API responses. Not persisted.
FlowfileData	Root model for flowfile serialization (YAML/JSON).
FlowfileGroup	Serialized representation of a visual node group (YAML/JSON).
FlowfileInputConnection	One keyed input edge of a dynamic-input node (per-edge target handle).
FlowfileNode	Node representation for flowfile serialization (YAML/JSON).
FlowfileSettings	Settings for flowfile serialization (YAML/JSON).
GroupBounds	Axis-aligned bounds of a group box, in absolute canvas coordinates.
GroupBoundsUpdate	A single group's new absolute bounds.
GroupInformation	Runtime representation of a visual node group (stored in FlowGraph._groups).
GroupMembershipRequest	Body for adding/removing nodes from a group.
NodeConnection	Represents a connection between two nodes in the flow.
NodeDefault	Defines default properties for a node type.
NodeEdge	Represents a connection (edge) between two nodes in the frontend.
NodeInformation	Stores the state and configuration of a specific node instance within a flow.
NodeInput	Represents a node as it is received from the frontend, including position.
NodePositionUpdate	A single node's new absolute canvas position.
NodeTag	Controlled vocabulary of palette search keywords.
NodeTemplate	Defines the template for a node type, specifying its UI and functional characteristics.
RawLogInput	Schema for a raw log message.
UpdateGroupRequest	Body for POST /editor/update_group/. All fields optional -> partial update.
UpdateLayoutRequest	Batch persistence of dragged node positions and/or group bounds (one drag-end -> one call).
VueFlowInput	Represents the complete graph structure from the Vue-based frontend.

Functions:

Name	Description
get_global_execution_location	Calculates the default execution location based on the global settings
get_settings_class_for_node_type	Get the settings class for a node type, supporting both standard and user-defined nodes.

CreateGroupRequest pydantic-model

Bases: BaseModel

Body for POST /editor/create_group/. Bounds are optional; computed from members if omitted.

Show JSON schema:

{
  "description": "Body for POST /editor/create_group/. Bounds are optional; computed from members if omitted.",
  "properties": {
    "node_ids": {
      "items": {
        "type": "integer"
      },
      "title": "Node Ids",
      "type": "array"
    },
    "name": {
      "default": "Group",
      "title": "Name",
      "type": "string"
    },
    "color": {
      "anyOf": [
        {
          "enum": [
            "slate",
            "blue",
            "green",
            "amber",
            "rose",
            "violet",
            "cyan"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Color"
    },
    "x_position": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "X Position"
    },
    "y_position": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Y Position"
    },
    "width": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Width"
    },
    "height": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Height"
    },
    "parent_group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Parent Group Id"
    },
    "child_group_ids": {
      "items": {
        "type": "integer"
      },
      "title": "Child Group Ids",
      "type": "array"
    }
  },
  "required": [
    "node_ids"
  ],
  "title": "CreateGroupRequest",
  "type": "object"
}

Fields:

• node_ids (list[int])
• name (str)
• color (GroupColor | None)
• x_position (float | None)
• y_position (float | None)
• width (float | None)
• height (float | None)
• parent_group_id (int | None)
• child_group_ids (list[int])

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class CreateGroupRequest(BaseModel):
    """Body for POST /editor/create_group/. Bounds are optional; computed from members if omitted."""

    node_ids: list[int]
    name: str = "Group"
    color: GroupColor | None = None
    x_position: float | None = None
    y_position: float | None = None
    width: float | None = None
    height: float | None = None
    parent_group_id: int | None = None  # nest the new group under this group
    child_group_ids: list[int] = Field(default_factory=list)  # existing groups to nest inside the new one

FlowGraphConfig pydantic-model

Bases: BaseModel

Configuration model for a flow graph's basic properties.

Attributes:

Name	Type	Description
flow_id	int	Unique identifier for the flow.
description	Optional[str]	A description of the flow.
save_location	Optional[str]	The location where the flow is saved.
name	str	The name of the flow.
path	str	The file path associated with the flow.
execution_mode	ExecutionModeLiteral	The mode of execution ('Development' or 'Performance').
execution_location	ExecutionLocationsLiteral	The location for execution ('local', 'remote').
max_parallel_workers	int	Maximum number of threads used for parallel node execution within a stage. Set to 1 to disable parallelism. Defaults to 4.
parameters	list[FlowParameter]	Flow-level parameters referenceable via ${name} syntax.

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    }
  },
  "description": "Configuration model for a flow graph's basic properties.\n\nAttributes:\n    flow_id (int): Unique identifier for the flow.\n    description (Optional[str]): A description of the flow.\n    save_location (Optional[str]): The location where the flow is saved.\n    name (str): The name of the flow.\n    path (str): The file path associated with the flow.\n    execution_mode (ExecutionModeLiteral): The mode of execution ('Development' or 'Performance').\n    execution_location (ExecutionLocationsLiteral): The location for execution ('local', 'remote').\n    max_parallel_workers (int): Maximum number of threads used for parallel node execution within a\n        stage. Set to 1 to disable parallelism. Defaults to 4.\n    parameters (list[FlowParameter]): Flow-level parameters referenceable via ${name} syntax.",
  "properties": {
    "flow_id": {
      "description": "Unique identifier for the flow.",
      "title": "Flow Id",
      "type": "integer"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Description"
    },
    "save_location": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Save Location"
    },
    "name": {
      "default": "",
      "title": "Name",
      "type": "string"
    },
    "path": {
      "default": "",
      "title": "Path",
      "type": "string"
    },
    "source_registration_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Catalog registration ID when running a registered flow.",
      "title": "Source Registration Id"
    },
    "execution_mode": {
      "default": "Performance",
      "enum": [
        "Development",
        "Performance"
      ],
      "title": "Execution Mode",
      "type": "string"
    },
    "execution_location": {
      "enum": [
        "local",
        "remote"
      ],
      "title": "Execution Location",
      "type": "string"
    },
    "max_parallel_workers": {
      "default": 4,
      "description": "Max threads for parallel node execution.",
      "minimum": 1,
      "title": "Max Parallel Workers",
      "type": "integer"
    },
    "parameters": {
      "description": "Flow-level parameters.",
      "items": {
        "$ref": "#/$defs/FlowParameter"
      },
      "title": "Parameters",
      "type": "array"
    }
  },
  "title": "FlowGraphConfig",
  "type": "object"
}

Fields:

• flow_id (int)
• description (str | None)
• save_location (str | None)
• name (str)
• path (str)
• source_registration_id (int | None)
• execution_mode (ExecutionModeLiteral)
• execution_location (ExecutionLocationsLiteral)
• max_parallel_workers (int)
• parameters (list[FlowParameter])

Validators:

• validate_execution_mode → execution_mode
• validate_and_set_execution_location → execution_location

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowGraphConfig(BaseModel):
    """
    Configuration model for a flow graph's basic properties.

    Attributes:
        flow_id (int): Unique identifier for the flow.
        description (Optional[str]): A description of the flow.
        save_location (Optional[str]): The location where the flow is saved.
        name (str): The name of the flow.
        path (str): The file path associated with the flow.
        execution_mode (ExecutionModeLiteral): The mode of execution ('Development' or 'Performance').
        execution_location (ExecutionLocationsLiteral): The location for execution ('local', 'remote').
        max_parallel_workers (int): Maximum number of threads used for parallel node execution within a
            stage. Set to 1 to disable parallelism. Defaults to 4.
        parameters (list[FlowParameter]): Flow-level parameters referenceable via ${name} syntax.
    """

    flow_id: int = Field(default_factory=create_unique_id, description="Unique identifier for the flow.")
    description: str | None = None
    save_location: str | None = None
    name: str = ""
    path: str = ""
    source_registration_id: int | None = Field(
        default=None,
        description="Catalog registration ID when running a registered flow.",
    )
    execution_mode: ExecutionModeLiteral = "Performance"
    execution_location: ExecutionLocationsLiteral = Field(default_factory=get_global_execution_location)
    max_parallel_workers: int = Field(default=4, ge=1, description="Max threads for parallel node execution.")
    parameters: list[FlowParameter] = Field(default_factory=list, description="Flow-level parameters.")

    @field_validator("execution_mode", mode="before")
    @classmethod
    def validate_execution_mode(cls, v: str) -> ExecutionModeLiteral:
        if v not in ("Development", "Performance"):
            return "Performance"
        return v

    @field_validator("execution_location", mode="before")
    def validate_and_set_execution_location(cls, v: ExecutionLocationsLiteral | None) -> ExecutionLocationsLiteral:
        """
        Validates and sets the execution location.
        1.  **If `None` is provided**: It defaults to the location determined by global settings.
        2.  **If a value is provided**: It checks if the value is compatible with the global
            settings. If not (e.g., requesting 'remote' when only 'local' is possible),
            it corrects the value to a compatible one.
        """
        if v is None:
            return get_global_execution_location()
        if v == "auto":
            return get_global_execution_location()

        return get_prio_execution_location(v, get_global_execution_location())

flow_id pydantic-field

Unique identifier for the flow.

max_parallel_workers = 4 pydantic-field

Max threads for parallel node execution.

parameters pydantic-field

Flow-level parameters.

source_registration_id = None pydantic-field

Catalog registration ID when running a registered flow.

validate_and_set_execution_location(v) pydantic-validator

Validates and sets the execution location. 1. If None is provided: It defaults to the location determined by global settings. 2. If a value is provided: It checks if the value is compatible with the global settings. If not (e.g., requesting 'remote' when only 'local' is possible), it corrects the value to a compatible one.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@field_validator("execution_location", mode="before")
def validate_and_set_execution_location(cls, v: ExecutionLocationsLiteral | None) -> ExecutionLocationsLiteral:
    """
    Validates and sets the execution location.
    1.  **If `None` is provided**: It defaults to the location determined by global settings.
    2.  **If a value is provided**: It checks if the value is compatible with the global
        settings. If not (e.g., requesting 'remote' when only 'local' is possible),
        it corrects the value to a compatible one.
    """
    if v is None:
        return get_global_execution_location()
    if v == "auto":
        return get_global_execution_location()

    return get_prio_execution_location(v, get_global_execution_location())

FlowInformation pydantic-model

Bases: BaseModel

Represents the complete state of a flow, including settings, nodes, and connections.

Attributes:

Name	Type	Description
flow_id	int	The unique ID of the flow.
flow_name	Optional[str]	The name of the flow.
flow_settings	FlowSettings	The settings for the flow.
data	Dict[int, NodeInformation]	A dictionary mapping node IDs to their information.
node_starts	List[int]	A list of starting node IDs.
node_connections	List[Tuple[int, int]]	A list of tuples representing connections between nodes.

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    },
    "FlowSettings": {
      "description": "Extends FlowGraphConfig with additional operational settings for a flow.\n\nAttributes:\n    auto_save (bool): Flag to enable or disable automatic saving.\n    modified_on (Optional[float]): Timestamp of the last modification.\n    show_detailed_progress (bool): Flag to show detailed progress during execution.\n    show_edge_labels (bool): Flag to show or hide named edge labels on connections.\n    is_running (bool): Indicates if the flow is currently running.\n    is_canceled (bool): Indicates if the flow execution has been canceled.\n    track_history (bool): Flag to enable or disable undo/redo history tracking.\n    validate_settings (bool): Flag to warn on nodes whose settings reference missing input columns.",
      "properties": {
        "flow_id": {
          "description": "Unique identifier for the flow.",
          "title": "Flow Id",
          "type": "integer"
        },
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Description"
        },
        "save_location": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Save Location"
        },
        "name": {
          "default": "",
          "title": "Name",
          "type": "string"
        },
        "path": {
          "default": "",
          "title": "Path",
          "type": "string"
        },
        "source_registration_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Catalog registration ID when running a registered flow.",
          "title": "Source Registration Id"
        },
        "execution_mode": {
          "default": "Performance",
          "enum": [
            "Development",
            "Performance"
          ],
          "title": "Execution Mode",
          "type": "string"
        },
        "execution_location": {
          "enum": [
            "local",
            "remote"
          ],
          "title": "Execution Location",
          "type": "string"
        },
        "max_parallel_workers": {
          "default": 4,
          "description": "Max threads for parallel node execution.",
          "minimum": 1,
          "title": "Max Parallel Workers",
          "type": "integer"
        },
        "parameters": {
          "description": "Flow-level parameters.",
          "items": {
            "$ref": "#/$defs/FlowParameter"
          },
          "title": "Parameters",
          "type": "array"
        },
        "auto_save": {
          "default": false,
          "title": "Auto Save",
          "type": "boolean"
        },
        "modified_on": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Modified On"
        },
        "show_detailed_progress": {
          "default": true,
          "title": "Show Detailed Progress",
          "type": "boolean"
        },
        "show_edge_labels": {
          "default": false,
          "title": "Show Edge Labels",
          "type": "boolean"
        },
        "is_running": {
          "default": false,
          "title": "Is Running",
          "type": "boolean"
        },
        "is_canceled": {
          "default": false,
          "title": "Is Canceled",
          "type": "boolean"
        },
        "track_history": {
          "default": true,
          "title": "Track History",
          "type": "boolean"
        },
        "validate_settings": {
          "default": true,
          "title": "Validate Settings",
          "type": "boolean"
        }
      },
      "title": "FlowSettings",
      "type": "object"
    },
    "FlowfileInputConnection": {
      "description": "One keyed input edge of a dynamic-input node (per-edge target handle).\n\nOnly nodes whose template sets ``dynamic_inputs`` serialize these; the same\nupstream node may legitimately appear twice with different handles.",
      "properties": {
        "from_id": {
          "title": "From Id",
          "type": "integer"
        },
        "input_handle": {
          "title": "Input Handle",
          "type": "string"
        },
        "source_handle": {
          "default": "output-0",
          "title": "Source Handle",
          "type": "string"
        }
      },
      "required": [
        "from_id",
        "input_handle"
      ],
      "title": "FlowfileInputConnection",
      "type": "object"
    },
    "GroupInformation": {
      "description": "Runtime representation of a visual node group (stored in FlowGraph._groups).",
      "properties": {
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "name": {
          "default": "Group",
          "title": "Name",
          "type": "string"
        },
        "color": {
          "anyOf": [
            {
              "enum": [
                "slate",
                "blue",
                "green",
                "amber",
                "rose",
                "violet",
                "cyan"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Color"
        },
        "x_position": {
          "default": 0.0,
          "title": "X Position",
          "type": "number"
        },
        "y_position": {
          "default": 0.0,
          "title": "Y Position",
          "type": "number"
        },
        "width": {
          "default": 400.0,
          "title": "Width",
          "type": "number"
        },
        "height": {
          "default": 250.0,
          "title": "Height",
          "type": "number"
        },
        "collapsed": {
          "default": false,
          "title": "Collapsed",
          "type": "boolean"
        },
        "parent_group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parent Group Id"
        }
      },
      "required": [
        "id"
      ],
      "title": "GroupInformation",
      "type": "object"
    },
    "NodeInformation": {
      "description": "Stores the state and configuration of a specific node instance within a flow.",
      "properties": {
        "id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Id"
        },
        "type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Type"
        },
        "is_setup": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Is Setup"
        },
        "is_start_node": {
          "default": false,
          "title": "Is Start Node",
          "type": "boolean"
        },
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "",
          "title": "Description"
        },
        "node_reference": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Node Reference"
        },
        "x_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": 0,
          "title": "X Position"
        },
        "y_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": 0,
          "title": "Y Position"
        },
        "group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Group Id"
        },
        "left_input_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Left Input Id"
        },
        "right_input_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Right Input Id"
        },
        "input_ids": {
          "anyOf": [
            {
              "items": {
                "type": "integer"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Input Ids"
        },
        "outputs": {
          "anyOf": [
            {
              "items": {
                "type": "integer"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Outputs"
        },
        "output_handles": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Handles"
        },
        "input_connections": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/FlowfileInputConnection"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Input Connections"
        },
        "setting_input": {
          "anyOf": [
            {},
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Setting Input"
        }
      },
      "title": "NodeInformation",
      "type": "object"
    }
  },
  "description": "Represents the complete state of a flow, including settings, nodes, and connections.\n\nAttributes:\n    flow_id (int): The unique ID of the flow.\n    flow_name (Optional[str]): The name of the flow.\n    flow_settings (FlowSettings): The settings for the flow.\n    data (Dict[int, NodeInformation]): A dictionary mapping node IDs to their information.\n    node_starts (List[int]): A list of starting node IDs.\n    node_connections (List[Tuple[int, int]]): A list of tuples representing connections between nodes.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "flow_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Flow Name"
    },
    "flow_settings": {
      "$ref": "#/$defs/FlowSettings"
    },
    "data": {
      "additionalProperties": {
        "$ref": "#/$defs/NodeInformation"
      },
      "default": {},
      "title": "Data",
      "type": "object"
    },
    "node_starts": {
      "items": {
        "type": "integer"
      },
      "title": "Node Starts",
      "type": "array"
    },
    "node_connections": {
      "default": [],
      "items": {
        "maxItems": 2,
        "minItems": 2,
        "prefixItems": [
          {
            "type": "integer"
          },
          {
            "type": "integer"
          }
        ],
        "type": "array"
      },
      "title": "Node Connections",
      "type": "array"
    },
    "groups": {
      "items": {
        "$ref": "#/$defs/GroupInformation"
      },
      "title": "Groups",
      "type": "array"
    }
  },
  "required": [
    "flow_id",
    "flow_settings",
    "node_starts"
  ],
  "title": "FlowInformation",
  "type": "object"
}

Fields:

• flow_id (int)
• flow_name (str | None)
• flow_settings (FlowSettings)
• data (dict[int, NodeInformation])
• node_starts (list[int])
• node_connections (list[tuple[int, int]])
• groups (list[GroupInformation])

Validators:

• ensure_string → flow_name

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowInformation(BaseModel):
    """
    Represents the complete state of a flow, including settings, nodes, and connections.

    Attributes:
        flow_id (int): The unique ID of the flow.
        flow_name (Optional[str]): The name of the flow.
        flow_settings (FlowSettings): The settings for the flow.
        data (Dict[int, NodeInformation]): A dictionary mapping node IDs to their information.
        node_starts (List[int]): A list of starting node IDs.
        node_connections (List[Tuple[int, int]]): A list of tuples representing connections between nodes.
    """

    flow_id: int
    flow_name: str | None = ""
    flow_settings: FlowSettings
    data: dict[int, NodeInformation] = {}
    node_starts: list[int]
    node_connections: list[tuple[int, int]] = []
    groups: list[GroupInformation] = Field(default_factory=list)

    @field_validator("flow_name", mode="before")
    def ensure_string(cls, v):
        """
        Validator to ensure the flow_name is always a string.
        :param v: The value to validate.
        :return: The value as a string, or an empty string if it's None.
        """
        return str(v) if v is not None else ""

ensure_string(v) pydantic-validator

Validator to ensure the flow_name is always a string. :param v: The value to validate. :return: The value as a string, or an empty string if it's None.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@field_validator("flow_name", mode="before")
def ensure_string(cls, v):
    """
    Validator to ensure the flow_name is always a string.
    :param v: The value to validate.
    :return: The value as a string, or an empty string if it's None.
    """
    return str(v) if v is not None else ""

FlowSettings pydantic-model

Bases: FlowGraphConfig

Extends FlowGraphConfig with additional operational settings for a flow.

Attributes:

Name	Type	Description
auto_save	bool	Flag to enable or disable automatic saving.
modified_on	Optional[float]	Timestamp of the last modification.
show_detailed_progress	bool	Flag to show detailed progress during execution.
show_edge_labels	bool	Flag to show or hide named edge labels on connections.
is_running	bool	Indicates if the flow is currently running.
is_canceled	bool	Indicates if the flow execution has been canceled.
track_history	bool	Flag to enable or disable undo/redo history tracking.
validate_settings	bool	Flag to warn on nodes whose settings reference missing input columns.

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    }
  },
  "description": "Extends FlowGraphConfig with additional operational settings for a flow.\n\nAttributes:\n    auto_save (bool): Flag to enable or disable automatic saving.\n    modified_on (Optional[float]): Timestamp of the last modification.\n    show_detailed_progress (bool): Flag to show detailed progress during execution.\n    show_edge_labels (bool): Flag to show or hide named edge labels on connections.\n    is_running (bool): Indicates if the flow is currently running.\n    is_canceled (bool): Indicates if the flow execution has been canceled.\n    track_history (bool): Flag to enable or disable undo/redo history tracking.\n    validate_settings (bool): Flag to warn on nodes whose settings reference missing input columns.",
  "properties": {
    "flow_id": {
      "description": "Unique identifier for the flow.",
      "title": "Flow Id",
      "type": "integer"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Description"
    },
    "save_location": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Save Location"
    },
    "name": {
      "default": "",
      "title": "Name",
      "type": "string"
    },
    "path": {
      "default": "",
      "title": "Path",
      "type": "string"
    },
    "source_registration_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Catalog registration ID when running a registered flow.",
      "title": "Source Registration Id"
    },
    "execution_mode": {
      "default": "Performance",
      "enum": [
        "Development",
        "Performance"
      ],
      "title": "Execution Mode",
      "type": "string"
    },
    "execution_location": {
      "enum": [
        "local",
        "remote"
      ],
      "title": "Execution Location",
      "type": "string"
    },
    "max_parallel_workers": {
      "default": 4,
      "description": "Max threads for parallel node execution.",
      "minimum": 1,
      "title": "Max Parallel Workers",
      "type": "integer"
    },
    "parameters": {
      "description": "Flow-level parameters.",
      "items": {
        "$ref": "#/$defs/FlowParameter"
      },
      "title": "Parameters",
      "type": "array"
    },
    "auto_save": {
      "default": false,
      "title": "Auto Save",
      "type": "boolean"
    },
    "modified_on": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modified On"
    },
    "show_detailed_progress": {
      "default": true,
      "title": "Show Detailed Progress",
      "type": "boolean"
    },
    "show_edge_labels": {
      "default": false,
      "title": "Show Edge Labels",
      "type": "boolean"
    },
    "is_running": {
      "default": false,
      "title": "Is Running",
      "type": "boolean"
    },
    "is_canceled": {
      "default": false,
      "title": "Is Canceled",
      "type": "boolean"
    },
    "track_history": {
      "default": true,
      "title": "Track History",
      "type": "boolean"
    },
    "validate_settings": {
      "default": true,
      "title": "Validate Settings",
      "type": "boolean"
    }
  },
  "title": "FlowSettings",
  "type": "object"
}

Fields:

• flow_id (int)
• description (str | None)
• save_location (str | None)
• name (str)
• path (str)
• source_registration_id (int | None)
• execution_mode (ExecutionModeLiteral)
• execution_location (ExecutionLocationsLiteral)
• max_parallel_workers (int)
• parameters (list[FlowParameter])
• auto_save (bool)
• modified_on (float | None)
• show_detailed_progress (bool)
• show_edge_labels (bool)
• is_running (bool)
• is_canceled (bool)
• track_history (bool)
• validate_settings (bool)

Validators:

• validate_execution_mode → execution_mode
• validate_and_set_execution_location → execution_location

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowSettings(FlowGraphConfig):
    """
    Extends FlowGraphConfig with additional operational settings for a flow.

    Attributes:
        auto_save (bool): Flag to enable or disable automatic saving.
        modified_on (Optional[float]): Timestamp of the last modification.
        show_detailed_progress (bool): Flag to show detailed progress during execution.
        show_edge_labels (bool): Flag to show or hide named edge labels on connections.
        is_running (bool): Indicates if the flow is currently running.
        is_canceled (bool): Indicates if the flow execution has been canceled.
        track_history (bool): Flag to enable or disable undo/redo history tracking.
        validate_settings (bool): Flag to warn on nodes whose settings reference missing input columns.
    """

    auto_save: bool = False
    modified_on: float | None = None
    show_detailed_progress: bool = True
    show_edge_labels: bool = False
    is_running: bool = False
    is_canceled: bool = False
    track_history: bool = True
    validate_settings: bool = True

    @classmethod
    def from_flow_settings_input(cls, flow_graph_config: FlowGraphConfig):
        """
        Creates a FlowSettings instance from a FlowGraphConfig instance.

        :param flow_graph_config: The base flow graph configuration.
        :return: A new instance of FlowSettings with data from flow_graph_config.
        """
        return cls.model_validate(flow_graph_config.model_dump())

flow_id pydantic-field

Unique identifier for the flow.

max_parallel_workers = 4 pydantic-field

Max threads for parallel node execution.

parameters pydantic-field

Flow-level parameters.

source_registration_id = None pydantic-field

Catalog registration ID when running a registered flow.

from_flow_settings_input(flow_graph_config) classmethod

Creates a FlowSettings instance from a FlowGraphConfig instance.

:param flow_graph_config: The base flow graph configuration. :return: A new instance of FlowSettings with data from flow_graph_config.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@classmethod
def from_flow_settings_input(cls, flow_graph_config: FlowGraphConfig):
    """
    Creates a FlowSettings instance from a FlowGraphConfig instance.

    :param flow_graph_config: The base flow graph configuration.
    :return: A new instance of FlowSettings with data from flow_graph_config.
    """
    return cls.model_validate(flow_graph_config.model_dump())

validate_and_set_execution_location(v) pydantic-validator

Validates and sets the execution location. 1. If None is provided: It defaults to the location determined by global settings. 2. If a value is provided: It checks if the value is compatible with the global settings. If not (e.g., requesting 'remote' when only 'local' is possible), it corrects the value to a compatible one.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@field_validator("execution_location", mode="before")
def validate_and_set_execution_location(cls, v: ExecutionLocationsLiteral | None) -> ExecutionLocationsLiteral:
    """
    Validates and sets the execution location.
    1.  **If `None` is provided**: It defaults to the location determined by global settings.
    2.  **If a value is provided**: It checks if the value is compatible with the global
        settings. If not (e.g., requesting 'remote' when only 'local' is possible),
        it corrects the value to a compatible one.
    """
    if v is None:
        return get_global_execution_location()
    if v == "auto":
        return get_global_execution_location()

    return get_prio_execution_location(v, get_global_execution_location())

FlowSettingsResponse pydantic-model

Bases: FlowSettings

FlowSettings plus runtime-only fields for API responses. Not persisted.

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    }
  },
  "description": "FlowSettings plus runtime-only fields for API responses. Not persisted.",
  "properties": {
    "flow_id": {
      "description": "Unique identifier for the flow.",
      "title": "Flow Id",
      "type": "integer"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Description"
    },
    "save_location": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Save Location"
    },
    "name": {
      "default": "",
      "title": "Name",
      "type": "string"
    },
    "path": {
      "default": "",
      "title": "Path",
      "type": "string"
    },
    "source_registration_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Catalog registration ID when running a registered flow.",
      "title": "Source Registration Id"
    },
    "execution_mode": {
      "default": "Performance",
      "enum": [
        "Development",
        "Performance"
      ],
      "title": "Execution Mode",
      "type": "string"
    },
    "execution_location": {
      "enum": [
        "local",
        "remote"
      ],
      "title": "Execution Location",
      "type": "string"
    },
    "max_parallel_workers": {
      "default": 4,
      "description": "Max threads for parallel node execution.",
      "minimum": 1,
      "title": "Max Parallel Workers",
      "type": "integer"
    },
    "parameters": {
      "description": "Flow-level parameters.",
      "items": {
        "$ref": "#/$defs/FlowParameter"
      },
      "title": "Parameters",
      "type": "array"
    },
    "auto_save": {
      "default": false,
      "title": "Auto Save",
      "type": "boolean"
    },
    "modified_on": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modified On"
    },
    "show_detailed_progress": {
      "default": true,
      "title": "Show Detailed Progress",
      "type": "boolean"
    },
    "show_edge_labels": {
      "default": false,
      "title": "Show Edge Labels",
      "type": "boolean"
    },
    "is_running": {
      "default": false,
      "title": "Is Running",
      "type": "boolean"
    },
    "is_canceled": {
      "default": false,
      "title": "Is Canceled",
      "type": "boolean"
    },
    "track_history": {
      "default": true,
      "title": "Track History",
      "type": "boolean"
    },
    "validate_settings": {
      "default": true,
      "title": "Validate Settings",
      "type": "boolean"
    },
    "has_unsaved_changes": {
      "default": false,
      "title": "Has Unsaved Changes",
      "type": "boolean"
    },
    "display_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Display Name"
    }
  },
  "title": "FlowSettingsResponse",
  "type": "object"
}

Fields:

• flow_id (int)
• description (str | None)
• save_location (str | None)
• name (str)
• path (str)
• source_registration_id (int | None)
• execution_mode (ExecutionModeLiteral)
• execution_location (ExecutionLocationsLiteral)
• max_parallel_workers (int)
• parameters (list[FlowParameter])
• auto_save (bool)
• modified_on (float | None)
• show_detailed_progress (bool)
• show_edge_labels (bool)
• is_running (bool)
• is_canceled (bool)
• track_history (bool)
• validate_settings (bool)
• has_unsaved_changes (bool)
• display_name (str | None)

Validators:

• validate_execution_mode → execution_mode
• validate_and_set_execution_location → execution_location

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowSettingsResponse(FlowSettings):
    """FlowSettings plus runtime-only fields for API responses. Not persisted."""

    has_unsaved_changes: bool = False
    display_name: str | None = None

flow_id pydantic-field

Unique identifier for the flow.

max_parallel_workers = 4 pydantic-field

Max threads for parallel node execution.

parameters pydantic-field

Flow-level parameters.

source_registration_id = None pydantic-field

Catalog registration ID when running a registered flow.

from_flow_settings_input(flow_graph_config) classmethod

Creates a FlowSettings instance from a FlowGraphConfig instance.

:param flow_graph_config: The base flow graph configuration. :return: A new instance of FlowSettings with data from flow_graph_config.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@classmethod
def from_flow_settings_input(cls, flow_graph_config: FlowGraphConfig):
    """
    Creates a FlowSettings instance from a FlowGraphConfig instance.

    :param flow_graph_config: The base flow graph configuration.
    :return: A new instance of FlowSettings with data from flow_graph_config.
    """
    return cls.model_validate(flow_graph_config.model_dump())

validate_and_set_execution_location(v) pydantic-validator

Validates and sets the execution location. 1. If None is provided: It defaults to the location determined by global settings. 2. If a value is provided: It checks if the value is compatible with the global settings. If not (e.g., requesting 'remote' when only 'local' is possible), it corrects the value to a compatible one.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

@field_validator("execution_location", mode="before")
def validate_and_set_execution_location(cls, v: ExecutionLocationsLiteral | None) -> ExecutionLocationsLiteral:
    """
    Validates and sets the execution location.
    1.  **If `None` is provided**: It defaults to the location determined by global settings.
    2.  **If a value is provided**: It checks if the value is compatible with the global
        settings. If not (e.g., requesting 'remote' when only 'local' is possible),
        it corrects the value to a compatible one.
    """
    if v is None:
        return get_global_execution_location()
    if v == "auto":
        return get_global_execution_location()

    return get_prio_execution_location(v, get_global_execution_location())

FlowfileData pydantic-model

Bases: BaseModel

Root model for flowfile serialization (YAML/JSON).

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    },
    "FlowfileGroup": {
      "description": "Serialized representation of a visual node group (YAML/JSON).",
      "properties": {
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "name": {
          "default": "Group",
          "title": "Name",
          "type": "string"
        },
        "color": {
          "anyOf": [
            {
              "enum": [
                "slate",
                "blue",
                "green",
                "amber",
                "rose",
                "violet",
                "cyan"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Color"
        },
        "x_position": {
          "default": 0.0,
          "title": "X Position",
          "type": "number"
        },
        "y_position": {
          "default": 0.0,
          "title": "Y Position",
          "type": "number"
        },
        "width": {
          "default": 400.0,
          "title": "Width",
          "type": "number"
        },
        "height": {
          "default": 250.0,
          "title": "Height",
          "type": "number"
        },
        "collapsed": {
          "default": false,
          "title": "Collapsed",
          "type": "boolean"
        },
        "parent_group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parent Group Id"
        }
      },
      "required": [
        "id"
      ],
      "title": "FlowfileGroup",
      "type": "object"
    },
    "FlowfileInputConnection": {
      "description": "One keyed input edge of a dynamic-input node (per-edge target handle).\n\nOnly nodes whose template sets ``dynamic_inputs`` serialize these; the same\nupstream node may legitimately appear twice with different handles.",
      "properties": {
        "from_id": {
          "title": "From Id",
          "type": "integer"
        },
        "input_handle": {
          "title": "Input Handle",
          "type": "string"
        },
        "source_handle": {
          "default": "output-0",
          "title": "Source Handle",
          "type": "string"
        }
      },
      "required": [
        "from_id",
        "input_handle"
      ],
      "title": "FlowfileInputConnection",
      "type": "object"
    },
    "FlowfileNode": {
      "description": "Node representation for flowfile serialization (YAML/JSON).",
      "properties": {
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "type": {
          "title": "Type",
          "type": "string"
        },
        "is_start_node": {
          "default": false,
          "title": "Is Start Node",
          "type": "boolean"
        },
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "",
          "title": "Description"
        },
        "node_reference": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Node Reference"
        },
        "x_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": 0,
          "title": "X Position"
        },
        "y_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": 0,
          "title": "Y Position"
        },
        "group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Group Id"
        },
        "left_input_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Left Input Id"
        },
        "right_input_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Right Input Id"
        },
        "input_ids": {
          "anyOf": [
            {
              "items": {
                "type": "integer"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Input Ids"
        },
        "outputs": {
          "anyOf": [
            {
              "items": {
                "type": "integer"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Outputs"
        },
        "output_handles": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Handles"
        },
        "input_connections": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/FlowfileInputConnection"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Input Connections"
        },
        "setting_input": {
          "anyOf": [
            {},
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Setting Input"
        }
      },
      "required": [
        "id",
        "type"
      ],
      "title": "FlowfileNode",
      "type": "object"
    },
    "FlowfileSettings": {
      "description": "Settings for flowfile serialization (YAML/JSON).\n\nExcludes runtime state fields like is_running, is_canceled, modified_on.",
      "properties": {
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Description"
        },
        "execution_mode": {
          "default": "Performance",
          "enum": [
            "Development",
            "Performance"
          ],
          "title": "Execution Mode",
          "type": "string"
        },
        "execution_location": {
          "default": "local",
          "enum": [
            "local",
            "remote"
          ],
          "title": "Execution Location",
          "type": "string"
        },
        "auto_save": {
          "default": false,
          "title": "Auto Save",
          "type": "boolean"
        },
        "show_detailed_progress": {
          "default": true,
          "title": "Show Detailed Progress",
          "type": "boolean"
        },
        "validate_settings": {
          "default": true,
          "title": "Validate Settings",
          "type": "boolean"
        },
        "max_parallel_workers": {
          "default": 4,
          "minimum": 1,
          "title": "Max Parallel Workers",
          "type": "integer"
        },
        "source_registration_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Source Registration Id"
        },
        "parameters": {
          "description": "Flow-level parameters.",
          "items": {
            "$ref": "#/$defs/FlowParameter"
          },
          "title": "Parameters",
          "type": "array"
        }
      },
      "title": "FlowfileSettings",
      "type": "object"
    }
  },
  "description": "Root model for flowfile serialization (YAML/JSON).",
  "properties": {
    "flowfile_version": {
      "title": "Flowfile Version",
      "type": "string"
    },
    "flowfile_id": {
      "title": "Flowfile Id",
      "type": "integer"
    },
    "flowfile_name": {
      "title": "Flowfile Name",
      "type": "string"
    },
    "flowfile_settings": {
      "$ref": "#/$defs/FlowfileSettings"
    },
    "nodes": {
      "items": {
        "$ref": "#/$defs/FlowfileNode"
      },
      "title": "Nodes",
      "type": "array"
    },
    "groups": {
      "items": {
        "$ref": "#/$defs/FlowfileGroup"
      },
      "title": "Groups",
      "type": "array"
    }
  },
  "required": [
    "flowfile_version",
    "flowfile_id",
    "flowfile_name",
    "flowfile_settings",
    "nodes"
  ],
  "title": "FlowfileData",
  "type": "object"
}

Fields:

• flowfile_version (str)
• flowfile_id (int)
• flowfile_name (str)
• flowfile_settings (FlowfileSettings)
• nodes (list[FlowfileNode])
• groups (list[FlowfileGroup])

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowfileData(BaseModel):
    """Root model for flowfile serialization (YAML/JSON)."""

    flowfile_version: str
    flowfile_id: int
    flowfile_name: str
    flowfile_settings: FlowfileSettings
    nodes: list[FlowfileNode]
    groups: list[FlowfileGroup] = Field(default_factory=list)

FlowfileGroup pydantic-model

Bases: _GroupFields

Serialized representation of a visual node group (YAML/JSON).

Show JSON schema:

{
  "description": "Serialized representation of a visual node group (YAML/JSON).",
  "properties": {
    "id": {
      "title": "Id",
      "type": "integer"
    },
    "name": {
      "default": "Group",
      "title": "Name",
      "type": "string"
    },
    "color": {
      "anyOf": [
        {
          "enum": [
            "slate",
            "blue",
            "green",
            "amber",
            "rose",
            "violet",
            "cyan"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Color"
    },
    "x_position": {
      "default": 0.0,
      "title": "X Position",
      "type": "number"
    },
    "y_position": {
      "default": 0.0,
      "title": "Y Position",
      "type": "number"
    },
    "width": {
      "default": 400.0,
      "title": "Width",
      "type": "number"
    },
    "height": {
      "default": 250.0,
      "title": "Height",
      "type": "number"
    },
    "collapsed": {
      "default": false,
      "title": "Collapsed",
      "type": "boolean"
    },
    "parent_group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Parent Group Id"
    }
  },
  "required": [
    "id"
  ],
  "title": "FlowfileGroup",
  "type": "object"
}

Fields:

• id (int)
• name (str)
• color (GroupColor | None)
• x_position (float)
• y_position (float)
• width (float)
• height (float)
• collapsed (bool)
• parent_group_id (int | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowfileGroup(_GroupFields):
    """Serialized representation of a visual node group (YAML/JSON)."""

FlowfileInputConnection pydantic-model

Bases: BaseModel

One keyed input edge of a dynamic-input node (per-edge target handle).

Only nodes whose template sets dynamic_inputs serialize these; the same upstream node may legitimately appear twice with different handles.

Show JSON schema:

{
  "description": "One keyed input edge of a dynamic-input node (per-edge target handle).\n\nOnly nodes whose template sets ``dynamic_inputs`` serialize these; the same\nupstream node may legitimately appear twice with different handles.",
  "properties": {
    "from_id": {
      "title": "From Id",
      "type": "integer"
    },
    "input_handle": {
      "title": "Input Handle",
      "type": "string"
    },
    "source_handle": {
      "default": "output-0",
      "title": "Source Handle",
      "type": "string"
    }
  },
  "required": [
    "from_id",
    "input_handle"
  ],
  "title": "FlowfileInputConnection",
  "type": "object"
}

Fields:

• from_id (int)
• input_handle (str)
• source_handle (str)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowfileInputConnection(BaseModel):
    """One keyed input edge of a dynamic-input node (per-edge target handle).

    Only nodes whose template sets ``dynamic_inputs`` serialize these; the same
    upstream node may legitimately appear twice with different handles.
    """

    from_id: int
    input_handle: str
    source_handle: str = "output-0"

FlowfileNode pydantic-model

Bases: BaseModel

Node representation for flowfile serialization (YAML/JSON).

Show JSON schema:

{
  "$defs": {
    "FlowfileInputConnection": {
      "description": "One keyed input edge of a dynamic-input node (per-edge target handle).\n\nOnly nodes whose template sets ``dynamic_inputs`` serialize these; the same\nupstream node may legitimately appear twice with different handles.",
      "properties": {
        "from_id": {
          "title": "From Id",
          "type": "integer"
        },
        "input_handle": {
          "title": "Input Handle",
          "type": "string"
        },
        "source_handle": {
          "default": "output-0",
          "title": "Source Handle",
          "type": "string"
        }
      },
      "required": [
        "from_id",
        "input_handle"
      ],
      "title": "FlowfileInputConnection",
      "type": "object"
    }
  },
  "description": "Node representation for flowfile serialization (YAML/JSON).",
  "properties": {
    "id": {
      "title": "Id",
      "type": "integer"
    },
    "type": {
      "title": "Type",
      "type": "string"
    },
    "is_start_node": {
      "default": false,
      "title": "Is Start Node",
      "type": "boolean"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "x_position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "X Position"
    },
    "y_position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Y Position"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "left_input_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Left Input Id"
    },
    "right_input_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Right Input Id"
    },
    "input_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Input Ids"
    },
    "outputs": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Outputs"
    },
    "output_handles": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Handles"
    },
    "input_connections": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/FlowfileInputConnection"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Input Connections"
    },
    "setting_input": {
      "anyOf": [
        {},
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Setting Input"
    }
  },
  "required": [
    "id",
    "type"
  ],
  "title": "FlowfileNode",
  "type": "object"
}

Fields:

• id (int)
• type (str)
• is_start_node (bool)
• description (str | None)
• node_reference (str | None)
• x_position (int | None)
• y_position (int | None)
• group_id (int | None)
• left_input_id (int | None)
• right_input_id (int | None)
• input_ids (list[int] | None)
• outputs (list[int] | None)
• output_handles (list[str] | None)
• input_connections (list[FlowfileInputConnection] | None)
• setting_input (Any | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowfileNode(BaseModel):
    """Node representation for flowfile serialization (YAML/JSON)."""

    id: int
    type: str
    is_start_node: bool = False
    description: str | None = ""
    node_reference: str | None = None  # Unique reference identifier for code generation
    x_position: int | None = 0
    y_position: int | None = 0
    group_id: int | None = None  # Visual group this node belongs to (organizational only)
    left_input_id: int | None = None
    right_input_id: int | None = None
    input_ids: list[int] | None = Field(default_factory=list)
    outputs: list[int] | None = Field(default_factory=list)
    # Parallel to ``outputs``: the source-side output handle for each connection
    # (e.g. ["output-0", "output-1"]). Older flowfiles omit this — loaders treat
    # missing entries as "output-0".
    output_handles: list[str] | None = None
    # Keyed edges for dynamic-input nodes; None for every other node type.
    input_connections: list[FlowfileInputConnection] | None = None
    setting_input: Any | None = None

    _setting_input_exclude: ClassVar[set] = {
        "flow_id",
        "node_id",
        "pos_x",
        "pos_y",
        "group_id",
        "is_setup",
        "description",
        "node_reference",
        "user_id",
        "is_flow_output",
        "is_user_defined",
        "depending_on_id",
        "depending_on_ids",
    }

    @field_serializer("setting_input")
    def serialize_setting_input(self, value, _info):
        if value is None:
            return None
        if isinstance(value, input_schema.NodePromise):
            return None
        if isinstance(value, dict):
            return value
        if hasattr(value, "to_yaml_dict"):
            return value.to_yaml_dict()
        if isinstance(value, input_schema.UserDefinedNode):
            # Persist the marker so flows reopen gracefully even when the node
            # type is no longer installed (settings-class resolution).
            data = value.model_dump(exclude=self._setting_input_exclude)
            data["is_user_defined"] = True
            return data
        return value.model_dump(exclude=self._setting_input_exclude)

FlowfileSettings pydantic-model

Bases: BaseModel

Settings for flowfile serialization (YAML/JSON).

Excludes runtime state fields like is_running, is_canceled, modified_on.

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    }
  },
  "description": "Settings for flowfile serialization (YAML/JSON).\n\nExcludes runtime state fields like is_running, is_canceled, modified_on.",
  "properties": {
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Description"
    },
    "execution_mode": {
      "default": "Performance",
      "enum": [
        "Development",
        "Performance"
      ],
      "title": "Execution Mode",
      "type": "string"
    },
    "execution_location": {
      "default": "local",
      "enum": [
        "local",
        "remote"
      ],
      "title": "Execution Location",
      "type": "string"
    },
    "auto_save": {
      "default": false,
      "title": "Auto Save",
      "type": "boolean"
    },
    "show_detailed_progress": {
      "default": true,
      "title": "Show Detailed Progress",
      "type": "boolean"
    },
    "validate_settings": {
      "default": true,
      "title": "Validate Settings",
      "type": "boolean"
    },
    "max_parallel_workers": {
      "default": 4,
      "minimum": 1,
      "title": "Max Parallel Workers",
      "type": "integer"
    },
    "source_registration_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Registration Id"
    },
    "parameters": {
      "description": "Flow-level parameters.",
      "items": {
        "$ref": "#/$defs/FlowParameter"
      },
      "title": "Parameters",
      "type": "array"
    }
  },
  "title": "FlowfileSettings",
  "type": "object"
}

Fields:

• description (str | None)
• execution_mode (ExecutionModeLiteral)
• execution_location (ExecutionLocationsLiteral)
• auto_save (bool)
• show_detailed_progress (bool)
• validate_settings (bool)
• max_parallel_workers (int)
• source_registration_id (int | None)
• parameters (list[FlowParameter])

Validators:

• validate_execution_mode → execution_mode

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class FlowfileSettings(BaseModel):
    """Settings for flowfile serialization (YAML/JSON).

    Excludes runtime state fields like is_running, is_canceled, modified_on.
    """

    description: str | None = None
    execution_mode: ExecutionModeLiteral = "Performance"
    execution_location: ExecutionLocationsLiteral = "local"
    auto_save: bool = False
    show_detailed_progress: bool = True
    validate_settings: bool = True
    max_parallel_workers: int = Field(default=4, ge=1)
    source_registration_id: int | None = None
    parameters: list[FlowParameter] = Field(default_factory=list, description="Flow-level parameters.")

    @field_validator("execution_mode", mode="before")
    @classmethod
    def validate_execution_mode(cls, v: str) -> ExecutionModeLiteral:
        if v not in ("Development", "Performance"):
            return "Performance"
        return v

parameters pydantic-field

Flow-level parameters.

GroupBounds

Bases: NamedTuple

Axis-aligned bounds of a group box, in absolute canvas coordinates.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class GroupBounds(NamedTuple):
    """Axis-aligned bounds of a group box, in absolute canvas coordinates."""

    x: float
    y: float
    width: float
    height: float

GroupBoundsUpdate pydantic-model

Bases: BaseModel

A single group's new absolute bounds.

Show JSON schema:

{
  "description": "A single group's new absolute bounds.",
  "properties": {
    "group_id": {
      "title": "Group Id",
      "type": "integer"
    },
    "x_position": {
      "title": "X Position",
      "type": "number"
    },
    "y_position": {
      "title": "Y Position",
      "type": "number"
    },
    "width": {
      "title": "Width",
      "type": "number"
    },
    "height": {
      "title": "Height",
      "type": "number"
    }
  },
  "required": [
    "group_id",
    "x_position",
    "y_position",
    "width",
    "height"
  ],
  "title": "GroupBoundsUpdate",
  "type": "object"
}

Fields:

• group_id (int)
• x_position (float)
• y_position (float)
• width (float)
• height (float)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class GroupBoundsUpdate(BaseModel):
    """A single group's new absolute bounds."""

    group_id: int
    x_position: float
    y_position: float
    width: float
    height: float

GroupInformation pydantic-model

Bases: _GroupFields

Runtime representation of a visual node group (stored in FlowGraph._groups).

Show JSON schema:

{
  "description": "Runtime representation of a visual node group (stored in FlowGraph._groups).",
  "properties": {
    "id": {
      "title": "Id",
      "type": "integer"
    },
    "name": {
      "default": "Group",
      "title": "Name",
      "type": "string"
    },
    "color": {
      "anyOf": [
        {
          "enum": [
            "slate",
            "blue",
            "green",
            "amber",
            "rose",
            "violet",
            "cyan"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Color"
    },
    "x_position": {
      "default": 0.0,
      "title": "X Position",
      "type": "number"
    },
    "y_position": {
      "default": 0.0,
      "title": "Y Position",
      "type": "number"
    },
    "width": {
      "default": 400.0,
      "title": "Width",
      "type": "number"
    },
    "height": {
      "default": 250.0,
      "title": "Height",
      "type": "number"
    },
    "collapsed": {
      "default": false,
      "title": "Collapsed",
      "type": "boolean"
    },
    "parent_group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Parent Group Id"
    }
  },
  "required": [
    "id"
  ],
  "title": "GroupInformation",
  "type": "object"
}

Fields:

• id (int)
• name (str)
• color (GroupColor | None)
• x_position (float)
• y_position (float)
• width (float)
• height (float)
• collapsed (bool)
• parent_group_id (int | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class GroupInformation(_GroupFields):
    """Runtime representation of a visual node group (stored in FlowGraph._groups)."""

GroupMembershipRequest pydantic-model

Bases: BaseModel

Body for adding/removing nodes from a group.

Show JSON schema:

{
  "description": "Body for adding/removing nodes from a group.",
  "properties": {
    "node_ids": {
      "items": {
        "type": "integer"
      },
      "title": "Node Ids",
      "type": "array"
    }
  },
  "required": [
    "node_ids"
  ],
  "title": "GroupMembershipRequest",
  "type": "object"
}

Fields:

• node_ids (list[int])

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class GroupMembershipRequest(BaseModel):
    """Body for adding/removing nodes from a group."""

    node_ids: list[int]

NodeConnection pydantic-model

Bases: BaseModel

Represents a connection between two nodes in the flow.

Attributes:

Name	Type	Description
from_node_id	int	The ID of the source node.
to_node_id	int	The ID of the target node.

Show JSON schema:

{
  "description": "Represents a connection between two nodes in the flow.\n\nAttributes:\n    from_node_id (int): The ID of the source node.\n    to_node_id (int): The ID of the target node.",
  "properties": {
    "from_node_id": {
      "title": "From Node Id",
      "type": "integer"
    },
    "to_node_id": {
      "title": "To Node Id",
      "type": "integer"
    }
  },
  "required": [
    "from_node_id",
    "to_node_id"
  ],
  "title": "NodeConnection",
  "type": "object"
}

Config:

• frozen: True

Fields:

• from_node_id (int)
• to_node_id (int)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeConnection(BaseModel):
    """
    Represents a connection between two nodes in the flow.

    Attributes:
        from_node_id (int): The ID of the source node.
        to_node_id (int): The ID of the target node.
    """

    model_config = ConfigDict(frozen=True)
    from_node_id: int
    to_node_id: int

NodeDefault pydantic-model

Bases: BaseModel

Defines default properties for a node type.

Attributes:

Name	Type	Description
node_name	str	The name of the node.
node_type	NodeTypeLiteral	The functional type of the node ('input', 'output', 'process').
transform_type	TransformTypeLiteral	The data transformation behavior ('narrow', 'wide', 'other').
has_default_settings	Optional[Any]	Indicates if the node has predefined default settings.

Show JSON schema:

{
  "description": "Defines default properties for a node type.\n\nAttributes:\n    node_name (str): The name of the node.\n    node_type (NodeTypeLiteral): The functional type of the node ('input', 'output', 'process').\n    transform_type (TransformTypeLiteral): The data transformation behavior ('narrow', 'wide', 'other').\n    has_default_settings (Optional[Any]): Indicates if the node has predefined default settings.",
  "properties": {
    "node_name": {
      "title": "Node Name",
      "type": "string"
    },
    "node_type": {
      "enum": [
        "input",
        "output",
        "process"
      ],
      "title": "Node Type",
      "type": "string"
    },
    "transform_type": {
      "enum": [
        "narrow",
        "wide",
        "other"
      ],
      "title": "Transform Type",
      "type": "string"
    },
    "has_default_settings": {
      "anyOf": [
        {},
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Has Default Settings"
    }
  },
  "required": [
    "node_name",
    "node_type",
    "transform_type"
  ],
  "title": "NodeDefault",
  "type": "object"
}

Fields:

• node_name (str)
• node_type (NodeTypeLiteral)
• transform_type (TransformTypeLiteral)
• has_default_settings (Any | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeDefault(BaseModel):
    """
    Defines default properties for a node type.

    Attributes:
        node_name (str): The name of the node.
        node_type (NodeTypeLiteral): The functional type of the node ('input', 'output', 'process').
        transform_type (TransformTypeLiteral): The data transformation behavior ('narrow', 'wide', 'other').
        has_default_settings (Optional[Any]): Indicates if the node has predefined default settings.
    """

    node_name: str
    node_type: NodeTypeLiteral
    transform_type: TransformTypeLiteral
    has_default_settings: Any | None = None

NodeEdge pydantic-model

Bases: BaseModel

Represents a connection (edge) between two nodes in the frontend.

Attributes:

Name	Type	Description
id	str	A unique identifier for the edge.
source	str	The ID of the source node.
target	str	The ID of the target node.
targetHandle	str	The specific input handle on the target node.
sourceHandle	str	The specific output handle on the source node.

Show JSON schema:

{
  "description": "Represents a connection (edge) between two nodes in the frontend.\n\nAttributes:\n    id (str): A unique identifier for the edge.\n    source (str): The ID of the source node.\n    target (str): The ID of the target node.\n    targetHandle (str): The specific input handle on the target node.\n    sourceHandle (str): The specific output handle on the source node.",
  "properties": {
    "id": {
      "title": "Id",
      "type": "string"
    },
    "source": {
      "title": "Source",
      "type": "string"
    },
    "target": {
      "title": "Target",
      "type": "string"
    },
    "targetHandle": {
      "title": "Targethandle",
      "type": "string"
    },
    "sourceHandle": {
      "title": "Sourcehandle",
      "type": "string"
    }
  },
  "required": [
    "id",
    "source",
    "target",
    "targetHandle",
    "sourceHandle"
  ],
  "title": "NodeEdge",
  "type": "object"
}

Config:

• coerce_numbers_to_str: True

Fields:

• id (str)
• source (str)
• target (str)
• targetHandle (str)
• sourceHandle (str)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeEdge(BaseModel):
    """
    Represents a connection (edge) between two nodes in the frontend.

    Attributes:
        id (str): A unique identifier for the edge.
        source (str): The ID of the source node.
        target (str): The ID of the target node.
        targetHandle (str): The specific input handle on the target node.
        sourceHandle (str): The specific output handle on the source node.
    """

    model_config = ConfigDict(coerce_numbers_to_str=True)
    id: str
    source: str
    target: str
    targetHandle: str
    sourceHandle: str

NodeInformation pydantic-model

Bases: BaseModel

Stores the state and configuration of a specific node instance within a flow.

Show JSON schema:

{
  "$defs": {
    "FlowfileInputConnection": {
      "description": "One keyed input edge of a dynamic-input node (per-edge target handle).\n\nOnly nodes whose template sets ``dynamic_inputs`` serialize these; the same\nupstream node may legitimately appear twice with different handles.",
      "properties": {
        "from_id": {
          "title": "From Id",
          "type": "integer"
        },
        "input_handle": {
          "title": "Input Handle",
          "type": "string"
        },
        "source_handle": {
          "default": "output-0",
          "title": "Source Handle",
          "type": "string"
        }
      },
      "required": [
        "from_id",
        "input_handle"
      ],
      "title": "FlowfileInputConnection",
      "type": "object"
    }
  },
  "description": "Stores the state and configuration of a specific node instance within a flow.",
  "properties": {
    "id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Id"
    },
    "type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Type"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Is Setup"
    },
    "is_start_node": {
      "default": false,
      "title": "Is Start Node",
      "type": "boolean"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "x_position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "X Position"
    },
    "y_position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Y Position"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "left_input_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Left Input Id"
    },
    "right_input_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Right Input Id"
    },
    "input_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Input Ids"
    },
    "outputs": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Outputs"
    },
    "output_handles": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Handles"
    },
    "input_connections": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/FlowfileInputConnection"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Input Connections"
    },
    "setting_input": {
      "anyOf": [
        {},
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Setting Input"
    }
  },
  "title": "NodeInformation",
  "type": "object"
}

Fields:

• id (int | None)
• type (str | None)
• is_setup (bool | None)
• is_start_node (bool)
• description (str | None)
• node_reference (str | None)
• x_position (int | None)
• y_position (int | None)
• group_id (int | None)
• left_input_id (int | None)
• right_input_id (int | None)
• input_ids (list[int] | None)
• outputs (list[int] | None)
• output_handles (list[str] | None)
• input_connections (list[FlowfileInputConnection] | None)
• setting_input (Any | None)

Validators:

• validate_setting_input → setting_input

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeInformation(BaseModel):
    """
    Stores the state and configuration of a specific node instance within a flow.
    """

    id: int | None = None
    type: str | None = None
    is_setup: bool | None = None
    is_start_node: bool = False
    description: str | None = ""
    node_reference: str | None = None  # Unique reference identifier for code generation
    x_position: int | None = 0
    y_position: int | None = 0
    group_id: int | None = None
    left_input_id: int | None = None
    right_input_id: int | None = None
    input_ids: list[int] | None = Field(default_factory=list)
    outputs: list[int] | None = Field(default_factory=list)
    output_handles: list[str] | None = None
    input_connections: list[FlowfileInputConnection] | None = None
    setting_input: Any | None = None

    @property
    def data(self) -> Any:
        return self.setting_input

    @property
    def main_input_ids(self) -> list[int] | None:
        return self.input_ids

    @field_validator("setting_input", mode="before")
    @classmethod
    def validate_setting_input(cls, v, info: ValidationInfo):
        if v is None:
            return None
        if isinstance(v, BaseModel):
            return v

        node_type = info.data.get("type")
        model_class = get_settings_class_for_node_type(node_type, v if isinstance(v, dict) else None)

        if model_class is None:
            raise ValueError(f"Unknown node type: {node_type}")

        if isinstance(v, model_class):
            return v

        return model_class.model_validate(v)

NodeInput pydantic-model

Bases: NodeTemplate

Represents a node as it is received from the frontend, including position.

Attributes:

Name	Type	Description
id	int	The unique ID of the node instance.
pos_x	float	The x-coordinate on the canvas.
pos_y	float	The y-coordinate on the canvas.
output_names	list[str] | None	Named outputs for multi-output nodes.
node_reference	str | None	Reference name used for code generation and input naming.

Show JSON schema:

{
  "$defs": {
    "ArtifactDecl": {
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Type"
        }
      },
      "required": [
        "name"
      ],
      "title": "ArtifactDecl",
      "type": "object"
    },
    "NodeTag": {
      "description": "Controlled vocabulary of palette search keywords.\n\nMatched (case-insensitive substring) against the user's query in the node palette so a\nnode surfaces by concept, format, or tool rather than only its display name\n(e.g. \"s3\" -> cloud reader/writer, \"sum\" -> formula and group by). As a ``str`` enum each\nmember serializes to its plain string value for the frontend.",
      "enum": [
        "csv",
        "excel",
        "parquet",
        "json",
        "file",
        "read",
        "write",
        "import",
        "export",
        "save",
        "delta",
        "api",
        "rest",
        "http",
        "external",
        "response",
        "pagination",
        "database",
        "sql",
        "query",
        "table",
        "postgres",
        "mysql",
        "sql server",
        "snowflake",
        "oracle",
        "sqlite",
        "redshift",
        "bigquery",
        "s3",
        "aws",
        "azure",
        "adls",
        "gcs",
        "blob",
        "bucket",
        "cloud",
        "catalog",
        "lakehouse",
        "time travel",
        "kafka",
        "redpanda",
        "streaming",
        "topic",
        "google analytics",
        "ga4",
        "analytics",
        "manual",
        "paste",
        "input",
        "select",
        "columns",
        "rename",
        "reorder",
        "projection",
        "filter",
        "where",
        "subset",
        "sample",
        "limit",
        "head",
        "formula",
        "expression",
        "calculate",
        "math",
        "concat",
        "transform",
        "group by",
        "aggregate",
        "sum",
        "mean",
        "average",
        "count",
        "min",
        "max",
        "median",
        "summarize",
        "record count",
        "rows",
        "window",
        "rolling",
        "cumulative",
        "rank",
        "partition",
        "lag",
        "lead",
        "join",
        "merge",
        "lookup",
        "vlookup",
        "inner",
        "outer",
        "cross join",
        "cartesian",
        "fuzzy",
        "similarity",
        "levenshtein",
        "union",
        "append",
        "wait",
        "dependency",
        "pivot",
        "crosstab",
        "unpivot",
        "melt",
        "reshape",
        "text to rows",
        "split",
        "explode",
        "unique",
        "dedupe",
        "distinct",
        "drop duplicates",
        "graph",
        "network",
        "cluster",
        "connected components",
        "record id",
        "row number",
        "index",
        "sort",
        "order",
        "ascending",
        "descending",
        "polars",
        "code",
        "python",
        "script",
        "kernel",
        "custom",
        "dataframe",
        "explore",
        "profile",
        "preview",
        "eda",
        "statistics",
        "visualize",
        "bar chart",
        "insight",
        "graphs",
        "ml",
        "machine learning",
        "train",
        "test",
        "model",
        "regression",
        "classification",
        "predict",
        "score",
        "evaluate",
        "metrics"
      ],
      "title": "NodeTag",
      "type": "string"
    }
  },
  "description": "Represents a node as it is received from the frontend, including position.\n\nAttributes:\n    id (int): The unique ID of the node instance.\n    pos_x (float): The x-coordinate on the canvas.\n    pos_y (float): The y-coordinate on the canvas.\n    output_names (list[str] | None): Named outputs for multi-output nodes.\n    node_reference (str | None): Reference name used for code generation and input naming.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "item": {
      "title": "Item",
      "type": "string"
    },
    "input": {
      "title": "Input",
      "type": "integer"
    },
    "output": {
      "title": "Output",
      "type": "integer"
    },
    "image": {
      "title": "Image",
      "type": "string"
    },
    "multi": {
      "default": false,
      "title": "Multi",
      "type": "boolean"
    },
    "node_type": {
      "enum": [
        "input",
        "output",
        "process"
      ],
      "title": "Node Type",
      "type": "string"
    },
    "transform_type": {
      "enum": [
        "narrow",
        "wide",
        "other"
      ],
      "title": "Transform Type",
      "type": "string"
    },
    "node_group": {
      "title": "Node Group",
      "type": "string"
    },
    "node_group_label": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Group Label"
    },
    "prod_ready": {
      "default": true,
      "title": "Prod Ready",
      "type": "boolean"
    },
    "can_be_start": {
      "default": false,
      "title": "Can Be Start",
      "type": "boolean"
    },
    "drawer_title": {
      "default": "Node title",
      "title": "Drawer Title",
      "type": "string"
    },
    "drawer_intro": {
      "default": "Drawer into",
      "title": "Drawer Intro",
      "type": "string"
    },
    "custom_node": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Custom Node"
    },
    "execution_environment": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Execution Environment"
    },
    "dependencies": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Dependencies"
    },
    "publishes": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/ArtifactDecl"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Publishes"
    },
    "laziness": {
      "default": "eager",
      "enum": [
        "lazy",
        "eager",
        "conditional"
      ],
      "title": "Laziness",
      "type": "string"
    },
    "output_names": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Names"
    },
    "input_labels": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Input Labels"
    },
    "dynamic_inputs": {
      "default": false,
      "title": "Dynamic Inputs",
      "type": "boolean"
    },
    "tags": {
      "items": {
        "$ref": "#/$defs/NodeTag"
      },
      "title": "Tags",
      "type": "array"
    },
    "id": {
      "title": "Id",
      "type": "integer"
    },
    "pos_x": {
      "title": "Pos X",
      "type": "number"
    },
    "pos_y": {
      "title": "Pos Y",
      "type": "number"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "input_names": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Input Names"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    }
  },
  "required": [
    "name",
    "item",
    "input",
    "output",
    "image",
    "node_type",
    "transform_type",
    "node_group",
    "id",
    "pos_x",
    "pos_y"
  ],
  "title": "NodeInput",
  "type": "object"
}

Fields:

• name (str)
• item (str)
• input (int)
• output (int)
• image (str)
• multi (bool)
• node_type (NodeTypeLiteral)
• transform_type (TransformTypeLiteral)
• node_group (str)
• node_group_label (str | None)
• prod_ready (bool)
• can_be_start (bool)
• drawer_title (str)
• drawer_intro (str)
• custom_node (bool | None)
• execution_environment (str | None)
• dependencies (list[str] | None)
• publishes (list[ArtifactDecl] | None)
• laziness (LazinessLiteral)
• input_labels (list[str] | None)
• dynamic_inputs (bool)
• tags (list[NodeTag])
• id (int)
• pos_x (float)
• pos_y (float)
• group_id (int | None)
• output_names (list[str] | None)
• input_names (list[str] | None)
• node_reference (str | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeInput(NodeTemplate):
    """
    Represents a node as it is received from the frontend, including position.

    Attributes:
        id (int): The unique ID of the node instance.
        pos_x (float): The x-coordinate on the canvas.
        pos_y (float): The y-coordinate on the canvas.
        output_names (list[str] | None): Named outputs for multi-output nodes.
        node_reference (str | None): Reference name used for code generation and input naming.
    """

    id: int
    pos_x: float
    pos_y: float
    group_id: int | None = None
    output_names: list[str] | None = None
    # Dynamic-input nodes: label per input handle, index i <-> "input-{i}"
    # (index 0 is the reserved parameter handle). None for static nodes.
    input_names: list[str] | None = None
    node_reference: str | None = None

NodePositionUpdate pydantic-model

Bases: BaseModel

A single node's new absolute canvas position.

Show JSON schema:

{
  "description": "A single node's new absolute canvas position.",
  "properties": {
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "pos_x": {
      "title": "Pos X",
      "type": "number"
    },
    "pos_y": {
      "title": "Pos Y",
      "type": "number"
    }
  },
  "required": [
    "node_id",
    "pos_x",
    "pos_y"
  ],
  "title": "NodePositionUpdate",
  "type": "object"
}

Fields:

• node_id (int)
• pos_x (float)
• pos_y (float)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodePositionUpdate(BaseModel):
    """A single node's new absolute canvas position."""

    node_id: int
    pos_x: float
    pos_y: float

NodeTag

Bases: str, Enum

Controlled vocabulary of palette search keywords.

Matched (case-insensitive substring) against the user's query in the node palette so a node surfaces by concept, format, or tool rather than only its display name (e.g. "s3" -> cloud reader/writer, "sum" -> formula and group by). As a str enum each member serializes to its plain string value for the frontend.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeTag(str, Enum):
    """Controlled vocabulary of palette search keywords.

    Matched (case-insensitive substring) against the user's query in the node palette so a
    node surfaces by concept, format, or tool rather than only its display name
    (e.g. "s3" -> cloud reader/writer, "sum" -> formula and group by). As a ``str`` enum each
    member serializes to its plain string value for the frontend.
    """

    # File formats & local IO
    CSV = "csv"
    EXCEL = "excel"
    PARQUET = "parquet"
    JSON = "json"
    FILE = "file"
    READ = "read"
    WRITE = "write"
    IMPORT = "import"
    EXPORT = "export"
    SAVE = "save"
    DELTA = "delta"

    # Connectivity & APIs
    API = "api"
    REST = "rest"
    HTTP = "http"
    EXTERNAL = "external"
    RESPONSE = "response"
    PAGINATION = "pagination"

    # Databases
    DATABASE = "database"
    SQL = "sql"
    QUERY = "query"
    TABLE = "table"
    POSTGRES = "postgres"
    MYSQL = "mysql"
    SQL_SERVER = "sql server"
    SNOWFLAKE = "snowflake"
    ORACLE = "oracle"
    SQLITE = "sqlite"
    REDSHIFT = "redshift"
    BIGQUERY = "bigquery"

    # Cloud storage
    S3 = "s3"
    AWS = "aws"
    AZURE = "azure"
    ADLS = "adls"
    GCS = "gcs"
    BLOB = "blob"
    BUCKET = "bucket"
    CLOUD = "cloud"

    # Catalog / lakehouse
    CATALOG = "catalog"
    LAKEHOUSE = "lakehouse"
    TIME_TRAVEL = "time travel"

    # Streaming
    KAFKA = "kafka"
    REDPANDA = "redpanda"
    STREAMING = "streaming"
    TOPIC = "topic"

    # Analytics sources
    GOOGLE_ANALYTICS = "google analytics"
    GA4 = "ga4"
    ANALYTICS = "analytics"

    # Data entry
    MANUAL = "manual"
    PASTE = "paste"
    INPUT = "input"

    # Column shaping
    SELECT = "select"
    COLUMNS = "columns"
    RENAME = "rename"
    REORDER = "reorder"
    PROJECTION = "projection"

    # Row selection
    FILTER = "filter"
    WHERE = "where"
    SUBSET = "subset"
    SAMPLE = "sample"
    LIMIT = "limit"
    HEAD = "head"

    # Formula / compute
    FORMULA = "formula"
    EXPRESSION = "expression"
    CALCULATE = "calculate"
    MATH = "math"
    CONCAT = "concat"
    TRANSFORM = "transform"

    # Aggregation
    GROUP_BY = "group by"
    AGGREGATE = "aggregate"
    SUM = "sum"
    MEAN = "mean"
    AVERAGE = "average"
    COUNT = "count"
    MIN = "min"
    MAX = "max"
    MEDIAN = "median"
    SUMMARIZE = "summarize"
    RECORD_COUNT = "record count"
    ROWS = "rows"

    # Window functions
    WINDOW = "window"
    ROLLING = "rolling"
    CUMULATIVE = "cumulative"
    RANK = "rank"
    PARTITION = "partition"
    LAG = "lag"
    LEAD = "lead"

    # Joins & combine
    JOIN = "join"
    MERGE = "merge"
    LOOKUP = "lookup"
    VLOOKUP = "vlookup"
    INNER = "inner"
    OUTER = "outer"
    CROSS_JOIN = "cross join"
    CARTESIAN = "cartesian"
    FUZZY = "fuzzy"
    SIMILARITY = "similarity"
    LEVENSHTEIN = "levenshtein"
    UNION = "union"
    APPEND = "append"
    WAIT = "wait"
    DEPENDENCY = "dependency"

    # Reshape
    PIVOT = "pivot"
    CROSSTAB = "crosstab"
    UNPIVOT = "unpivot"
    MELT = "melt"
    RESHAPE = "reshape"
    TEXT_TO_ROWS = "text to rows"
    SPLIT = "split"
    EXPLODE = "explode"

    # Deduplication
    UNIQUE = "unique"
    DEDUPE = "dedupe"
    DISTINCT = "distinct"
    DROP_DUPLICATES = "drop duplicates"

    # Graph
    GRAPH = "graph"
    NETWORK = "network"
    CLUSTER = "cluster"
    CONNECTED_COMPONENTS = "connected components"

    # Identifiers & ordering
    RECORD_ID = "record id"
    ROW_NUMBER = "row number"
    INDEX = "index"
    SORT = "sort"
    ORDER = "order"
    ASCENDING = "ascending"
    DESCENDING = "descending"

    # Code
    POLARS = "polars"
    CODE = "code"
    PYTHON = "python"
    SCRIPT = "script"
    KERNEL = "kernel"
    CUSTOM = "custom"
    DATAFRAME = "dataframe"

    # Explore
    EXPLORE = "explore"
    PROFILE = "profile"
    PREVIEW = "preview"
    EDA = "eda"
    STATISTICS = "statistics"
    VISUALIZE = "visualize"
    BAR_CHART = "bar chart"
    INSIGHT = "insight"
    GRAPHS = "graphs"

    # Machine learning
    ML = "ml"
    MACHINE_LEARNING = "machine learning"
    TRAIN = "train"
    TEST = "test"
    MODEL = "model"
    REGRESSION = "regression"
    CLASSIFICATION = "classification"
    PREDICT = "predict"
    SCORE = "score"
    EVALUATE = "evaluate"
    METRICS = "metrics"

NodeTemplate pydantic-model

Bases: BaseModel

Defines the template for a node type, specifying its UI and functional characteristics.

Attributes:

Name	Type	Description
name	str	The display name of the node.
item	str	The unique identifier for the node type.
input	int	The number of required input connections.
output	int	The number of output connections.
image	str	The filename of the icon for the node.
multi	bool	Whether the node accepts multiple main input connections.
node_group	str	The category group the node belongs to (e.g., 'input', 'transform').
prod_ready	bool	Whether the node is considered production-ready.
can_be_start	bool	Whether the node can be a starting point in a flow.

Show JSON schema:

{
  "$defs": {
    "ArtifactDecl": {
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Type"
        }
      },
      "required": [
        "name"
      ],
      "title": "ArtifactDecl",
      "type": "object"
    },
    "NodeTag": {
      "description": "Controlled vocabulary of palette search keywords.\n\nMatched (case-insensitive substring) against the user's query in the node palette so a\nnode surfaces by concept, format, or tool rather than only its display name\n(e.g. \"s3\" -> cloud reader/writer, \"sum\" -> formula and group by). As a ``str`` enum each\nmember serializes to its plain string value for the frontend.",
      "enum": [
        "csv",
        "excel",
        "parquet",
        "json",
        "file",
        "read",
        "write",
        "import",
        "export",
        "save",
        "delta",
        "api",
        "rest",
        "http",
        "external",
        "response",
        "pagination",
        "database",
        "sql",
        "query",
        "table",
        "postgres",
        "mysql",
        "sql server",
        "snowflake",
        "oracle",
        "sqlite",
        "redshift",
        "bigquery",
        "s3",
        "aws",
        "azure",
        "adls",
        "gcs",
        "blob",
        "bucket",
        "cloud",
        "catalog",
        "lakehouse",
        "time travel",
        "kafka",
        "redpanda",
        "streaming",
        "topic",
        "google analytics",
        "ga4",
        "analytics",
        "manual",
        "paste",
        "input",
        "select",
        "columns",
        "rename",
        "reorder",
        "projection",
        "filter",
        "where",
        "subset",
        "sample",
        "limit",
        "head",
        "formula",
        "expression",
        "calculate",
        "math",
        "concat",
        "transform",
        "group by",
        "aggregate",
        "sum",
        "mean",
        "average",
        "count",
        "min",
        "max",
        "median",
        "summarize",
        "record count",
        "rows",
        "window",
        "rolling",
        "cumulative",
        "rank",
        "partition",
        "lag",
        "lead",
        "join",
        "merge",
        "lookup",
        "vlookup",
        "inner",
        "outer",
        "cross join",
        "cartesian",
        "fuzzy",
        "similarity",
        "levenshtein",
        "union",
        "append",
        "wait",
        "dependency",
        "pivot",
        "crosstab",
        "unpivot",
        "melt",
        "reshape",
        "text to rows",
        "split",
        "explode",
        "unique",
        "dedupe",
        "distinct",
        "drop duplicates",
        "graph",
        "network",
        "cluster",
        "connected components",
        "record id",
        "row number",
        "index",
        "sort",
        "order",
        "ascending",
        "descending",
        "polars",
        "code",
        "python",
        "script",
        "kernel",
        "custom",
        "dataframe",
        "explore",
        "profile",
        "preview",
        "eda",
        "statistics",
        "visualize",
        "bar chart",
        "insight",
        "graphs",
        "ml",
        "machine learning",
        "train",
        "test",
        "model",
        "regression",
        "classification",
        "predict",
        "score",
        "evaluate",
        "metrics"
      ],
      "title": "NodeTag",
      "type": "string"
    }
  },
  "description": "Defines the template for a node type, specifying its UI and functional characteristics.\n\nAttributes:\n    name (str): The display name of the node.\n    item (str): The unique identifier for the node type.\n    input (int): The number of required input connections.\n    output (int): The number of output connections.\n    image (str): The filename of the icon for the node.\n    multi (bool): Whether the node accepts multiple main input connections.\n    node_group (str): The category group the node belongs to (e.g., 'input', 'transform').\n    prod_ready (bool): Whether the node is considered production-ready.\n    can_be_start (bool): Whether the node can be a starting point in a flow.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "item": {
      "title": "Item",
      "type": "string"
    },
    "input": {
      "title": "Input",
      "type": "integer"
    },
    "output": {
      "title": "Output",
      "type": "integer"
    },
    "image": {
      "title": "Image",
      "type": "string"
    },
    "multi": {
      "default": false,
      "title": "Multi",
      "type": "boolean"
    },
    "node_type": {
      "enum": [
        "input",
        "output",
        "process"
      ],
      "title": "Node Type",
      "type": "string"
    },
    "transform_type": {
      "enum": [
        "narrow",
        "wide",
        "other"
      ],
      "title": "Transform Type",
      "type": "string"
    },
    "node_group": {
      "title": "Node Group",
      "type": "string"
    },
    "node_group_label": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Group Label"
    },
    "prod_ready": {
      "default": true,
      "title": "Prod Ready",
      "type": "boolean"
    },
    "can_be_start": {
      "default": false,
      "title": "Can Be Start",
      "type": "boolean"
    },
    "drawer_title": {
      "default": "Node title",
      "title": "Drawer Title",
      "type": "string"
    },
    "drawer_intro": {
      "default": "Drawer into",
      "title": "Drawer Intro",
      "type": "string"
    },
    "custom_node": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Custom Node"
    },
    "execution_environment": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Execution Environment"
    },
    "dependencies": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Dependencies"
    },
    "publishes": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/ArtifactDecl"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Publishes"
    },
    "laziness": {
      "default": "eager",
      "enum": [
        "lazy",
        "eager",
        "conditional"
      ],
      "title": "Laziness",
      "type": "string"
    },
    "output_names": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Names"
    },
    "input_labels": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Input Labels"
    },
    "dynamic_inputs": {
      "default": false,
      "title": "Dynamic Inputs",
      "type": "boolean"
    },
    "tags": {
      "items": {
        "$ref": "#/$defs/NodeTag"
      },
      "title": "Tags",
      "type": "array"
    }
  },
  "required": [
    "name",
    "item",
    "input",
    "output",
    "image",
    "node_type",
    "transform_type",
    "node_group"
  ],
  "title": "NodeTemplate",
  "type": "object"
}

Fields:

• name (str)
• item (str)
• input (int)
• output (int)
• image (str)
• multi (bool)
• node_type (NodeTypeLiteral)
• transform_type (TransformTypeLiteral)
• node_group (str)
• node_group_label (str | None)
• prod_ready (bool)
• can_be_start (bool)
• drawer_title (str)
• drawer_intro (str)
• custom_node (bool | None)
• execution_environment (str | None)
• dependencies (list[str] | None)
• publishes (list[ArtifactDecl] | None)
• laziness (LazinessLiteral)
• output_names (list[str] | None)
• input_labels (list[str] | None)
• dynamic_inputs (bool)
• tags (list[NodeTag])

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class NodeTemplate(BaseModel):
    """
    Defines the template for a node type, specifying its UI and functional characteristics.

    Attributes:
        name (str): The display name of the node.
        item (str): The unique identifier for the node type.
        input (int): The number of required input connections.
        output (int): The number of output connections.
        image (str): The filename of the icon for the node.
        multi (bool): Whether the node accepts multiple main input connections.
        node_group (str): The category group the node belongs to (e.g., 'input', 'transform').
        prod_ready (bool): Whether the node is considered production-ready.
        can_be_start (bool): Whether the node can be a starting point in a flow.
    """

    name: str
    item: str
    input: int
    output: int
    image: str
    multi: bool = False
    node_type: NodeTypeLiteral
    transform_type: TransformTypeLiteral
    node_group: str
    # Display name for dynamic (custom-category) palette groups; None for built-ins.
    node_group_label: str | None = None
    prod_ready: bool = True
    can_be_start: bool = False
    drawer_title: str = "Node title"
    drawer_intro: str = "Drawer into"
    custom_node: bool | None = False
    execution_environment: str | None = None
    dependencies: list[str] | None = None
    publishes: list[ArtifactDecl] | None = None
    laziness: LazinessLiteral = "eager"
    output_names: list[str] | None = None
    # Display-only names for the canvas input handles, index i <-> "input-{i}".
    # Purely cosmetic: unlike output_names these are never dict keys and nothing
    # reads them at execution time. Distinct from NodeInput.input_names, which is
    # the per-instance dynamic-handle list (run_flow) with a reserved index 0.
    input_labels: list[str] | None = None
    # Per-instance input handles (run_flow): connections are keyed by target
    # handle instead of collapsing onto input-0. See flow_node/input_handles.py.
    dynamic_inputs: bool = False
    tags: list[NodeTag] = Field(default_factory=list)

RawLogInput pydantic-model

Bases: BaseModel

Schema for a raw log message.

Attributes:

Name	Type	Description
flowfile_flow_id	int	The ID of the flow that generated the log.
log_message	str	The content of the log message.
log_type	Literal['INFO', 'WARNING', 'ERROR']	The type of log.
node_id	int | None	Optional node ID to attribute the log to.
extra	Optional[dict]	Extra context data for the log.

Show JSON schema:

{
  "description": "Schema for a raw log message.\n\nAttributes:\n    flowfile_flow_id (int): The ID of the flow that generated the log.\n    log_message (str): The content of the log message.\n    log_type (Literal[\"INFO\", \"WARNING\", \"ERROR\"]): The type of log.\n    node_id (int | None): Optional node ID to attribute the log to.\n    extra (Optional[dict]): Extra context data for the log.",
  "properties": {
    "flowfile_flow_id": {
      "title": "Flowfile Flow Id",
      "type": "integer"
    },
    "log_message": {
      "title": "Log Message",
      "type": "string"
    },
    "log_type": {
      "enum": [
        "INFO",
        "WARNING",
        "ERROR"
      ],
      "title": "Log Type",
      "type": "string"
    },
    "node_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Id"
    },
    "extra": {
      "anyOf": [
        {
          "additionalProperties": true,
          "type": "object"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Extra"
    }
  },
  "required": [
    "flowfile_flow_id",
    "log_message",
    "log_type"
  ],
  "title": "RawLogInput",
  "type": "object"
}

Fields:

• flowfile_flow_id (int)
• log_message (str)
• log_type (Literal['INFO', 'WARNING', 'ERROR'])
• node_id (int | None)
• extra (dict | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class RawLogInput(BaseModel):
    """
    Schema for a raw log message.

    Attributes:
        flowfile_flow_id (int): The ID of the flow that generated the log.
        log_message (str): The content of the log message.
        log_type (Literal["INFO", "WARNING", "ERROR"]): The type of log.
        node_id (int | None): Optional node ID to attribute the log to.
        extra (Optional[dict]): Extra context data for the log.
    """

    flowfile_flow_id: int
    log_message: str
    log_type: Literal["INFO", "WARNING", "ERROR"]
    node_id: int | None = None
    extra: dict | None = None

UpdateGroupRequest pydantic-model

Bases: BaseModel

Body for POST /editor/update_group/. All fields optional -> partial update.

Show JSON schema:

{
  "description": "Body for POST /editor/update_group/. All fields optional -> partial update.",
  "properties": {
    "name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Name"
    },
    "color": {
      "anyOf": [
        {
          "enum": [
            "slate",
            "blue",
            "green",
            "amber",
            "rose",
            "violet",
            "cyan"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Color"
    },
    "x_position": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "X Position"
    },
    "y_position": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Y Position"
    },
    "width": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Width"
    },
    "height": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Height"
    },
    "collapsed": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Collapsed"
    }
  },
  "title": "UpdateGroupRequest",
  "type": "object"
}

Fields:

• name (str | None)
• color (GroupColor | None)
• x_position (float | None)
• y_position (float | None)
• width (float | None)
• height (float | None)
• collapsed (bool | None)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class UpdateGroupRequest(BaseModel):
    """Body for POST /editor/update_group/. All fields optional -> partial update."""

    name: str | None = None
    color: GroupColor | None = None
    x_position: float | None = None
    y_position: float | None = None
    width: float | None = None
    height: float | None = None
    collapsed: bool | None = None

UpdateLayoutRequest pydantic-model

Bases: BaseModel

Batch persistence of dragged node positions and/or group bounds (one drag-end -> one call).

Show JSON schema:

{
  "$defs": {
    "GroupBoundsUpdate": {
      "description": "A single group's new absolute bounds.",
      "properties": {
        "group_id": {
          "title": "Group Id",
          "type": "integer"
        },
        "x_position": {
          "title": "X Position",
          "type": "number"
        },
        "y_position": {
          "title": "Y Position",
          "type": "number"
        },
        "width": {
          "title": "Width",
          "type": "number"
        },
        "height": {
          "title": "Height",
          "type": "number"
        }
      },
      "required": [
        "group_id",
        "x_position",
        "y_position",
        "width",
        "height"
      ],
      "title": "GroupBoundsUpdate",
      "type": "object"
    },
    "NodePositionUpdate": {
      "description": "A single node's new absolute canvas position.",
      "properties": {
        "node_id": {
          "title": "Node Id",
          "type": "integer"
        },
        "pos_x": {
          "title": "Pos X",
          "type": "number"
        },
        "pos_y": {
          "title": "Pos Y",
          "type": "number"
        }
      },
      "required": [
        "node_id",
        "pos_x",
        "pos_y"
      ],
      "title": "NodePositionUpdate",
      "type": "object"
    }
  },
  "description": "Batch persistence of dragged node positions and/or group bounds (one drag-end -> one call).",
  "properties": {
    "node_positions": {
      "items": {
        "$ref": "#/$defs/NodePositionUpdate"
      },
      "title": "Node Positions",
      "type": "array"
    },
    "group_bounds": {
      "items": {
        "$ref": "#/$defs/GroupBoundsUpdate"
      },
      "title": "Group Bounds",
      "type": "array"
    },
    "record_history": {
      "default": true,
      "title": "Record History",
      "type": "boolean"
    }
  },
  "title": "UpdateLayoutRequest",
  "type": "object"
}

Fields:

• node_positions (list[NodePositionUpdate])
• group_bounds (list[GroupBoundsUpdate])
• record_history (bool)

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class UpdateLayoutRequest(BaseModel):
    """Batch persistence of dragged node positions and/or group bounds (one drag-end -> one call)."""

    node_positions: list[NodePositionUpdate] = Field(default_factory=list)
    group_bounds: list[GroupBoundsUpdate] = Field(default_factory=list)
    # False -> apply without a new undo entry (folds into a preceding op's snapshot).
    record_history: bool = True

VueFlowInput pydantic-model

Bases: BaseModel

Represents the complete graph structure from the Vue-based frontend.

Attributes:

Name	Type	Description
node_edges	List[NodeEdge]	A list of all edges in the graph.
node_inputs	List[NodeInput]	A list of all nodes in the graph.

Show JSON schema:

{
  "$defs": {
    "ArtifactDecl": {
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Type"
        }
      },
      "required": [
        "name"
      ],
      "title": "ArtifactDecl",
      "type": "object"
    },
    "FlowfileGroup": {
      "description": "Serialized representation of a visual node group (YAML/JSON).",
      "properties": {
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "name": {
          "default": "Group",
          "title": "Name",
          "type": "string"
        },
        "color": {
          "anyOf": [
            {
              "enum": [
                "slate",
                "blue",
                "green",
                "amber",
                "rose",
                "violet",
                "cyan"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Color"
        },
        "x_position": {
          "default": 0.0,
          "title": "X Position",
          "type": "number"
        },
        "y_position": {
          "default": 0.0,
          "title": "Y Position",
          "type": "number"
        },
        "width": {
          "default": 400.0,
          "title": "Width",
          "type": "number"
        },
        "height": {
          "default": 250.0,
          "title": "Height",
          "type": "number"
        },
        "collapsed": {
          "default": false,
          "title": "Collapsed",
          "type": "boolean"
        },
        "parent_group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parent Group Id"
        }
      },
      "required": [
        "id"
      ],
      "title": "FlowfileGroup",
      "type": "object"
    },
    "NodeEdge": {
      "description": "Represents a connection (edge) between two nodes in the frontend.\n\nAttributes:\n    id (str): A unique identifier for the edge.\n    source (str): The ID of the source node.\n    target (str): The ID of the target node.\n    targetHandle (str): The specific input handle on the target node.\n    sourceHandle (str): The specific output handle on the source node.",
      "properties": {
        "id": {
          "title": "Id",
          "type": "string"
        },
        "source": {
          "title": "Source",
          "type": "string"
        },
        "target": {
          "title": "Target",
          "type": "string"
        },
        "targetHandle": {
          "title": "Targethandle",
          "type": "string"
        },
        "sourceHandle": {
          "title": "Sourcehandle",
          "type": "string"
        }
      },
      "required": [
        "id",
        "source",
        "target",
        "targetHandle",
        "sourceHandle"
      ],
      "title": "NodeEdge",
      "type": "object"
    },
    "NodeInput": {
      "description": "Represents a node as it is received from the frontend, including position.\n\nAttributes:\n    id (int): The unique ID of the node instance.\n    pos_x (float): The x-coordinate on the canvas.\n    pos_y (float): The y-coordinate on the canvas.\n    output_names (list[str] | None): Named outputs for multi-output nodes.\n    node_reference (str | None): Reference name used for code generation and input naming.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "item": {
          "title": "Item",
          "type": "string"
        },
        "input": {
          "title": "Input",
          "type": "integer"
        },
        "output": {
          "title": "Output",
          "type": "integer"
        },
        "image": {
          "title": "Image",
          "type": "string"
        },
        "multi": {
          "default": false,
          "title": "Multi",
          "type": "boolean"
        },
        "node_type": {
          "enum": [
            "input",
            "output",
            "process"
          ],
          "title": "Node Type",
          "type": "string"
        },
        "transform_type": {
          "enum": [
            "narrow",
            "wide",
            "other"
          ],
          "title": "Transform Type",
          "type": "string"
        },
        "node_group": {
          "title": "Node Group",
          "type": "string"
        },
        "node_group_label": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Node Group Label"
        },
        "prod_ready": {
          "default": true,
          "title": "Prod Ready",
          "type": "boolean"
        },
        "can_be_start": {
          "default": false,
          "title": "Can Be Start",
          "type": "boolean"
        },
        "drawer_title": {
          "default": "Node title",
          "title": "Drawer Title",
          "type": "string"
        },
        "drawer_intro": {
          "default": "Drawer into",
          "title": "Drawer Intro",
          "type": "string"
        },
        "custom_node": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": false,
          "title": "Custom Node"
        },
        "execution_environment": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Execution Environment"
        },
        "dependencies": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Dependencies"
        },
        "publishes": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/ArtifactDecl"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Publishes"
        },
        "laziness": {
          "default": "eager",
          "enum": [
            "lazy",
            "eager",
            "conditional"
          ],
          "title": "Laziness",
          "type": "string"
        },
        "output_names": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Names"
        },
        "input_labels": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Input Labels"
        },
        "dynamic_inputs": {
          "default": false,
          "title": "Dynamic Inputs",
          "type": "boolean"
        },
        "tags": {
          "items": {
            "$ref": "#/$defs/NodeTag"
          },
          "title": "Tags",
          "type": "array"
        },
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "pos_x": {
          "title": "Pos X",
          "type": "number"
        },
        "pos_y": {
          "title": "Pos Y",
          "type": "number"
        },
        "group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Group Id"
        },
        "input_names": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Input Names"
        },
        "node_reference": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Node Reference"
        }
      },
      "required": [
        "name",
        "item",
        "input",
        "output",
        "image",
        "node_type",
        "transform_type",
        "node_group",
        "id",
        "pos_x",
        "pos_y"
      ],
      "title": "NodeInput",
      "type": "object"
    },
    "NodeTag": {
      "description": "Controlled vocabulary of palette search keywords.\n\nMatched (case-insensitive substring) against the user's query in the node palette so a\nnode surfaces by concept, format, or tool rather than only its display name\n(e.g. \"s3\" -> cloud reader/writer, \"sum\" -> formula and group by). As a ``str`` enum each\nmember serializes to its plain string value for the frontend.",
      "enum": [
        "csv",
        "excel",
        "parquet",
        "json",
        "file",
        "read",
        "write",
        "import",
        "export",
        "save",
        "delta",
        "api",
        "rest",
        "http",
        "external",
        "response",
        "pagination",
        "database",
        "sql",
        "query",
        "table",
        "postgres",
        "mysql",
        "sql server",
        "snowflake",
        "oracle",
        "sqlite",
        "redshift",
        "bigquery",
        "s3",
        "aws",
        "azure",
        "adls",
        "gcs",
        "blob",
        "bucket",
        "cloud",
        "catalog",
        "lakehouse",
        "time travel",
        "kafka",
        "redpanda",
        "streaming",
        "topic",
        "google analytics",
        "ga4",
        "analytics",
        "manual",
        "paste",
        "input",
        "select",
        "columns",
        "rename",
        "reorder",
        "projection",
        "filter",
        "where",
        "subset",
        "sample",
        "limit",
        "head",
        "formula",
        "expression",
        "calculate",
        "math",
        "concat",
        "transform",
        "group by",
        "aggregate",
        "sum",
        "mean",
        "average",
        "count",
        "min",
        "max",
        "median",
        "summarize",
        "record count",
        "rows",
        "window",
        "rolling",
        "cumulative",
        "rank",
        "partition",
        "lag",
        "lead",
        "join",
        "merge",
        "lookup",
        "vlookup",
        "inner",
        "outer",
        "cross join",
        "cartesian",
        "fuzzy",
        "similarity",
        "levenshtein",
        "union",
        "append",
        "wait",
        "dependency",
        "pivot",
        "crosstab",
        "unpivot",
        "melt",
        "reshape",
        "text to rows",
        "split",
        "explode",
        "unique",
        "dedupe",
        "distinct",
        "drop duplicates",
        "graph",
        "network",
        "cluster",
        "connected components",
        "record id",
        "row number",
        "index",
        "sort",
        "order",
        "ascending",
        "descending",
        "polars",
        "code",
        "python",
        "script",
        "kernel",
        "custom",
        "dataframe",
        "explore",
        "profile",
        "preview",
        "eda",
        "statistics",
        "visualize",
        "bar chart",
        "insight",
        "graphs",
        "ml",
        "machine learning",
        "train",
        "test",
        "model",
        "regression",
        "classification",
        "predict",
        "score",
        "evaluate",
        "metrics"
      ],
      "title": "NodeTag",
      "type": "string"
    }
  },
  "description": "Represents the complete graph structure from the Vue-based frontend.\n\nAttributes:\n    node_edges (List[NodeEdge]): A list of all edges in the graph.\n    node_inputs (List[NodeInput]): A list of all nodes in the graph.",
  "properties": {
    "node_edges": {
      "items": {
        "$ref": "#/$defs/NodeEdge"
      },
      "title": "Node Edges",
      "type": "array"
    },
    "node_inputs": {
      "items": {
        "$ref": "#/$defs/NodeInput"
      },
      "title": "Node Inputs",
      "type": "array"
    },
    "groups": {
      "items": {
        "$ref": "#/$defs/FlowfileGroup"
      },
      "title": "Groups",
      "type": "array"
    }
  },
  "required": [
    "node_edges",
    "node_inputs"
  ],
  "title": "VueFlowInput",
  "type": "object"
}

Fields:

• node_edges (list[NodeEdge])
• node_inputs (list[NodeInput])
• groups (list[FlowfileGroup])

Source code in flowfile_core/flowfile_core/schemas/schemas.py

class VueFlowInput(BaseModel):
    """

    Represents the complete graph structure from the Vue-based frontend.

    Attributes:
        node_edges (List[NodeEdge]): A list of all edges in the graph.
        node_inputs (List[NodeInput]): A list of all nodes in the graph.
    """

    node_edges: list[NodeEdge]
    node_inputs: list[NodeInput]
    groups: list[FlowfileGroup] = Field(default_factory=list)

get_global_execution_location()

Calculates the default execution location based on the global settings Returns

---

ExecutionLocationsLiteral where the current

Source code in flowfile_core/flowfile_core/schemas/schemas.py

def get_global_execution_location() -> ExecutionLocationsLiteral:
    """
    Calculates the default execution location based on the global settings
    Returns
    -------
    ExecutionLocationsLiteral where the current
    """
    if OFFLOAD_TO_WORKER:
        return "remote"
    return "local"

get_settings_class_for_node_type(node_type, setting_data=None)

Get the settings class for a node type, supporting both standard and user-defined nodes.

setting_data (the raw stored settings dict, when available) lets flows referencing a custom node that is missing from the store still resolve to UserDefinedNode instead of failing as an unknown type.

Source code in flowfile_core/flowfile_core/schemas/schemas.py

def get_settings_class_for_node_type(node_type: str, setting_data: dict | None = None):
    """Get the settings class for a node type, supporting both standard and user-defined nodes.

    ``setting_data`` (the raw stored settings dict, when available) lets flows
    referencing a custom node that is missing from the store still resolve to
    ``UserDefinedNode`` instead of failing as an unknown type.
    """
    # A user-defined node whose key collides with a built-in type must still
    # resolve to UserDefinedNode. Built-in schemas drop ``is_user_defined`` on
    # serialization, so a truthy marker means a genuine custom node — it wins
    # over the built-in type->schema mapping.
    if isinstance(setting_data, dict) and setting_data.get("is_user_defined"):
        return input_schema.UserDefinedNode
    model_class = NODE_TYPE_TO_SETTINGS_CLASS.get(node_type)
    if model_class is not None:
        return model_class
    if node_type in _get_custom_node_store():
        return input_schema.UserDefinedNode
    if isinstance(setting_data, dict) and "settings" in setting_data:
        return input_schema.UserDefinedNode
    return None

input_schema

flowfile_core.schemas.input_schema

Classes:

Name	Description
ApplyModelSettings	Settings payload for the Apply Model node.
CatalogWriteSettings	Settings for writing data to the catalog.
DatabaseConnection	Defines the connection parameters for a database.
DatabaseSettings	Defines settings for reading from a database, either via table or query.
DatabaseWriteSettings	Defines settings for writing data to a database table.
EvaluateModelSettings	Settings payload for the Evaluate Model node.
ExternalSource	Base model for data coming from a predefined external source.
FullDatabaseConnection	A complete database connection model including the secret password.
FullDatabaseConnectionInterface	A database connection model intended for UI display, omitting the password.
GoogleAnalyticsFilter	A single filter applied to a GA4 dimension or metric.
GoogleAnalyticsOrderBy	A single sort entry applied to the GA4 report.
GoogleAnalyticsSettings	UI settings for a Google Analytics 4 reader node.
InputAvroTable	Defines settings for reading an Avro file.
InputCsvTable	Defines settings for reading a CSV file.
InputExcelTable	Defines settings for reading an Excel file.
InputIpcTable	Defines settings for reading an Arrow IPC/Feather file.
InputJsonTable	Defines settings for reading a JSON file.
InputNdjsonTable	Defines settings for reading a newline-delimited JSON file.
InputParquetTable	Defines settings for reading a Parquet file.
InputTableBase	Base settings for input file operations.
KafkaSourceSettings	Configuration for reading from a Kafka/Redpanda topic.
MinimalFieldInfo	Represents the most basic information about a data field (column).
NewDirectory	Defines the information required to create a new directory.
NodeApiResponse	Settings for a node that marks its input as the body of an HTTP API response.
NodeApplyModel	Score data using a previously trained model artifact.
NodeBase	Base model for all nodes in a FlowGraph. Contains common metadata.
NodeCatalogReader	Settings for a node that reads a table from the catalog.
NodeCatalogWriter	Settings for a node that writes its input to the catalog.
NodeCloudStorageReader	Settings for a node that reads from a cloud storage service (S3, GCS, etc.).
NodeCloudStorageWriter	Settings for a node that writes to a cloud storage service.
NodeConnection	Represents a connection (edge) between two nodes in the graph.
NodeCrossJoin	Settings for a node that performs a cross join.
NodeDatabaseReader	Settings for a node that reads from a database.
NodeDatabaseWriter	Settings for a node that writes data to a database.
NodeDatasource	Base settings for a node that acts as a data source.
NodeDescription	A simple model for updating a node's description text.
NodeDynamicRename	Settings for a node that renames many columns at once via a single rule.
NodeEvaluateModel	Compute model-quality metrics by comparing actual and predicted columns.
NodeExploreData	Settings for a node that provides an interactive data exploration interface.
NodeExternalSource	Settings for a node that connects to a registered external data source.
NodeFilter	Settings for a node that filters rows based on a condition.
NodeFlowInput	Named source placeholder inside a subflow.
NodeFlowOutput	Named passthrough sink marking a subflow output; multiple allowed per flow.
NodeFormula	Settings for a node that applies a formula to create/modify a column.
NodeFuzzyMatch	Settings for a node that performs a fuzzy join based on string similarity.
NodeGoogleAnalyticsReader	Settings for a node that reads from a Google Analytics 4 property.
NodeGraphSolver	Settings for a node that solves graph-based problems (e.g., connected components).
NodeGroupBy	Settings for a node that performs a group-by and aggregation operation.
NodeInputConnection	Represents the input side of a connection between two nodes.
NodeJoin	Settings for a node that performs a standard SQL-style join.
NodeKafkaSource	Settings for a node that reads from a Kafka or Redpanda topic.
NodeManualInput	Settings for a node that allows direct data entry in the UI.
NodeMultiInput	A base model for any node that takes multiple data inputs.
NodeOutput	Settings for a node that writes its input to a file.
NodeOutputConnection	Represents the output side of a connection between two nodes.
NodePivot	Settings for a node that pivots data from a long to a wide format.
NodePolarsCode	Settings for a node that executes arbitrary user-provided Polars code.
NodePromise	A placeholder node for an operation that has not yet been configured.
NodePythonScript	Node that executes Python code on a kernel container.
NodeRandomSplit	Settings for a node that randomly partitions rows into N labeled outputs.
NodeRead	Settings for a node that reads data from a file.
NodeRecordCount	Settings for a node that counts the number of records.
NodeRecordId	Settings for a node that adds a unique record ID column.
NodeRestApiReader	Settings for a node that reads from a REST API.
NodeRunFlow	Settings for a node that executes a catalog-registered flow as a subflow.
NodeSample	Settings for a node that samples a subset of the data.
NodeSelect	Settings for a node that selects, renames, and reorders columns.
NodeSingleInput	A base model for any node that takes a single data input.
NodeSort	Settings for a node that sorts the data by one or more columns.
NodeSqlQuery	Settings for a node that executes a SQL query against connected data sources.
NodeTextToRows	Settings for a node that splits a text column into multiple rows.
NodeTrainModel	Train an ML model (regression or classification) and optionally publish it to the catalog.
NodeUnion	Settings for a node that concatenates multiple data inputs.
NodeUnique	Settings for a node that returns the unique rows from the data.
NodeUnpivot	Settings for a node that unpivots data from a wide to a long format.
NodeWaitFor	Pass-through node that enforces ordering on extra dependency inputs.
NodeWindowFunctions	Settings for a node that adds rolling, cumulative, rank or tile columns.
NotebookCell	A single cell in the notebook editor.
OutputAvroTable	Defines settings for writing an Avro file.
OutputCsvTable	Defines settings for writing a CSV file.
OutputExcelTable	Defines settings for writing an Excel file.
OutputFieldConfig	Configuration for output field validation and transformation behavior.
OutputFieldInfo	Field information with optional default value for output field configuration.
OutputIpcTable	Defines settings for writing an Arrow IPC/Feather file.
OutputNdjsonTable	Defines settings for writing a newline-delimited JSON file.
OutputParquetTable	Defines settings for writing a Parquet file.
OutputSettings	Defines the complete settings for an output node.
PythonScriptInput	Settings for Python code execution on a kernel.
RandomSplitGroup	A single output partition in a random split.
RawData	Represents data in a raw, columnar format for manual input.
ReceivedTable	Model for defining a table received from an external source.
RemoveItem	Represents a single item to be removed from a directory or list.
RemoveItemsInput	Defines a list of items to be removed.
RestApiAuthSettings	Authentication settings for a REST API reader node.
RestApiPaginationSettings	Pagination strategy and parameters for a REST API reader node.
RestApiSettings	UI settings for a REST API reader node.
RunFlowParameterBinding	How one subflow parameter gets its value for a run_flow execution.
SampleUsers	Settings for generating a sample dataset of users.
Scd2Settings	Slowly-changing-dimension type 2 configuration for a catalog write.
SubflowReference	Reference to a catalog-registered flow.
TrainModelSettings	Settings payload for the Train Model node.
UserDefinedNode	Settings for a node that contains the user defined node information

ApplyModelSettings pydantic-model

Bases: BaseModel

Settings payload for the Apply Model node.

Two model sources are supported:

• "upstream" (default): pick a Train Model node from somewhere in this flow's upstream chain. The model file is read from the flow's cache directory using the train node's id — works at design time, no catalog round-trip needed.
• "catalog": fall back to the existing catalog lookup by name/version.

Show JSON schema:

{
  "description": "Settings payload for the Apply Model node.\n\nTwo model sources are supported:\n\n- ``\"upstream\"`` (default): pick a Train Model node from somewhere in this\n  flow's upstream chain. The model file is read from the flow's cache\n  directory using the train node's id \u2014 works at design time, no catalog\n  round-trip needed.\n- ``\"catalog\"``: fall back to the existing catalog lookup by name/version.",
  "properties": {
    "source": {
      "default": "upstream",
      "enum": [
        "upstream",
        "catalog"
      ],
      "title": "Source",
      "type": "string"
    },
    "upstream_node_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Upstream Node Id"
    },
    "model_name": {
      "default": "",
      "title": "Model Name",
      "type": "string"
    },
    "model_version": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Model Version"
    },
    "namespace_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Id"
    },
    "namespace_full_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Full Name"
    },
    "output_column": {
      "default": "prediction",
      "title": "Output Column",
      "type": "string"
    }
  },
  "title": "ApplyModelSettings",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• source (Literal['upstream', 'catalog'])
• upstream_node_id (int | None)
• model_name (str)
• model_version (int | None)
• namespace_id (int | None)
• namespace_full_name (str | None)
• output_column (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class ApplyModelSettings(BaseModel):
    """Settings payload for the Apply Model node.

    Two model sources are supported:

    - ``"upstream"`` (default): pick a Train Model node from somewhere in this
      flow's upstream chain. The model file is read from the flow's cache
      directory using the train node's id — works at design time, no catalog
      round-trip needed.
    - ``"catalog"``: fall back to the existing catalog lookup by name/version.
    """

    model_config = ConfigDict(protected_namespaces=())

    source: Literal["upstream", "catalog"] = "upstream"

    # source="upstream" — id of an upstream Train Model node in the same flow.
    upstream_node_id: int | None = None

    # source="catalog"
    model_name: str = ""
    model_version: int | None = None
    namespace_id: int | None = None
    namespace_full_name: str | None = None  # portable "catalog.schema"; resolved name-first, id is fallback

    output_column: str = "prediction"

CatalogWriteSettings pydantic-model

Bases: BaseModel

Settings for writing data to the catalog.

The target namespace is referenced name-first: namespace_full_name ("catalog.schema") is the portable reference that survives recreation on another machine; namespace_id is a numeric fallback for flows saved before names were stored.

Show JSON schema:

{
  "$defs": {
    "Scd2Settings": {
      "description": "Slowly-changing-dimension type 2 configuration for a catalog write.\n\nThe business key is ``CatalogWriteSettings.merge_keys`` \u2014 this block only carries the\nchange-detection scope and the names of the four generated columns. It is persisted verbatim\nonto the catalog table record (``CatalogTable.scd2_config``) so a reader can filter history\nwithout ever reading a writer node's settings.",
      "properties": {
        "compare_columns": {
          "items": {
            "type": "string"
          },
          "title": "Compare Columns",
          "type": "array"
        },
        "full_snapshot": {
          "default": false,
          "title": "Full Snapshot",
          "type": "boolean"
        },
        "partition_on_current": {
          "default": true,
          "title": "Partition On Current",
          "type": "boolean"
        },
        "surrogate_key_column": {
          "default": "sk",
          "title": "Surrogate Key Column",
          "type": "string"
        },
        "valid_from_column": {
          "default": "valid_from",
          "title": "Valid From Column",
          "type": "string"
        },
        "valid_to_column": {
          "default": "valid_to",
          "title": "Valid To Column",
          "type": "string"
        },
        "is_current_column": {
          "default": "is_current",
          "title": "Is Current Column",
          "type": "string"
        }
      },
      "title": "Scd2Settings",
      "type": "object"
    }
  },
  "description": "Settings for writing data to the catalog.\n\nThe target namespace is referenced name-first: ``namespace_full_name`` (``\"catalog.schema\"``) is\nthe portable reference that survives recreation on another machine; ``namespace_id`` is a numeric\nfallback for flows saved before names were stored.",
  "properties": {
    "table_name": {
      "default": "",
      "title": "Table Name",
      "type": "string"
    },
    "namespace_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Id"
    },
    "namespace_full_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Full Name"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Description"
    },
    "write_mode": {
      "default": "overwrite",
      "enum": [
        "overwrite",
        "error",
        "append",
        "upsert",
        "update",
        "delete",
        "scd2",
        "virtual"
      ],
      "title": "Write Mode",
      "type": "string"
    },
    "merge_keys": {
      "items": {
        "type": "string"
      },
      "title": "Merge Keys",
      "type": "array"
    },
    "partition_by": {
      "items": {
        "type": "string"
      },
      "title": "Partition By",
      "type": "array"
    },
    "scd2": {
      "anyOf": [
        {
          "$ref": "#/$defs/Scd2Settings"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "title": "CatalogWriteSettings",
  "type": "object"
}

Fields:

• table_name (str)
• namespace_id (int | None)
• namespace_full_name (str | None)
• description (str | None)
• write_mode (Literal['overwrite', 'error', 'append', 'upsert', 'update', 'delete', 'scd2', 'virtual'])
• merge_keys (list[str])
• partition_by (list[str])
• scd2 (Scd2Settings | None)

Validators:

• _validate_merge_keys

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class CatalogWriteSettings(BaseModel):
    """Settings for writing data to the catalog.

    The target namespace is referenced name-first: ``namespace_full_name`` (``"catalog.schema"``) is
    the portable reference that survives recreation on another machine; ``namespace_id`` is a numeric
    fallback for flows saved before names were stored.
    """

    table_name: str = ""
    namespace_id: int | None = None
    namespace_full_name: str | None = None
    description: str | None = None
    write_mode: Literal["overwrite", "error", "append", "upsert", "update", "delete", "scd2", "virtual"] = "overwrite"
    merge_keys: list[str] = Field(default_factory=list)
    partition_by: list[str] = Field(default_factory=list)
    # A dangling block on a non-scd2 physical mode is tolerated: the UI keeps it while toggling modes.
    scd2: Scd2Settings | None = None

    @model_validator(mode="after")
    def _validate_merge_keys(self) -> "CatalogWriteSettings":
        if self.write_mode in ("upsert", "update", "delete", "scd2") and not self.merge_keys:
            raise ValueError(f"merge_keys must be non-empty when write_mode is '{self.write_mode}'")
        if self.partition_by and self.write_mode == "virtual":
            raise ValueError("partition_by is not allowed for virtual tables")
        if self.write_mode == "virtual" and self.scd2 is not None:
            raise ValueError("scd2 settings are not allowed for virtual tables")
        if self.write_mode == "scd2":
            cfg = self.scd2 or Scd2Settings()
            if len(set(self.merge_keys)) != len(self.merge_keys):
                raise ValueError("merge_keys must not contain duplicates when write_mode is 'scd2'")
            clash = sorted(set(self.merge_keys) & set(cfg.system_columns))
            if clash:
                raise ValueError(f"merge_keys may not name SCD2 system columns: {clash}")
            hot = sorted(set(self.partition_by) & {cfg.surrogate_key_column, cfg.valid_from_column})
            if hot:
                raise ValueError(f"partition_by may not use high-cardinality SCD2 columns: {hot}")
        return self

DatabaseConnection pydantic-model

Bases: BaseModel

Defines the connection parameters for a database.

Show JSON schema:

{
  "description": "Defines the connection parameters for a database.",
  "properties": {
    "database_type": {
      "default": "postgresql",
      "title": "Database Type",
      "type": "string"
    },
    "username": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Username"
    },
    "password_ref": {
      "anyOf": [
        {
          "description": "An ID referencing an encrypted secret.",
          "maxLength": 100,
          "minLength": 1,
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Password Ref"
    },
    "host": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Host"
    },
    "port": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Port"
    },
    "database": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Database"
    },
    "url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Url"
    }
  },
  "title": "DatabaseConnection",
  "type": "object"
}

Fields:

• database_type (str)
• username (str | None)
• password_ref (SecretRef | None)
• host (str | None)
• port (int | None)
• database (str | None)
• url (str | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class DatabaseConnection(BaseModel):
    """Defines the connection parameters for a database."""

    database_type: str = "postgresql"
    username: str | None = None
    password_ref: SecretRef | None = None
    host: str | None = None
    port: int | None = None
    database: str | None = None
    url: str | None = None

    @field_validator("database_type")
    @classmethod
    def known_database_type(cls, v: str) -> str:
        from shared.db_dialects import KNOWN_DIALECT_NAMES

        low = v.lower()
        if low not in KNOWN_DIALECT_NAMES:
            raise ValueError(f"Unsupported database type '{v}'. Supported types: {', '.join(KNOWN_DIALECT_NAMES)}")
        return low

    @field_validator("password_ref", mode="before")
    @classmethod
    def empty_string_to_none(cls, v):
        if v == "":
            return None
        return v

DatabaseSettings pydantic-model

Bases: BaseModel

Defines settings for reading from a database, either via table or query.

Show JSON schema:

{
  "$defs": {
    "DatabaseConnection": {
      "description": "Defines the connection parameters for a database.",
      "properties": {
        "database_type": {
          "default": "postgresql",
          "title": "Database Type",
          "type": "string"
        },
        "username": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Username"
        },
        "password_ref": {
          "anyOf": [
            {
              "description": "An ID referencing an encrypted secret.",
              "maxLength": 100,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Password Ref"
        },
        "host": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Host"
        },
        "port": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Port"
        },
        "database": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database"
        },
        "url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Url"
        }
      },
      "title": "DatabaseConnection",
      "type": "object"
    }
  },
  "description": "Defines settings for reading from a database, either via table or query.",
  "properties": {
    "connection_mode": {
      "anyOf": [
        {
          "enum": [
            "inline",
            "reference"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "inline",
      "title": "Connection Mode"
    },
    "database_connection": {
      "anyOf": [
        {
          "$ref": "#/$defs/DatabaseConnection"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "database_connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Database Connection Name"
    },
    "schema_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Schema Name"
    },
    "table_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Table Name"
    },
    "query": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Query"
    },
    "query_mode": {
      "default": "table",
      "enum": [
        "query",
        "table",
        "reference"
      ],
      "title": "Query Mode",
      "type": "string"
    }
  },
  "title": "DatabaseSettings",
  "type": "object"
}

Fields:

• connection_mode (Literal['inline', 'reference'] | None)
• database_connection (DatabaseConnection | None)
• database_connection_name (str | None)
• schema_name (str | None)
• table_name (str | None)
• query (str | None)
• query_mode (Literal['query', 'table', 'reference'])

Validators:

• validate_sql_identifier → table_name, schema_name
• validate_table_or_query

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class DatabaseSettings(BaseModel):
    """Defines settings for reading from a database, either via table or query."""

    connection_mode: Literal["inline", "reference"] | None = "inline"
    database_connection: DatabaseConnection | None = None
    database_connection_name: str | None = None
    schema_name: str | None = None
    table_name: str | None = None
    query: str | None = None
    query_mode: Literal["query", "table", "reference"] = "table"

    @field_validator("table_name", "schema_name", mode="before")
    @classmethod
    def validate_sql_identifier(cls, v):
        if v is not None and v != "":
            parts = v.split(".")
            for part in parts:
                if not part or not re.match(r"^[a-zA-Z_][a-zA-Z0-9_]*$", part):
                    raise ValueError(
                        f"Invalid SQL identifier: '{v}'. " f"Only letters, numbers, and underscores are allowed."
                    )
        return v

    @model_validator(mode="after")
    def validate_table_or_query(self):
        if (not self.table_name and not self.query) and self.query_mode == "inline":
            raise ValueError("Either 'table_name' or 'query' must be provided")

        if self.connection_mode == "inline" and self.database_connection is None:
            raise ValueError("When 'connection_mode' is 'inline', 'database_connection' must be provided")

        if self.connection_mode == "reference" and not self.database_connection_name:
            raise ValueError("When 'connection_mode' is 'reference', 'database_connection_name' must be provided")

        return self

DatabaseWriteSettings pydantic-model

Bases: BaseModel

Defines settings for writing data to a database table.

Show JSON schema:

{
  "$defs": {
    "DatabaseConnection": {
      "description": "Defines the connection parameters for a database.",
      "properties": {
        "database_type": {
          "default": "postgresql",
          "title": "Database Type",
          "type": "string"
        },
        "username": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Username"
        },
        "password_ref": {
          "anyOf": [
            {
              "description": "An ID referencing an encrypted secret.",
              "maxLength": 100,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Password Ref"
        },
        "host": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Host"
        },
        "port": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Port"
        },
        "database": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database"
        },
        "url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Url"
        }
      },
      "title": "DatabaseConnection",
      "type": "object"
    }
  },
  "description": "Defines settings for writing data to a database table.",
  "properties": {
    "connection_mode": {
      "anyOf": [
        {
          "enum": [
            "inline",
            "reference"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "inline",
      "title": "Connection Mode"
    },
    "database_connection": {
      "anyOf": [
        {
          "$ref": "#/$defs/DatabaseConnection"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "database_connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Database Connection Name"
    },
    "table_name": {
      "title": "Table Name",
      "type": "string"
    },
    "schema_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Schema Name"
    },
    "if_exists": {
      "anyOf": [
        {
          "enum": [
            "append",
            "replace",
            "fail"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "append",
      "title": "If Exists"
    }
  },
  "required": [
    "table_name"
  ],
  "title": "DatabaseWriteSettings",
  "type": "object"
}

Fields:

• connection_mode (Literal['inline', 'reference'] | None)
• database_connection (DatabaseConnection | None)
• database_connection_name (str | None)
• table_name (str)
• schema_name (str | None)
• if_exists (Literal['append', 'replace', 'fail'] | None)

Validators:

• validate_sql_identifier → table_name, schema_name

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class DatabaseWriteSettings(BaseModel):
    """Defines settings for writing data to a database table."""

    connection_mode: Literal["inline", "reference"] | None = "inline"
    database_connection: DatabaseConnection | None = None
    database_connection_name: str | None = None
    table_name: str
    schema_name: str | None = None
    if_exists: Literal["append", "replace", "fail"] | None = "append"

    @field_validator("table_name", "schema_name", mode="before")
    @classmethod
    def validate_sql_identifier(cls, v):
        if v is not None and v != "":
            parts = v.split(".")
            for part in parts:
                if not part or not re.match(r"^[a-zA-Z_][a-zA-Z0-9_]*$", part):
                    raise ValueError(
                        f"Invalid SQL identifier: '{v}'. " f"Only letters, numbers, and underscores are allowed."
                    )
        return v

EvaluateModelSettings pydantic-model

Bases: BaseModel

Settings payload for the Evaluate Model node.

Decoupled from any specific Train/Apply pair: takes a dataframe that already contains both the actual target column and a prediction column and emits a long-form (metric, value) frame. Reusable on training, test, or hold-out splits.

task_type="auto" resolves the metric set from an upstream Train Model node when one is configured; otherwise defaults to regression.

Show JSON schema:

{
  "description": "Settings payload for the Evaluate Model node.\n\nDecoupled from any specific Train/Apply pair: takes a dataframe that\nalready contains both the actual target column and a prediction column\nand emits a long-form ``(metric, value)`` frame. Reusable on training,\ntest, or hold-out splits.\n\n``task_type=\"auto\"`` resolves the metric set from an upstream Train\nModel node when one is configured; otherwise defaults to ``regression``.",
  "properties": {
    "actual_column": {
      "default": "",
      "title": "Actual Column",
      "type": "string"
    },
    "predicted_column": {
      "default": "prediction",
      "title": "Predicted Column",
      "type": "string"
    },
    "task_type": {
      "default": "auto",
      "enum": [
        "auto",
        "regression",
        "classification"
      ],
      "title": "Task Type",
      "type": "string"
    },
    "upstream_train_node_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Upstream Train Node Id"
    }
  },
  "title": "EvaluateModelSettings",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• actual_column (str)
• predicted_column (str)
• task_type (Literal['auto', 'regression', 'classification'])
• upstream_train_node_id (int | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class EvaluateModelSettings(BaseModel):
    """Settings payload for the Evaluate Model node.

    Decoupled from any specific Train/Apply pair: takes a dataframe that
    already contains both the actual target column and a prediction column
    and emits a long-form ``(metric, value)`` frame. Reusable on training,
    test, or hold-out splits.

    ``task_type="auto"`` resolves the metric set from an upstream Train
    Model node when one is configured; otherwise defaults to ``regression``.
    """

    model_config = ConfigDict(protected_namespaces=())

    actual_column: str = ""
    predicted_column: str = "prediction"
    task_type: Literal["auto", "regression", "classification"] = "auto"
    upstream_train_node_id: int | None = None

ExternalSource pydantic-model

Bases: BaseModel

Base model for data coming from a predefined external source.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    }
  },
  "description": "Base model for data coming from a predefined external source.",
  "properties": {
    "orientation": {
      "default": "row",
      "title": "Orientation",
      "type": "string"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "title": "ExternalSource",
  "type": "object"
}

Fields:

• orientation (str)
• fields (list[MinimalFieldInfo] | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class ExternalSource(BaseModel):
    """Base model for data coming from a predefined external source."""

    orientation: str = "row"
    fields: list[MinimalFieldInfo] | None = None

FullDatabaseConnection pydantic-model

Bases: BaseModel

A complete database connection model including the secret password.

Show JSON schema:

{
  "description": "A complete database connection model including the secret password.",
  "properties": {
    "connection_name": {
      "title": "Connection Name",
      "type": "string"
    },
    "database_type": {
      "default": "postgresql",
      "title": "Database Type",
      "type": "string"
    },
    "username": {
      "title": "Username",
      "type": "string"
    },
    "password": {
      "format": "password",
      "title": "Password",
      "type": "string",
      "writeOnly": true
    },
    "host": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Host"
    },
    "port": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Port"
    },
    "database": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Database"
    },
    "ssl_enabled": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Ssl Enabled"
    },
    "url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Url"
    }
  },
  "required": [
    "connection_name",
    "username",
    "password"
  ],
  "title": "FullDatabaseConnection",
  "type": "object"
}

Fields:

• connection_name (str)
• database_type (str)
• username (str)
• password (SecretStr)
• host (str | None)
• port (int | None)
• database (str | None)
• ssl_enabled (bool | None)
• url (str | None)

Validators:

• normalize_database_type → database_type

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class FullDatabaseConnection(BaseModel):
    """A complete database connection model including the secret password."""

    connection_name: str
    database_type: str = "postgresql"
    username: str
    password: SecretStr
    host: str | None = None
    port: int | None = None
    database: str | None = None
    ssl_enabled: bool | None = False
    url: str | None = None

    @field_validator("database_type")
    @classmethod
    def normalize_database_type(cls, v: str) -> str:
        # lowercase only, no vocabulary check: legacy stored types (e.g. redshift) must keep loading
        return v.lower()

FullDatabaseConnectionInterface pydantic-model

Bases: BaseModel

A database connection model intended for UI display, omitting the password.

Show JSON schema:

{
  "$defs": {
    "AccessInfo": {
      "description": "How the requesting user can access a resource; attached to list/detail responses.",
      "properties": {
        "is_owner": {
          "title": "Is Owner",
          "type": "boolean"
        },
        "access_level": {
          "enum": [
            "owner",
            "manage",
            "use"
          ],
          "title": "Access Level",
          "type": "string"
        },
        "shared_by": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Shared By"
        }
      },
      "required": [
        "is_owner",
        "access_level"
      ],
      "title": "AccessInfo",
      "type": "object"
    }
  },
  "description": "A database connection model intended for UI display, omitting the password.",
  "properties": {
    "connection_name": {
      "title": "Connection Name",
      "type": "string"
    },
    "database_type": {
      "default": "postgresql",
      "title": "Database Type",
      "type": "string"
    },
    "username": {
      "title": "Username",
      "type": "string"
    },
    "host": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Host"
    },
    "port": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Port"
    },
    "database": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Database"
    },
    "ssl_enabled": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Ssl Enabled"
    },
    "url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Url"
    },
    "id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Id"
    },
    "access": {
      "anyOf": [
        {
          "$ref": "#/$defs/AccessInfo"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "connection_name",
    "username"
  ],
  "title": "FullDatabaseConnectionInterface",
  "type": "object"
}

Fields:

• connection_name (str)
• database_type (str)
• username (str)
• host (str | None)
• port (int | None)
• database (str | None)
• ssl_enabled (bool | None)
• url (str | None)
• id (int | None)
• access (AccessInfo | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class FullDatabaseConnectionInterface(BaseModel):
    """A database connection model intended for UI display, omitting the password."""

    connection_name: str
    database_type: str = "postgresql"
    username: str
    host: str | None = None
    port: int | None = None
    database: str | None = None
    ssl_enabled: bool | None = False
    url: str | None = None
    id: int | None = None
    access: AccessInfo | None = None

GoogleAnalyticsFilter pydantic-model

Bases: BaseModel

A single filter applied to a GA4 dimension or metric.

field must match one of the selected dimensions or metrics; the worker auto-routes the filter into either the request's dimension_filter (for string-typed dimensions) or metric_filter (for numeric-typed metrics).

Supported operators — strings (dimensions): - equals, not_equals - contains, begins_with, ends_with - regex (full regex match) - in_list, not_in_list (comma-separated value)

Supported operators — numeric (metrics): - equals, not_equals - less_than, less_equal, greater_than, greater_equal - between (comma-separated "low,high")

Multiple filters on the same kind are AND-combined. String matching is case-insensitive by default (case_sensitive=False below).

Show JSON schema:

{
  "description": "A single filter applied to a GA4 dimension or metric.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nauto-routes the filter into either the request's ``dimension_filter`` (for\nstring-typed dimensions) or ``metric_filter`` (for numeric-typed metrics).\n\nSupported operators \u2014 strings (dimensions):\n  - ``equals``, ``not_equals``\n  - ``contains``, ``begins_with``, ``ends_with``\n  - ``regex``  (full regex match)\n  - ``in_list``, ``not_in_list``  (comma-separated ``value``)\n\nSupported operators \u2014 numeric (metrics):\n  - ``equals``, ``not_equals``\n  - ``less_than``, ``less_equal``, ``greater_than``, ``greater_equal``\n  - ``between``  (comma-separated ``\"low,high\"``)\n\nMultiple filters on the same kind are AND-combined. String matching is\ncase-insensitive by default (``case_sensitive=False`` below).",
  "properties": {
    "field": {
      "title": "Field",
      "type": "string"
    },
    "operator": {
      "title": "Operator",
      "type": "string"
    },
    "value": {
      "default": "",
      "title": "Value",
      "type": "string"
    },
    "case_sensitive": {
      "default": false,
      "title": "Case Sensitive",
      "type": "boolean"
    }
  },
  "required": [
    "field",
    "operator"
  ],
  "title": "GoogleAnalyticsFilter",
  "type": "object"
}

Fields:

• field (str)
• operator (str)
• value (str)
• case_sensitive (bool)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class GoogleAnalyticsFilter(BaseModel):
    """A single filter applied to a GA4 dimension or metric.

    ``field`` must match one of the selected dimensions or metrics; the worker
    auto-routes the filter into either the request's ``dimension_filter`` (for
    string-typed dimensions) or ``metric_filter`` (for numeric-typed metrics).

    Supported operators — strings (dimensions):
      - ``equals``, ``not_equals``
      - ``contains``, ``begins_with``, ``ends_with``
      - ``regex``  (full regex match)
      - ``in_list``, ``not_in_list``  (comma-separated ``value``)

    Supported operators — numeric (metrics):
      - ``equals``, ``not_equals``
      - ``less_than``, ``less_equal``, ``greater_than``, ``greater_equal``
      - ``between``  (comma-separated ``"low,high"``)

    Multiple filters on the same kind are AND-combined. String matching is
    case-insensitive by default (``case_sensitive=False`` below).
    """

    field: str
    operator: str
    value: str = ""
    case_sensitive: bool = False

GoogleAnalyticsOrderBy pydantic-model

Bases: BaseModel

A single sort entry applied to the GA4 report.

field must match one of the selected dimensions or metrics; the worker routes it into a DimensionOrderBy or MetricOrderBy accordingly. descending=True produces a descending sort. Sort entries are applied in list order.

Show JSON schema:

{
  "description": "A single sort entry applied to the GA4 report.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nroutes it into a ``DimensionOrderBy`` or ``MetricOrderBy`` accordingly.\n``descending=True`` produces a descending sort. Sort entries are applied in\nlist order.",
  "properties": {
    "field": {
      "title": "Field",
      "type": "string"
    },
    "descending": {
      "default": false,
      "title": "Descending",
      "type": "boolean"
    }
  },
  "required": [
    "field"
  ],
  "title": "GoogleAnalyticsOrderBy",
  "type": "object"
}

Fields:

• field (str)
• descending (bool)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class GoogleAnalyticsOrderBy(BaseModel):
    """A single sort entry applied to the GA4 report.

    ``field`` must match one of the selected dimensions or metrics; the worker
    routes it into a ``DimensionOrderBy`` or ``MetricOrderBy`` accordingly.
    ``descending=True`` produces a descending sort. Sort entries are applied in
    list order.
    """

    field: str
    descending: bool = False

GoogleAnalyticsSettings pydantic-model

Bases: BaseModel

UI settings for a Google Analytics 4 reader node.

Credentials are NOT stored inline: ga_connection_name is a reference to a Google Analytics connection managed under /ga_connections (whose service-account JSON is encrypted at rest).

Show JSON schema:

{
  "$defs": {
    "GoogleAnalyticsFilter": {
      "description": "A single filter applied to a GA4 dimension or metric.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nauto-routes the filter into either the request's ``dimension_filter`` (for\nstring-typed dimensions) or ``metric_filter`` (for numeric-typed metrics).\n\nSupported operators \u2014 strings (dimensions):\n  - ``equals``, ``not_equals``\n  - ``contains``, ``begins_with``, ``ends_with``\n  - ``regex``  (full regex match)\n  - ``in_list``, ``not_in_list``  (comma-separated ``value``)\n\nSupported operators \u2014 numeric (metrics):\n  - ``equals``, ``not_equals``\n  - ``less_than``, ``less_equal``, ``greater_than``, ``greater_equal``\n  - ``between``  (comma-separated ``\"low,high\"``)\n\nMultiple filters on the same kind are AND-combined. String matching is\ncase-insensitive by default (``case_sensitive=False`` below).",
      "properties": {
        "field": {
          "title": "Field",
          "type": "string"
        },
        "operator": {
          "title": "Operator",
          "type": "string"
        },
        "value": {
          "default": "",
          "title": "Value",
          "type": "string"
        },
        "case_sensitive": {
          "default": false,
          "title": "Case Sensitive",
          "type": "boolean"
        }
      },
      "required": [
        "field",
        "operator"
      ],
      "title": "GoogleAnalyticsFilter",
      "type": "object"
    },
    "GoogleAnalyticsOrderBy": {
      "description": "A single sort entry applied to the GA4 report.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nroutes it into a ``DimensionOrderBy`` or ``MetricOrderBy`` accordingly.\n``descending=True`` produces a descending sort. Sort entries are applied in\nlist order.",
      "properties": {
        "field": {
          "title": "Field",
          "type": "string"
        },
        "descending": {
          "default": false,
          "title": "Descending",
          "type": "boolean"
        }
      },
      "required": [
        "field"
      ],
      "title": "GoogleAnalyticsOrderBy",
      "type": "object"
    }
  },
  "description": "UI settings for a Google Analytics 4 reader node.\n\nCredentials are NOT stored inline: ``ga_connection_name`` is a reference to\na Google Analytics connection managed under ``/ga_connections`` (whose\nservice-account JSON is encrypted at rest).",
  "properties": {
    "ga_connection_name": {
      "title": "Ga Connection Name",
      "type": "string"
    },
    "property_id": {
      "title": "Property Id",
      "type": "string"
    },
    "start_date": {
      "default": "7daysAgo",
      "title": "Start Date",
      "type": "string"
    },
    "end_date": {
      "default": "yesterday",
      "title": "End Date",
      "type": "string"
    },
    "metrics": {
      "items": {
        "type": "string"
      },
      "title": "Metrics",
      "type": "array"
    },
    "dimensions": {
      "items": {
        "type": "string"
      },
      "title": "Dimensions",
      "type": "array"
    },
    "limit": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Limit"
    },
    "filters": {
      "items": {
        "$ref": "#/$defs/GoogleAnalyticsFilter"
      },
      "title": "Filters",
      "type": "array"
    },
    "order_bys": {
      "items": {
        "$ref": "#/$defs/GoogleAnalyticsOrderBy"
      },
      "title": "Order Bys",
      "type": "array"
    }
  },
  "required": [
    "ga_connection_name",
    "property_id"
  ],
  "title": "GoogleAnalyticsSettings",
  "type": "object"
}

Fields:

• ga_connection_name (str)
• property_id (str)
• start_date (str)
• end_date (str)
• metrics (list[str])
• dimensions (list[str])
• limit (int | None)
• filters (list[GoogleAnalyticsFilter])
• order_bys (list[GoogleAnalyticsOrderBy])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class GoogleAnalyticsSettings(BaseModel):
    """UI settings for a Google Analytics 4 reader node.

    Credentials are NOT stored inline: ``ga_connection_name`` is a reference to
    a Google Analytics connection managed under ``/ga_connections`` (whose
    service-account JSON is encrypted at rest).
    """

    ga_connection_name: str
    property_id: str
    start_date: str = "7daysAgo"
    end_date: str = "yesterday"
    metrics: list[str] = Field(default_factory=list)
    dimensions: list[str] = Field(default_factory=list)
    # ``None`` means "fetch everything the report returns".
    limit: int | None = None
    # Row-level filters. See ``GoogleAnalyticsFilter`` for the operator list.
    # Multiple filters across the same category (dimension vs metric) are AND-combined.
    filters: list[GoogleAnalyticsFilter] = Field(default_factory=list)
    # Sort entries applied in list order. Each ``field`` must be one of the
    # selected metrics or dimensions; the worker raises a clear ``ValueError``
    # otherwise.
    order_bys: list[GoogleAnalyticsOrderBy] = Field(default_factory=list)

InputAvroTable pydantic-model

Bases: InputTableBase

Defines settings for reading an Avro file.

Show JSON schema:

{
  "description": "Defines settings for reading an Avro file.",
  "properties": {
    "file_type": {
      "const": "avro",
      "default": "avro",
      "title": "File Type",
      "type": "string"
    }
  },
  "title": "InputAvroTable",
  "type": "object"
}

Fields:

• file_type (Literal['avro'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputAvroTable(InputTableBase):
    """Defines settings for reading an Avro file."""

    file_type: Literal["avro"] = "avro"

InputCsvTable pydantic-model

Bases: InputTableBase

Defines settings for reading a CSV file.

Show JSON schema:

{
  "description": "Defines settings for reading a CSV file.",
  "properties": {
    "file_type": {
      "const": "csv",
      "default": "csv",
      "title": "File Type",
      "type": "string"
    },
    "reference": {
      "default": "",
      "title": "Reference",
      "type": "string"
    },
    "starting_from_line": {
      "default": 0,
      "title": "Starting From Line",
      "type": "integer"
    },
    "delimiter": {
      "default": ",",
      "title": "Delimiter",
      "type": "string"
    },
    "has_headers": {
      "default": true,
      "title": "Has Headers",
      "type": "boolean"
    },
    "encoding": {
      "default": "utf-8",
      "title": "Encoding",
      "type": "string"
    },
    "parquet_ref": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Parquet Ref"
    },
    "row_delimiter": {
      "default": "\n",
      "title": "Row Delimiter",
      "type": "string"
    },
    "quote_char": {
      "default": "\"",
      "title": "Quote Char",
      "type": "string"
    },
    "infer_schema_length": {
      "default": 10000,
      "title": "Infer Schema Length",
      "type": "integer"
    },
    "infer_schema": {
      "default": true,
      "title": "Infer Schema",
      "type": "boolean"
    },
    "truncate_ragged_lines": {
      "default": false,
      "title": "Truncate Ragged Lines",
      "type": "boolean"
    },
    "ignore_errors": {
      "default": false,
      "title": "Ignore Errors",
      "type": "boolean"
    }
  },
  "title": "InputCsvTable",
  "type": "object"
}

Fields:

• file_type (Literal['csv'])
• reference (str)
• starting_from_line (int)
• delimiter (str)
• has_headers (bool)
• encoding (str)
• parquet_ref (str | None)
• row_delimiter (str)
• quote_char (str)
• infer_schema_length (int)
• infer_schema (bool)
• truncate_ragged_lines (bool)
• ignore_errors (bool)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputCsvTable(InputTableBase):
    """Defines settings for reading a CSV file."""

    file_type: Literal["csv"] = "csv"
    reference: str = ""
    starting_from_line: int = 0
    delimiter: str = ","
    has_headers: bool = True
    encoding: str = "utf-8"
    parquet_ref: str | None = None
    row_delimiter: str = "\n"
    quote_char: str = '"'
    infer_schema_length: int = 10_000
    infer_schema: bool = True
    truncate_ragged_lines: bool = False
    ignore_errors: bool = False

InputExcelTable pydantic-model

Bases: InputTableBase

Defines settings for reading an Excel file.

Show JSON schema:

{
  "description": "Defines settings for reading an Excel file.",
  "properties": {
    "file_type": {
      "const": "excel",
      "default": "excel",
      "title": "File Type",
      "type": "string"
    },
    "sheet_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Sheet Name"
    },
    "start_row": {
      "default": 0,
      "title": "Start Row",
      "type": "integer"
    },
    "start_column": {
      "default": 0,
      "title": "Start Column",
      "type": "integer"
    },
    "end_row": {
      "default": 0,
      "title": "End Row",
      "type": "integer"
    },
    "end_column": {
      "default": 0,
      "title": "End Column",
      "type": "integer"
    },
    "has_headers": {
      "default": true,
      "title": "Has Headers",
      "type": "boolean"
    },
    "type_inference": {
      "default": false,
      "title": "Type Inference",
      "type": "boolean"
    }
  },
  "title": "InputExcelTable",
  "type": "object"
}

Fields:

• file_type (Literal['excel'])
• sheet_name (str | None)
• start_row (int)
• start_column (int)
• end_row (int)
• end_column (int)
• has_headers (bool)
• type_inference (bool)

Validators:

• validate_range_values

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputExcelTable(InputTableBase):
    """Defines settings for reading an Excel file."""

    file_type: Literal["excel"] = "excel"
    sheet_name: str | None = None
    start_row: int = 0
    start_column: int = 0
    end_row: int = 0
    end_column: int = 0
    has_headers: bool = True
    type_inference: bool = False

    @model_validator(mode="after")
    def validate_range_values(self):
        """Validates that the Excel cell range is logical."""
        for attribute in [self.start_row, self.start_column, self.end_row, self.end_column]:
            if not isinstance(attribute, int) or attribute < 0:
                raise ValueError("Row and column indices must be non-negative integers")
        if (self.end_row > 0 and self.start_row > self.end_row) or (
            self.end_column > 0 and self.start_column > self.end_column
        ):
            raise ValueError("Start row/column must not be greater than end row/column")
        return self

validate_range_values() pydantic-validator

Validates that the Excel cell range is logical.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@model_validator(mode="after")
def validate_range_values(self):
    """Validates that the Excel cell range is logical."""
    for attribute in [self.start_row, self.start_column, self.end_row, self.end_column]:
        if not isinstance(attribute, int) or attribute < 0:
            raise ValueError("Row and column indices must be non-negative integers")
    if (self.end_row > 0 and self.start_row > self.end_row) or (
        self.end_column > 0 and self.start_column > self.end_column
    ):
        raise ValueError("Start row/column must not be greater than end row/column")
    return self

InputIpcTable pydantic-model

Bases: InputTableBase

Defines settings for reading an Arrow IPC/Feather file.

Show JSON schema:

{
  "description": "Defines settings for reading an Arrow IPC/Feather file.",
  "properties": {
    "file_type": {
      "const": "ipc",
      "default": "ipc",
      "title": "File Type",
      "type": "string"
    }
  },
  "title": "InputIpcTable",
  "type": "object"
}

Fields:

• file_type (Literal['ipc'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputIpcTable(InputTableBase):
    """Defines settings for reading an Arrow IPC/Feather file."""

    file_type: Literal["ipc"] = "ipc"

InputJsonTable pydantic-model

Bases: InputCsvTable

Defines settings for reading a JSON file.

Show JSON schema:

{
  "description": "Defines settings for reading a JSON file.",
  "properties": {
    "file_type": {
      "const": "json",
      "default": "json",
      "title": "File Type",
      "type": "string"
    },
    "reference": {
      "default": "",
      "title": "Reference",
      "type": "string"
    },
    "starting_from_line": {
      "default": 0,
      "title": "Starting From Line",
      "type": "integer"
    },
    "delimiter": {
      "default": ",",
      "title": "Delimiter",
      "type": "string"
    },
    "has_headers": {
      "default": true,
      "title": "Has Headers",
      "type": "boolean"
    },
    "encoding": {
      "default": "utf-8",
      "title": "Encoding",
      "type": "string"
    },
    "parquet_ref": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Parquet Ref"
    },
    "row_delimiter": {
      "default": "\n",
      "title": "Row Delimiter",
      "type": "string"
    },
    "quote_char": {
      "default": "\"",
      "title": "Quote Char",
      "type": "string"
    },
    "infer_schema_length": {
      "default": 10000,
      "title": "Infer Schema Length",
      "type": "integer"
    },
    "infer_schema": {
      "default": true,
      "title": "Infer Schema",
      "type": "boolean"
    },
    "truncate_ragged_lines": {
      "default": false,
      "title": "Truncate Ragged Lines",
      "type": "boolean"
    },
    "ignore_errors": {
      "default": false,
      "title": "Ignore Errors",
      "type": "boolean"
    }
  },
  "title": "InputJsonTable",
  "type": "object"
}

Fields:

• reference (str)
• starting_from_line (int)
• delimiter (str)
• has_headers (bool)
• encoding (str)
• parquet_ref (str | None)
• row_delimiter (str)
• quote_char (str)
• infer_schema_length (int)
• infer_schema (bool)
• truncate_ragged_lines (bool)
• ignore_errors (bool)
• file_type (Literal['json'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputJsonTable(InputCsvTable):
    """Defines settings for reading a JSON file."""

    file_type: Literal["json"] = "json"

InputNdjsonTable pydantic-model

Bases: InputTableBase

Defines settings for reading a newline-delimited JSON file.

Show JSON schema:

{
  "description": "Defines settings for reading a newline-delimited JSON file.",
  "properties": {
    "file_type": {
      "const": "ndjson",
      "default": "ndjson",
      "title": "File Type",
      "type": "string"
    }
  },
  "title": "InputNdjsonTable",
  "type": "object"
}

Fields:

• file_type (Literal['ndjson'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputNdjsonTable(InputTableBase):
    """Defines settings for reading a newline-delimited JSON file."""

    file_type: Literal["ndjson"] = "ndjson"

InputParquetTable pydantic-model

Bases: InputTableBase

Defines settings for reading a Parquet file.

Show JSON schema:

{
  "description": "Defines settings for reading a Parquet file.",
  "properties": {
    "file_type": {
      "const": "parquet",
      "default": "parquet",
      "title": "File Type",
      "type": "string"
    }
  },
  "title": "InputParquetTable",
  "type": "object"
}

Fields:

• file_type (Literal['parquet'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputParquetTable(InputTableBase):
    """Defines settings for reading a Parquet file."""

    file_type: Literal["parquet"] = "parquet"

InputTableBase pydantic-model

Bases: BaseModel

Base settings for input file operations.

Show JSON schema:

{
  "description": "Base settings for input file operations.",
  "properties": {
    "file_type": {
      "title": "File Type",
      "type": "string"
    }
  },
  "required": [
    "file_type"
  ],
  "title": "InputTableBase",
  "type": "object"
}

Fields:

• file_type (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class InputTableBase(BaseModel):
    """Base settings for input file operations."""

    file_type: str  # Will be overridden with Literal in subclasses

KafkaSourceSettings pydantic-model

Bases: BaseModel

Configuration for reading from a Kafka/Redpanda topic.

Show JSON schema:

{
  "description": "Configuration for reading from a Kafka/Redpanda topic.",
  "properties": {
    "kafka_connection_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Kafka Connection Id"
    },
    "kafka_connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Kafka Connection Name"
    },
    "topic_name": {
      "default": "",
      "title": "Topic Name",
      "type": "string"
    },
    "value_format": {
      "const": "json",
      "default": "json",
      "title": "Value Format",
      "type": "string"
    },
    "sync_name": {
      "default": "",
      "title": "Sync Name",
      "type": "string"
    },
    "start_offset": {
      "default": "latest",
      "enum": [
        "earliest",
        "latest"
      ],
      "title": "Start Offset",
      "type": "string"
    },
    "max_messages": {
      "default": 100000,
      "title": "Max Messages",
      "type": "integer"
    },
    "poll_timeout_seconds": {
      "default": 30.0,
      "title": "Poll Timeout Seconds",
      "type": "number"
    }
  },
  "title": "KafkaSourceSettings",
  "type": "object"
}

Fields:

• kafka_connection_id (int | None)
• kafka_connection_name (str | None)
• topic_name (str)
• value_format (Literal['json'])
• sync_name (str)
• start_offset (Literal['earliest', 'latest'])
• max_messages (int)
• poll_timeout_seconds (float)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class KafkaSourceSettings(BaseModel):
    """Configuration for reading from a Kafka/Redpanda topic."""

    kafka_connection_id: int | None = None
    kafka_connection_name: str | None = None
    topic_name: str = ""
    value_format: Literal["json"] = "json"
    sync_name: str = ""  # unique key for offset tracking between runs
    start_offset: Literal["earliest", "latest"] = "latest"
    max_messages: int = 100_000
    poll_timeout_seconds: float = 30.0

MinimalFieldInfo pydantic-model

Bases: BaseModel

Represents the most basic information about a data field (column).

Show JSON schema:

{
  "description": "Represents the most basic information about a data field (column).",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "data_type": {
      "default": "String",
      "title": "Data Type",
      "type": "string"
    }
  },
  "required": [
    "name"
  ],
  "title": "MinimalFieldInfo",
  "type": "object"
}

Fields:

• name (str)
• data_type (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class MinimalFieldInfo(BaseModel):
    """Represents the most basic information about a data field (column)."""

    name: str
    data_type: str = "String"

NewDirectory pydantic-model

Bases: BaseModel

Defines the information required to create a new directory.

Show JSON schema:

{
  "description": "Defines the information required to create a new directory.",
  "properties": {
    "source_path": {
      "title": "Source Path",
      "type": "string"
    },
    "dir_name": {
      "title": "Dir Name",
      "type": "string"
    }
  },
  "required": [
    "source_path",
    "dir_name"
  ],
  "title": "NewDirectory",
  "type": "object"
}

Fields:

• source_path (str)
• dir_name (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NewDirectory(BaseModel):
    """Defines the information required to create a new directory."""

    source_path: str
    dir_name: str

NodeApiResponse pydantic-model

Bases: NodeSingleInput

Settings for a node that marks its input as the body of an HTTP API response.

This node is a sink (one input, no output). When the flow is published as an API endpoint, the data flowing into this node is serialized and returned to the caller. During interactive runs it is a pass-through (its result equals its input), so previews keep working.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that marks its input as the body of an HTTP API response.\n\nThis node is a sink (one input, no output). When the flow is published as an\nAPI endpoint, the data flowing into this node is serialized and returned to the\ncaller. During interactive runs it is a pass-through (its result equals its\ninput), so previews keep working.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "orientation": {
      "default": "records",
      "enum": [
        "records",
        "columns"
      ],
      "title": "Orientation",
      "type": "string"
    },
    "max_rows": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Max Rows"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeApiResponse",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• orientation (Literal['records', 'columns'])
• max_rows (int | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeApiResponse(NodeSingleInput):
    """Settings for a node that marks its input as the body of an HTTP API response.

    This node is a sink (one input, no output). When the flow is published as an
    API endpoint, the data flowing into this node is serialized and returned to the
    caller. During interactive runs it is a pass-through (its result equals its
    input), so previews keep working.
    """

    orientation: Literal["records", "columns"] = "records"
    max_rows: int | None = None

    def get_default_description(self) -> str:
        """Describes the API response shape."""
        limit = f", max {self.max_rows} rows" if self.max_rows else ""
        return f"API response ({self.orientation}{limit})"

get_default_description()

Describes the API response shape.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the API response shape."""
    limit = f", max {self.max_rows} rows" if self.max_rows else ""
    return f"API response ({self.orientation}{limit})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeApplyModel pydantic-model

Bases: NodeSingleInput

Score data using a previously trained model artifact.

Show JSON schema:

{
  "$defs": {
    "ApplyModelSettings": {
      "description": "Settings payload for the Apply Model node.\n\nTwo model sources are supported:\n\n- ``\"upstream\"`` (default): pick a Train Model node from somewhere in this\n  flow's upstream chain. The model file is read from the flow's cache\n  directory using the train node's id \u2014 works at design time, no catalog\n  round-trip needed.\n- ``\"catalog\"``: fall back to the existing catalog lookup by name/version.",
      "properties": {
        "source": {
          "default": "upstream",
          "enum": [
            "upstream",
            "catalog"
          ],
          "title": "Source",
          "type": "string"
        },
        "upstream_node_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Upstream Node Id"
        },
        "model_name": {
          "default": "",
          "title": "Model Name",
          "type": "string"
        },
        "model_version": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Model Version"
        },
        "namespace_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Id"
        },
        "namespace_full_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Full Name"
        },
        "output_column": {
          "default": "prediction",
          "title": "Output Column",
          "type": "string"
        }
      },
      "title": "ApplyModelSettings",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Score data using a previously trained model artifact.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "apply_input": {
      "$ref": "#/$defs/ApplyModelSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeApplyModel",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• apply_input (ApplyModelSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeApplyModel(NodeSingleInput):
    """Score data using a previously trained model artifact."""

    model_config = ConfigDict(protected_namespaces=())

    apply_input: ApplyModelSettings = Field(default_factory=ApplyModelSettings)

    def get_default_description(self) -> str:
        s = self.apply_input
        if s.model_name:
            ver = f" v{s.model_version}" if s.model_version is not None else ""
            return f"Apply '{s.model_name}'{ver} -> {s.output_column}"
        return "Apply Model"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeBase pydantic-model

Bases: BaseModel

Base model for all nodes in a FlowGraph. Contains common metadata.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Base model for all nodes in a FlowGraph. Contains common metadata.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeBase",
  "type": "object"
}

Config:

• arbitrary_types_allowed: True

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeBase(BaseModel):
    """Base model for all nodes in a FlowGraph. Contains common metadata."""

    model_config = ConfigDict(arbitrary_types_allowed=True)
    flow_id: int
    node_id: int
    cache_results: bool | None = False
    pos_x: float | None = 0
    pos_y: float | None = 0
    group_id: int | None = None  # Visual group membership (organizational only; no execution impact)
    is_setup: bool | None = True
    description: str | None = ""
    node_reference: str | None = None  # Unique reference identifier for code generation (lowercase, no spaces)
    user_id: int | None = None
    is_flow_output: bool | None = False
    is_user_defined: bool | None = False  # Indicator if the node is a user defined node
    output_field_config: OutputFieldConfig | None = None

    @field_validator("node_reference", mode="before")
    @classmethod
    def validate_node_reference(cls, v):
        """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
        if v is None or v == "":
            return None
        if not isinstance(v, str):
            raise ValueError("node_reference must be a string")
        if " " in v:
            raise ValueError("node_reference cannot contain spaces")
        if v != v.lower():
            raise ValueError("node_reference must be lowercase")
        if not _SAFE_IDENTIFIER_RE.match(v):
            raise ValueError(
                "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
            )
        return v

    def get_default_description(self) -> str:
        """Generates a human-readable description based on the node's configured content.

        Subclasses override this to provide meaningful descriptions.
        Returns an empty string by default.
        """
        return ""

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeCatalogReader pydantic-model

Bases: NodeBase

Settings for a node that reads a table from the catalog.

Resolution priority at runtime: catalog_table_id > catalog_full_table_name > (catalog_table_name, catalog_namespace_id). The qualified form (catalog_full_table_name = "schema.table") is the preferred human-facing identifier when an id isn't available.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads a table from the catalog.\n\nResolution priority at runtime: ``catalog_table_id`` > ``catalog_full_table_name`` >\n``(catalog_table_name, catalog_namespace_id)``. The qualified form\n(``catalog_full_table_name`` = ``\"schema.table\"``) is the preferred human-facing\nidentifier when an id isn't available.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "catalog_table_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Catalog Table Id"
    },
    "catalog_full_table_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Catalog Full Table Name"
    },
    "catalog_table_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Catalog Table Name"
    },
    "catalog_namespace_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Catalog Namespace Id"
    },
    "delta_version": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Delta Version"
    },
    "scd2_view": {
      "anyOf": [
        {
          "enum": [
            "active",
            "all",
            "active_at"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Scd2 View"
    },
    "scd2_as_of": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Scd2 As Of"
    },
    "sql_query": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Sql Query"
    },
    "is_virtual_optimized": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Is Virtual Optimized"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeCatalogReader",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• catalog_table_id (int | None)
• catalog_full_table_name (str | None)
• catalog_table_name (str | None)
• catalog_namespace_id (int | None)
• delta_version (int | None)
• scd2_view (Literal['active', 'all', 'active_at'] | None)
• scd2_as_of (str | None)
• sql_query (str | None)
• is_virtual_optimized (bool | None)

Validators:

• validate_node_reference → node_reference
• _validate_scd2_view

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeCatalogReader(NodeBase):
    """Settings for a node that reads a table from the catalog.

    Resolution priority at runtime: ``catalog_table_id`` > ``catalog_full_table_name`` >
    ``(catalog_table_name, catalog_namespace_id)``. The qualified form
    (``catalog_full_table_name`` = ``"schema.table"``) is the preferred human-facing
    identifier when an id isn't available.
    """

    catalog_table_id: int | None = None
    catalog_full_table_name: str | None = None
    catalog_table_name: str | None = None
    catalog_namespace_id: int | None = None
    delta_version: int | None = None
    # SCD2 history view. Only honoured when the resolved table carries an ``scd2_config`` record;
    # ``None`` means no filter (every version) and is ignored for non-SCD2 tables.
    scd2_view: Literal["active", "all", "active_at"] | None = None
    scd2_as_of: str | None = None  # ISO-8601 instant, required when scd2_view == "active_at"
    sql_query: str | None = None
    is_virtual_optimized: bool | None = None

    @model_validator(mode="after")
    def _validate_scd2_view(self) -> "NodeCatalogReader":
        if self.scd2_view == "active_at" and not self.scd2_as_of:
            raise ValueError("scd2_as_of is required when scd2_view is 'active_at'")
        if self.scd2_as_of:
            try:
                # Python 3.10's fromisoformat rejects the trailing "Z" the UI's DateTimePicker emits.
                datetime.fromisoformat(self.scd2_as_of.replace("Z", "+00:00"))
            except ValueError as exc:
                raise ValueError(f"scd2_as_of must be an ISO-8601 datetime: {self.scd2_as_of!r}") from exc
        return self

    def get_default_description(self) -> str:
        if self.sql_query:
            first_line = self.sql_query.strip().split("\n")[0]
            if len(first_line) > 80:
                first_line = first_line[:77] + "..."
            return f"SQL: {first_line}"
        display = self.catalog_full_table_name or self.catalog_table_name
        if display:
            suffix = f" (v{self.delta_version})" if self.delta_version is not None else ""
            if self.scd2_view == "active":
                suffix += " [active]"
            elif self.scd2_view == "active_at" and self.scd2_as_of:
                suffix += f" [as of {self.scd2_as_of}]"
            return f"Catalog: {display}{suffix}"
        return "Read from Catalog"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeCatalogWriter pydantic-model

Bases: NodeSingleInput

Settings for a node that writes its input to the catalog.

Show JSON schema:

{
  "$defs": {
    "CatalogWriteSettings": {
      "description": "Settings for writing data to the catalog.\n\nThe target namespace is referenced name-first: ``namespace_full_name`` (``\"catalog.schema\"``) is\nthe portable reference that survives recreation on another machine; ``namespace_id`` is a numeric\nfallback for flows saved before names were stored.",
      "properties": {
        "table_name": {
          "default": "",
          "title": "Table Name",
          "type": "string"
        },
        "namespace_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Id"
        },
        "namespace_full_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Full Name"
        },
        "description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Description"
        },
        "write_mode": {
          "default": "overwrite",
          "enum": [
            "overwrite",
            "error",
            "append",
            "upsert",
            "update",
            "delete",
            "scd2",
            "virtual"
          ],
          "title": "Write Mode",
          "type": "string"
        },
        "merge_keys": {
          "items": {
            "type": "string"
          },
          "title": "Merge Keys",
          "type": "array"
        },
        "partition_by": {
          "items": {
            "type": "string"
          },
          "title": "Partition By",
          "type": "array"
        },
        "scd2": {
          "anyOf": [
            {
              "$ref": "#/$defs/Scd2Settings"
            },
            {
              "type": "null"
            }
          ],
          "default": null
        }
      },
      "title": "CatalogWriteSettings",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "Scd2Settings": {
      "description": "Slowly-changing-dimension type 2 configuration for a catalog write.\n\nThe business key is ``CatalogWriteSettings.merge_keys`` \u2014 this block only carries the\nchange-detection scope and the names of the four generated columns. It is persisted verbatim\nonto the catalog table record (``CatalogTable.scd2_config``) so a reader can filter history\nwithout ever reading a writer node's settings.",
      "properties": {
        "compare_columns": {
          "items": {
            "type": "string"
          },
          "title": "Compare Columns",
          "type": "array"
        },
        "full_snapshot": {
          "default": false,
          "title": "Full Snapshot",
          "type": "boolean"
        },
        "partition_on_current": {
          "default": true,
          "title": "Partition On Current",
          "type": "boolean"
        },
        "surrogate_key_column": {
          "default": "sk",
          "title": "Surrogate Key Column",
          "type": "string"
        },
        "valid_from_column": {
          "default": "valid_from",
          "title": "Valid From Column",
          "type": "string"
        },
        "valid_to_column": {
          "default": "valid_to",
          "title": "Valid To Column",
          "type": "string"
        },
        "is_current_column": {
          "default": "is_current",
          "title": "Is Current Column",
          "type": "string"
        }
      },
      "title": "Scd2Settings",
      "type": "object"
    }
  },
  "description": "Settings for a node that writes its input to the catalog.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "catalog_write_settings": {
      "$ref": "#/$defs/CatalogWriteSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeCatalogWriter",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• catalog_write_settings (CatalogWriteSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeCatalogWriter(NodeSingleInput):
    """Settings for a node that writes its input to the catalog."""

    catalog_write_settings: CatalogWriteSettings = Field(default_factory=CatalogWriteSettings)

    def get_default_description(self) -> str:
        s = self.catalog_write_settings
        return f"Catalog: {s.table_name}" if s.table_name else "Write to Catalog"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeCloudStorageReader pydantic-model

Bases: NodeBase

Settings for a node that reads from a cloud storage service (S3, GCS, etc.).

Show JSON schema:

{
  "$defs": {
    "CloudStorageReadSettings": {
      "description": "Settings for reading from cloud storage",
      "properties": {
        "auth_mode": {
          "default": "auto",
          "enum": [
            "access_key",
            "iam_role",
            "service_principal",
            "managed_identity",
            "sas_token",
            "aws-cli",
            "env_vars",
            "service_account",
            "auto"
          ],
          "title": "Auth Mode",
          "type": "string"
        },
        "connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Connection Name"
        },
        "resource_path": {
          "title": "Resource Path",
          "type": "string"
        },
        "scan_mode": {
          "default": "single_file",
          "enum": [
            "single_file",
            "directory"
          ],
          "title": "Scan Mode",
          "type": "string"
        },
        "file_format": {
          "default": "parquet",
          "enum": [
            "csv",
            "parquet",
            "json",
            "delta",
            "iceberg"
          ],
          "title": "File Format",
          "type": "string"
        },
        "csv_has_header": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": true,
          "title": "Csv Has Header"
        },
        "csv_delimiter": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": ",",
          "title": "Csv Delimiter"
        },
        "csv_encoding": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "utf8",
          "title": "Csv Encoding"
        },
        "delta_version": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Delta Version"
        }
      },
      "required": [
        "resource_path"
      ],
      "title": "CloudStorageReadSettings",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads from a cloud storage service (S3, GCS, etc.).",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "cloud_storage_settings": {
      "$ref": "#/$defs/CloudStorageReadSettings"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "cloud_storage_settings"
  ],
  "title": "NodeCloudStorageReader",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• cloud_storage_settings (CloudStorageReadSettings)
• fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeCloudStorageReader(NodeBase):
    """Settings for a node that reads from a cloud storage service (S3, GCS, etc.)."""

    cloud_storage_settings: CloudStorageReadSettings
    fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        """Describes the cloud storage source."""
        cs = self.cloud_storage_settings
        return f"Read {cs.resource_path} ({cs.file_format})"

get_default_description()

Describes the cloud storage source.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the cloud storage source."""
    cs = self.cloud_storage_settings
    return f"Read {cs.resource_path} ({cs.file_format})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeCloudStorageWriter pydantic-model

Bases: NodeSingleInput

Settings for a node that writes to a cloud storage service.

Show JSON schema:

{
  "$defs": {
    "CloudStorageWriteSettings": {
      "description": "Settings for writing to cloud storage",
      "properties": {
        "resource_path": {
          "title": "Resource Path",
          "type": "string"
        },
        "write_mode": {
          "default": "overwrite",
          "enum": [
            "overwrite",
            "append"
          ],
          "title": "Write Mode",
          "type": "string"
        },
        "file_format": {
          "default": "parquet",
          "enum": [
            "csv",
            "parquet",
            "json",
            "delta"
          ],
          "title": "File Format",
          "type": "string"
        },
        "parquet_compression": {
          "default": "snappy",
          "enum": [
            "snappy",
            "gzip",
            "brotli",
            "lz4",
            "zstd"
          ],
          "title": "Parquet Compression",
          "type": "string"
        },
        "csv_delimiter": {
          "default": ",",
          "title": "Csv Delimiter",
          "type": "string"
        },
        "csv_encoding": {
          "default": "utf8",
          "title": "Csv Encoding",
          "type": "string"
        },
        "partition_by": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Partition By"
        },
        "auth_mode": {
          "default": "auto",
          "enum": [
            "access_key",
            "iam_role",
            "service_principal",
            "managed_identity",
            "sas_token",
            "aws-cli",
            "env_vars",
            "service_account",
            "auto"
          ],
          "title": "Auth Mode",
          "type": "string"
        },
        "connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Connection Name"
        }
      },
      "required": [
        "resource_path"
      ],
      "title": "CloudStorageWriteSettings",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that writes to a cloud storage service.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "cloud_storage_settings": {
      "$ref": "#/$defs/CloudStorageWriteSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "cloud_storage_settings"
  ],
  "title": "NodeCloudStorageWriter",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• cloud_storage_settings (CloudStorageWriteSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeCloudStorageWriter(NodeSingleInput):
    """Settings for a node that writes to a cloud storage service."""

    cloud_storage_settings: CloudStorageWriteSettings

    def get_default_description(self) -> str:
        """Describes the cloud storage write target."""
        cs = self.cloud_storage_settings
        return f"Write to {cs.resource_path} ({cs.file_format})"

get_default_description()

Describes the cloud storage write target.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the cloud storage write target."""
    cs = self.cloud_storage_settings
    return f"Write to {cs.resource_path} ({cs.file_format})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeConnection pydantic-model

Bases: BaseModel

Represents a connection (edge) between two nodes in the graph.

Show JSON schema:

{
  "$defs": {
    "NodeInputConnection": {
      "description": "Represents the input side of a connection between two nodes.",
      "properties": {
        "node_id": {
          "title": "Node Id",
          "type": "integer"
        },
        "connection_class": {
          "enum": [
            "input-0",
            "input-1",
            "input-2",
            "input-3",
            "input-4",
            "input-5",
            "input-6",
            "input-7",
            "input-8",
            "input-9"
          ],
          "title": "Connection Class",
          "type": "string"
        }
      },
      "required": [
        "node_id",
        "connection_class"
      ],
      "title": "NodeInputConnection",
      "type": "object"
    },
    "NodeOutputConnection": {
      "description": "Represents the output side of a connection between two nodes.",
      "properties": {
        "node_id": {
          "title": "Node Id",
          "type": "integer"
        },
        "connection_class": {
          "enum": [
            "output-0",
            "output-1",
            "output-2",
            "output-3",
            "output-4",
            "output-5",
            "output-6",
            "output-7",
            "output-8",
            "output-9"
          ],
          "title": "Connection Class",
          "type": "string"
        }
      },
      "required": [
        "node_id",
        "connection_class"
      ],
      "title": "NodeOutputConnection",
      "type": "object"
    }
  },
  "description": "Represents a connection (edge) between two nodes in the graph.",
  "properties": {
    "input_connection": {
      "$ref": "#/$defs/NodeInputConnection"
    },
    "output_connection": {
      "$ref": "#/$defs/NodeOutputConnection"
    }
  },
  "required": [
    "input_connection",
    "output_connection"
  ],
  "title": "NodeConnection",
  "type": "object"
}

Fields:

• input_connection (NodeInputConnection)
• output_connection (NodeOutputConnection)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeConnection(BaseModel):
    """Represents a connection (edge) between two nodes in the graph."""

    input_connection: NodeInputConnection
    output_connection: NodeOutputConnection

    @classmethod
    def create_from_simple_input(
        cls,
        from_id: int,
        to_id: int,
        input_type: InputType = "input-0",
        output_handle: OutputConnectionClass = "output-0",
    ):
        """Creates a standard connection between two nodes."""
        match input_type:
            case "main":
                connection_class: InputConnectionClass = "input-0"
            case "right":
                connection_class: InputConnectionClass = "input-1"
            case "left":
                connection_class: InputConnectionClass = "input-2"
            case _:
                connection_class: InputConnectionClass = "input-0"
        node_input = NodeInputConnection(node_id=to_id, connection_class=connection_class)
        node_output = NodeOutputConnection(node_id=from_id, connection_class=output_handle)
        return cls(input_connection=node_input, output_connection=node_output)

create_from_simple_input(from_id, to_id, input_type='input-0', output_handle='output-0') classmethod

Creates a standard connection between two nodes.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@classmethod
def create_from_simple_input(
    cls,
    from_id: int,
    to_id: int,
    input_type: InputType = "input-0",
    output_handle: OutputConnectionClass = "output-0",
):
    """Creates a standard connection between two nodes."""
    match input_type:
        case "main":
            connection_class: InputConnectionClass = "input-0"
        case "right":
            connection_class: InputConnectionClass = "input-1"
        case "left":
            connection_class: InputConnectionClass = "input-2"
        case _:
            connection_class: InputConnectionClass = "input-0"
    node_input = NodeInputConnection(node_id=to_id, connection_class=connection_class)
    node_output = NodeOutputConnection(node_id=from_id, connection_class=output_handle)
    return cls(input_connection=node_input, output_connection=node_output)

NodeCrossJoin pydantic-model

Bases: NodeMultiInput

Settings for a node that performs a cross join.

Show JSON schema:

{
  "$defs": {
    "CrossJoinInput": {
      "description": "Data model for cross join operations.",
      "properties": {
        "left_select": {
          "$ref": "#/$defs/JoinInputs"
        },
        "right_select": {
          "$ref": "#/$defs/JoinInputs"
        }
      },
      "required": [
        "left_select",
        "right_select"
      ],
      "title": "CrossJoinInput",
      "type": "object"
    },
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that performs a cross join.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "auto_generate_selection": {
      "default": true,
      "title": "Auto Generate Selection",
      "type": "boolean"
    },
    "verify_integrity": {
      "default": true,
      "title": "Verify Integrity",
      "type": "boolean"
    },
    "cross_join_input": {
      "$ref": "#/$defs/CrossJoinInput"
    },
    "auto_keep_all": {
      "default": true,
      "title": "Auto Keep All",
      "type": "boolean"
    },
    "auto_keep_right": {
      "default": true,
      "title": "Auto Keep Right",
      "type": "boolean"
    },
    "auto_keep_left": {
      "default": true,
      "title": "Auto Keep Left",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "cross_join_input"
  ],
  "title": "NodeCrossJoin",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• auto_generate_selection (bool)
• verify_integrity (bool)
• cross_join_input (CrossJoinInput)
• auto_keep_all (bool)
• auto_keep_right (bool)
• auto_keep_left (bool)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeCrossJoin(NodeMultiInput):
    """Settings for a node that performs a cross join."""

    auto_generate_selection: bool = True
    verify_integrity: bool = True
    cross_join_input: transform_schema.CrossJoinInput
    auto_keep_all: bool = True
    auto_keep_right: bool = True
    auto_keep_left: bool = True

    def get_default_description(self) -> str:
        """Describes the cross join."""
        return "Cross join"

    def to_yaml_dict(self) -> NodeCrossJoinYaml:
        """Converts the cross join node settings to a dictionary for YAML serialization."""
        result: NodeCrossJoinYaml = {
            "cache_results": self.cache_results,
            "auto_generate_selection": self.auto_generate_selection,
            "verify_integrity": self.verify_integrity,
            "cross_join_input": self.cross_join_input.to_yaml_dict(),
            "auto_keep_all": self.auto_keep_all,
            "auto_keep_right": self.auto_keep_right,
            "auto_keep_left": self.auto_keep_left,
        }
        if self.output_field_config:
            result["output_field_config"] = {
                "enabled": self.output_field_config.enabled,
                "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
                "validate_data_types": self.output_field_config.validate_data_types,
                "fields": [
                    {
                        "name": f.name,
                        "data_type": f.data_type,
                        "default_value": f.default_value,
                    }
                    for f in self.output_field_config.fields
                ],
            }
        return result

get_default_description()

Describes the cross join.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the cross join."""
    return "Cross join"

to_yaml_dict()

Converts the cross join node settings to a dictionary for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> NodeCrossJoinYaml:
    """Converts the cross join node settings to a dictionary for YAML serialization."""
    result: NodeCrossJoinYaml = {
        "cache_results": self.cache_results,
        "auto_generate_selection": self.auto_generate_selection,
        "verify_integrity": self.verify_integrity,
        "cross_join_input": self.cross_join_input.to_yaml_dict(),
        "auto_keep_all": self.auto_keep_all,
        "auto_keep_right": self.auto_keep_right,
        "auto_keep_left": self.auto_keep_left,
    }
    if self.output_field_config:
        result["output_field_config"] = {
            "enabled": self.output_field_config.enabled,
            "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
            "validate_data_types": self.output_field_config.validate_data_types,
            "fields": [
                {
                    "name": f.name,
                    "data_type": f.data_type,
                    "default_value": f.default_value,
                }
                for f in self.output_field_config.fields
            ],
        }
    return result

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeDatabaseReader pydantic-model

Bases: NodeBase

Settings for a node that reads from a database.

Show JSON schema:

{
  "$defs": {
    "DatabaseConnection": {
      "description": "Defines the connection parameters for a database.",
      "properties": {
        "database_type": {
          "default": "postgresql",
          "title": "Database Type",
          "type": "string"
        },
        "username": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Username"
        },
        "password_ref": {
          "anyOf": [
            {
              "description": "An ID referencing an encrypted secret.",
              "maxLength": 100,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Password Ref"
        },
        "host": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Host"
        },
        "port": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Port"
        },
        "database": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database"
        },
        "url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Url"
        }
      },
      "title": "DatabaseConnection",
      "type": "object"
    },
    "DatabaseSettings": {
      "description": "Defines settings for reading from a database, either via table or query.",
      "properties": {
        "connection_mode": {
          "anyOf": [
            {
              "enum": [
                "inline",
                "reference"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "inline",
          "title": "Connection Mode"
        },
        "database_connection": {
          "anyOf": [
            {
              "$ref": "#/$defs/DatabaseConnection"
            },
            {
              "type": "null"
            }
          ],
          "default": null
        },
        "database_connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database Connection Name"
        },
        "schema_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Schema Name"
        },
        "table_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Table Name"
        },
        "query": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Query"
        },
        "query_mode": {
          "default": "table",
          "enum": [
            "query",
            "table",
            "reference"
          ],
          "title": "Query Mode",
          "type": "string"
        }
      },
      "title": "DatabaseSettings",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads from a database.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "database_settings": {
      "$ref": "#/$defs/DatabaseSettings"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "database_settings"
  ],
  "title": "NodeDatabaseReader",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• database_settings (DatabaseSettings)
• fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeDatabaseReader(NodeBase):
    """Settings for a node that reads from a database."""

    database_settings: DatabaseSettings
    fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        """Describes the database source."""
        ds = self.database_settings
        if ds.query_mode == "table" and ds.table_name:
            table = f"{ds.schema_name}.{ds.table_name}" if ds.schema_name else ds.table_name
            return f"Read from {table}"
        if ds.query_mode == "query" and ds.query:
            q = ds.query
            if len(q) > 60:
                q = q[:57] + "..."
            return f"Query: {q}"
        return ""

get_default_description()

Describes the database source.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the database source."""
    ds = self.database_settings
    if ds.query_mode == "table" and ds.table_name:
        table = f"{ds.schema_name}.{ds.table_name}" if ds.schema_name else ds.table_name
        return f"Read from {table}"
    if ds.query_mode == "query" and ds.query:
        q = ds.query
        if len(q) > 60:
            q = q[:57] + "..."
        return f"Query: {q}"
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeDatabaseWriter pydantic-model

Bases: NodeSingleInput

Settings for a node that writes data to a database.

Show JSON schema:

{
  "$defs": {
    "DatabaseConnection": {
      "description": "Defines the connection parameters for a database.",
      "properties": {
        "database_type": {
          "default": "postgresql",
          "title": "Database Type",
          "type": "string"
        },
        "username": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Username"
        },
        "password_ref": {
          "anyOf": [
            {
              "description": "An ID referencing an encrypted secret.",
              "maxLength": 100,
              "minLength": 1,
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Password Ref"
        },
        "host": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Host"
        },
        "port": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Port"
        },
        "database": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database"
        },
        "url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Url"
        }
      },
      "title": "DatabaseConnection",
      "type": "object"
    },
    "DatabaseWriteSettings": {
      "description": "Defines settings for writing data to a database table.",
      "properties": {
        "connection_mode": {
          "anyOf": [
            {
              "enum": [
                "inline",
                "reference"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "inline",
          "title": "Connection Mode"
        },
        "database_connection": {
          "anyOf": [
            {
              "$ref": "#/$defs/DatabaseConnection"
            },
            {
              "type": "null"
            }
          ],
          "default": null
        },
        "database_connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Database Connection Name"
        },
        "table_name": {
          "title": "Table Name",
          "type": "string"
        },
        "schema_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Schema Name"
        },
        "if_exists": {
          "anyOf": [
            {
              "enum": [
                "append",
                "replace",
                "fail"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "append",
          "title": "If Exists"
        }
      },
      "required": [
        "table_name"
      ],
      "title": "DatabaseWriteSettings",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that writes data to a database.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "database_write_settings": {
      "$ref": "#/$defs/DatabaseWriteSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "database_write_settings"
  ],
  "title": "NodeDatabaseWriter",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• database_write_settings (DatabaseWriteSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeDatabaseWriter(NodeSingleInput):
    """Settings for a node that writes data to a database."""

    database_write_settings: DatabaseWriteSettings

    def get_default_description(self) -> str:
        """Describes the database write target."""
        dw = self.database_write_settings
        table = f"{dw.schema_name}.{dw.table_name}" if dw.schema_name else dw.table_name
        return f"Write to {table} ({dw.if_exists})"

get_default_description()

Describes the database write target.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the database write target."""
    dw = self.database_write_settings
    table = f"{dw.schema_name}.{dw.table_name}" if dw.schema_name else dw.table_name
    return f"Write to {table} ({dw.if_exists})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeDatasource pydantic-model

Bases: NodeBase

Base settings for a node that acts as a data source.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Base settings for a node that acts as a data source.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "file_ref": {
      "default": null,
      "title": "File Ref",
      "type": "string"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeDatasource",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• file_ref (str)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeDatasource(NodeBase):
    """Base settings for a node that acts as a data source."""

    file_ref: str = None

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeDescription pydantic-model

Bases: BaseModel

A simple model for updating a node's description text.

Show JSON schema:

{
  "description": "A simple model for updating a node's description text.",
  "properties": {
    "description": {
      "default": "",
      "title": "Description",
      "type": "string"
    }
  },
  "title": "NodeDescription",
  "type": "object"
}

Fields:

• description (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeDescription(BaseModel):
    """A simple model for updating a node's description text."""

    description: str = ""

NodeDynamicRename pydantic-model

Bases: NodeSingleInput

Settings for a node that renames many columns at once via a single rule.

Show JSON schema:

{
  "$defs": {
    "DynamicRenameInput": {
      "description": "Defines settings for a dynamic rename operation.\n\nApplies a single rule (prefix / suffix / formula / first_row) to a set of selected\ncolumns, rather than requiring the user to rename columns one-by-one.\n\nIn formula mode, the flowfile formula syntax is evaluated with `[column_name]`\nbound to each target column's current name; for example `uppercase([column_name])`\nor `\"v2_\" + [column_name]`.\n\nIn first_row mode, the first row of the incoming table is promoted to column\nheaders and then dropped from the data. Non-string values are coerced to `str`;\nnull or empty values raise an error. Selection filters still apply \u2014 only selected\ncolumns are renamed, but the first row is always dropped.",
      "properties": {
        "rename_mode": {
          "default": "prefix",
          "enum": [
            "prefix",
            "suffix",
            "formula",
            "first_row"
          ],
          "title": "Rename Mode",
          "type": "string"
        },
        "prefix": {
          "default": "",
          "title": "Prefix",
          "type": "string"
        },
        "suffix": {
          "default": "",
          "title": "Suffix",
          "type": "string"
        },
        "formula": {
          "default": "",
          "expression": true,
          "title": "Formula",
          "type": "string"
        },
        "selection_mode": {
          "default": "all",
          "enum": [
            "all",
            "list",
            "data_type"
          ],
          "title": "Selection Mode",
          "type": "string"
        },
        "selected_columns": {
          "items": {
            "type": "string"
          },
          "title": "Selected Columns",
          "type": "array"
        },
        "selected_data_type": {
          "anyOf": [
            {
              "enum": [
                "Numeric",
                "String",
                "Date",
                "Other",
                "Boolean",
                "Binary",
                "Complex"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Selected Data Type"
        }
      },
      "title": "DynamicRenameInput",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that renames many columns at once via a single rule.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "dynamic_rename_input": {
      "$ref": "#/$defs/DynamicRenameInput"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeDynamicRename",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• dynamic_rename_input (DynamicRenameInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeDynamicRename(NodeSingleInput):
    """Settings for a node that renames many columns at once via a single rule."""

    dynamic_rename_input: transform_schema.DynamicRenameInput = Field(
        default_factory=transform_schema.DynamicRenameInput
    )

    def get_default_description(self) -> str:
        """Describes the dynamic rename rule."""
        s = self.dynamic_rename_input
        if s.rename_mode == "prefix" and s.prefix:
            rule = f"prefix '{s.prefix}'"
        elif s.rename_mode == "suffix" and s.suffix:
            rule = f"suffix '{s.suffix}'"
        elif s.rename_mode == "formula" and s.formula:
            rule = f"formula {s.formula}"
        elif s.rename_mode == "first_row":
            rule = "promote first row to headers"
        else:
            return ""
        if s.selection_mode == "all":
            scope = "all columns"
        elif s.selection_mode == "list":
            scope = f"{len(s.selected_columns)} column(s)"
        else:
            scope = f"{s.selected_data_type or '(none)'} columns"
        return f"{rule} on {scope}"

get_default_description()

Describes the dynamic rename rule.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the dynamic rename rule."""
    s = self.dynamic_rename_input
    if s.rename_mode == "prefix" and s.prefix:
        rule = f"prefix '{s.prefix}'"
    elif s.rename_mode == "suffix" and s.suffix:
        rule = f"suffix '{s.suffix}'"
    elif s.rename_mode == "formula" and s.formula:
        rule = f"formula {s.formula}"
    elif s.rename_mode == "first_row":
        rule = "promote first row to headers"
    else:
        return ""
    if s.selection_mode == "all":
        scope = "all columns"
    elif s.selection_mode == "list":
        scope = f"{len(s.selected_columns)} column(s)"
    else:
        scope = f"{s.selected_data_type or '(none)'} columns"
    return f"{rule} on {scope}"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeEvaluateModel pydantic-model

Bases: NodeSingleInput

Compute model-quality metrics by comparing actual and predicted columns.

Show JSON schema:

{
  "$defs": {
    "EvaluateModelSettings": {
      "description": "Settings payload for the Evaluate Model node.\n\nDecoupled from any specific Train/Apply pair: takes a dataframe that\nalready contains both the actual target column and a prediction column\nand emits a long-form ``(metric, value)`` frame. Reusable on training,\ntest, or hold-out splits.\n\n``task_type=\"auto\"`` resolves the metric set from an upstream Train\nModel node when one is configured; otherwise defaults to ``regression``.",
      "properties": {
        "actual_column": {
          "default": "",
          "title": "Actual Column",
          "type": "string"
        },
        "predicted_column": {
          "default": "prediction",
          "title": "Predicted Column",
          "type": "string"
        },
        "task_type": {
          "default": "auto",
          "enum": [
            "auto",
            "regression",
            "classification"
          ],
          "title": "Task Type",
          "type": "string"
        },
        "upstream_train_node_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Upstream Train Node Id"
        }
      },
      "title": "EvaluateModelSettings",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Compute model-quality metrics by comparing actual and predicted columns.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "evaluate_input": {
      "$ref": "#/$defs/EvaluateModelSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeEvaluateModel",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• evaluate_input (EvaluateModelSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeEvaluateModel(NodeSingleInput):
    """Compute model-quality metrics by comparing actual and predicted columns."""

    model_config = ConfigDict(protected_namespaces=())

    evaluate_input: EvaluateModelSettings = Field(default_factory=EvaluateModelSettings)

    def get_default_description(self) -> str:
        s = self.evaluate_input
        if s.actual_column and s.predicted_column:
            return f"Evaluate {s.predicted_column} vs {s.actual_column}"
        return "Evaluate Model"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeExploreData pydantic-model

Bases: NodeBase

Settings for a node that provides an interactive data exploration interface.

Show JSON schema:

{
  "$defs": {
    "DataModel": {
      "properties": {
        "data": {
          "items": {
            "additionalProperties": true,
            "type": "object"
          },
          "title": "Data",
          "type": "array"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/MutField"
          },
          "title": "Fields",
          "type": "array"
        }
      },
      "required": [
        "data",
        "fields"
      ],
      "title": "DataModel",
      "type": "object"
    },
    "GraphicWalkerInput": {
      "properties": {
        "dataModel": {
          "$ref": "#/$defs/DataModel"
        },
        "is_initial": {
          "default": true,
          "title": "Is Initial",
          "type": "boolean"
        },
        "specList": {
          "anyOf": [
            {
              "items": {},
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Speclist"
        }
      },
      "title": "GraphicWalkerInput",
      "type": "object"
    },
    "MutField": {
      "properties": {
        "fid": {
          "title": "Fid",
          "type": "string"
        },
        "key": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Key"
        },
        "name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Name"
        },
        "basename": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Basename"
        },
        "disable": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": false,
          "title": "Disable"
        },
        "semanticType": {
          "title": "Semantictype",
          "type": "string"
        },
        "analyticType": {
          "enum": [
            "measure",
            "dimension"
          ],
          "title": "Analytictype",
          "type": "string"
        },
        "path": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Path"
        },
        "offset": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Offset"
        }
      },
      "required": [
        "fid",
        "semanticType",
        "analyticType"
      ],
      "title": "MutField",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that provides an interactive data exploration interface.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "graphic_walker_input": {
      "anyOf": [
        {
          "$ref": "#/$defs/GraphicWalkerInput"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeExploreData",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• graphic_walker_input (GraphicWalkerInput | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeExploreData(NodeBase):
    """Settings for a node that provides an interactive data exploration interface."""

    graphic_walker_input: gs_schemas.GraphicWalkerInput | None = None

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeExternalSource pydantic-model

Bases: NodeBase

Settings for a node that connects to a registered external data source.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SampleUsers": {
      "description": "Settings for generating a sample dataset of users.",
      "properties": {
        "orientation": {
          "default": "row",
          "title": "Orientation",
          "type": "string"
        },
        "fields": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/MinimalFieldInfo"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Fields"
        },
        "SAMPLE_USERS": {
          "title": "Sample Users",
          "type": "boolean"
        },
        "class_name": {
          "default": "sample_users",
          "title": "Class Name",
          "type": "string"
        },
        "size": {
          "default": 100,
          "title": "Size",
          "type": "integer"
        }
      },
      "required": [
        "SAMPLE_USERS"
      ],
      "title": "SampleUsers",
      "type": "object"
    }
  },
  "description": "Settings for a node that connects to a registered external data source.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "identifier": {
      "title": "Identifier",
      "type": "string"
    },
    "source_settings": {
      "$ref": "#/$defs/SampleUsers"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "identifier",
    "source_settings"
  ],
  "title": "NodeExternalSource",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• identifier (str)
• source_settings (SampleUsers)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeExternalSource(NodeBase):
    """Settings for a node that connects to a registered external data source."""

    identifier: str
    source_settings: SampleUsers

    def get_default_description(self) -> str:
        """Describes the external source."""
        return self.identifier

get_default_description()

Describes the external source.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the external source."""
    return self.identifier

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeFilter pydantic-model

Bases: NodeSingleInput

Settings for a node that filters rows based on a condition.

Show JSON schema:

{
  "$defs": {
    "BasicFilter": {
      "description": "Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').\n\nAttributes:\n    field: The column name to filter on.\n    operator: The comparison operator (FilterOperator enum value or symbol).\n    value: The value to compare against.\n    value2: Second value for BETWEEN operator (optional).",
      "properties": {
        "field": {
          "default": "",
          "title": "Field",
          "type": "string"
        },
        "operator": {
          "anyOf": [
            {
              "$ref": "#/$defs/FilterOperator"
            },
            {
              "type": "string"
            }
          ],
          "default": "equals",
          "title": "Operator"
        },
        "value": {
          "default": "",
          "title": "Value",
          "type": "string"
        },
        "value2": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Value2"
        },
        "filter_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Filter Type"
        },
        "filter_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Filter Value"
        }
      },
      "title": "BasicFilter",
      "type": "object"
    },
    "FilterInput": {
      "description": "Defines the settings for a filter operation, supporting basic or advanced (expression-based) modes.\n\nAttributes:\n    mode: The filter mode - \"basic\" or \"advanced\".\n    basic_filter: The basic filter configuration (used when mode=\"basic\").\n    advanced_filter: The advanced filter expression string (used when mode=\"advanced\").",
      "properties": {
        "mode": {
          "default": "basic",
          "enum": [
            "basic",
            "advanced"
          ],
          "title": "Mode",
          "type": "string"
        },
        "basic_filter": {
          "anyOf": [
            {
              "$ref": "#/$defs/BasicFilter"
            },
            {
              "type": "null"
            }
          ],
          "default": null
        },
        "advanced_filter": {
          "default": "",
          "expression": true,
          "title": "Advanced Filter",
          "type": "string"
        },
        "filter_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Filter Type"
        }
      },
      "title": "FilterInput",
      "type": "object"
    },
    "FilterOperator": {
      "description": "Supported filter comparison operators.",
      "enum": [
        "equals",
        "not_equals",
        "greater_than",
        "greater_than_or_equals",
        "less_than",
        "less_than_or_equals",
        "contains",
        "not_contains",
        "starts_with",
        "ends_with",
        "is_null",
        "is_not_null",
        "in",
        "not_in",
        "between"
      ],
      "title": "FilterOperator",
      "type": "string"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that filters rows based on a condition.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "filter_input": {
      "$ref": "#/$defs/FilterInput"
    },
    "split_mode": {
      "default": false,
      "title": "Split Mode",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "filter_input"
  ],
  "title": "NodeFilter",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• filter_input (FilterInput)
• split_mode (bool)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeFilter(NodeSingleInput):
    """Settings for a node that filters rows based on a condition."""

    filter_input: transform_schema.FilterInput
    # When True the node emits two outputs: "pass" (output-0, matching rows)
    # and "fail" (output-1, non-matching rows). Default preserves
    # single-output behaviour for existing flows.
    split_mode: bool = False

    @property
    def output_names(self) -> list[str] | None:
        """Declared output handles, so the canvas can build both without the drawer.

        The template's ``output`` stays 1 for backwards compatibility; the canvas
        sizes the handle list off ``max(output, len(output_names))``. Matches the
        keys ``FlowDataEngine.filter_split`` produces at run time.
        """
        return ["pass", "fail"] if self.split_mode else None

    def get_default_description(self) -> str:
        """Describes the filter condition."""
        fi = self.filter_input
        if fi.mode == "advanced" and fi.advanced_filter:
            expr = fi.advanced_filter
            if len(expr) > 80:
                expr = expr[:77] + "..."
            return expr
        if fi.mode == "basic" and fi.basic_filter:
            bf = fi.basic_filter
            if not bf.field:
                return ""
            op = bf.operator
            op_str = op.to_symbol() if hasattr(op, "to_symbol") else str(op)
            if op_str in ("is_null", "is_not_null"):
                return f"{bf.field} {op_str}"
            if op_str == "between" and bf.value2:
                return f"{bf.field} between {bf.value} and {bf.value2}"
            return f"{bf.field} {op_str} {bf.value}"
        return ""

output_names property

Declared output handles, so the canvas can build both without the drawer.

The template's output stays 1 for backwards compatibility; the canvas sizes the handle list off max(output, len(output_names)). Matches the keys FlowDataEngine.filter_split produces at run time.

get_default_description()

Describes the filter condition.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the filter condition."""
    fi = self.filter_input
    if fi.mode == "advanced" and fi.advanced_filter:
        expr = fi.advanced_filter
        if len(expr) > 80:
            expr = expr[:77] + "..."
        return expr
    if fi.mode == "basic" and fi.basic_filter:
        bf = fi.basic_filter
        if not bf.field:
            return ""
        op = bf.operator
        op_str = op.to_symbol() if hasattr(op, "to_symbol") else str(op)
        if op_str in ("is_null", "is_not_null"):
            return f"{bf.field} {op_str}"
        if op_str == "between" and bf.value2:
            return f"{bf.field} between {bf.value} and {bf.value2}"
        return f"{bf.field} {op_str} {bf.value}"
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeFlowInput pydantic-model

Bases: NodeManualInput

Named source placeholder inside a subflow.

Extends NodeManualInput so the sample data reuses the manual-input settings shape and editor. Standalone runs serve raw_data_format (empty frame when blank); a parent run_flow node injects real data at execution time.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RawData": {
      "description": "Represents data in a raw, columnar format for manual input.",
      "properties": {
        "columns": {
          "description": "Schema in column order. The i-th MinimalFieldInfo describes the values in data[i].",
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "title": "Columns",
          "type": "array"
        },
        "data": {
          "description": "Columnar layout: data[i] is the list of values for columns[i], in column order. len(data) must equal len(columns); each inner list has the same length (one entry per row). For two rows of {name, age}, emit [[\"Alice\", \"Bob\"], [30, 25]] \u2014 NOT [[\"Alice\", 30], [\"Bob\", 25]]. Reading rows back is `data[col_idx][row_idx]`.",
          "items": {
            "items": {},
            "type": "array"
          },
          "title": "Data",
          "type": "array"
        }
      },
      "required": [
        "columns",
        "data"
      ],
      "title": "RawData",
      "type": "object"
    }
  },
  "description": "Named source placeholder inside a subflow.\n\nExtends NodeManualInput so the sample data reuses the manual-input settings\nshape and editor. Standalone runs serve ``raw_data_format`` (empty frame when\nblank); a parent run_flow node injects real data at execution time.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "raw_data_format": {
      "$ref": "#/$defs/RawData"
    },
    "input_name": {
      "default": "input",
      "title": "Input Name",
      "type": "string"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "raw_data_format"
  ],
  "title": "NodeFlowInput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• raw_data_format (RawData)
• input_name (str)

Validators:

• validate_node_reference → node_reference
• _coerce_none_raw_data_format
• _validate_input_name → input_name

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeFlowInput(NodeManualInput):
    """Named source placeholder inside a subflow.

    Extends NodeManualInput so the sample data reuses the manual-input settings
    shape and editor. Standalone runs serve ``raw_data_format`` (empty frame when
    blank); a parent run_flow node injects real data at execution time.
    """

    input_name: str = "input"

    @field_validator("input_name")
    @classmethod
    def _validate_input_name(cls, v: str) -> str:
        return _validate_port_name(v, "flow input")

    def get_default_description(self) -> str:
        return f"Subflow input '{self.input_name}'"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeFlowOutput pydantic-model

Bases: NodeSingleInput

Named passthrough sink marking a subflow output; multiple allowed per flow.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Named passthrough sink marking a subflow output; multiple allowed per flow.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "output_name": {
      "default": "output",
      "title": "Output Name",
      "type": "string"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeFlowOutput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• output_name (str)

Validators:

• validate_node_reference → node_reference
• _validate_output_name → output_name

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeFlowOutput(NodeSingleInput):
    """Named passthrough sink marking a subflow output; multiple allowed per flow."""

    output_name: str = "output"

    @field_validator("output_name")
    @classmethod
    def _validate_output_name(cls, v: str) -> str:
        return _validate_port_name(v, "flow output")

    def get_default_description(self) -> str:
        return f"Subflow output '{self.output_name}'"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeFormula pydantic-model

Bases: NodeSingleInput

Settings for a node that applies a formula to create/modify a column.

Show JSON schema:

{
  "$defs": {
    "DataType": {
      "description": "Specific data types for fine-grained control.",
      "enum": [
        "Int8",
        "Int16",
        "Int32",
        "Int64",
        "Int128",
        "UInt8",
        "UInt16",
        "UInt32",
        "UInt64",
        "UInt128",
        "Float16",
        "Float32",
        "Float64",
        "Decimal",
        "String",
        "Categorical",
        "Date",
        "Datetime",
        "Time",
        "Duration",
        "Boolean",
        "Binary",
        "List",
        "Struct",
        "Array"
      ],
      "title": "DataType",
      "type": "string"
    },
    "FieldInput": {
      "description": "Represents a single field with its name and data type, typically for defining an output column.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "anyOf": [
            {
              "$ref": "#/$defs/DataType"
            },
            {
              "const": "Auto",
              "type": "string"
            },
            {
              "enum": [
                "Int8",
                "Int16",
                "Int32",
                "Int64",
                "Int128",
                "UInt8",
                "UInt16",
                "UInt32",
                "UInt64",
                "UInt128",
                "Float16",
                "Float32",
                "Float64",
                "Decimal",
                "String",
                "Date",
                "Datetime",
                "Time",
                "Duration",
                "Boolean",
                "Binary",
                "List",
                "Struct",
                "Array",
                "Integer",
                "Double",
                "Utf8"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "Auto",
          "title": "Data Type"
        }
      },
      "required": [
        "name"
      ],
      "title": "FieldInput",
      "type": "object"
    },
    "FunctionInput": {
      "description": "Defines a formula to be applied, including the output field information.",
      "properties": {
        "field": {
          "$ref": "#/$defs/FieldInput"
        },
        "function": {
          "expression": true,
          "title": "Function",
          "type": "string"
        }
      },
      "required": [
        "field",
        "function"
      ],
      "title": "FunctionInput",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that applies a formula to create/modify a column.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "function": {
      "$ref": "#/$defs/FunctionInput",
      "default": null
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeFormula",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• function (FunctionInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeFormula(NodeSingleInput):
    """Settings for a node that applies a formula to create/modify a column."""

    function: transform_schema.FunctionInput = None

    def get_default_description(self) -> str:
        """Describes the formula being applied."""
        if self.function is None:
            return ""
        name = self.function.field.name if self.function.field else ""
        expr = self.function.function or ""
        if len(expr) > 60:
            expr = expr[:57] + "..."
        return f"{name} = {expr}" if name else expr

get_default_description()

Describes the formula being applied.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the formula being applied."""
    if self.function is None:
        return ""
    name = self.function.field.name if self.function.field else ""
    expr = self.function.function or ""
    if len(expr) > 60:
        expr = expr[:57] + "..."
    return f"{name} = {expr}" if name else expr

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeFuzzyMatch pydantic-model

Bases: NodeJoin

Settings for a node that performs a fuzzy join based on string similarity.

Show JSON schema:

{
  "$defs": {
    "FuzzyMapping": {
      "description": "Represents the configuration for a fuzzy string match between two columns.\n\nThis class defines all the necessary parameters to perform a fuzzy join,\nincluding the columns to match, the specific algorithm to use, and the\nsimilarity threshold required to consider two strings a match.\n\nIt generates a default name for the output score column if one is not\nprovided.\n\nAttributes:\n    left_col (str): The name of the column in the left dataframe to join on.\n    right_col (str): The name of the column in the right dataframe to join on.\n    threshold_score (float): The similarity score threshold required for a\n        match, typically on a scale of 0 to 100. Defaults to 80.0.\n    fuzzy_type (FuzzyTypeLiteral): The string-matching algorithm to use.\n        Defaults to \"levenshtein\".\n    perc_unique (float): A parameter that may be used to assess column\n        uniqueness before performing a costly fuzzy match. Defaults to 0.0.\n    output_column_name (str | None): The name for the new column that will\n        contain the calculated fuzzy match score. If None, a name is\n        generated automatically in the format 'fuzzy_score_{left_col}_{right_col}'.\n    valid (bool): A flag to indicate whether this mapping is active and should\n        be used in a join operation. Defaults to True.\n    reversed_threshold_score (float): A property that converts the 0-100\n        threshold score into a 0.0-1.0 distance score, where 0.0 is a\n        perfect match.",
      "properties": {
        "left_col": {
          "title": "Left Col",
          "type": "string"
        },
        "right_col": {
          "title": "Right Col",
          "type": "string"
        },
        "threshold_score": {
          "default": 80.0,
          "title": "Threshold Score",
          "type": "number"
        },
        "fuzzy_type": {
          "default": "levenshtein",
          "enum": [
            "levenshtein",
            "jaro",
            "jaro_winkler",
            "hamming",
            "damerau_levenshtein",
            "indel"
          ],
          "title": "Fuzzy Type",
          "type": "string"
        },
        "perc_unique": {
          "default": 0.0,
          "title": "Perc Unique",
          "type": "number"
        },
        "output_column_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Column Name"
        },
        "valid": {
          "default": true,
          "title": "Valid",
          "type": "boolean"
        }
      },
      "required": [
        "left_col",
        "right_col"
      ],
      "title": "FuzzyMapping",
      "type": "object"
    },
    "FuzzyMatchInput": {
      "description": "Data model for fuzzy matching join operations.",
      "properties": {
        "join_mapping": {
          "items": {
            "$ref": "#/$defs/FuzzyMapping"
          },
          "title": "Join Mapping",
          "type": "array"
        },
        "left_select": {
          "$ref": "#/$defs/JoinInputs"
        },
        "right_select": {
          "$ref": "#/$defs/JoinInputs"
        },
        "how": {
          "default": "inner",
          "enum": [
            "inner",
            "left",
            "right",
            "full",
            "semi",
            "anti",
            "cross",
            "outer"
          ],
          "title": "How",
          "type": "string"
        },
        "aggregate_output": {
          "default": false,
          "title": "Aggregate Output",
          "type": "boolean"
        }
      },
      "required": [
        "join_mapping",
        "left_select",
        "right_select"
      ],
      "title": "FuzzyMatchInput",
      "type": "object"
    },
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that performs a fuzzy join based on string similarity.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "auto_generate_selection": {
      "default": true,
      "title": "Auto Generate Selection",
      "type": "boolean"
    },
    "verify_integrity": {
      "default": true,
      "title": "Verify Integrity",
      "type": "boolean"
    },
    "join_input": {
      "$ref": "#/$defs/FuzzyMatchInput"
    },
    "auto_keep_all": {
      "default": true,
      "title": "Auto Keep All",
      "type": "boolean"
    },
    "auto_keep_right": {
      "default": true,
      "title": "Auto Keep Right",
      "type": "boolean"
    },
    "auto_keep_left": {
      "default": true,
      "title": "Auto Keep Left",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "join_input"
  ],
  "title": "NodeFuzzyMatch",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• auto_generate_selection (bool)
• verify_integrity (bool)
• auto_keep_all (bool)
• auto_keep_right (bool)
• auto_keep_left (bool)
• join_input (FuzzyMatchInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeFuzzyMatch(NodeJoin):
    """Settings for a node that performs a fuzzy join based on string similarity."""

    join_input: transform_schema.FuzzyMatchInput

    def get_default_description(self) -> str:
        """Describes the fuzzy match join."""
        ji = self.join_input
        how = ji.how
        if ji.join_mapping:
            keys = [
                f"{fm.left_col} ~ {fm.right_col}" if fm.left_col != fm.right_col else fm.left_col
                for fm in ji.join_mapping[:3]
            ]
            key_str = ", ".join(keys)
            if len(ji.join_mapping) > 3:
                key_str += f" (+{len(ji.join_mapping) - 3} more)"
            return f"Fuzzy {how} join on {key_str}"
        return f"Fuzzy {how} join"

    def to_yaml_dict(self) -> NodeFuzzyMatchYaml:
        """Converts the fuzzy match node settings to a dictionary for YAML serialization."""
        result: NodeFuzzyMatchYaml = {
            "cache_results": self.cache_results,
            "auto_generate_selection": self.auto_generate_selection,
            "verify_integrity": self.verify_integrity,
            "join_input": self.join_input.to_yaml_dict(),
            "auto_keep_all": self.auto_keep_all,
            "auto_keep_right": self.auto_keep_right,
            "auto_keep_left": self.auto_keep_left,
        }
        if self.output_field_config:
            result["output_field_config"] = {
                "enabled": self.output_field_config.enabled,
                "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
                "validate_data_types": self.output_field_config.validate_data_types,
                "fields": [
                    {
                        "name": f.name,
                        "data_type": f.data_type,
                        "default_value": f.default_value,
                    }
                    for f in self.output_field_config.fields
                ],
            }
        return result

get_default_description()

Describes the fuzzy match join.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the fuzzy match join."""
    ji = self.join_input
    how = ji.how
    if ji.join_mapping:
        keys = [
            f"{fm.left_col} ~ {fm.right_col}" if fm.left_col != fm.right_col else fm.left_col
            for fm in ji.join_mapping[:3]
        ]
        key_str = ", ".join(keys)
        if len(ji.join_mapping) > 3:
            key_str += f" (+{len(ji.join_mapping) - 3} more)"
        return f"Fuzzy {how} join on {key_str}"
    return f"Fuzzy {how} join"

to_yaml_dict()

Converts the fuzzy match node settings to a dictionary for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> NodeFuzzyMatchYaml:
    """Converts the fuzzy match node settings to a dictionary for YAML serialization."""
    result: NodeFuzzyMatchYaml = {
        "cache_results": self.cache_results,
        "auto_generate_selection": self.auto_generate_selection,
        "verify_integrity": self.verify_integrity,
        "join_input": self.join_input.to_yaml_dict(),
        "auto_keep_all": self.auto_keep_all,
        "auto_keep_right": self.auto_keep_right,
        "auto_keep_left": self.auto_keep_left,
    }
    if self.output_field_config:
        result["output_field_config"] = {
            "enabled": self.output_field_config.enabled,
            "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
            "validate_data_types": self.output_field_config.validate_data_types,
            "fields": [
                {
                    "name": f.name,
                    "data_type": f.data_type,
                    "default_value": f.default_value,
                }
                for f in self.output_field_config.fields
            ],
        }
    return result

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeGoogleAnalyticsReader pydantic-model

Bases: NodeBase

Settings for a node that reads from a Google Analytics 4 property.

Show JSON schema:

{
  "$defs": {
    "GoogleAnalyticsFilter": {
      "description": "A single filter applied to a GA4 dimension or metric.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nauto-routes the filter into either the request's ``dimension_filter`` (for\nstring-typed dimensions) or ``metric_filter`` (for numeric-typed metrics).\n\nSupported operators \u2014 strings (dimensions):\n  - ``equals``, ``not_equals``\n  - ``contains``, ``begins_with``, ``ends_with``\n  - ``regex``  (full regex match)\n  - ``in_list``, ``not_in_list``  (comma-separated ``value``)\n\nSupported operators \u2014 numeric (metrics):\n  - ``equals``, ``not_equals``\n  - ``less_than``, ``less_equal``, ``greater_than``, ``greater_equal``\n  - ``between``  (comma-separated ``\"low,high\"``)\n\nMultiple filters on the same kind are AND-combined. String matching is\ncase-insensitive by default (``case_sensitive=False`` below).",
      "properties": {
        "field": {
          "title": "Field",
          "type": "string"
        },
        "operator": {
          "title": "Operator",
          "type": "string"
        },
        "value": {
          "default": "",
          "title": "Value",
          "type": "string"
        },
        "case_sensitive": {
          "default": false,
          "title": "Case Sensitive",
          "type": "boolean"
        }
      },
      "required": [
        "field",
        "operator"
      ],
      "title": "GoogleAnalyticsFilter",
      "type": "object"
    },
    "GoogleAnalyticsOrderBy": {
      "description": "A single sort entry applied to the GA4 report.\n\n``field`` must match one of the selected dimensions or metrics; the worker\nroutes it into a ``DimensionOrderBy`` or ``MetricOrderBy`` accordingly.\n``descending=True`` produces a descending sort. Sort entries are applied in\nlist order.",
      "properties": {
        "field": {
          "title": "Field",
          "type": "string"
        },
        "descending": {
          "default": false,
          "title": "Descending",
          "type": "boolean"
        }
      },
      "required": [
        "field"
      ],
      "title": "GoogleAnalyticsOrderBy",
      "type": "object"
    },
    "GoogleAnalyticsSettings": {
      "description": "UI settings for a Google Analytics 4 reader node.\n\nCredentials are NOT stored inline: ``ga_connection_name`` is a reference to\na Google Analytics connection managed under ``/ga_connections`` (whose\nservice-account JSON is encrypted at rest).",
      "properties": {
        "ga_connection_name": {
          "title": "Ga Connection Name",
          "type": "string"
        },
        "property_id": {
          "title": "Property Id",
          "type": "string"
        },
        "start_date": {
          "default": "7daysAgo",
          "title": "Start Date",
          "type": "string"
        },
        "end_date": {
          "default": "yesterday",
          "title": "End Date",
          "type": "string"
        },
        "metrics": {
          "items": {
            "type": "string"
          },
          "title": "Metrics",
          "type": "array"
        },
        "dimensions": {
          "items": {
            "type": "string"
          },
          "title": "Dimensions",
          "type": "array"
        },
        "limit": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Limit"
        },
        "filters": {
          "items": {
            "$ref": "#/$defs/GoogleAnalyticsFilter"
          },
          "title": "Filters",
          "type": "array"
        },
        "order_bys": {
          "items": {
            "$ref": "#/$defs/GoogleAnalyticsOrderBy"
          },
          "title": "Order Bys",
          "type": "array"
        }
      },
      "required": [
        "ga_connection_name",
        "property_id"
      ],
      "title": "GoogleAnalyticsSettings",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads from a Google Analytics 4 property.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "google_analytics_settings": {
      "$ref": "#/$defs/GoogleAnalyticsSettings"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "google_analytics_settings"
  ],
  "title": "NodeGoogleAnalyticsReader",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• google_analytics_settings (GoogleAnalyticsSettings)
• fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeGoogleAnalyticsReader(NodeBase):
    """Settings for a node that reads from a Google Analytics 4 property."""

    google_analytics_settings: GoogleAnalyticsSettings
    fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        """Describes the GA4 query."""
        s = self.google_analytics_settings
        pieces = []
        if s.property_id:
            pieces.append(f"property {s.property_id}")
        if s.metrics:
            metrics_preview = ", ".join(s.metrics[:3])
            if len(s.metrics) > 3:
                metrics_preview += f" (+{len(s.metrics) - 3} more)"
            pieces.append(f"metrics: {metrics_preview}")
        if s.start_date and s.end_date:
            pieces.append(f"{s.start_date} .. {s.end_date}")
        return " | ".join(pieces)

get_default_description()

Describes the GA4 query.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the GA4 query."""
    s = self.google_analytics_settings
    pieces = []
    if s.property_id:
        pieces.append(f"property {s.property_id}")
    if s.metrics:
        metrics_preview = ", ".join(s.metrics[:3])
        if len(s.metrics) > 3:
            metrics_preview += f" (+{len(s.metrics) - 3} more)"
        pieces.append(f"metrics: {metrics_preview}")
    if s.start_date and s.end_date:
        pieces.append(f"{s.start_date} .. {s.end_date}")
    return " | ".join(pieces)

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeGraphSolver pydantic-model

Bases: NodeSingleInput

Settings for a node that solves graph-based problems (e.g., connected components).

Show JSON schema:

{
  "$defs": {
    "GraphSolverInput": {
      "description": "Defines settings for a graph-solving operation (e.g., finding connected components).",
      "properties": {
        "col_from": {
          "title": "Col From",
          "type": "string"
        },
        "col_to": {
          "title": "Col To",
          "type": "string"
        },
        "output_column_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "graph_group",
          "title": "Output Column Name"
        }
      },
      "required": [
        "col_from",
        "col_to"
      ],
      "title": "GraphSolverInput",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that solves graph-based problems (e.g., connected components).",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "graph_solver_input": {
      "$ref": "#/$defs/GraphSolverInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "graph_solver_input"
  ],
  "title": "NodeGraphSolver",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• graph_solver_input (GraphSolverInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeGraphSolver(NodeSingleInput):
    """Settings for a node that solves graph-based problems (e.g., connected components)."""

    graph_solver_input: transform_schema.GraphSolverInput

    def get_default_description(self) -> str:
        """Describes the graph solver operation."""
        g = self.graph_solver_input
        return f"{g.col_from} -> {g.col_to} as '{g.output_column_name}'"

get_default_description()

Describes the graph solver operation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the graph solver operation."""
    g = self.graph_solver_input
    return f"{g.col_from} -> {g.col_to} as '{g.output_column_name}'"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeGroupBy pydantic-model

Bases: NodeSingleInput

Settings for a node that performs a group-by and aggregation operation.

Show JSON schema:

{
  "$defs": {
    "AggColl": {
      "description": "A data class that represents a single aggregation operation for a group by operation.\n\nAttributes\n----------\nold_name : str\n    The name of the column in the original DataFrame to be aggregated.\n\nagg : str\n    The aggregation function to use. This can be a string representing a built-in function or a custom function.\n\nnew_name : Optional[str]\n    The name of the resulting aggregated column in the output DataFrame. If not provided, it will default to the\n    old_name appended with the aggregation function.\n\noutput_type : Optional[str]\n    The type of the output values of the aggregation. If not provided, it is inferred from the aggregation function\n    using the `get_func_type_mapping` function.\n\nExample\n--------\nagg_col = AggColl(\n    old_name='col1',\n    agg='sum',\n    new_name='sum_col1',\n    output_type='float'\n)",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "agg": {
          "title": "Agg",
          "type": "string"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "output_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Type"
        }
      },
      "required": [
        "old_name",
        "agg"
      ],
      "title": "AggColl",
      "type": "object"
    },
    "GroupByInput": {
      "description": "A data class that represents the input for a group by operation.\n\nAttributes\n----------\nagg_cols : List[AggColl]\n    A list of `AggColl` objects that specify the aggregation operations to perform on the DataFrame columns\n    after grouping. Each `AggColl` object should specify the column to be aggregated and the aggregation\n    function to use.\n\nExample\n--------\ngroup_by_input = GroupByInput(\n    agg_cols=[AggColl(old_name='ix', agg='groupby'), AggColl(old_name='groups', agg='groupby'),\n              AggColl(old_name='col1', agg='sum'), AggColl(old_name='col2', agg='mean')]\n)",
      "properties": {
        "agg_cols": {
          "items": {
            "$ref": "#/$defs/AggColl"
          },
          "title": "Agg Cols",
          "type": "array"
        }
      },
      "required": [
        "agg_cols"
      ],
      "title": "GroupByInput",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that performs a group-by and aggregation operation.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "groupby_input": {
      "$ref": "#/$defs/GroupByInput",
      "default": null
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeGroupBy",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• groupby_input (GroupByInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeGroupBy(NodeSingleInput):
    """Settings for a node that performs a group-by and aggregation operation."""

    groupby_input: transform_schema.GroupByInput = None

    def get_default_description(self) -> str:
        """Describes the group-by columns and aggregations."""
        if self.groupby_input is None or not self.groupby_input.agg_cols:
            return ""
        group_cols = [a.old_name for a in self.groupby_input.agg_cols if a.agg == "groupby"]
        agg_cols = [a for a in self.groupby_input.agg_cols if a.agg != "groupby"]
        parts = []
        if group_cols:
            cols_str = ", ".join(group_cols[:3])
            if len(group_cols) > 3:
                cols_str += f" (+{len(group_cols) - 3} more)"
            parts.append(f"By {cols_str}")
        if agg_cols:
            agg_strs = [f"{a.agg}({a.old_name})" for a in agg_cols[:3]]
            if len(agg_cols) > 3:
                agg_strs.append(f"+{len(agg_cols) - 3} more")
            parts.append(", ".join(agg_strs))
        return ": ".join(parts) if parts else ""

get_default_description()

Describes the group-by columns and aggregations.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the group-by columns and aggregations."""
    if self.groupby_input is None or not self.groupby_input.agg_cols:
        return ""
    group_cols = [a.old_name for a in self.groupby_input.agg_cols if a.agg == "groupby"]
    agg_cols = [a for a in self.groupby_input.agg_cols if a.agg != "groupby"]
    parts = []
    if group_cols:
        cols_str = ", ".join(group_cols[:3])
        if len(group_cols) > 3:
            cols_str += f" (+{len(group_cols) - 3} more)"
        parts.append(f"By {cols_str}")
    if agg_cols:
        agg_strs = [f"{a.agg}({a.old_name})" for a in agg_cols[:3]]
        if len(agg_cols) > 3:
            agg_strs.append(f"+{len(agg_cols) - 3} more")
        parts.append(", ".join(agg_strs))
    return ": ".join(parts) if parts else ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeInputConnection pydantic-model

Bases: BaseModel

Represents the input side of a connection between two nodes.

Show JSON schema:

{
  "description": "Represents the input side of a connection between two nodes.",
  "properties": {
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "connection_class": {
      "enum": [
        "input-0",
        "input-1",
        "input-2",
        "input-3",
        "input-4",
        "input-5",
        "input-6",
        "input-7",
        "input-8",
        "input-9"
      ],
      "title": "Connection Class",
      "type": "string"
    }
  },
  "required": [
    "node_id",
    "connection_class"
  ],
  "title": "NodeInputConnection",
  "type": "object"
}

Fields:

• node_id (int)
• connection_class (InputConnectionClass)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeInputConnection(BaseModel):
    """Represents the input side of a connection between two nodes."""

    node_id: int
    connection_class: InputConnectionClass

    def get_node_input_connection_type(self) -> Literal["main", "right", "left"]:
        """Determines the semantic type of the input (e.g., for a join)."""
        match self.connection_class:
            case "input-0":
                return "main"
            case "input-1":
                return "right"
            case "input-2":
                return "left"
            case _:
                raise ValueError(f"Unexpected connection_class: {self.connection_class}")

get_node_input_connection_type()

Determines the semantic type of the input (e.g., for a join).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_node_input_connection_type(self) -> Literal["main", "right", "left"]:
    """Determines the semantic type of the input (e.g., for a join)."""
    match self.connection_class:
        case "input-0":
            return "main"
        case "input-1":
            return "right"
        case "input-2":
            return "left"
        case _:
            raise ValueError(f"Unexpected connection_class: {self.connection_class}")

NodeJoin pydantic-model

Bases: NodeMultiInput

Settings for a node that performs a standard SQL-style join.

Show JSON schema:

{
  "$defs": {
    "JoinInput": {
      "description": "Data model for standard SQL-style join operations.",
      "properties": {
        "join_mapping": {
          "items": {
            "$ref": "#/$defs/JoinMap"
          },
          "title": "Join Mapping",
          "type": "array"
        },
        "left_select": {
          "$ref": "#/$defs/JoinInputs"
        },
        "right_select": {
          "$ref": "#/$defs/JoinInputs"
        },
        "how": {
          "default": "inner",
          "enum": [
            "inner",
            "left",
            "right",
            "full",
            "semi",
            "anti",
            "outer"
          ],
          "title": "How",
          "type": "string"
        }
      },
      "required": [
        "join_mapping",
        "left_select",
        "right_select"
      ],
      "title": "JoinInput",
      "type": "object"
    },
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "JoinMap": {
      "description": "Defines a single mapping between a left and right column for a join key.",
      "properties": {
        "left_col": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Left Col"
        },
        "right_col": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Right Col"
        }
      },
      "title": "JoinMap",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that performs a standard SQL-style join.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "auto_generate_selection": {
      "default": true,
      "title": "Auto Generate Selection",
      "type": "boolean"
    },
    "verify_integrity": {
      "default": true,
      "title": "Verify Integrity",
      "type": "boolean"
    },
    "join_input": {
      "$ref": "#/$defs/JoinInput"
    },
    "auto_keep_all": {
      "default": true,
      "title": "Auto Keep All",
      "type": "boolean"
    },
    "auto_keep_right": {
      "default": true,
      "title": "Auto Keep Right",
      "type": "boolean"
    },
    "auto_keep_left": {
      "default": true,
      "title": "Auto Keep Left",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "join_input"
  ],
  "title": "NodeJoin",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• auto_generate_selection (bool)
• verify_integrity (bool)
• join_input (JoinInput)
• auto_keep_all (bool)
• auto_keep_right (bool)
• auto_keep_left (bool)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeJoin(NodeMultiInput):
    """Settings for a node that performs a standard SQL-style join."""

    auto_generate_selection: bool = True
    verify_integrity: bool = True
    join_input: transform_schema.JoinInput
    auto_keep_all: bool = True
    auto_keep_right: bool = True
    auto_keep_left: bool = True

    def get_default_description(self) -> str:
        """Describes the join type and key columns."""
        ji = self.join_input
        how = ji.how
        if ji.join_mapping:
            keys = [
                f"{jm.left_col} = {jm.right_col}" if jm.left_col != jm.right_col else jm.left_col
                for jm in ji.join_mapping[:3]
            ]
            key_str = ", ".join(keys)
            if len(ji.join_mapping) > 3:
                key_str += f" (+{len(ji.join_mapping) - 3} more)"
            return f"{how} join on {key_str}"
        return f"{how} join"

    def to_yaml_dict(self) -> NodeJoinYaml:
        """Converts the join node settings to a dictionary for YAML serialization."""
        result: NodeJoinYaml = {
            "cache_results": self.cache_results,
            "auto_generate_selection": self.auto_generate_selection,
            "verify_integrity": self.verify_integrity,
            "join_input": self.join_input.to_yaml_dict(),
            "auto_keep_all": self.auto_keep_all,
            "auto_keep_right": self.auto_keep_right,
            "auto_keep_left": self.auto_keep_left,
        }
        if self.output_field_config:
            result["output_field_config"] = {
                "enabled": self.output_field_config.enabled,
                "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
                "validate_data_types": self.output_field_config.validate_data_types,
                "fields": [
                    {
                        "name": f.name,
                        "data_type": f.data_type,
                        "default_value": f.default_value,
                    }
                    for f in self.output_field_config.fields
                ],
            }
        return result

get_default_description()

Describes the join type and key columns.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the join type and key columns."""
    ji = self.join_input
    how = ji.how
    if ji.join_mapping:
        keys = [
            f"{jm.left_col} = {jm.right_col}" if jm.left_col != jm.right_col else jm.left_col
            for jm in ji.join_mapping[:3]
        ]
        key_str = ", ".join(keys)
        if len(ji.join_mapping) > 3:
            key_str += f" (+{len(ji.join_mapping) - 3} more)"
        return f"{how} join on {key_str}"
    return f"{how} join"

to_yaml_dict()

Converts the join node settings to a dictionary for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> NodeJoinYaml:
    """Converts the join node settings to a dictionary for YAML serialization."""
    result: NodeJoinYaml = {
        "cache_results": self.cache_results,
        "auto_generate_selection": self.auto_generate_selection,
        "verify_integrity": self.verify_integrity,
        "join_input": self.join_input.to_yaml_dict(),
        "auto_keep_all": self.auto_keep_all,
        "auto_keep_right": self.auto_keep_right,
        "auto_keep_left": self.auto_keep_left,
    }
    if self.output_field_config:
        result["output_field_config"] = {
            "enabled": self.output_field_config.enabled,
            "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
            "validate_data_types": self.output_field_config.validate_data_types,
            "fields": [
                {
                    "name": f.name,
                    "data_type": f.data_type,
                    "default_value": f.default_value,
                }
                for f in self.output_field_config.fields
            ],
        }
    return result

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeKafkaSource pydantic-model

Bases: NodeBase

Settings for a node that reads from a Kafka or Redpanda topic.

Show JSON schema:

{
  "$defs": {
    "KafkaSourceSettings": {
      "description": "Configuration for reading from a Kafka/Redpanda topic.",
      "properties": {
        "kafka_connection_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Kafka Connection Id"
        },
        "kafka_connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Kafka Connection Name"
        },
        "topic_name": {
          "default": "",
          "title": "Topic Name",
          "type": "string"
        },
        "value_format": {
          "const": "json",
          "default": "json",
          "title": "Value Format",
          "type": "string"
        },
        "sync_name": {
          "default": "",
          "title": "Sync Name",
          "type": "string"
        },
        "start_offset": {
          "default": "latest",
          "enum": [
            "earliest",
            "latest"
          ],
          "title": "Start Offset",
          "type": "string"
        },
        "max_messages": {
          "default": 100000,
          "title": "Max Messages",
          "type": "integer"
        },
        "poll_timeout_seconds": {
          "default": 30.0,
          "title": "Poll Timeout Seconds",
          "type": "number"
        }
      },
      "title": "KafkaSourceSettings",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads from a Kafka or Redpanda topic.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "kafka_settings": {
      "$ref": "#/$defs/KafkaSourceSettings",
      "default": {
        "kafka_connection_id": null,
        "kafka_connection_name": null,
        "topic_name": "",
        "value_format": "json",
        "sync_name": "",
        "start_offset": "latest",
        "max_messages": 100000,
        "poll_timeout_seconds": 30.0
      }
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeKafkaSource",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• kafka_settings (KafkaSourceSettings)
• fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeKafkaSource(NodeBase):
    """Settings for a node that reads from a Kafka or Redpanda topic."""

    kafka_settings: KafkaSourceSettings = KafkaSourceSettings()
    fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        ks = self.kafka_settings
        if ks.topic_name:
            return f"Kafka: {ks.topic_name}"
        return "Kafka Source"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeManualInput pydantic-model

Bases: NodeBase

Settings for a node that allows direct data entry in the UI.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RawData": {
      "description": "Represents data in a raw, columnar format for manual input.",
      "properties": {
        "columns": {
          "description": "Schema in column order. The i-th MinimalFieldInfo describes the values in data[i].",
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "title": "Columns",
          "type": "array"
        },
        "data": {
          "description": "Columnar layout: data[i] is the list of values for columns[i], in column order. len(data) must equal len(columns); each inner list has the same length (one entry per row). For two rows of {name, age}, emit [[\"Alice\", \"Bob\"], [30, 25]] \u2014 NOT [[\"Alice\", 30], [\"Bob\", 25]]. Reading rows back is `data[col_idx][row_idx]`.",
          "items": {
            "items": {},
            "type": "array"
          },
          "title": "Data",
          "type": "array"
        }
      },
      "required": [
        "columns",
        "data"
      ],
      "title": "RawData",
      "type": "object"
    }
  },
  "description": "Settings for a node that allows direct data entry in the UI.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "raw_data_format": {
      "$ref": "#/$defs/RawData"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "raw_data_format"
  ],
  "title": "NodeManualInput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• raw_data_format (RawData)

Validators:

• validate_node_reference → node_reference
• _coerce_none_raw_data_format

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeManualInput(NodeBase):
    """Settings for a node that allows direct data entry in the UI."""

    raw_data_format: RawData

    @model_validator(mode="before")
    @classmethod
    def _coerce_none_raw_data_format(cls, values):
        if isinstance(values, dict) and values.get("raw_data_format") is None:
            return {**values, "raw_data_format": {"columns": [], "data": []}}
        return values

    def get_default_description(self) -> str:
        """Describes the manual input columns."""
        if self.raw_data_format and self.raw_data_format.columns:
            cols = [c.name for c in self.raw_data_format.columns[:5]]
            desc = ", ".join(cols)
            if len(self.raw_data_format.columns) > 5:
                desc += f" (+{len(self.raw_data_format.columns) - 5} more)"
            num_rows = (
                len(self.raw_data_format.data[0]) if self.raw_data_format.data and self.raw_data_format.data[0] else 0
            )
            return f"{len(self.raw_data_format.columns)} cols, {num_rows} rows: {desc}"
        return ""

get_default_description()

Describes the manual input columns.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the manual input columns."""
    if self.raw_data_format and self.raw_data_format.columns:
        cols = [c.name for c in self.raw_data_format.columns[:5]]
        desc = ", ".join(cols)
        if len(self.raw_data_format.columns) > 5:
            desc += f" (+{len(self.raw_data_format.columns) - 5} more)"
        num_rows = (
            len(self.raw_data_format.data[0]) if self.raw_data_format.data and self.raw_data_format.data[0] else 0
        )
        return f"{len(self.raw_data_format.columns)} cols, {num_rows} rows: {desc}"
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeMultiInput pydantic-model

Bases: NodeBase

A base model for any node that takes multiple data inputs.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "A base model for any node that takes multiple data inputs.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeMultiInput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeMultiInput(NodeBase):
    """A base model for any node that takes multiple data inputs."""

    depending_on_ids: list[int] | None = Field(default_factory=list)

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeOutput pydantic-model

Bases: NodeSingleInput

Settings for a node that writes its input to a file.

Show JSON schema:

{
  "$defs": {
    "OutputAvroTable": {
      "description": "Defines settings for writing an Avro file.",
      "properties": {
        "file_type": {
          "const": "avro",
          "default": "avro",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "snappy",
            "deflate"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputAvroTable",
      "type": "object"
    },
    "OutputCsvTable": {
      "description": "Defines settings for writing a CSV file.",
      "properties": {
        "file_type": {
          "const": "csv",
          "default": "csv",
          "title": "File Type",
          "type": "string"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        }
      },
      "title": "OutputCsvTable",
      "type": "object"
    },
    "OutputExcelTable": {
      "description": "Defines settings for writing an Excel file.",
      "properties": {
        "file_type": {
          "const": "excel",
          "default": "excel",
          "title": "File Type",
          "type": "string"
        },
        "sheet_name": {
          "default": "Sheet1",
          "title": "Sheet Name",
          "type": "string"
        }
      },
      "title": "OutputExcelTable",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "OutputIpcTable": {
      "description": "Defines settings for writing an Arrow IPC/Feather file.",
      "properties": {
        "file_type": {
          "const": "ipc",
          "default": "ipc",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "lz4",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputIpcTable",
      "type": "object"
    },
    "OutputNdjsonTable": {
      "description": "Defines settings for writing a newline-delimited JSON file.",
      "properties": {
        "file_type": {
          "const": "ndjson",
          "default": "ndjson",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "gzip",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputNdjsonTable",
      "type": "object"
    },
    "OutputParquetTable": {
      "description": "Defines settings for writing a Parquet file.",
      "properties": {
        "file_type": {
          "const": "parquet",
          "default": "parquet",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "zstd",
          "enum": [
            "lz4",
            "uncompressed",
            "snappy",
            "gzip",
            "brotli",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputParquetTable",
      "type": "object"
    },
    "OutputSettings": {
      "description": "Defines the complete settings for an output node.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "directory": {
          "title": "Directory",
          "type": "string"
        },
        "file_type": {
          "title": "File Type",
          "type": "string"
        },
        "fields": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Fields"
        },
        "write_mode": {
          "default": "overwrite",
          "title": "Write Mode",
          "type": "string"
        },
        "table_settings": {
          "discriminator": {
            "mapping": {
              "avro": "#/$defs/OutputAvroTable",
              "csv": "#/$defs/OutputCsvTable",
              "excel": "#/$defs/OutputExcelTable",
              "ipc": "#/$defs/OutputIpcTable",
              "ndjson": "#/$defs/OutputNdjsonTable",
              "parquet": "#/$defs/OutputParquetTable"
            },
            "propertyName": "file_type"
          },
          "oneOf": [
            {
              "$ref": "#/$defs/OutputCsvTable"
            },
            {
              "$ref": "#/$defs/OutputParquetTable"
            },
            {
              "$ref": "#/$defs/OutputExcelTable"
            },
            {
              "$ref": "#/$defs/OutputIpcTable"
            },
            {
              "$ref": "#/$defs/OutputNdjsonTable"
            },
            {
              "$ref": "#/$defs/OutputAvroTable"
            }
          ],
          "title": "Table Settings"
        },
        "abs_file_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Abs File Path"
        }
      },
      "required": [
        "name",
        "directory",
        "file_type",
        "table_settings"
      ],
      "title": "OutputSettings",
      "type": "object"
    }
  },
  "description": "Settings for a node that writes its input to a file.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "output_settings": {
      "$ref": "#/$defs/OutputSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "output_settings"
  ],
  "title": "NodeOutput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• output_settings (OutputSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeOutput(NodeSingleInput):
    """Settings for a node that writes its input to a file."""

    output_settings: OutputSettings

    def get_default_description(self) -> str:
        """Describes the output file target."""
        o = self.output_settings
        return f"{o.name} ({o.file_type})"

    def to_yaml_dict(self) -> NodeOutputYaml:
        """Converts the output node settings to a dictionary for YAML serialization."""
        result: NodeOutputYaml = {
            "cache_results": self.cache_results,
            "output_settings": self.output_settings.to_yaml_dict(),
        }
        if self.output_field_config:
            result["output_field_config"] = {
                "enabled": self.output_field_config.enabled,
                "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
                "validate_data_types": self.output_field_config.validate_data_types,
                "fields": [
                    {
                        "name": f.name,
                        "data_type": f.data_type,
                        "default_value": f.default_value,
                    }
                    for f in self.output_field_config.fields
                ],
            }
        return result

get_default_description()

Describes the output file target.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the output file target."""
    o = self.output_settings
    return f"{o.name} ({o.file_type})"

to_yaml_dict()

Converts the output node settings to a dictionary for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> NodeOutputYaml:
    """Converts the output node settings to a dictionary for YAML serialization."""
    result: NodeOutputYaml = {
        "cache_results": self.cache_results,
        "output_settings": self.output_settings.to_yaml_dict(),
    }
    if self.output_field_config:
        result["output_field_config"] = {
            "enabled": self.output_field_config.enabled,
            "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
            "validate_data_types": self.output_field_config.validate_data_types,
            "fields": [
                {
                    "name": f.name,
                    "data_type": f.data_type,
                    "default_value": f.default_value,
                }
                for f in self.output_field_config.fields
            ],
        }
    return result

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeOutputConnection pydantic-model

Bases: BaseModel

Represents the output side of a connection between two nodes.

Show JSON schema:

{
  "description": "Represents the output side of a connection between two nodes.",
  "properties": {
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "connection_class": {
      "enum": [
        "output-0",
        "output-1",
        "output-2",
        "output-3",
        "output-4",
        "output-5",
        "output-6",
        "output-7",
        "output-8",
        "output-9"
      ],
      "title": "Connection Class",
      "type": "string"
    }
  },
  "required": [
    "node_id",
    "connection_class"
  ],
  "title": "NodeOutputConnection",
  "type": "object"
}

Fields:

• node_id (int)
• connection_class (OutputConnectionClass)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeOutputConnection(BaseModel):
    """Represents the output side of a connection between two nodes."""

    node_id: int
    connection_class: OutputConnectionClass

NodePivot pydantic-model

Bases: NodeSingleInput

Settings for a node that pivots data from a long to a wide format.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "PivotInput": {
      "description": "Defines the settings for a pivot (long-to-wide) operation.",
      "properties": {
        "index_columns": {
          "items": {
            "type": "string"
          },
          "title": "Index Columns",
          "type": "array"
        },
        "pivot_column": {
          "title": "Pivot Column",
          "type": "string"
        },
        "value_col": {
          "title": "Value Col",
          "type": "string"
        },
        "aggregations": {
          "items": {
            "type": "string"
          },
          "title": "Aggregations",
          "type": "array"
        }
      },
      "required": [
        "index_columns",
        "pivot_column",
        "value_col",
        "aggregations"
      ],
      "title": "PivotInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that pivots data from a long to a wide format.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "pivot_input": {
      "$ref": "#/$defs/PivotInput",
      "default": null
    },
    "output_fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodePivot",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• pivot_input (PivotInput)
• output_fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodePivot(NodeSingleInput):
    """Settings for a node that pivots data from a long to a wide format."""

    pivot_input: transform_schema.PivotInput = None
    output_fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        """Describes the pivot operation."""
        if self.pivot_input is None:
            return ""
        p = self.pivot_input
        aggs = ", ".join(p.aggregations[:2]) if p.aggregations else ""
        if len(p.aggregations) > 2:
            aggs += f" (+{len(p.aggregations) - 2} more)"
        return f"Pivot {p.value_col} by {p.pivot_column} ({aggs})"

get_default_description()

Describes the pivot operation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the pivot operation."""
    if self.pivot_input is None:
        return ""
    p = self.pivot_input
    aggs = ", ".join(p.aggregations[:2]) if p.aggregations else ""
    if len(p.aggregations) > 2:
        aggs += f" (+{len(p.aggregations) - 2} more)"
    return f"Pivot {p.value_col} by {p.pivot_column} ({aggs})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodePolarsCode pydantic-model

Bases: NodeMultiInput

Settings for a node that executes arbitrary user-provided Polars code.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "PolarsCodeInput": {
      "description": "A simple container for a string of user-provided Polars code to be executed.",
      "properties": {
        "polars_code": {
          "title": "Polars Code",
          "type": "string"
        }
      },
      "required": [
        "polars_code"
      ],
      "title": "PolarsCodeInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that executes arbitrary user-provided Polars code.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "polars_code_input": {
      "$ref": "#/$defs/PolarsCodeInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "polars_code_input"
  ],
  "title": "NodePolarsCode",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• polars_code_input (PolarsCodeInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodePolarsCode(NodeMultiInput):
    """Settings for a node that executes arbitrary user-provided Polars code."""

    polars_code_input: transform_schema.PolarsCodeInput

    def get_default_description(self) -> str:
        """Describes the Polars code snippet."""
        code = self.polars_code_input.polars_code
        first_line = code.strip().split("\n")[0] if code else ""
        if len(first_line) > 80:
            first_line = first_line[:77] + "..."
        return first_line

get_default_description()

Describes the Polars code snippet.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the Polars code snippet."""
    code = self.polars_code_input.polars_code
    first_line = code.strip().split("\n")[0] if code else ""
    if len(first_line) > 80:
        first_line = first_line[:77] + "..."
    return first_line

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodePromise pydantic-model

Bases: NodeBase

A placeholder node for an operation that has not yet been configured.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "A placeholder node for an operation that has not yet been configured.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "default": false,
      "title": "Is Setup",
      "type": "boolean"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "node_type": {
      "title": "Node Type",
      "type": "string"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "node_type"
  ],
  "title": "NodePromise",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• is_setup (bool)
• node_type (str)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodePromise(NodeBase):
    """A placeholder node for an operation that has not yet been configured."""

    is_setup: bool = False
    node_type: str

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodePythonScript pydantic-model

Bases: NodeMultiInput

Node that executes Python code on a kernel container.

Show JSON schema:

{
  "$defs": {
    "NotebookCell": {
      "description": "A single cell in the notebook editor.\n\nNote: Cell output (stdout, display_outputs, errors) is handled entirely\non the frontend and is not persisted. Only id and code are stored.",
      "properties": {
        "id": {
          "title": "Id",
          "type": "string"
        },
        "code": {
          "default": "",
          "title": "Code",
          "type": "string"
        }
      },
      "required": [
        "id"
      ],
      "title": "NotebookCell",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "PythonScriptInput": {
      "description": "Settings for Python code execution on a kernel.",
      "properties": {
        "code": {
          "default": "",
          "title": "Code",
          "type": "string"
        },
        "kernel_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Kernel Id"
        },
        "cells": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/NotebookCell"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Cells"
        }
      },
      "title": "PythonScriptInput",
      "type": "object"
    }
  },
  "description": "Node that executes Python code on a kernel container.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "python_script_input": {
      "$ref": "#/$defs/PythonScriptInput",
      "default": {
        "code": "",
        "kernel_id": null,
        "cells": null
      }
    },
    "output_names": {
      "items": {
        "type": "string"
      },
      "title": "Output Names",
      "type": "array"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodePythonScript",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• python_script_input (PythonScriptInput)
• output_names (list[str])

Validators:

• validate_node_reference → node_reference
• validate_output_names → output_names

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodePythonScript(NodeMultiInput):
    """Node that executes Python code on a kernel container."""

    python_script_input: PythonScriptInput = PythonScriptInput()
    output_names: list[str] = Field(default_factory=lambda: ["main"])

    @field_validator("output_names")
    @classmethod
    def validate_output_names(cls, v: list[str]) -> list[str]:
        return _validate_output_names(v)

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRandomSplit pydantic-model

Bases: NodeSingleInput

Settings for a node that randomly partitions rows into N labeled outputs.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RandomSplitGroup": {
      "description": "A single output partition in a random split.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "percentage": {
          "title": "Percentage",
          "type": "number"
        }
      },
      "required": [
        "name",
        "percentage"
      ],
      "title": "RandomSplitGroup",
      "type": "object"
    }
  },
  "description": "Settings for a node that randomly partitions rows into N labeled outputs.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "splits": {
      "items": {
        "$ref": "#/$defs/RandomSplitGroup"
      },
      "title": "Splits",
      "type": "array"
    },
    "seed": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Seed"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeRandomSplit",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• splits (list[RandomSplitGroup])
• seed (int | None)

Validators:

• validate_node_reference → node_reference
• _validate_splits

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRandomSplit(NodeSingleInput):
    """Settings for a node that randomly partitions rows into N labeled outputs."""

    splits: list[RandomSplitGroup] = Field(
        default_factory=lambda: [
            RandomSplitGroup(name="train", percentage=80.0),
            RandomSplitGroup(name="test", percentage=20.0),
        ]
    )
    seed: int | None = None

    @model_validator(mode="after")
    def _validate_splits(self) -> "NodeRandomSplit":
        if not self.splits:
            raise ValueError("At least one split is required")
        if len(self.splits) > 10:
            raise ValueError("At most 10 splits are supported")
        names = [s.name for s in self.splits]
        if len(set(names)) != len(names):
            raise ValueError("Split names must be unique")
        for s in self.splits:
            if not s.name or not s.name[0].isalpha() or not all(c.isalnum() or c == "_" for c in s.name):
                raise ValueError(
                    f"Invalid split name: {s.name!r} (must start with a letter; alphanumeric/underscore only)"
                )
            if s.percentage <= 0:
                raise ValueError(f"Split {s.name!r} percentage must be > 0")
        if abs(sum(s.percentage for s in self.splits) - 100.0) > 0.01:
            raise ValueError("Split percentages must sum to 100")
        return self

    @property
    def output_names(self) -> list[str]:
        return [s.name for s in self.splits]

    def get_default_description(self) -> str:
        return " / ".join(f"{s.name} {s.percentage:g}%" for s in self.splits)

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRead pydantic-model

Bases: NodeBase

Settings for a node that reads data from a file.

Show JSON schema:

{
  "$defs": {
    "InputAvroTable": {
      "description": "Defines settings for reading an Avro file.",
      "properties": {
        "file_type": {
          "const": "avro",
          "default": "avro",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputAvroTable",
      "type": "object"
    },
    "InputCsvTable": {
      "description": "Defines settings for reading a CSV file.",
      "properties": {
        "file_type": {
          "const": "csv",
          "default": "csv",
          "title": "File Type",
          "type": "string"
        },
        "reference": {
          "default": "",
          "title": "Reference",
          "type": "string"
        },
        "starting_from_line": {
          "default": 0,
          "title": "Starting From Line",
          "type": "integer"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        },
        "parquet_ref": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parquet Ref"
        },
        "row_delimiter": {
          "default": "\n",
          "title": "Row Delimiter",
          "type": "string"
        },
        "quote_char": {
          "default": "\"",
          "title": "Quote Char",
          "type": "string"
        },
        "infer_schema_length": {
          "default": 10000,
          "title": "Infer Schema Length",
          "type": "integer"
        },
        "infer_schema": {
          "default": true,
          "title": "Infer Schema",
          "type": "boolean"
        },
        "truncate_ragged_lines": {
          "default": false,
          "title": "Truncate Ragged Lines",
          "type": "boolean"
        },
        "ignore_errors": {
          "default": false,
          "title": "Ignore Errors",
          "type": "boolean"
        }
      },
      "title": "InputCsvTable",
      "type": "object"
    },
    "InputExcelTable": {
      "description": "Defines settings for reading an Excel file.",
      "properties": {
        "file_type": {
          "const": "excel",
          "default": "excel",
          "title": "File Type",
          "type": "string"
        },
        "sheet_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Sheet Name"
        },
        "start_row": {
          "default": 0,
          "title": "Start Row",
          "type": "integer"
        },
        "start_column": {
          "default": 0,
          "title": "Start Column",
          "type": "integer"
        },
        "end_row": {
          "default": 0,
          "title": "End Row",
          "type": "integer"
        },
        "end_column": {
          "default": 0,
          "title": "End Column",
          "type": "integer"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "type_inference": {
          "default": false,
          "title": "Type Inference",
          "type": "boolean"
        }
      },
      "title": "InputExcelTable",
      "type": "object"
    },
    "InputIpcTable": {
      "description": "Defines settings for reading an Arrow IPC/Feather file.",
      "properties": {
        "file_type": {
          "const": "ipc",
          "default": "ipc",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputIpcTable",
      "type": "object"
    },
    "InputJsonTable": {
      "description": "Defines settings for reading a JSON file.",
      "properties": {
        "file_type": {
          "const": "json",
          "default": "json",
          "title": "File Type",
          "type": "string"
        },
        "reference": {
          "default": "",
          "title": "Reference",
          "type": "string"
        },
        "starting_from_line": {
          "default": 0,
          "title": "Starting From Line",
          "type": "integer"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        },
        "parquet_ref": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parquet Ref"
        },
        "row_delimiter": {
          "default": "\n",
          "title": "Row Delimiter",
          "type": "string"
        },
        "quote_char": {
          "default": "\"",
          "title": "Quote Char",
          "type": "string"
        },
        "infer_schema_length": {
          "default": 10000,
          "title": "Infer Schema Length",
          "type": "integer"
        },
        "infer_schema": {
          "default": true,
          "title": "Infer Schema",
          "type": "boolean"
        },
        "truncate_ragged_lines": {
          "default": false,
          "title": "Truncate Ragged Lines",
          "type": "boolean"
        },
        "ignore_errors": {
          "default": false,
          "title": "Ignore Errors",
          "type": "boolean"
        }
      },
      "title": "InputJsonTable",
      "type": "object"
    },
    "InputNdjsonTable": {
      "description": "Defines settings for reading a newline-delimited JSON file.",
      "properties": {
        "file_type": {
          "const": "ndjson",
          "default": "ndjson",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputNdjsonTable",
      "type": "object"
    },
    "InputParquetTable": {
      "description": "Defines settings for reading a Parquet file.",
      "properties": {
        "file_type": {
          "const": "parquet",
          "default": "parquet",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputParquetTable",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "ReceivedTable": {
      "description": "Model for defining a table received from an external source.",
      "properties": {
        "id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Id"
        },
        "name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Name"
        },
        "path": {
          "title": "Path",
          "type": "string"
        },
        "directory": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Directory"
        },
        "analysis_file_available": {
          "default": false,
          "title": "Analysis File Available",
          "type": "boolean"
        },
        "status": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Status"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "abs_file_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Abs File Path"
        },
        "file_type": {
          "enum": [
            "csv",
            "json",
            "parquet",
            "excel",
            "ipc",
            "ndjson",
            "avro"
          ],
          "title": "File Type",
          "type": "string"
        },
        "table_settings": {
          "discriminator": {
            "mapping": {
              "avro": "#/$defs/InputAvroTable",
              "csv": "#/$defs/InputCsvTable",
              "excel": "#/$defs/InputExcelTable",
              "ipc": "#/$defs/InputIpcTable",
              "json": "#/$defs/InputJsonTable",
              "ndjson": "#/$defs/InputNdjsonTable",
              "parquet": "#/$defs/InputParquetTable"
            },
            "propertyName": "file_type"
          },
          "oneOf": [
            {
              "$ref": "#/$defs/InputCsvTable"
            },
            {
              "$ref": "#/$defs/InputJsonTable"
            },
            {
              "$ref": "#/$defs/InputParquetTable"
            },
            {
              "$ref": "#/$defs/InputExcelTable"
            },
            {
              "$ref": "#/$defs/InputIpcTable"
            },
            {
              "$ref": "#/$defs/InputNdjsonTable"
            },
            {
              "$ref": "#/$defs/InputAvroTable"
            }
          ],
          "title": "Table Settings"
        }
      },
      "required": [
        "path",
        "file_type",
        "table_settings"
      ],
      "title": "ReceivedTable",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads data from a file.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "received_file": {
      "$ref": "#/$defs/ReceivedTable"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "received_file"
  ],
  "title": "NodeRead",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• received_file (ReceivedTable)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRead(NodeBase):
    """Settings for a node that reads data from a file."""

    received_file: ReceivedTable

    def get_default_description(self) -> str:
        """Describes the file being read."""
        rf = self.received_file
        name = rf.name or Path(rf.path).name
        return f"{name} ({rf.file_type})"

get_default_description()

Describes the file being read.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the file being read."""
    rf = self.received_file
    name = rf.name or Path(rf.path).name
    return f"{name} ({rf.file_type})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRecordCount pydantic-model

Bases: NodeSingleInput

Settings for a node that counts the number of records.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that counts the number of records.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeRecordCount",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRecordCount(NodeSingleInput):
    """Settings for a node that counts the number of records."""

    pass

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRecordId pydantic-model

Bases: NodeSingleInput

Settings for a node that adds a unique record ID column.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RecordIdInput": {
      "description": "Defines settings for adding a record ID (row number) column to the data.",
      "properties": {
        "output_column_name": {
          "default": "record_id",
          "title": "Output Column Name",
          "type": "string"
        },
        "offset": {
          "default": 1,
          "title": "Offset",
          "type": "integer"
        },
        "group_by": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": false,
          "title": "Group By"
        },
        "group_by_columns": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Group By Columns"
        }
      },
      "title": "RecordIdInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that adds a unique record ID column.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "record_id_input": {
      "$ref": "#/$defs/RecordIdInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "record_id_input"
  ],
  "title": "NodeRecordId",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• record_id_input (RecordIdInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRecordId(NodeSingleInput):
    """Settings for a node that adds a unique record ID column."""

    record_id_input: transform_schema.RecordIdInput

    def get_default_description(self) -> str:
        """Describes the record ID column being added."""
        r = self.record_id_input
        desc = f"Add column '{r.output_column_name}'"
        if r.group_by and r.group_by_columns:
            cols = ", ".join(r.group_by_columns[:3])
            desc += f" per group ({cols})"
        return desc

get_default_description()

Describes the record ID column being added.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the record ID column being added."""
    r = self.record_id_input
    desc = f"Add column '{r.output_column_name}'"
    if r.group_by and r.group_by_columns:
        cols = ", ".join(r.group_by_columns[:3])
        desc += f" per group ({cols})"
    return desc

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRestApiReader pydantic-model

Bases: NodeBase

Settings for a node that reads from a REST API.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RestApiAuthSettings": {
      "description": "Authentication settings for a REST API reader node.\n\nThe credential (API key / bearer token / basic password, per ``auth_type``)\nis NOT stored inline. ``secret_name`` references a secret in the user's\nsecret store \u2014 created once via the Secrets manager and reusable across\nnodes \u2014 mirroring how the database reader references a stored password. The\n``.flowfile`` persists only the reference name, never the credential itself.\n\n``secret`` is an optional inline plaintext for programmatic use\n(``flowfile_frame.read_api``); it is encrypted with the master key and\ncleared, never persisted.",
      "properties": {
        "auth_type": {
          "default": "none",
          "enum": [
            "none",
            "api_key",
            "bearer",
            "basic"
          ],
          "title": "Auth Type",
          "type": "string"
        },
        "api_key_name": {
          "default": "X-API-Key",
          "title": "Api Key Name",
          "type": "string"
        },
        "api_key_location": {
          "default": "header",
          "enum": [
            "header",
            "query"
          ],
          "title": "Api Key Location",
          "type": "string"
        },
        "basic_username": {
          "default": "",
          "title": "Basic Username",
          "type": "string"
        },
        "secret_name": {
          "default": "",
          "title": "Secret Name",
          "type": "string"
        },
        "secret": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Secret"
        }
      },
      "title": "RestApiAuthSettings",
      "type": "object"
    },
    "RestApiPaginationSettings": {
      "description": "Pagination strategy and parameters for a REST API reader node.",
      "properties": {
        "pagination_type": {
          "default": "none",
          "enum": [
            "none",
            "offset",
            "page",
            "cursor"
          ],
          "title": "Pagination Type",
          "type": "string"
        },
        "offset_param": {
          "default": "offset",
          "title": "Offset Param",
          "type": "string"
        },
        "limit_param": {
          "default": "limit",
          "title": "Limit Param",
          "type": "string"
        },
        "page_size": {
          "default": 100,
          "title": "Page Size",
          "type": "integer"
        },
        "page_param": {
          "default": "page",
          "title": "Page Param",
          "type": "string"
        },
        "start_page": {
          "default": 1,
          "title": "Start Page",
          "type": "integer"
        },
        "cursor_param": {
          "default": "cursor",
          "title": "Cursor Param",
          "type": "string"
        },
        "cursor_location": {
          "default": "body",
          "enum": [
            "body",
            "header"
          ],
          "title": "Cursor Location",
          "type": "string"
        },
        "cursor_response_path": {
          "default": "",
          "title": "Cursor Response Path",
          "type": "string"
        },
        "initial_cursor": {
          "default": "",
          "title": "Initial Cursor",
          "type": "string"
        },
        "max_pages": {
          "default": 1000,
          "title": "Max Pages",
          "type": "integer"
        },
        "max_records": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Max Records"
        },
        "page_delay_seconds": {
          "default": 0.0,
          "title": "Page Delay Seconds",
          "type": "number"
        }
      },
      "title": "RestApiPaginationSettings",
      "type": "object"
    },
    "RestApiSettings": {
      "description": "UI settings for a REST API reader node.\n\nSecrets are stored inline but encrypted (see ``RestApiAuthSettings``). JSON\nis the only supported response format; ``record_path`` is a dot-path that\nlocates the record array within the response body (empty = top-level).",
      "properties": {
        "url": {
          "default": "",
          "title": "Url",
          "type": "string"
        },
        "method": {
          "default": "GET",
          "enum": [
            "GET",
            "POST"
          ],
          "title": "Method",
          "type": "string"
        },
        "headers": {
          "additionalProperties": {
            "type": "string"
          },
          "title": "Headers",
          "type": "object"
        },
        "query_params": {
          "additionalProperties": {
            "type": "string"
          },
          "title": "Query Params",
          "type": "object"
        },
        "json_body": {
          "anyOf": [
            {},
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Json Body"
        },
        "auth": {
          "$ref": "#/$defs/RestApiAuthSettings"
        },
        "pagination": {
          "$ref": "#/$defs/RestApiPaginationSettings"
        },
        "record_path": {
          "default": "",
          "title": "Record Path",
          "type": "string"
        },
        "timeout_seconds": {
          "default": 30.0,
          "title": "Timeout Seconds",
          "type": "number"
        },
        "max_retries": {
          "default": 3,
          "title": "Max Retries",
          "type": "integer"
        }
      },
      "title": "RestApiSettings",
      "type": "object"
    }
  },
  "description": "Settings for a node that reads from a REST API.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "rest_api_settings": {
      "$ref": "#/$defs/RestApiSettings"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "rest_api_settings"
  ],
  "title": "NodeRestApiReader",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• rest_api_settings (RestApiSettings)
• fields (list[MinimalFieldInfo] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRestApiReader(NodeBase):
    """Settings for a node that reads from a REST API."""

    rest_api_settings: RestApiSettings
    fields: list[MinimalFieldInfo] | None = None

    def get_default_description(self) -> str:
        """Describes the REST API request."""
        s = self.rest_api_settings
        pieces: list[str] = []
        if s.url:
            pieces.append(f"{s.method} {s.url}")
        if s.pagination and s.pagination.pagination_type != "none":
            pieces.append(f"paginated: {s.pagination.pagination_type}")
        return " | ".join(pieces)

get_default_description()

Describes the REST API request.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the REST API request."""
    s = self.rest_api_settings
    pieces: list[str] = []
    if s.url:
        pieces.append(f"{s.method} {s.url}")
    if s.pagination and s.pagination.pagination_type != "none":
        pieces.append(f"paginated: {s.pagination.pagination_type}")
    return " | ".join(pieces)

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeRunFlow pydantic-model

Bases: NodeBase

Settings for a node that executes a catalog-registered flow as a subflow.

input_slots/output_slots persist the last-synced subflow interface (flow_input/flow_output names in interface order); connections are keyed to handles positionally against them (handle input-{i+1} <-> input_slots[i]; input-0 is the reserved parameter-data handle).

Show JSON schema:

{
  "$defs": {
    "FlowParameter": {
      "description": "A single flow-level parameter that can be referenced via ${name} syntax.\n\n``default_value`` stays a string for file-format stability; ``typed_default``\nyields the coerced Python value used for whole-field ``${name}`` injection.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "default_value": {
          "default": "",
          "title": "Default Value",
          "type": "string"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "type": {
          "default": "string",
          "enum": [
            "string",
            "integer",
            "float",
            "boolean",
            "enum"
          ],
          "title": "Type",
          "type": "string"
        },
        "enum_values": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Enum Values"
        }
      },
      "required": [
        "name"
      ],
      "title": "FlowParameter",
      "type": "object"
    },
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "RunFlowParameterBinding": {
      "description": "How one subflow parameter gets its value for a run_flow execution.",
      "properties": {
        "parameter_name": {
          "title": "Parameter Name",
          "type": "string"
        },
        "source": {
          "default": "default",
          "enum": [
            "default",
            "constant",
            "column"
          ],
          "title": "Source",
          "type": "string"
        },
        "constant_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Constant Value"
        },
        "column_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Column Name"
        }
      },
      "required": [
        "parameter_name"
      ],
      "title": "RunFlowParameterBinding",
      "type": "object"
    },
    "SubflowReference": {
      "description": "Reference to a catalog-registered flow.\n\n``registration_id`` is the primary reference; ``flow_uuid`` is stamped\nserver-side and used to repair a dangling id; ``flow_path`` is display-only.",
      "properties": {
        "registration_id": {
          "title": "Registration Id",
          "type": "integer"
        },
        "flow_uuid": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Flow Uuid"
        },
        "flow_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Flow Path"
        }
      },
      "required": [
        "registration_id"
      ],
      "title": "SubflowReference",
      "type": "object"
    }
  },
  "description": "Settings for a node that executes a catalog-registered flow as a subflow.\n\n``input_slots``/``output_slots`` persist the last-synced subflow interface\n(flow_input/flow_output names in interface order); connections are keyed to\nhandles positionally against them (handle ``input-{i+1}`` <-> input_slots[i];\n``input-0`` is the reserved parameter-data handle).",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "flow_reference": {
      "$ref": "#/$defs/SubflowReference"
    },
    "input_slots": {
      "items": {
        "type": "string"
      },
      "title": "Input Slots",
      "type": "array"
    },
    "output_slots": {
      "items": {
        "type": "string"
      },
      "title": "Output Slots",
      "type": "array"
    },
    "parameter_specs": {
      "items": {
        "$ref": "#/$defs/FlowParameter"
      },
      "title": "Parameter Specs",
      "type": "array"
    },
    "parameter_bindings": {
      "items": {
        "$ref": "#/$defs/RunFlowParameterBinding"
      },
      "title": "Parameter Bindings",
      "type": "array"
    },
    "iteration_mode": {
      "default": "first_value",
      "enum": [
        "first_value",
        "iterate"
      ],
      "title": "Iteration Mode",
      "type": "string"
    },
    "append_run_metadata": {
      "default": true,
      "title": "Append Run Metadata",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "flow_reference"
  ],
  "title": "NodeRunFlow",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• flow_reference (SubflowReference)
• input_slots (list[str])
• output_slots (list[str])
• parameter_specs (list[FlowParameter])
• parameter_bindings (list[RunFlowParameterBinding])
• iteration_mode (Literal['first_value', 'iterate'])
• append_run_metadata (bool)

Validators:

• validate_node_reference → node_reference
• _validate_slots_and_bindings

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeRunFlow(NodeBase):
    """Settings for a node that executes a catalog-registered flow as a subflow.

    ``input_slots``/``output_slots`` persist the last-synced subflow interface
    (flow_input/flow_output names in interface order); connections are keyed to
    handles positionally against them (handle ``input-{i+1}`` <-> input_slots[i];
    ``input-0`` is the reserved parameter-data handle).
    """

    flow_reference: SubflowReference
    input_slots: list[str] = Field(default_factory=list)
    output_slots: list[str] = Field(default_factory=list)
    parameter_specs: list[FlowParameter] = Field(default_factory=list)
    parameter_bindings: list[RunFlowParameterBinding] = Field(default_factory=list)
    iteration_mode: Literal["first_value", "iterate"] = "first_value"
    append_run_metadata: bool = True

    @property
    def input_names(self) -> list[str]:
        """Handle labels, index i <-> handle input-i (index 0 = parameter handle).

        An empty label at index 0 tells the frontend to hide the parameter
        handle (the subflow has no parameters); data handles keep input-1..N.
        """
        param_label = "Parameters" if (self.parameter_specs or self.parameter_bindings) else ""
        return [param_label, *self.input_slots]

    @property
    def output_names(self) -> list[str]:
        return list(self.output_slots)

    @model_validator(mode="after")
    def _validate_slots_and_bindings(self) -> "NodeRunFlow":
        if len(self.input_slots) > 9:
            raise ValueError("Subflows with more than 9 data inputs are not supported")
        if len(self.output_slots) > 10:
            raise ValueError("Subflows with more than 10 outputs are not supported")
        if len(set(self.input_slots)) != len(self.input_slots):
            raise ValueError("input_slots must be unique")
        if len(set(self.output_slots)) != len(self.output_slots):
            raise ValueError("output_slots must be unique")
        binding_names = [b.parameter_name for b in self.parameter_bindings]
        if len(set(binding_names)) != len(binding_names):
            raise ValueError("Duplicate parameter bindings")
        return self

    def get_default_description(self) -> str:
        name = self.flow_reference.flow_path or f"registration {self.flow_reference.registration_id}"
        mode = "per row" if self.iteration_mode == "iterate" else "once"
        return f"Run flow {Path(name).stem if self.flow_reference.flow_path else name} ({mode})"

input_names property

Handle labels, index i <-> handle input-i (index 0 = parameter handle).

An empty label at index 0 tells the frontend to hide the parameter handle (the subflow has no parameters); data handles keep input-1..N.

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeSample pydantic-model

Bases: NodeSingleInput

Settings for a node that samples a subset of the data.

sample_method selects between a cheap top-N slice and a uniform random sample. It defaults to "first" so flows saved before random sampling existed (which only carry sample_size) keep their exact behaviour. fraction is a percentage and only read by "random_fraction"; seed only by the two random methods, where None means a fresh permutation on every run.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that samples a subset of the data.\n\n``sample_method`` selects between a cheap top-N slice and a uniform random\nsample. It defaults to ``\"first\"`` so flows saved before random sampling\nexisted (which only carry ``sample_size``) keep their exact behaviour.\n``fraction`` is a percentage and only read by ``\"random_fraction\"``;\n``seed`` only by the two random methods, where ``None`` means a fresh\npermutation on every run.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "sample_method": {
      "default": "first",
      "enum": [
        "first",
        "random",
        "random_fraction"
      ],
      "title": "Sample Method",
      "type": "string"
    },
    "sample_size": {
      "default": 1000,
      "title": "Sample Size",
      "type": "integer"
    },
    "fraction": {
      "default": 10.0,
      "title": "Fraction",
      "type": "number"
    },
    "seed": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Seed"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeSample",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• sample_method (SampleMethod)
• sample_size (int)
• fraction (float)
• seed (int | None)

Validators:

• validate_node_reference → node_reference
• _validate_sample

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeSample(NodeSingleInput):
    """Settings for a node that samples a subset of the data.

    ``sample_method`` selects between a cheap top-N slice and a uniform random
    sample. It defaults to ``"first"`` so flows saved before random sampling
    existed (which only carry ``sample_size``) keep their exact behaviour.
    ``fraction`` is a percentage and only read by ``"random_fraction"``;
    ``seed`` only by the two random methods, where ``None`` means a fresh
    permutation on every run.
    """

    sample_method: SampleMethod = "first"
    sample_size: int = 1000
    fraction: float = 10.0
    seed: int | None = None

    @model_validator(mode="after")
    def _validate_sample(self) -> "NodeSample":
        # sample_size is deliberately unvalidated: the pre-existing UI allowed 0
        # and saved flows carrying it must keep loading.
        if self.sample_method == "random_fraction" and not 0 < self.fraction <= 100:
            raise ValueError("fraction must be greater than 0 and at most 100")
        return self

    def get_default_description(self) -> str:
        """Describes the sampling method and size."""
        if self.sample_method == "random":
            return f"Random {self.sample_size} rows"
        if self.sample_method == "random_fraction":
            return f"Random {self.fraction:g}% of rows"
        return f"First {self.sample_size} rows"

get_default_description()

Describes the sampling method and size.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the sampling method and size."""
    if self.sample_method == "random":
        return f"Random {self.sample_size} rows"
    if self.sample_method == "random_fraction":
        return f"Random {self.fraction:g}% of rows"
    return f"First {self.sample_size} rows"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeSelect pydantic-model

Bases: NodeSingleInput

Settings for a node that selects, renames, and reorders columns.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that selects, renames, and reorders columns.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "keep_missing": {
      "default": true,
      "title": "Keep Missing",
      "type": "boolean"
    },
    "select_input": {
      "items": {
        "$ref": "#/$defs/SelectInput"
      },
      "title": "Select Input",
      "type": "array"
    },
    "sorted_by": {
      "anyOf": [
        {
          "enum": [
            "none",
            "asc",
            "desc"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "none",
      "title": "Sorted By"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeSelect",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• keep_missing (bool)
• select_input (list[SelectInput])
• sorted_by (Literal['none', 'asc', 'desc'] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeSelect(NodeSingleInput):
    """Settings for a node that selects, renames, and reorders columns."""

    keep_missing: bool = True
    select_input: list[transform_schema.SelectInput] = Field(default_factory=list)
    sorted_by: Literal["none", "asc", "desc"] | None = "none"

    def get_default_description(self) -> str:
        """Describes column selections, renames, and drops."""
        if not self.select_input:
            return ""
        parts = []
        renames = [s for s in self.select_input if s.old_name != s.new_name and s.keep]
        drops = [s for s in self.select_input if not s.keep]
        type_changes = [s for s in self.select_input if s.data_type_change and s.keep]
        if renames:
            rename_strs = [f"{r.old_name} -> {r.new_name}" for r in renames[:3]]
            parts.append("Rename: " + ", ".join(rename_strs))
            if len(renames) > 3:
                parts[-1] += f" (+{len(renames) - 3} more)"
        if drops:
            drop_names = [d.old_name for d in drops[:3]]
            parts.append("Drop: " + ", ".join(drop_names))
            if len(drops) > 3:
                parts[-1] += f" (+{len(drops) - 3} more)"
        if type_changes and not renames and not drops:
            cast_strs = [f"{t.old_name} to {t.data_type}" for t in type_changes[:3]]
            parts.append("Cast: " + ", ".join(cast_strs))
            if len(type_changes) > 3:
                parts[-1] += f" (+{len(type_changes) - 3} more)"
        return "; ".join(parts) if parts else ""

    def to_yaml_dict(self) -> NodeSelectYaml:
        """Converts the select node settings to a dictionary for YAML serialization."""
        result: NodeSelectYaml = {
            "cache_results": bool(self.cache_results),
            "keep_missing": self.keep_missing,
            "select_input": [s.to_yaml_dict() for s in self.select_input],
            "sorted_by": self.sorted_by,
        }
        if self.output_field_config:
            result["output_field_config"] = {
                "enabled": self.output_field_config.enabled,
                "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
                "validate_data_types": self.output_field_config.validate_data_types,
                "fields": [
                    {
                        "name": f.name,
                        "data_type": f.data_type,
                        "default_value": f.default_value,
                    }
                    for f in self.output_field_config.fields
                ],
            }
        return result

get_default_description()

Describes column selections, renames, and drops.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes column selections, renames, and drops."""
    if not self.select_input:
        return ""
    parts = []
    renames = [s for s in self.select_input if s.old_name != s.new_name and s.keep]
    drops = [s for s in self.select_input if not s.keep]
    type_changes = [s for s in self.select_input if s.data_type_change and s.keep]
    if renames:
        rename_strs = [f"{r.old_name} -> {r.new_name}" for r in renames[:3]]
        parts.append("Rename: " + ", ".join(rename_strs))
        if len(renames) > 3:
            parts[-1] += f" (+{len(renames) - 3} more)"
    if drops:
        drop_names = [d.old_name for d in drops[:3]]
        parts.append("Drop: " + ", ".join(drop_names))
        if len(drops) > 3:
            parts[-1] += f" (+{len(drops) - 3} more)"
    if type_changes and not renames and not drops:
        cast_strs = [f"{t.old_name} to {t.data_type}" for t in type_changes[:3]]
        parts.append("Cast: " + ", ".join(cast_strs))
        if len(type_changes) > 3:
            parts[-1] += f" (+{len(type_changes) - 3} more)"
    return "; ".join(parts) if parts else ""

to_yaml_dict()

Converts the select node settings to a dictionary for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> NodeSelectYaml:
    """Converts the select node settings to a dictionary for YAML serialization."""
    result: NodeSelectYaml = {
        "cache_results": bool(self.cache_results),
        "keep_missing": self.keep_missing,
        "select_input": [s.to_yaml_dict() for s in self.select_input],
        "sorted_by": self.sorted_by,
    }
    if self.output_field_config:
        result["output_field_config"] = {
            "enabled": self.output_field_config.enabled,
            "validation_mode_behavior": self.output_field_config.validation_mode_behavior,
            "validate_data_types": self.output_field_config.validate_data_types,
            "fields": [
                {
                    "name": f.name,
                    "data_type": f.data_type,
                    "default_value": f.default_value,
                }
                for f in self.output_field_config.fields
            ],
        }
    return result

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeSingleInput pydantic-model

Bases: NodeBase

A base model for any node that takes a single data input.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "A base model for any node that takes a single data input.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeSingleInput",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeSingleInput(NodeBase):
    """A base model for any node that takes a single data input."""

    depending_on_id: int | None = -1

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeSort pydantic-model

Bases: NodeSingleInput

Settings for a node that sorts the data by one or more columns.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SortByInput": {
      "description": "Defines a single sort condition on a column, including the direction.",
      "properties": {
        "column": {
          "title": "Column",
          "type": "string"
        },
        "how": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "asc",
          "title": "How"
        }
      },
      "required": [
        "column"
      ],
      "title": "SortByInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that sorts the data by one or more columns.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "sort_input": {
      "items": {
        "$ref": "#/$defs/SortByInput"
      },
      "title": "Sort Input",
      "type": "array"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeSort",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• sort_input (list[SortByInput])

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeSort(NodeSingleInput):
    """Settings for a node that sorts the data by one or more columns."""

    sort_input: list[transform_schema.SortByInput] = Field(default_factory=list)

    def get_default_description(self) -> str:
        """Describes the sort columns and directions."""
        if not self.sort_input:
            return ""
        parts = [f"{s.column} {s.how or 'asc'}" for s in self.sort_input[:3]]
        desc = "Sort by " + ", ".join(parts)
        if len(self.sort_input) > 3:
            desc += f" (+{len(self.sort_input) - 3} more)"
        return desc

get_default_description()

Describes the sort columns and directions.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the sort columns and directions."""
    if not self.sort_input:
        return ""
    parts = [f"{s.column} {s.how or 'asc'}" for s in self.sort_input[:3]]
    desc = "Sort by " + ", ".join(parts)
    if len(self.sort_input) > 3:
        desc += f" (+{len(self.sort_input) - 3} more)"
    return desc

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeSqlQuery pydantic-model

Bases: NodeMultiInput

Settings for a node that executes a SQL query against connected data sources.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SqlQueryInput": {
      "description": "A container for a SQL query to execute against connected data sources.\n\nNote: ``sql_code`` is *not* validated at schema-construction time. Construction\nis a passive shape-check; the unsafe-SQL gate lives at the executor seam in\n``execute_sql_query`` (and is also enforced by the underlying\n``validate_sql_query`` utility callers can use directly). Validating here too\nwould block legitimate non-AI callers from drafting/testing SQL before\nexecution.",
      "properties": {
        "sql_code": {
          "title": "Sql Code",
          "type": "string"
        }
      },
      "required": [
        "sql_code"
      ],
      "title": "SqlQueryInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that executes a SQL query against connected data sources.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "sql_query_input": {
      "$ref": "#/$defs/SqlQueryInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "sql_query_input"
  ],
  "title": "NodeSqlQuery",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• sql_query_input (SqlQueryInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeSqlQuery(NodeMultiInput):
    """Settings for a node that executes a SQL query against connected data sources."""

    sql_query_input: transform_schema.SqlQueryInput

    def get_default_description(self) -> str:
        """Describes the SQL query snippet."""
        code = self.sql_query_input.sql_code
        first_line = code.strip().split("\n")[0] if code else ""
        if len(first_line) > 80:
            first_line = first_line[:77] + "..."
        return first_line

get_default_description()

Describes the SQL query snippet.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the SQL query snippet."""
    code = self.sql_query_input.sql_code
    first_line = code.strip().split("\n")[0] if code else ""
    if len(first_line) > 80:
        first_line = first_line[:77] + "..."
    return first_line

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeTextToRows pydantic-model

Bases: NodeSingleInput

Settings for a node that splits a text column into multiple rows.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "TextToRowsInput": {
      "description": "Defines settings for splitting a text column into multiple rows based on a delimiter.",
      "properties": {
        "column_to_split": {
          "title": "Column To Split",
          "type": "string"
        },
        "output_column_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Column Name"
        },
        "split_by_fixed_value": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": true,
          "title": "Split By Fixed Value"
        },
        "split_fixed_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": ",",
          "title": "Split Fixed Value"
        },
        "split_by_column": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Split By Column"
        }
      },
      "required": [
        "column_to_split"
      ],
      "title": "TextToRowsInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that splits a text column into multiple rows.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "text_to_rows_input": {
      "$ref": "#/$defs/TextToRowsInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "text_to_rows_input"
  ],
  "title": "NodeTextToRows",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• text_to_rows_input (TextToRowsInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeTextToRows(NodeSingleInput):
    """Settings for a node that splits a text column into multiple rows."""

    text_to_rows_input: transform_schema.TextToRowsInput

    def get_default_description(self) -> str:
        """Describes the text-to-rows split operation."""
        t = self.text_to_rows_input
        delim = t.split_fixed_value if t.split_by_fixed_value else t.split_by_column
        return f"Split {t.column_to_split} by '{delim}'"

get_default_description()

Describes the text-to-rows split operation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the text-to-rows split operation."""
    t = self.text_to_rows_input
    delim = t.split_fixed_value if t.split_by_fixed_value else t.split_by_column
    return f"Split {t.column_to_split} by '{delim}'"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeTrainModel pydantic-model

Bases: NodeSingleInput

Train an ML model (regression or classification) and optionally publish it to the catalog.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "TrainModelSettings": {
      "description": "Settings payload for the Train Model node.\n\n``params`` is a flat dict so the form-driven hyperparameter UI doesn't need\na discriminated union \u2014 the worker validates against the algorithm-specific\nPydantic class via ``shared.ml.trainers.get_trainer(model_type).params_class``.\n\nThe trained model is always written to a flow-scoped path keyed off this\nnode's id so downstream Apply Model nodes in the same flow can read it\nwithout first publishing to the catalog. Set ``publish_to_catalog=True``\nto additionally store the artifact in the catalog (with a stable\ncross-run name + version).",
      "properties": {
        "target_column": {
          "default": "",
          "title": "Target Column",
          "type": "string"
        },
        "feature_columns": {
          "items": {
            "type": "string"
          },
          "title": "Feature Columns",
          "type": "array"
        },
        "model_type": {
          "default": "linear_regression",
          "title": "Model Type",
          "type": "string"
        },
        "params": {
          "additionalProperties": true,
          "title": "Params",
          "type": "object"
        },
        "publish_to_catalog": {
          "default": false,
          "title": "Publish To Catalog",
          "type": "boolean"
        },
        "model_name": {
          "default": "",
          "title": "Model Name",
          "type": "string"
        },
        "namespace_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Id"
        },
        "namespace_full_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Namespace Full Name"
        },
        "catalog_description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Catalog Description"
        },
        "catalog_tags": {
          "items": {
            "type": "string"
          },
          "title": "Catalog Tags",
          "type": "array"
        }
      },
      "title": "TrainModelSettings",
      "type": "object"
    }
  },
  "description": "Train an ML model (regression or classification) and optionally publish it to the catalog.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "train_input": {
      "$ref": "#/$defs/TrainModelSettings"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeTrainModel",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• train_input (TrainModelSettings)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeTrainModel(NodeSingleInput):
    """Train an ML model (regression or classification) and optionally publish it to the catalog."""

    model_config = ConfigDict(protected_namespaces=())

    train_input: TrainModelSettings = Field(default_factory=TrainModelSettings)

    def get_default_description(self) -> str:
        s = self.train_input
        if s.model_name and s.target_column:
            return f"Train {s.model_type} '{s.model_name}' on {s.target_column}"
        return "Train Model"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeUnion pydantic-model

Bases: NodeMultiInput

Settings for a node that concatenates multiple data inputs.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "UnionInput": {
      "description": "Defines settings for a union (concatenation) operation.",
      "properties": {
        "mode": {
          "default": "relaxed",
          "enum": [
            "selective",
            "relaxed"
          ],
          "title": "Mode",
          "type": "string"
        }
      },
      "title": "UnionInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that concatenates multiple data inputs.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "union_input": {
      "$ref": "#/$defs/UnionInput"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeUnion",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• union_input (UnionInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeUnion(NodeMultiInput):
    """Settings for a node that concatenates multiple data inputs."""

    union_input: transform_schema.UnionInput = Field(default_factory=transform_schema.UnionInput)

    def get_default_description(self) -> str:
        """Describes the union mode."""
        return f"Union ({self.union_input.mode})"

get_default_description()

Describes the union mode.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the union mode."""
    return f"Union ({self.union_input.mode})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeUnique pydantic-model

Bases: NodeSingleInput

Settings for a node that returns the unique rows from the data.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "UniqueInput": {
      "description": "Defines settings for a uniqueness operation, specifying columns and which row to keep.",
      "properties": {
        "columns": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Columns"
        },
        "strategy": {
          "default": "any",
          "enum": [
            "first",
            "last",
            "any",
            "none"
          ],
          "title": "Strategy",
          "type": "string"
        }
      },
      "title": "UniqueInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that returns the unique rows from the data.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "unique_input": {
      "$ref": "#/$defs/UniqueInput"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "unique_input"
  ],
  "title": "NodeUnique",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• unique_input (UniqueInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeUnique(NodeSingleInput):
    """Settings for a node that returns the unique rows from the data."""

    unique_input: transform_schema.UniqueInput

    def get_default_description(self) -> str:
        """Describes the uniqueness operation."""
        u = self.unique_input
        if u.columns:
            cols = ", ".join(u.columns[:3])
            if len(u.columns) > 3:
                cols += f" (+{len(u.columns) - 3} more)"
            return f"Unique by {cols} (keep {u.strategy})"
        return f"Unique rows (keep {u.strategy})"

get_default_description()

Describes the uniqueness operation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the uniqueness operation."""
    u = self.unique_input
    if u.columns:
        cols = ", ".join(u.columns[:3])
        if len(u.columns) > 3:
            cols += f" (+{len(u.columns) - 3} more)"
        return f"Unique by {cols} (keep {u.strategy})"
    return f"Unique rows (keep {u.strategy})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeUnpivot pydantic-model

Bases: NodeSingleInput

Settings for a node that unpivots data from a wide to a long format.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "UnpivotInput": {
      "description": "Defines settings for an unpivot (wide-to-long) operation.",
      "properties": {
        "index_columns": {
          "items": {
            "type": "string"
          },
          "title": "Index Columns",
          "type": "array"
        },
        "value_columns": {
          "items": {
            "type": "string"
          },
          "title": "Value Columns",
          "type": "array"
        },
        "data_type_selector": {
          "anyOf": [
            {
              "enum": [
                "float",
                "all",
                "date",
                "numeric",
                "string"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type Selector"
        },
        "data_type_selector_mode": {
          "default": "column",
          "enum": [
            "data_type",
            "column"
          ],
          "title": "Data Type Selector Mode",
          "type": "string"
        }
      },
      "title": "UnpivotInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that unpivots data from a wide to a long format.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "unpivot_input": {
      "$ref": "#/$defs/UnpivotInput",
      "default": null
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeUnpivot",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• unpivot_input (UnpivotInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeUnpivot(NodeSingleInput):
    """Settings for a node that unpivots data from a wide to a long format."""

    unpivot_input: transform_schema.UnpivotInput = None

    def get_default_description(self) -> str:
        """Describes the unpivot operation."""
        if self.unpivot_input is None:
            return ""
        u = self.unpivot_input
        if u.value_columns:
            cols = ", ".join(u.value_columns[:3])
            if len(u.value_columns) > 3:
                cols += f" (+{len(u.value_columns) - 3} more)"
            return f"Unpivot {cols}"
        if u.data_type_selector:
            return f"Unpivot {u.data_type_selector} columns"
        return "Unpivot"

get_default_description()

Describes the unpivot operation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the unpivot operation."""
    if self.unpivot_input is None:
        return ""
    u = self.unpivot_input
    if u.value_columns:
        cols = ", ".join(u.value_columns[:3])
        if len(u.value_columns) > 3:
            cols += f" (+{len(u.value_columns) - 3} more)"
        return f"Unpivot {cols}"
    if u.data_type_selector:
        return f"Unpivot {u.data_type_selector} columns"
    return "Unpivot"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeWaitFor pydantic-model

Bases: NodeMultiInput

Pass-through node that enforces ordering on extra dependency inputs.

The first input flows through unchanged; the others have to complete before this node runs but their data is discarded. Useful for enforcing "Apply Model must wait for Train Model" without otherwise coupling their data.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Pass-through node that enforces ordering on extra dependency inputs.\n\nThe first input flows through unchanged; the others have to complete before\nthis node runs but their data is discarded. Useful for enforcing \"Apply\nModel must wait for Train Model\" without otherwise coupling their data.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeWaitFor",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeWaitFor(NodeMultiInput):
    """Pass-through node that enforces ordering on extra dependency inputs.

    The first input flows through unchanged; the others have to complete before
    this node runs but their data is discarded. Useful for enforcing "Apply
    Model must wait for Train Model" without otherwise coupling their data.
    """

    def get_default_description(self) -> str:
        n = len(self.depending_on_ids or [])
        if n <= 1:
            return "Wait For"
        return f"Wait For ({n - 1} dep{'s' if n > 2 else ''})"

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NodeWindowFunctions pydantic-model

Bases: NodeSingleInput

Settings for a node that adds rolling, cumulative, rank or tile columns.

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    },
    "SortByInput": {
      "description": "Defines a single sort condition on a column, including the direction.",
      "properties": {
        "column": {
          "title": "Column",
          "type": "string"
        },
        "how": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "asc",
          "title": "How"
        }
      },
      "required": [
        "column"
      ],
      "title": "SortByInput",
      "type": "object"
    },
    "WindowFunctionInput": {
      "description": "A single window-function operation producing one new column.\n\n`column` is the source column for rolling, cumulative and rank functions.\nFor `tile`, `column` is ignored (ordering comes from the outer\n``WindowFunctionsInput.order_by``).",
      "properties": {
        "column": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Column"
        },
        "function": {
          "enum": [
            "rolling_sum",
            "rolling_mean",
            "rolling_min",
            "rolling_max",
            "rolling_std",
            "cum_sum",
            "cum_count",
            "cum_min",
            "cum_max",
            "rank",
            "tile"
          ],
          "title": "Function",
          "type": "string"
        },
        "new_column_name": {
          "title": "New Column Name",
          "type": "string"
        },
        "window_size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Window Size"
        },
        "min_periods": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Min Periods"
        },
        "edge_behavior": {
          "anyOf": [
            {
              "enum": [
                "require_full",
                "partial",
                "fill_zero"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "require_full",
          "title": "Edge Behavior"
        },
        "number_of_groups": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Groups"
        },
        "rank_method": {
          "anyOf": [
            {
              "enum": [
                "ordinal",
                "dense",
                "min",
                "max",
                "average"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "ordinal",
          "title": "Rank Method"
        },
        "output_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Type"
        }
      },
      "required": [
        "function",
        "new_column_name"
      ],
      "title": "WindowFunctionInput",
      "type": "object"
    },
    "WindowFunctionsInput": {
      "description": "Defines the settings for a window-functions node.\n\nAttributes\n----------\npartition_by : list[str]\n    Optional list of columns to partition by (equivalent to ``.over(...)``).\norder_by : list[SortByInput]\n    Ordering within each partition. Required for rolling and tile\n    functions; optional (but usually wanted) for cumulative functions.\nwindow_functions : list[WindowFunctionInput]\n    Ordered list of per-column window operations to apply. Each produces\n    one new column; all are applied in a single ``with_columns`` call.",
      "properties": {
        "partition_by": {
          "items": {
            "type": "string"
          },
          "title": "Partition By",
          "type": "array"
        },
        "order_by": {
          "items": {
            "$ref": "#/$defs/SortByInput"
          },
          "title": "Order By",
          "type": "array"
        },
        "window_functions": {
          "items": {
            "$ref": "#/$defs/WindowFunctionInput"
          },
          "title": "Window Functions",
          "type": "array"
        }
      },
      "title": "WindowFunctionsInput",
      "type": "object"
    }
  },
  "description": "Settings for a node that adds rolling, cumulative, rank or tile columns.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": -1,
      "title": "Depending On Id"
    },
    "window_input": {
      "$ref": "#/$defs/WindowFunctionsInput"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "NodeWindowFunctions",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_id (int | None)
• window_input (WindowFunctionsInput)

Validators:

• validate_node_reference → node_reference

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NodeWindowFunctions(NodeSingleInput):
    """Settings for a node that adds rolling, cumulative, rank or tile columns."""

    window_input: transform_schema.WindowFunctionsInput = Field(default_factory=transform_schema.WindowFunctionsInput)

    def get_default_description(self) -> str:
        """Describes the configured window functions."""
        if self.window_input is None or not self.window_input.window_functions:
            return ""
        parts: list[str] = []
        if self.window_input.partition_by:
            cols = ", ".join(self.window_input.partition_by[:3])
            if len(self.window_input.partition_by) > 3:
                cols += f" (+{len(self.window_input.partition_by) - 3} more)"
            parts.append(f"By {cols}")
        ops = self.window_input.window_functions
        op_strs = [f"{w.function}({w.column or ''})".replace("()", "()") for w in ops[:3]]
        if len(ops) > 3:
            op_strs.append(f"+{len(ops) - 3} more")
        parts.append(", ".join(op_strs))
        return ": ".join(parts)

get_default_description()

Describes the configured window functions.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Describes the configured window functions."""
    if self.window_input is None or not self.window_input.window_functions:
        return ""
    parts: list[str] = []
    if self.window_input.partition_by:
        cols = ", ".join(self.window_input.partition_by[:3])
        if len(self.window_input.partition_by) > 3:
            cols += f" (+{len(self.window_input.partition_by) - 3} more)"
        parts.append(f"By {cols}")
    ops = self.window_input.window_functions
    op_strs = [f"{w.function}({w.column or ''})".replace("()", "()") for w in ops[:3]]
    if len(ops) > 3:
        op_strs.append(f"+{len(ops) - 3} more")
    parts.append(", ".join(op_strs))
    return ": ".join(parts)

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

NotebookCell pydantic-model

Bases: BaseModel

A single cell in the notebook editor.

Note: Cell output (stdout, display_outputs, errors) is handled entirely on the frontend and is not persisted. Only id and code are stored.

Show JSON schema:

{
  "description": "A single cell in the notebook editor.\n\nNote: Cell output (stdout, display_outputs, errors) is handled entirely\non the frontend and is not persisted. Only id and code are stored.",
  "properties": {
    "id": {
      "title": "Id",
      "type": "string"
    },
    "code": {
      "default": "",
      "title": "Code",
      "type": "string"
    }
  },
  "required": [
    "id"
  ],
  "title": "NotebookCell",
  "type": "object"
}

Fields:

• id (str)
• code (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class NotebookCell(BaseModel):
    """A single cell in the notebook editor.

    Note: Cell output (stdout, display_outputs, errors) is handled entirely
    on the frontend and is not persisted. Only id and code are stored.
    """

    id: str
    code: str = ""

OutputAvroTable pydantic-model

Bases: BaseModel

Defines settings for writing an Avro file.

Show JSON schema:

{
  "description": "Defines settings for writing an Avro file.",
  "properties": {
    "file_type": {
      "const": "avro",
      "default": "avro",
      "title": "File Type",
      "type": "string"
    },
    "compression": {
      "default": "uncompressed",
      "enum": [
        "uncompressed",
        "snappy",
        "deflate"
      ],
      "title": "Compression",
      "type": "string"
    }
  },
  "title": "OutputAvroTable",
  "type": "object"
}

Fields:

• file_type (Literal['avro'])
• compression (Literal['uncompressed', 'snappy', 'deflate'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputAvroTable(BaseModel):
    """Defines settings for writing an Avro file."""

    file_type: Literal["avro"] = "avro"
    compression: Literal["uncompressed", "snappy", "deflate"] = "uncompressed"

OutputCsvTable pydantic-model

Bases: BaseModel

Defines settings for writing a CSV file.

Show JSON schema:

{
  "description": "Defines settings for writing a CSV file.",
  "properties": {
    "file_type": {
      "const": "csv",
      "default": "csv",
      "title": "File Type",
      "type": "string"
    },
    "delimiter": {
      "default": ",",
      "title": "Delimiter",
      "type": "string"
    },
    "encoding": {
      "default": "utf-8",
      "title": "Encoding",
      "type": "string"
    }
  },
  "title": "OutputCsvTable",
  "type": "object"
}

Fields:

• file_type (Literal['csv'])
• delimiter (str)
• encoding (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputCsvTable(BaseModel):
    """Defines settings for writing a CSV file."""

    file_type: Literal["csv"] = "csv"
    delimiter: str = ","
    encoding: str = "utf-8"

OutputExcelTable pydantic-model

Bases: BaseModel

Defines settings for writing an Excel file.

Show JSON schema:

{
  "description": "Defines settings for writing an Excel file.",
  "properties": {
    "file_type": {
      "const": "excel",
      "default": "excel",
      "title": "File Type",
      "type": "string"
    },
    "sheet_name": {
      "default": "Sheet1",
      "title": "Sheet Name",
      "type": "string"
    }
  },
  "title": "OutputExcelTable",
  "type": "object"
}

Fields:

• file_type (Literal['excel'])
• sheet_name (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputExcelTable(BaseModel):
    """Defines settings for writing an Excel file."""

    file_type: Literal["excel"] = "excel"
    sheet_name: str = "Sheet1"

OutputFieldConfig pydantic-model

Bases: BaseModel

Configuration for output field validation and transformation behavior.

Show JSON schema:

{
  "$defs": {
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Configuration for output field validation and transformation behavior.",
  "properties": {
    "enabled": {
      "default": false,
      "title": "Enabled",
      "type": "boolean"
    },
    "validation_mode_behavior": {
      "default": "select_only",
      "enum": [
        "add_missing",
        "add_missing_keep_extra",
        "raise_on_missing",
        "select_only"
      ],
      "title": "Validation Mode Behavior",
      "type": "string"
    },
    "fields": {
      "items": {
        "$ref": "#/$defs/OutputFieldInfo"
      },
      "title": "Fields",
      "type": "array"
    },
    "validate_data_types": {
      "default": false,
      "title": "Validate Data Types",
      "type": "boolean"
    }
  },
  "title": "OutputFieldConfig",
  "type": "object"
}

Fields:

• enabled (bool)
• validation_mode_behavior (Literal['add_missing', 'add_missing_keep_extra', 'raise_on_missing', 'select_only'])
• fields (list[OutputFieldInfo])
• validate_data_types (bool)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputFieldConfig(BaseModel):
    """Configuration for output field validation and transformation behavior."""

    enabled: bool = False
    validation_mode_behavior: Literal[
        "add_missing",  # Add missing fields with defaults, remove extra columns
        "add_missing_keep_extra",  # Add missing fields with defaults, keep all incoming columns
        "raise_on_missing",  # Raise error if any fields are missing
        "select_only",  # Select only specified fields, skip missing silently
    ] = "select_only"
    fields: list[OutputFieldInfo] = Field(default_factory=list)
    validate_data_types: bool = False  # Enable data type validation without casting

OutputFieldInfo pydantic-model

Bases: BaseModel

Field information with optional default value for output field configuration.

Show JSON schema:

{
  "description": "Field information with optional default value for output field configuration.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "data_type": {
      "default": "String",
      "enum": [
        "Int8",
        "Int16",
        "Int32",
        "Int64",
        "Int128",
        "UInt8",
        "UInt16",
        "UInt32",
        "UInt64",
        "UInt128",
        "Float16",
        "Float32",
        "Float64",
        "Decimal",
        "String",
        "Date",
        "Datetime",
        "Time",
        "Duration",
        "Boolean",
        "Binary",
        "List",
        "Struct",
        "Array",
        "Integer",
        "Double",
        "Utf8"
      ],
      "title": "Data Type",
      "type": "string"
    },
    "default_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Default Value"
    }
  },
  "required": [
    "name"
  ],
  "title": "OutputFieldInfo",
  "type": "object"
}

Fields:

• name (str)
• data_type (DataTypeStr)
• default_value (str | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputFieldInfo(BaseModel):
    """Field information with optional default value for output field configuration."""

    name: str
    data_type: DataTypeStr = "String"
    default_value: str | None = None  # Can be a literal value or expression

OutputIpcTable pydantic-model

Bases: BaseModel

Defines settings for writing an Arrow IPC/Feather file.

Show JSON schema:

{
  "description": "Defines settings for writing an Arrow IPC/Feather file.",
  "properties": {
    "file_type": {
      "const": "ipc",
      "default": "ipc",
      "title": "File Type",
      "type": "string"
    },
    "compression": {
      "default": "uncompressed",
      "enum": [
        "uncompressed",
        "lz4",
        "zstd"
      ],
      "title": "Compression",
      "type": "string"
    }
  },
  "title": "OutputIpcTable",
  "type": "object"
}

Fields:

• file_type (Literal['ipc'])
• compression (Literal['uncompressed', 'lz4', 'zstd'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputIpcTable(BaseModel):
    """Defines settings for writing an Arrow IPC/Feather file."""

    file_type: Literal["ipc"] = "ipc"
    compression: Literal["uncompressed", "lz4", "zstd"] = "uncompressed"

OutputNdjsonTable pydantic-model

Bases: BaseModel

Defines settings for writing a newline-delimited JSON file.

Show JSON schema:

{
  "description": "Defines settings for writing a newline-delimited JSON file.",
  "properties": {
    "file_type": {
      "const": "ndjson",
      "default": "ndjson",
      "title": "File Type",
      "type": "string"
    },
    "compression": {
      "default": "uncompressed",
      "enum": [
        "uncompressed",
        "gzip",
        "zstd"
      ],
      "title": "Compression",
      "type": "string"
    }
  },
  "title": "OutputNdjsonTable",
  "type": "object"
}

Fields:

• file_type (Literal['ndjson'])
• compression (Literal['uncompressed', 'gzip', 'zstd'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputNdjsonTable(BaseModel):
    """Defines settings for writing a newline-delimited JSON file."""

    file_type: Literal["ndjson"] = "ndjson"
    compression: Literal["uncompressed", "gzip", "zstd"] = "uncompressed"

OutputParquetTable pydantic-model

Bases: BaseModel

Defines settings for writing a Parquet file.

Show JSON schema:

{
  "description": "Defines settings for writing a Parquet file.",
  "properties": {
    "file_type": {
      "const": "parquet",
      "default": "parquet",
      "title": "File Type",
      "type": "string"
    },
    "compression": {
      "default": "zstd",
      "enum": [
        "lz4",
        "uncompressed",
        "snappy",
        "gzip",
        "brotli",
        "zstd"
      ],
      "title": "Compression",
      "type": "string"
    }
  },
  "title": "OutputParquetTable",
  "type": "object"
}

Fields:

• file_type (Literal['parquet'])
• compression (Literal['lz4', 'uncompressed', 'snappy', 'gzip', 'brotli', 'zstd'])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputParquetTable(BaseModel):
    """Defines settings for writing a Parquet file."""

    file_type: Literal["parquet"] = "parquet"
    # Polars default for write_parquet/sink_parquet
    compression: Literal["lz4", "uncompressed", "snappy", "gzip", "brotli", "zstd"] = "zstd"

OutputSettings pydantic-model

Bases: BaseModel

Defines the complete settings for an output node.

Show JSON schema:

{
  "$defs": {
    "OutputAvroTable": {
      "description": "Defines settings for writing an Avro file.",
      "properties": {
        "file_type": {
          "const": "avro",
          "default": "avro",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "snappy",
            "deflate"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputAvroTable",
      "type": "object"
    },
    "OutputCsvTable": {
      "description": "Defines settings for writing a CSV file.",
      "properties": {
        "file_type": {
          "const": "csv",
          "default": "csv",
          "title": "File Type",
          "type": "string"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        }
      },
      "title": "OutputCsvTable",
      "type": "object"
    },
    "OutputExcelTable": {
      "description": "Defines settings for writing an Excel file.",
      "properties": {
        "file_type": {
          "const": "excel",
          "default": "excel",
          "title": "File Type",
          "type": "string"
        },
        "sheet_name": {
          "default": "Sheet1",
          "title": "Sheet Name",
          "type": "string"
        }
      },
      "title": "OutputExcelTable",
      "type": "object"
    },
    "OutputIpcTable": {
      "description": "Defines settings for writing an Arrow IPC/Feather file.",
      "properties": {
        "file_type": {
          "const": "ipc",
          "default": "ipc",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "lz4",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputIpcTable",
      "type": "object"
    },
    "OutputNdjsonTable": {
      "description": "Defines settings for writing a newline-delimited JSON file.",
      "properties": {
        "file_type": {
          "const": "ndjson",
          "default": "ndjson",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "uncompressed",
          "enum": [
            "uncompressed",
            "gzip",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputNdjsonTable",
      "type": "object"
    },
    "OutputParquetTable": {
      "description": "Defines settings for writing a Parquet file.",
      "properties": {
        "file_type": {
          "const": "parquet",
          "default": "parquet",
          "title": "File Type",
          "type": "string"
        },
        "compression": {
          "default": "zstd",
          "enum": [
            "lz4",
            "uncompressed",
            "snappy",
            "gzip",
            "brotli",
            "zstd"
          ],
          "title": "Compression",
          "type": "string"
        }
      },
      "title": "OutputParquetTable",
      "type": "object"
    }
  },
  "description": "Defines the complete settings for an output node.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "directory": {
      "title": "Directory",
      "type": "string"
    },
    "file_type": {
      "title": "File Type",
      "type": "string"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Fields"
    },
    "write_mode": {
      "default": "overwrite",
      "title": "Write Mode",
      "type": "string"
    },
    "table_settings": {
      "discriminator": {
        "mapping": {
          "avro": "#/$defs/OutputAvroTable",
          "csv": "#/$defs/OutputCsvTable",
          "excel": "#/$defs/OutputExcelTable",
          "ipc": "#/$defs/OutputIpcTable",
          "ndjson": "#/$defs/OutputNdjsonTable",
          "parquet": "#/$defs/OutputParquetTable"
        },
        "propertyName": "file_type"
      },
      "oneOf": [
        {
          "$ref": "#/$defs/OutputCsvTable"
        },
        {
          "$ref": "#/$defs/OutputParquetTable"
        },
        {
          "$ref": "#/$defs/OutputExcelTable"
        },
        {
          "$ref": "#/$defs/OutputIpcTable"
        },
        {
          "$ref": "#/$defs/OutputNdjsonTable"
        },
        {
          "$ref": "#/$defs/OutputAvroTable"
        }
      ],
      "title": "Table Settings"
    },
    "abs_file_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Abs File Path"
    }
  },
  "required": [
    "name",
    "directory",
    "file_type",
    "table_settings"
  ],
  "title": "OutputSettings",
  "type": "object"
}

Fields:

• name (str)
• directory (str)
• file_type (str)
• fields (list[str] | None)
• write_mode (str)
• table_settings (OutputTableSettings)
• abs_file_path (str | None)

Validators:

• validate_table_settings → table_settings
• populate_abs_file_path

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class OutputSettings(BaseModel):
    """Defines the complete settings for an output node."""

    name: str
    directory: str
    file_type: str  # This drives which table_settings to use
    fields: list[str] | None = Field(default_factory=list)
    write_mode: str = "overwrite"
    table_settings: OutputTableSettings
    abs_file_path: str | None = None

    def to_yaml_dict(self) -> OutputSettingsYaml:
        """Converts the output settings to a dictionary suitable for YAML serialization."""
        result: OutputSettingsYaml = {
            "name": self.name,
            "directory": self.directory,
            "file_type": self.file_type,
            "write_mode": self.write_mode,
        }
        if self.abs_file_path:
            result["abs_file_path"] = self.abs_file_path
        if self.fields:
            result["fields"] = self.fields
        # Only include table_settings if it has non-default values beyond file_type
        ts_dict = self.table_settings.model_dump(exclude={"file_type"})
        # Drop compression when it equals the format default so YAML stays minimal
        # and the "omit table_settings when all-default" invariant still holds.
        compression_field = type(self.table_settings).model_fields.get("compression")
        if (
            "compression" in ts_dict
            and compression_field is not None
            and ts_dict["compression"] == compression_field.default
        ):
            ts_dict.pop("compression")
        if any(v for v in ts_dict.values()):
            result["table_settings"] = ts_dict
        return result

    @property
    def sheet_name(self) -> str | None:
        if self.file_type == "excel":
            return self.table_settings.sheet_name

    @property
    def delimiter(self) -> str | None:
        if self.file_type == "csv":
            return self.table_settings.delimiter

    @property
    def compression(self) -> str | None:
        return getattr(self.table_settings, "compression", None)

    @field_validator("table_settings", mode="before")
    @classmethod
    def validate_table_settings(cls, v, info: ValidationInfo):
        """Ensures table_settings matches the file_type."""
        if v is None:
            file_type = info.data.get("file_type", "csv")
            match file_type:
                case "csv":
                    return OutputCsvTable()
                case "parquet":
                    return OutputParquetTable()
                case "excel":
                    return OutputExcelTable()
                case "ipc":
                    return OutputIpcTable()
                case "ndjson":
                    return OutputNdjsonTable()
                case "avro":
                    return OutputAvroTable()
                case _:
                    return OutputCsvTable()

        if isinstance(v, dict) and "file_type" not in v:
            v["file_type"] = info.data.get("file_type", "csv")

        return v

    def set_absolute_filepath(self):
        """Resolves the output directory and name into an absolute path."""
        base_path = Path(self.directory)
        if not base_path.is_absolute():
            base_path = Path.cwd() / base_path
        if self.name and self.name not in base_path.name:
            base_path = base_path / self.name
        self.abs_file_path = str(base_path.resolve())

    @model_validator(mode="after")
    def populate_abs_file_path(self):
        """Ensures the absolute file path is populated after validation."""
        self.set_absolute_filepath()
        return self

populate_abs_file_path() pydantic-validator

Ensures the absolute file path is populated after validation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@model_validator(mode="after")
def populate_abs_file_path(self):
    """Ensures the absolute file path is populated after validation."""
    self.set_absolute_filepath()
    return self

set_absolute_filepath()

Resolves the output directory and name into an absolute path.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def set_absolute_filepath(self):
    """Resolves the output directory and name into an absolute path."""
    base_path = Path(self.directory)
    if not base_path.is_absolute():
        base_path = Path.cwd() / base_path
    if self.name and self.name not in base_path.name:
        base_path = base_path / self.name
    self.abs_file_path = str(base_path.resolve())

to_yaml_dict()

Converts the output settings to a dictionary suitable for YAML serialization.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_yaml_dict(self) -> OutputSettingsYaml:
    """Converts the output settings to a dictionary suitable for YAML serialization."""
    result: OutputSettingsYaml = {
        "name": self.name,
        "directory": self.directory,
        "file_type": self.file_type,
        "write_mode": self.write_mode,
    }
    if self.abs_file_path:
        result["abs_file_path"] = self.abs_file_path
    if self.fields:
        result["fields"] = self.fields
    # Only include table_settings if it has non-default values beyond file_type
    ts_dict = self.table_settings.model_dump(exclude={"file_type"})
    # Drop compression when it equals the format default so YAML stays minimal
    # and the "omit table_settings when all-default" invariant still holds.
    compression_field = type(self.table_settings).model_fields.get("compression")
    if (
        "compression" in ts_dict
        and compression_field is not None
        and ts_dict["compression"] == compression_field.default
    ):
        ts_dict.pop("compression")
    if any(v for v in ts_dict.values()):
        result["table_settings"] = ts_dict
    return result

validate_table_settings(v, info) pydantic-validator

Ensures table_settings matches the file_type.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("table_settings", mode="before")
@classmethod
def validate_table_settings(cls, v, info: ValidationInfo):
    """Ensures table_settings matches the file_type."""
    if v is None:
        file_type = info.data.get("file_type", "csv")
        match file_type:
            case "csv":
                return OutputCsvTable()
            case "parquet":
                return OutputParquetTable()
            case "excel":
                return OutputExcelTable()
            case "ipc":
                return OutputIpcTable()
            case "ndjson":
                return OutputNdjsonTable()
            case "avro":
                return OutputAvroTable()
            case _:
                return OutputCsvTable()

    if isinstance(v, dict) and "file_type" not in v:
        v["file_type"] = info.data.get("file_type", "csv")

    return v

PythonScriptInput pydantic-model

Bases: BaseModel

Settings for Python code execution on a kernel.

Show JSON schema:

{
  "$defs": {
    "NotebookCell": {
      "description": "A single cell in the notebook editor.\n\nNote: Cell output (stdout, display_outputs, errors) is handled entirely\non the frontend and is not persisted. Only id and code are stored.",
      "properties": {
        "id": {
          "title": "Id",
          "type": "string"
        },
        "code": {
          "default": "",
          "title": "Code",
          "type": "string"
        }
      },
      "required": [
        "id"
      ],
      "title": "NotebookCell",
      "type": "object"
    }
  },
  "description": "Settings for Python code execution on a kernel.",
  "properties": {
    "code": {
      "default": "",
      "title": "Code",
      "type": "string"
    },
    "kernel_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Kernel Id"
    },
    "cells": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/NotebookCell"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Cells"
    }
  },
  "title": "PythonScriptInput",
  "type": "object"
}

Fields:

• code (str)
• kernel_id (str | None)
• cells (list[NotebookCell] | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class PythonScriptInput(BaseModel):
    """Settings for Python code execution on a kernel."""

    code: str = ""
    kernel_id: str | None = None
    cells: list[NotebookCell] | None = None

RandomSplitGroup pydantic-model

Bases: BaseModel

A single output partition in a random split.

Show JSON schema:

{
  "description": "A single output partition in a random split.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "percentage": {
      "title": "Percentage",
      "type": "number"
    }
  },
  "required": [
    "name",
    "percentage"
  ],
  "title": "RandomSplitGroup",
  "type": "object"
}

Fields:

• name (str)
• percentage (float)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RandomSplitGroup(BaseModel):
    """A single output partition in a random split."""

    name: str
    percentage: float

RawData pydantic-model

Bases: BaseModel

Represents data in a raw, columnar format for manual input.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    }
  },
  "description": "Represents data in a raw, columnar format for manual input.",
  "properties": {
    "columns": {
      "description": "Schema in column order. The i-th MinimalFieldInfo describes the values in data[i].",
      "items": {
        "$ref": "#/$defs/MinimalFieldInfo"
      },
      "title": "Columns",
      "type": "array"
    },
    "data": {
      "description": "Columnar layout: data[i] is the list of values for columns[i], in column order. len(data) must equal len(columns); each inner list has the same length (one entry per row). For two rows of {name, age}, emit [[\"Alice\", \"Bob\"], [30, 25]] \u2014 NOT [[\"Alice\", 30], [\"Bob\", 25]]. Reading rows back is `data[col_idx][row_idx]`.",
      "items": {
        "items": {},
        "type": "array"
      },
      "title": "Data",
      "type": "array"
    }
  },
  "required": [
    "columns",
    "data"
  ],
  "title": "RawData",
  "type": "object"
}

Fields:

• columns (list[MinimalFieldInfo])
• data (list[list])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RawData(BaseModel):
    """Represents data in a raw, columnar format for manual input."""

    columns: list[MinimalFieldInfo] = Field(
        ...,
        description=("Schema in column order. The i-th MinimalFieldInfo describes the values " "in data[i]."),
    )
    data: list[list] = Field(
        ...,
        description=(
            "Columnar layout: data[i] is the list of values for columns[i], in column "
            "order. len(data) must equal len(columns); each inner list has the same "
            "length (one entry per row). For two rows of {name, age}, emit "
            '[["Alice", "Bob"], [30, 25]] — NOT [["Alice", 30], ["Bob", 25]]. '
            "Reading rows back is `data[col_idx][row_idx]`."
        ),
    )

    @staticmethod
    def _infer_data_type(column_values: list) -> str:
        # standardize_col_dtype leaves int/float columns mixed on purpose, so promote
        # them to Float64 — typing by the first value would truncate floats on cast.
        types = {type(v) for v in column_values if v is not None}
        if not types:
            return str(pl.String())
        if types == {int, float}:
            return str(pl.Float64())
        return str(pl.DataType.from_python(next(iter(types))))

    @classmethod
    def from_pylist(cls, pylist: list[dict]):
        """Creates a RawData object from a list of Python dictionaries."""
        if len(pylist) == 0:
            return cls(columns=[], data=[])
        pylist = ensure_similarity_dicts(pylist)
        values = [standardize_col_dtype([vv for vv in c]) for c in zip(*(r.values() for r in pylist), strict=False)]
        columns = [
            MinimalFieldInfo(name=name, data_type=cls._infer_data_type(column_values))
            for name, column_values in zip(pylist[0].keys(), values, strict=True)
        ]
        return cls(columns=columns, data=values)

    @classmethod
    def from_pydict(cls, pydict: dict[str, list]):
        """Creates a RawData object from a dictionary of lists."""
        if len(pydict) == 0:
            return cls(columns=[], data=[])
        values = [standardize_col_dtype(column_values) for column_values in pydict.values()]
        columns = [
            MinimalFieldInfo(name=name, data_type=cls._infer_data_type(column_values))
            for name, column_values in zip(pydict.keys(), values, strict=True)
        ]
        return cls(columns=columns, data=values)

    def to_pylist(self) -> list[dict]:
        """Converts the RawData object back into a list of Python dictionaries."""
        return [{c.name: self.data[ci][ri] for ci, c in enumerate(self.columns)} for ri in range(len(self.data[0]))]

columns pydantic-field

Schema in column order. The i-th MinimalFieldInfo describes the values in data[i].

data pydantic-field

Columnar layout: data[i] is the list of values for columns[i], in column order. len(data) must equal len(columns); each inner list has the same length (one entry per row). For two rows of {name, age}, emit [["Alice", "Bob"], [30, 25]] — NOT [["Alice", 30], ["Bob", 25]]. Reading rows back is data[col_idx][row_idx].

from_pydict(pydict) classmethod

Creates a RawData object from a dictionary of lists.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@classmethod
def from_pydict(cls, pydict: dict[str, list]):
    """Creates a RawData object from a dictionary of lists."""
    if len(pydict) == 0:
        return cls(columns=[], data=[])
    values = [standardize_col_dtype(column_values) for column_values in pydict.values()]
    columns = [
        MinimalFieldInfo(name=name, data_type=cls._infer_data_type(column_values))
        for name, column_values in zip(pydict.keys(), values, strict=True)
    ]
    return cls(columns=columns, data=values)

from_pylist(pylist) classmethod

Creates a RawData object from a list of Python dictionaries.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@classmethod
def from_pylist(cls, pylist: list[dict]):
    """Creates a RawData object from a list of Python dictionaries."""
    if len(pylist) == 0:
        return cls(columns=[], data=[])
    pylist = ensure_similarity_dicts(pylist)
    values = [standardize_col_dtype([vv for vv in c]) for c in zip(*(r.values() for r in pylist), strict=False)]
    columns = [
        MinimalFieldInfo(name=name, data_type=cls._infer_data_type(column_values))
        for name, column_values in zip(pylist[0].keys(), values, strict=True)
    ]
    return cls(columns=columns, data=values)

to_pylist()

Converts the RawData object back into a list of Python dictionaries.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def to_pylist(self) -> list[dict]:
    """Converts the RawData object back into a list of Python dictionaries."""
    return [{c.name: self.data[ci][ri] for ci, c in enumerate(self.columns)} for ri in range(len(self.data[0]))]

ReceivedTable pydantic-model

Bases: BaseModel

Model for defining a table received from an external source.

Show JSON schema:

{
  "$defs": {
    "InputAvroTable": {
      "description": "Defines settings for reading an Avro file.",
      "properties": {
        "file_type": {
          "const": "avro",
          "default": "avro",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputAvroTable",
      "type": "object"
    },
    "InputCsvTable": {
      "description": "Defines settings for reading a CSV file.",
      "properties": {
        "file_type": {
          "const": "csv",
          "default": "csv",
          "title": "File Type",
          "type": "string"
        },
        "reference": {
          "default": "",
          "title": "Reference",
          "type": "string"
        },
        "starting_from_line": {
          "default": 0,
          "title": "Starting From Line",
          "type": "integer"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        },
        "parquet_ref": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parquet Ref"
        },
        "row_delimiter": {
          "default": "\n",
          "title": "Row Delimiter",
          "type": "string"
        },
        "quote_char": {
          "default": "\"",
          "title": "Quote Char",
          "type": "string"
        },
        "infer_schema_length": {
          "default": 10000,
          "title": "Infer Schema Length",
          "type": "integer"
        },
        "infer_schema": {
          "default": true,
          "title": "Infer Schema",
          "type": "boolean"
        },
        "truncate_ragged_lines": {
          "default": false,
          "title": "Truncate Ragged Lines",
          "type": "boolean"
        },
        "ignore_errors": {
          "default": false,
          "title": "Ignore Errors",
          "type": "boolean"
        }
      },
      "title": "InputCsvTable",
      "type": "object"
    },
    "InputExcelTable": {
      "description": "Defines settings for reading an Excel file.",
      "properties": {
        "file_type": {
          "const": "excel",
          "default": "excel",
          "title": "File Type",
          "type": "string"
        },
        "sheet_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Sheet Name"
        },
        "start_row": {
          "default": 0,
          "title": "Start Row",
          "type": "integer"
        },
        "start_column": {
          "default": 0,
          "title": "Start Column",
          "type": "integer"
        },
        "end_row": {
          "default": 0,
          "title": "End Row",
          "type": "integer"
        },
        "end_column": {
          "default": 0,
          "title": "End Column",
          "type": "integer"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "type_inference": {
          "default": false,
          "title": "Type Inference",
          "type": "boolean"
        }
      },
      "title": "InputExcelTable",
      "type": "object"
    },
    "InputIpcTable": {
      "description": "Defines settings for reading an Arrow IPC/Feather file.",
      "properties": {
        "file_type": {
          "const": "ipc",
          "default": "ipc",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputIpcTable",
      "type": "object"
    },
    "InputJsonTable": {
      "description": "Defines settings for reading a JSON file.",
      "properties": {
        "file_type": {
          "const": "json",
          "default": "json",
          "title": "File Type",
          "type": "string"
        },
        "reference": {
          "default": "",
          "title": "Reference",
          "type": "string"
        },
        "starting_from_line": {
          "default": 0,
          "title": "Starting From Line",
          "type": "integer"
        },
        "delimiter": {
          "default": ",",
          "title": "Delimiter",
          "type": "string"
        },
        "has_headers": {
          "default": true,
          "title": "Has Headers",
          "type": "boolean"
        },
        "encoding": {
          "default": "utf-8",
          "title": "Encoding",
          "type": "string"
        },
        "parquet_ref": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parquet Ref"
        },
        "row_delimiter": {
          "default": "\n",
          "title": "Row Delimiter",
          "type": "string"
        },
        "quote_char": {
          "default": "\"",
          "title": "Quote Char",
          "type": "string"
        },
        "infer_schema_length": {
          "default": 10000,
          "title": "Infer Schema Length",
          "type": "integer"
        },
        "infer_schema": {
          "default": true,
          "title": "Infer Schema",
          "type": "boolean"
        },
        "truncate_ragged_lines": {
          "default": false,
          "title": "Truncate Ragged Lines",
          "type": "boolean"
        },
        "ignore_errors": {
          "default": false,
          "title": "Ignore Errors",
          "type": "boolean"
        }
      },
      "title": "InputJsonTable",
      "type": "object"
    },
    "InputNdjsonTable": {
      "description": "Defines settings for reading a newline-delimited JSON file.",
      "properties": {
        "file_type": {
          "const": "ndjson",
          "default": "ndjson",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputNdjsonTable",
      "type": "object"
    },
    "InputParquetTable": {
      "description": "Defines settings for reading a Parquet file.",
      "properties": {
        "file_type": {
          "const": "parquet",
          "default": "parquet",
          "title": "File Type",
          "type": "string"
        }
      },
      "title": "InputParquetTable",
      "type": "object"
    },
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    }
  },
  "description": "Model for defining a table received from an external source.",
  "properties": {
    "id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Id"
    },
    "name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Name"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "directory": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Directory"
    },
    "analysis_file_available": {
      "default": false,
      "title": "Analysis File Available",
      "type": "boolean"
    },
    "status": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Status"
    },
    "fields": {
      "items": {
        "$ref": "#/$defs/MinimalFieldInfo"
      },
      "title": "Fields",
      "type": "array"
    },
    "abs_file_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Abs File Path"
    },
    "file_type": {
      "enum": [
        "csv",
        "json",
        "parquet",
        "excel",
        "ipc",
        "ndjson",
        "avro"
      ],
      "title": "File Type",
      "type": "string"
    },
    "table_settings": {
      "discriminator": {
        "mapping": {
          "avro": "#/$defs/InputAvroTable",
          "csv": "#/$defs/InputCsvTable",
          "excel": "#/$defs/InputExcelTable",
          "ipc": "#/$defs/InputIpcTable",
          "json": "#/$defs/InputJsonTable",
          "ndjson": "#/$defs/InputNdjsonTable",
          "parquet": "#/$defs/InputParquetTable"
        },
        "propertyName": "file_type"
      },
      "oneOf": [
        {
          "$ref": "#/$defs/InputCsvTable"
        },
        {
          "$ref": "#/$defs/InputJsonTable"
        },
        {
          "$ref": "#/$defs/InputParquetTable"
        },
        {
          "$ref": "#/$defs/InputExcelTable"
        },
        {
          "$ref": "#/$defs/InputIpcTable"
        },
        {
          "$ref": "#/$defs/InputNdjsonTable"
        },
        {
          "$ref": "#/$defs/InputAvroTable"
        }
      ],
      "title": "Table Settings"
    }
  },
  "required": [
    "path",
    "file_type",
    "table_settings"
  ],
  "title": "ReceivedTable",
  "type": "object"
}

Fields:

• id (int | None)
• name (str | None)
• path (str)
• directory (str | None)
• analysis_file_available (bool)
• status (str | None)
• fields (list[MinimalFieldInfo])
• abs_file_path (str | None)
• file_type (Literal['csv', 'json', 'parquet', 'excel', 'ipc', 'ndjson', 'avro'])
• table_settings (InputTableSettings)

Validators:

• set_default_table_settings
• populate_abs_file_path

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class ReceivedTable(BaseModel):
    """Model for defining a table received from an external source."""

    # Metadata fields
    id: int | None = None
    name: str | None = None
    path: str  # This can be an absolute or relative path
    directory: str | None = None
    analysis_file_available: bool = False
    status: str | None = None
    fields: list[MinimalFieldInfo] = Field(default_factory=list)
    abs_file_path: str | None = None

    file_type: Literal["csv", "json", "parquet", "excel", "ipc", "ndjson", "avro"]

    table_settings: InputTableSettings

    @classmethod
    def create_from_path(
        cls, path: str, file_type: Literal["csv", "json", "parquet", "excel", "ipc", "ndjson", "avro"] = "csv"
    ):
        """Creates an instance from a file path string."""
        filename = Path(path).name

        settings_map = {
            "csv": InputCsvTable(),
            "json": InputJsonTable(),
            "parquet": InputParquetTable(),
            "excel": InputExcelTable(),
            "ipc": InputIpcTable(),
            "ndjson": InputNdjsonTable(),
            "avro": InputAvroTable(),
        }

        return cls(
            name=filename, path=path, file_type=file_type, table_settings=settings_map.get(file_type, InputCsvTable())
        )

    @property
    def file_path(self) -> str:
        """Constructs the full file path from the directory and name."""
        if self.name and self.name not in self.path:
            return os.path.join(self.path, self.name)
        else:
            return self.path

    def set_absolute_filepath(self):
        """Resolves the path to an absolute file path."""
        if is_url(self.path):
            self.abs_file_path = self.path
            return
        base_path = Path(self.path).expanduser()
        if not base_path.is_absolute():
            base_path = Path.cwd() / base_path
        if self.name and self.name not in base_path.name:
            base_path = base_path / self.name
        self.abs_file_path = str(base_path.resolve())

    @model_validator(mode="before")
    @classmethod
    def set_default_table_settings(cls, data):
        """Create default table_settings based on file_type if not provided."""
        if isinstance(data, dict):
            if "table_settings" not in data or data["table_settings"] is None:
                data["table_settings"] = {}

            if isinstance(data["table_settings"], dict) and "file_type" not in data["table_settings"]:
                data["table_settings"]["file_type"] = data.get("file_type", "csv")
        return data

    @model_validator(mode="after")
    def populate_abs_file_path(self):
        """Ensures the absolute file path is populated after validation."""
        if not self.abs_file_path:
            self.set_absolute_filepath()
        return self

file_path property

Constructs the full file path from the directory and name.

create_from_path(path, file_type='csv') classmethod

Creates an instance from a file path string.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@classmethod
def create_from_path(
    cls, path: str, file_type: Literal["csv", "json", "parquet", "excel", "ipc", "ndjson", "avro"] = "csv"
):
    """Creates an instance from a file path string."""
    filename = Path(path).name

    settings_map = {
        "csv": InputCsvTable(),
        "json": InputJsonTable(),
        "parquet": InputParquetTable(),
        "excel": InputExcelTable(),
        "ipc": InputIpcTable(),
        "ndjson": InputNdjsonTable(),
        "avro": InputAvroTable(),
    }

    return cls(
        name=filename, path=path, file_type=file_type, table_settings=settings_map.get(file_type, InputCsvTable())
    )

populate_abs_file_path() pydantic-validator

Ensures the absolute file path is populated after validation.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@model_validator(mode="after")
def populate_abs_file_path(self):
    """Ensures the absolute file path is populated after validation."""
    if not self.abs_file_path:
        self.set_absolute_filepath()
    return self

set_absolute_filepath()

Resolves the path to an absolute file path.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def set_absolute_filepath(self):
    """Resolves the path to an absolute file path."""
    if is_url(self.path):
        self.abs_file_path = self.path
        return
    base_path = Path(self.path).expanduser()
    if not base_path.is_absolute():
        base_path = Path.cwd() / base_path
    if self.name and self.name not in base_path.name:
        base_path = base_path / self.name
    self.abs_file_path = str(base_path.resolve())

set_default_table_settings(data) pydantic-validator

Create default table_settings based on file_type if not provided.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@model_validator(mode="before")
@classmethod
def set_default_table_settings(cls, data):
    """Create default table_settings based on file_type if not provided."""
    if isinstance(data, dict):
        if "table_settings" not in data or data["table_settings"] is None:
            data["table_settings"] = {}

        if isinstance(data["table_settings"], dict) and "file_type" not in data["table_settings"]:
            data["table_settings"]["file_type"] = data.get("file_type", "csv")
    return data

RemoveItem pydantic-model

Bases: BaseModel

Represents a single item to be removed from a directory or list.

Show JSON schema:

{
  "description": "Represents a single item to be removed from a directory or list.",
  "properties": {
    "path": {
      "title": "Path",
      "type": "string"
    },
    "id": {
      "default": -1,
      "title": "Id",
      "type": "integer"
    }
  },
  "required": [
    "path"
  ],
  "title": "RemoveItem",
  "type": "object"
}

Fields:

• path (str)
• id (int)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RemoveItem(BaseModel):
    """Represents a single item to be removed from a directory or list."""

    path: str
    id: int = -1

RemoveItemsInput pydantic-model

Bases: BaseModel

Defines a list of items to be removed.

Show JSON schema:

{
  "$defs": {
    "RemoveItem": {
      "description": "Represents a single item to be removed from a directory or list.",
      "properties": {
        "path": {
          "title": "Path",
          "type": "string"
        },
        "id": {
          "default": -1,
          "title": "Id",
          "type": "integer"
        }
      },
      "required": [
        "path"
      ],
      "title": "RemoveItem",
      "type": "object"
    }
  },
  "description": "Defines a list of items to be removed.",
  "properties": {
    "paths": {
      "items": {
        "$ref": "#/$defs/RemoveItem"
      },
      "title": "Paths",
      "type": "array"
    },
    "source_path": {
      "title": "Source Path",
      "type": "string"
    }
  },
  "required": [
    "paths",
    "source_path"
  ],
  "title": "RemoveItemsInput",
  "type": "object"
}

Fields:

• paths (list[RemoveItem])
• source_path (str)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RemoveItemsInput(BaseModel):
    """Defines a list of items to be removed."""

    paths: list[RemoveItem]
    source_path: str

RestApiAuthSettings pydantic-model

Bases: BaseModel

Authentication settings for a REST API reader node.

The credential (API key / bearer token / basic password, per auth_type) is NOT stored inline. secret_name references a secret in the user's secret store — created once via the Secrets manager and reusable across nodes — mirroring how the database reader references a stored password. The .flowfile persists only the reference name, never the credential itself.

secret is an optional inline plaintext for programmatic use (flowfile_frame.read_api); it is encrypted with the master key and cleared, never persisted.

Show JSON schema:

{
  "description": "Authentication settings for a REST API reader node.\n\nThe credential (API key / bearer token / basic password, per ``auth_type``)\nis NOT stored inline. ``secret_name`` references a secret in the user's\nsecret store \u2014 created once via the Secrets manager and reusable across\nnodes \u2014 mirroring how the database reader references a stored password. The\n``.flowfile`` persists only the reference name, never the credential itself.\n\n``secret`` is an optional inline plaintext for programmatic use\n(``flowfile_frame.read_api``); it is encrypted with the master key and\ncleared, never persisted.",
  "properties": {
    "auth_type": {
      "default": "none",
      "enum": [
        "none",
        "api_key",
        "bearer",
        "basic"
      ],
      "title": "Auth Type",
      "type": "string"
    },
    "api_key_name": {
      "default": "X-API-Key",
      "title": "Api Key Name",
      "type": "string"
    },
    "api_key_location": {
      "default": "header",
      "enum": [
        "header",
        "query"
      ],
      "title": "Api Key Location",
      "type": "string"
    },
    "basic_username": {
      "default": "",
      "title": "Basic Username",
      "type": "string"
    },
    "secret_name": {
      "default": "",
      "title": "Secret Name",
      "type": "string"
    },
    "secret": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Secret"
    }
  },
  "title": "RestApiAuthSettings",
  "type": "object"
}

Fields:

• auth_type (Literal['none', 'api_key', 'bearer', 'basic'])
• api_key_name (str)
• api_key_location (Literal['header', 'query'])
• basic_username (str)
• secret_name (str)
• secret (str | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RestApiAuthSettings(BaseModel):
    """Authentication settings for a REST API reader node.

    The credential (API key / bearer token / basic password, per ``auth_type``)
    is NOT stored inline. ``secret_name`` references a secret in the user's
    secret store — created once via the Secrets manager and reusable across
    nodes — mirroring how the database reader references a stored password. The
    ``.flowfile`` persists only the reference name, never the credential itself.

    ``secret`` is an optional inline plaintext for programmatic use
    (``flowfile_frame.read_api``); it is encrypted with the master key and
    cleared, never persisted.
    """

    auth_type: Literal["none", "api_key", "bearer", "basic"] = "none"
    # API key placement
    api_key_name: str = "X-API-Key"
    api_key_location: Literal["header", "query"] = "header"
    # Basic auth username (not secret)
    basic_username: str = ""
    # Reference to a stored secret holding the credential value (empty = none).
    secret_name: str = ""
    # Optional inline plaintext for programmatic use; encrypted, never persisted.
    secret: str | None = None

RestApiPaginationSettings pydantic-model

Bases: BaseModel

Pagination strategy and parameters for a REST API reader node.

Show JSON schema:

{
  "description": "Pagination strategy and parameters for a REST API reader node.",
  "properties": {
    "pagination_type": {
      "default": "none",
      "enum": [
        "none",
        "offset",
        "page",
        "cursor"
      ],
      "title": "Pagination Type",
      "type": "string"
    },
    "offset_param": {
      "default": "offset",
      "title": "Offset Param",
      "type": "string"
    },
    "limit_param": {
      "default": "limit",
      "title": "Limit Param",
      "type": "string"
    },
    "page_size": {
      "default": 100,
      "title": "Page Size",
      "type": "integer"
    },
    "page_param": {
      "default": "page",
      "title": "Page Param",
      "type": "string"
    },
    "start_page": {
      "default": 1,
      "title": "Start Page",
      "type": "integer"
    },
    "cursor_param": {
      "default": "cursor",
      "title": "Cursor Param",
      "type": "string"
    },
    "cursor_location": {
      "default": "body",
      "enum": [
        "body",
        "header"
      ],
      "title": "Cursor Location",
      "type": "string"
    },
    "cursor_response_path": {
      "default": "",
      "title": "Cursor Response Path",
      "type": "string"
    },
    "initial_cursor": {
      "default": "",
      "title": "Initial Cursor",
      "type": "string"
    },
    "max_pages": {
      "default": 1000,
      "title": "Max Pages",
      "type": "integer"
    },
    "max_records": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Max Records"
    },
    "page_delay_seconds": {
      "default": 0.0,
      "title": "Page Delay Seconds",
      "type": "number"
    }
  },
  "title": "RestApiPaginationSettings",
  "type": "object"
}

Fields:

• pagination_type (Literal['none', 'offset', 'page', 'cursor'])
• offset_param (str)
• limit_param (str)
• page_size (int)
• page_param (str)
• start_page (int)
• cursor_param (str)
• cursor_location (Literal['body', 'header'])
• cursor_response_path (str)
• initial_cursor (str)
• max_pages (int)
• max_records (int | None)
• page_delay_seconds (float)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RestApiPaginationSettings(BaseModel):
    """Pagination strategy and parameters for a REST API reader node."""

    pagination_type: Literal["none", "offset", "page", "cursor"] = "none"
    # offset / limit
    offset_param: str = "offset"
    limit_param: str = "limit"
    page_size: int = 100
    # page number
    page_param: str = "page"
    start_page: int = 1
    # cursor / next-page token
    cursor_param: str = "cursor"
    cursor_location: Literal["body", "header"] = "body"
    cursor_response_path: str = ""
    initial_cursor: str = ""
    # safety caps
    max_pages: int = 1000
    max_records: int | None = None
    page_delay_seconds: float = 0.0

RestApiSettings pydantic-model

Bases: BaseModel

UI settings for a REST API reader node.

Secrets are stored inline but encrypted (see RestApiAuthSettings). JSON is the only supported response format; record_path is a dot-path that locates the record array within the response body (empty = top-level).

Show JSON schema:

{
  "$defs": {
    "RestApiAuthSettings": {
      "description": "Authentication settings for a REST API reader node.\n\nThe credential (API key / bearer token / basic password, per ``auth_type``)\nis NOT stored inline. ``secret_name`` references a secret in the user's\nsecret store \u2014 created once via the Secrets manager and reusable across\nnodes \u2014 mirroring how the database reader references a stored password. The\n``.flowfile`` persists only the reference name, never the credential itself.\n\n``secret`` is an optional inline plaintext for programmatic use\n(``flowfile_frame.read_api``); it is encrypted with the master key and\ncleared, never persisted.",
      "properties": {
        "auth_type": {
          "default": "none",
          "enum": [
            "none",
            "api_key",
            "bearer",
            "basic"
          ],
          "title": "Auth Type",
          "type": "string"
        },
        "api_key_name": {
          "default": "X-API-Key",
          "title": "Api Key Name",
          "type": "string"
        },
        "api_key_location": {
          "default": "header",
          "enum": [
            "header",
            "query"
          ],
          "title": "Api Key Location",
          "type": "string"
        },
        "basic_username": {
          "default": "",
          "title": "Basic Username",
          "type": "string"
        },
        "secret_name": {
          "default": "",
          "title": "Secret Name",
          "type": "string"
        },
        "secret": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Secret"
        }
      },
      "title": "RestApiAuthSettings",
      "type": "object"
    },
    "RestApiPaginationSettings": {
      "description": "Pagination strategy and parameters for a REST API reader node.",
      "properties": {
        "pagination_type": {
          "default": "none",
          "enum": [
            "none",
            "offset",
            "page",
            "cursor"
          ],
          "title": "Pagination Type",
          "type": "string"
        },
        "offset_param": {
          "default": "offset",
          "title": "Offset Param",
          "type": "string"
        },
        "limit_param": {
          "default": "limit",
          "title": "Limit Param",
          "type": "string"
        },
        "page_size": {
          "default": 100,
          "title": "Page Size",
          "type": "integer"
        },
        "page_param": {
          "default": "page",
          "title": "Page Param",
          "type": "string"
        },
        "start_page": {
          "default": 1,
          "title": "Start Page",
          "type": "integer"
        },
        "cursor_param": {
          "default": "cursor",
          "title": "Cursor Param",
          "type": "string"
        },
        "cursor_location": {
          "default": "body",
          "enum": [
            "body",
            "header"
          ],
          "title": "Cursor Location",
          "type": "string"
        },
        "cursor_response_path": {
          "default": "",
          "title": "Cursor Response Path",
          "type": "string"
        },
        "initial_cursor": {
          "default": "",
          "title": "Initial Cursor",
          "type": "string"
        },
        "max_pages": {
          "default": 1000,
          "title": "Max Pages",
          "type": "integer"
        },
        "max_records": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Max Records"
        },
        "page_delay_seconds": {
          "default": 0.0,
          "title": "Page Delay Seconds",
          "type": "number"
        }
      },
      "title": "RestApiPaginationSettings",
      "type": "object"
    }
  },
  "description": "UI settings for a REST API reader node.\n\nSecrets are stored inline but encrypted (see ``RestApiAuthSettings``). JSON\nis the only supported response format; ``record_path`` is a dot-path that\nlocates the record array within the response body (empty = top-level).",
  "properties": {
    "url": {
      "default": "",
      "title": "Url",
      "type": "string"
    },
    "method": {
      "default": "GET",
      "enum": [
        "GET",
        "POST"
      ],
      "title": "Method",
      "type": "string"
    },
    "headers": {
      "additionalProperties": {
        "type": "string"
      },
      "title": "Headers",
      "type": "object"
    },
    "query_params": {
      "additionalProperties": {
        "type": "string"
      },
      "title": "Query Params",
      "type": "object"
    },
    "json_body": {
      "anyOf": [
        {},
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Json Body"
    },
    "auth": {
      "$ref": "#/$defs/RestApiAuthSettings"
    },
    "pagination": {
      "$ref": "#/$defs/RestApiPaginationSettings"
    },
    "record_path": {
      "default": "",
      "title": "Record Path",
      "type": "string"
    },
    "timeout_seconds": {
      "default": 30.0,
      "title": "Timeout Seconds",
      "type": "number"
    },
    "max_retries": {
      "default": 3,
      "title": "Max Retries",
      "type": "integer"
    }
  },
  "title": "RestApiSettings",
  "type": "object"
}

Fields:

• url (str)
• method (Literal['GET', 'POST'])
• headers (dict[str, str])
• query_params (dict[str, str])
• json_body (Any | None)
• auth (RestApiAuthSettings)
• pagination (RestApiPaginationSettings)
• record_path (str)
• timeout_seconds (float)
• max_retries (int)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RestApiSettings(BaseModel):
    """UI settings for a REST API reader node.

    Secrets are stored inline but encrypted (see ``RestApiAuthSettings``). JSON
    is the only supported response format; ``record_path`` is a dot-path that
    locates the record array within the response body (empty = top-level).
    """

    url: str = ""
    method: Literal["GET", "POST"] = "GET"
    headers: dict[str, str] = Field(default_factory=dict)
    query_params: dict[str, str] = Field(default_factory=dict)
    json_body: Any | None = None
    auth: RestApiAuthSettings = Field(default_factory=RestApiAuthSettings)
    pagination: RestApiPaginationSettings = Field(default_factory=RestApiPaginationSettings)
    record_path: str = ""
    timeout_seconds: float = 30.0
    max_retries: int = 3

RunFlowParameterBinding pydantic-model

Bases: BaseModel

How one subflow parameter gets its value for a run_flow execution.

Show JSON schema:

{
  "description": "How one subflow parameter gets its value for a run_flow execution.",
  "properties": {
    "parameter_name": {
      "title": "Parameter Name",
      "type": "string"
    },
    "source": {
      "default": "default",
      "enum": [
        "default",
        "constant",
        "column"
      ],
      "title": "Source",
      "type": "string"
    },
    "constant_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Constant Value"
    },
    "column_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Column Name"
    }
  },
  "required": [
    "parameter_name"
  ],
  "title": "RunFlowParameterBinding",
  "type": "object"
}

Fields:

• parameter_name (str)
• source (Literal['default', 'constant', 'column'])
• constant_value (str | None)
• column_name (str | None)

Validators:

• _validate_source_value

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class RunFlowParameterBinding(BaseModel):
    """How one subflow parameter gets its value for a run_flow execution."""

    parameter_name: str
    source: Literal["default", "constant", "column"] = "default"
    constant_value: str | None = None
    column_name: str | None = None

    @model_validator(mode="after")
    def _validate_source_value(self) -> "RunFlowParameterBinding":
        if self.source == "constant" and self.constant_value is None:
            raise ValueError(f"parameter '{self.parameter_name}': constant binding requires constant_value")
        if self.source == "column" and not self.column_name:
            raise ValueError(f"parameter '{self.parameter_name}': column binding requires column_name")
        return self

SampleUsers pydantic-model

Bases: ExternalSource

Settings for generating a sample dataset of users.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for generating a sample dataset of users.",
  "properties": {
    "orientation": {
      "default": "row",
      "title": "Orientation",
      "type": "string"
    },
    "fields": {
      "anyOf": [
        {
          "items": {
            "$ref": "#/$defs/MinimalFieldInfo"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Fields"
    },
    "SAMPLE_USERS": {
      "title": "Sample Users",
      "type": "boolean"
    },
    "class_name": {
      "default": "sample_users",
      "title": "Class Name",
      "type": "string"
    },
    "size": {
      "default": 100,
      "title": "Size",
      "type": "integer"
    }
  },
  "required": [
    "SAMPLE_USERS"
  ],
  "title": "SampleUsers",
  "type": "object"
}

Fields:

• orientation (str)
• fields (list[MinimalFieldInfo] | None)
• SAMPLE_USERS (bool)
• class_name (str)
• size (int)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class SampleUsers(ExternalSource):
    """Settings for generating a sample dataset of users."""

    SAMPLE_USERS: bool
    class_name: str = "sample_users"
    size: int = 100

Scd2Settings pydantic-model

Bases: BaseModel

Slowly-changing-dimension type 2 configuration for a catalog write.

The business key is CatalogWriteSettings.merge_keys — this block only carries the change-detection scope and the names of the four generated columns. It is persisted verbatim onto the catalog table record (CatalogTable.scd2_config) so a reader can filter history without ever reading a writer node's settings.

Show JSON schema:

{
  "description": "Slowly-changing-dimension type 2 configuration for a catalog write.\n\nThe business key is ``CatalogWriteSettings.merge_keys`` \u2014 this block only carries the\nchange-detection scope and the names of the four generated columns. It is persisted verbatim\nonto the catalog table record (``CatalogTable.scd2_config``) so a reader can filter history\nwithout ever reading a writer node's settings.",
  "properties": {
    "compare_columns": {
      "items": {
        "type": "string"
      },
      "title": "Compare Columns",
      "type": "array"
    },
    "full_snapshot": {
      "default": false,
      "title": "Full Snapshot",
      "type": "boolean"
    },
    "partition_on_current": {
      "default": true,
      "title": "Partition On Current",
      "type": "boolean"
    },
    "surrogate_key_column": {
      "default": "sk",
      "title": "Surrogate Key Column",
      "type": "string"
    },
    "valid_from_column": {
      "default": "valid_from",
      "title": "Valid From Column",
      "type": "string"
    },
    "valid_to_column": {
      "default": "valid_to",
      "title": "Valid To Column",
      "type": "string"
    },
    "is_current_column": {
      "default": "is_current",
      "title": "Is Current Column",
      "type": "string"
    }
  },
  "title": "Scd2Settings",
  "type": "object"
}

Fields:

• compare_columns (list[str])
• full_snapshot (bool)
• partition_on_current (bool)
• surrogate_key_column (str)
• valid_from_column (str)
• valid_to_column (str)
• is_current_column (str)

Validators:

• _validate_scd2_columns

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class Scd2Settings(BaseModel):
    """Slowly-changing-dimension type 2 configuration for a catalog write.

    The business key is ``CatalogWriteSettings.merge_keys`` — this block only carries the
    change-detection scope and the names of the four generated columns. It is persisted verbatim
    onto the catalog table record (``CatalogTable.scd2_config``) so a reader can filter history
    without ever reading a writer node's settings.
    """

    compare_columns: list[str] = Field(default_factory=list)  # empty => every non-key, non-system column
    full_snapshot: bool = False  # True => current keys absent from the input get end-dated
    partition_on_current: bool = True  # partition new tables by the is-current column (creation-time)
    surrogate_key_column: str = "sk"
    valid_from_column: str = "valid_from"
    valid_to_column: str = "valid_to"
    is_current_column: str = "is_current"

    @property
    def system_columns(self) -> list[str]:
        """The four generated column names, in the primitive's canonical order."""
        return [
            self.surrogate_key_column,
            self.valid_from_column,
            self.valid_to_column,
            self.is_current_column,
        ]

    @model_validator(mode="after")
    def _validate_scd2_columns(self) -> "Scd2Settings":
        cols = self.system_columns
        if any((not c) or c != c.strip() for c in cols):
            raise ValueError("SCD2 column names must be non-empty and free of surrounding whitespace")
        # Server-side mirror of the writer form's CATALOG_NAME_PATTERN: these four names are
        # generated, not data, and they are spliced into the Delta merge's SQL identifiers.
        illegal = [c for c in cols if not _SCD2_COLUMN_NAME_RE.match(c)]
        if illegal:
            raise ValueError(
                f"SCD2 column name(s) {illegal} may only contain letters, digits, '_' and '-'. "
                f"Rename the generated column(s) in the catalog writer settings."
            )
        if len(set(cols)) != len(cols):
            raise ValueError(f"SCD2 column names must be unique, got {cols}")
        if len(set(self.compare_columns)) != len(self.compare_columns):
            raise ValueError("compare_columns must not contain duplicates")
        overlap = sorted(set(self.compare_columns) & set(cols))
        if overlap:
            raise ValueError(f"compare_columns may not name SCD2 system columns: {overlap}")
        return self

system_columns property

The four generated column names, in the primitive's canonical order.

SubflowReference pydantic-model

Bases: BaseModel

Reference to a catalog-registered flow.

registration_id is the primary reference; flow_uuid is stamped server-side and used to repair a dangling id; flow_path is display-only.

Show JSON schema:

{
  "description": "Reference to a catalog-registered flow.\n\n``registration_id`` is the primary reference; ``flow_uuid`` is stamped\nserver-side and used to repair a dangling id; ``flow_path`` is display-only.",
  "properties": {
    "registration_id": {
      "title": "Registration Id",
      "type": "integer"
    },
    "flow_uuid": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Flow Uuid"
    },
    "flow_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Flow Path"
    }
  },
  "required": [
    "registration_id"
  ],
  "title": "SubflowReference",
  "type": "object"
}

Fields:

• registration_id (int)
• flow_uuid (str | None)
• flow_path (str | None)

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class SubflowReference(BaseModel):
    """Reference to a catalog-registered flow.

    ``registration_id`` is the primary reference; ``flow_uuid`` is stamped
    server-side and used to repair a dangling id; ``flow_path`` is display-only.
    """

    registration_id: int
    flow_uuid: str | None = None
    flow_path: str | None = None

TrainModelSettings pydantic-model

Bases: BaseModel

Settings payload for the Train Model node.

params is a flat dict so the form-driven hyperparameter UI doesn't need a discriminated union — the worker validates against the algorithm-specific Pydantic class via shared.ml.trainers.get_trainer(model_type).params_class.

The trained model is always written to a flow-scoped path keyed off this node's id so downstream Apply Model nodes in the same flow can read it without first publishing to the catalog. Set publish_to_catalog=True to additionally store the artifact in the catalog (with a stable cross-run name + version).

Show JSON schema:

{
  "description": "Settings payload for the Train Model node.\n\n``params`` is a flat dict so the form-driven hyperparameter UI doesn't need\na discriminated union \u2014 the worker validates against the algorithm-specific\nPydantic class via ``shared.ml.trainers.get_trainer(model_type).params_class``.\n\nThe trained model is always written to a flow-scoped path keyed off this\nnode's id so downstream Apply Model nodes in the same flow can read it\nwithout first publishing to the catalog. Set ``publish_to_catalog=True``\nto additionally store the artifact in the catalog (with a stable\ncross-run name + version).",
  "properties": {
    "target_column": {
      "default": "",
      "title": "Target Column",
      "type": "string"
    },
    "feature_columns": {
      "items": {
        "type": "string"
      },
      "title": "Feature Columns",
      "type": "array"
    },
    "model_type": {
      "default": "linear_regression",
      "title": "Model Type",
      "type": "string"
    },
    "params": {
      "additionalProperties": true,
      "title": "Params",
      "type": "object"
    },
    "publish_to_catalog": {
      "default": false,
      "title": "Publish To Catalog",
      "type": "boolean"
    },
    "model_name": {
      "default": "",
      "title": "Model Name",
      "type": "string"
    },
    "namespace_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Id"
    },
    "namespace_full_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Namespace Full Name"
    },
    "catalog_description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Catalog Description"
    },
    "catalog_tags": {
      "items": {
        "type": "string"
      },
      "title": "Catalog Tags",
      "type": "array"
    }
  },
  "title": "TrainModelSettings",
  "type": "object"
}

Config:

• protected_namespaces: ()

Fields:

• target_column (str)
• feature_columns (list[str])
• model_type (str)
• params (dict[str, Any])
• publish_to_catalog (bool)
• model_name (str)
• namespace_id (int | None)
• namespace_full_name (str | None)
• catalog_description (str | None)
• catalog_tags (list[str])

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class TrainModelSettings(BaseModel):
    """Settings payload for the Train Model node.

    ``params`` is a flat dict so the form-driven hyperparameter UI doesn't need
    a discriminated union — the worker validates against the algorithm-specific
    Pydantic class via ``shared.ml.trainers.get_trainer(model_type).params_class``.

    The trained model is always written to a flow-scoped path keyed off this
    node's id so downstream Apply Model nodes in the same flow can read it
    without first publishing to the catalog. Set ``publish_to_catalog=True``
    to additionally store the artifact in the catalog (with a stable
    cross-run name + version).
    """

    model_config = ConfigDict(protected_namespaces=())

    target_column: str = ""
    feature_columns: list[str] = Field(default_factory=list)
    model_type: str = "linear_regression"
    params: dict[str, Any] = Field(default_factory=dict)

    # Catalog publishing — opt-in.
    publish_to_catalog: bool = False
    model_name: str = ""  # required when publish_to_catalog=True
    namespace_id: int | None = None
    namespace_full_name: str | None = None  # portable "catalog.schema"; resolved name-first, id is fallback
    catalog_description: str | None = None
    catalog_tags: list[str] = Field(default_factory=list)

UserDefinedNode pydantic-model

Bases: NodeMultiInput

Settings for a node that contains the user defined node information

Show JSON schema:

{
  "$defs": {
    "OutputFieldConfig": {
      "description": "Configuration for output field validation and transformation behavior.",
      "properties": {
        "enabled": {
          "default": false,
          "title": "Enabled",
          "type": "boolean"
        },
        "validation_mode_behavior": {
          "default": "select_only",
          "enum": [
            "add_missing",
            "add_missing_keep_extra",
            "raise_on_missing",
            "select_only"
          ],
          "title": "Validation Mode Behavior",
          "type": "string"
        },
        "fields": {
          "items": {
            "$ref": "#/$defs/OutputFieldInfo"
          },
          "title": "Fields",
          "type": "array"
        },
        "validate_data_types": {
          "default": false,
          "title": "Validate Data Types",
          "type": "boolean"
        }
      },
      "title": "OutputFieldConfig",
      "type": "object"
    },
    "OutputFieldInfo": {
      "description": "Field information with optional default value for output field configuration.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "title": "Data Type",
          "type": "string"
        },
        "default_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Default Value"
        }
      },
      "required": [
        "name"
      ],
      "title": "OutputFieldInfo",
      "type": "object"
    }
  },
  "description": "Settings for a node that contains the user defined node information",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "cache_results": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Cache Results"
    },
    "pos_x": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos X"
    },
    "pos_y": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": 0,
      "title": "Pos Y"
    },
    "group_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Group Id"
    },
    "is_setup": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Is Setup"
    },
    "description": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "",
      "title": "Description"
    },
    "node_reference": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Reference"
    },
    "user_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "User Id"
    },
    "is_flow_output": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is Flow Output"
    },
    "is_user_defined": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Is User Defined"
    },
    "output_field_config": {
      "anyOf": [
        {
          "$ref": "#/$defs/OutputFieldConfig"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "depending_on_ids": {
      "anyOf": [
        {
          "items": {
            "type": "integer"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Depending On Ids"
    },
    "settings": {
      "additionalProperties": {
        "additionalProperties": true,
        "type": "object"
      },
      "title": "Settings",
      "type": "object"
    },
    "kernel_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Kernel Id"
    },
    "output_names": {
      "items": {
        "type": "string"
      },
      "title": "Output Names",
      "type": "array"
    },
    "node_source_hash": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Source Hash"
    },
    "settings_format_version": {
      "default": 1,
      "title": "Settings Format Version",
      "type": "integer"
    }
  },
  "required": [
    "flow_id",
    "node_id"
  ],
  "title": "UserDefinedNode",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• cache_results (bool | None)
• pos_x (float | None)
• pos_y (float | None)
• group_id (int | None)
• is_setup (bool | None)
• description (str | None)
• node_reference (str | None)
• user_id (int | None)
• is_flow_output (bool | None)
• is_user_defined (bool | None)
• output_field_config (OutputFieldConfig | None)
• depending_on_ids (list[int] | None)
• settings (dict[str, dict[str, Any]])
• kernel_id (str | None)
• output_names (list[str])
• node_source_hash (str | None)
• settings_format_version (int)

Validators:

• validate_node_reference → node_reference
• _coerce_legacy_settings → settings
• validate_output_names → output_names

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

class UserDefinedNode(NodeMultiInput):
    """Settings for a node that contains the user defined node information"""

    settings: dict[str, dict[str, Any]] = Field(default_factory=dict)
    kernel_id: str | None = None
    output_names: list[str] = Field(default_factory=lambda: ["main"])
    # sha256 of the node's .py file at add time; source edits invalidate the node hash/cache.
    node_source_hash: str | None = None
    settings_format_version: int = 1

    @field_validator("settings", mode="before")
    @classmethod
    def _coerce_legacy_settings(cls, v):
        """Coerce legacy payloads (settings=None / non-dict) into the {section: {component: value}} envelope."""
        if v is None or not isinstance(v, dict):
            return {}
        return {section: (values if isinstance(values, dict) else {"value": values}) for section, values in v.items()}

    @field_validator("output_names")
    @classmethod
    def validate_output_names(cls, v: list[str]) -> list[str]:
        return _validate_output_names(v)

get_default_description()

Generates a human-readable description based on the node's configured content.

Subclasses override this to provide meaningful descriptions. Returns an empty string by default.

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

def get_default_description(self) -> str:
    """Generates a human-readable description based on the node's configured content.

    Subclasses override this to provide meaningful descriptions.
    Returns an empty string by default.
    """
    return ""

validate_node_reference(v) pydantic-validator

Validates that node_reference is a safe identifier (lowercase letters, digits, underscores).

Source code in flowfile_core/flowfile_core/schemas/input_schema.py

@field_validator("node_reference", mode="before")
@classmethod
def validate_node_reference(cls, v):
    """Validates that node_reference is a safe identifier (lowercase letters, digits, underscores)."""
    if v is None or v == "":
        return None
    if not isinstance(v, str):
        raise ValueError("node_reference must be a string")
    if " " in v:
        raise ValueError("node_reference cannot contain spaces")
    if v != v.lower():
        raise ValueError("node_reference must be lowercase")
    if not _SAFE_IDENTIFIER_RE.match(v):
        raise ValueError(
            "node_reference must start with a letter and contain only lowercase letters, digits, and underscores"
        )
    return v

transform_schema

flowfile_core.schemas.transform_schema

Classes:

Name	Description
AggColl	A data class that represents a single aggregation operation for a group by operation.
BasicFilter	Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').
CrossJoinInput	Data model for cross join operations.
CrossJoinInputManager	Manager for cross join operations.
DynamicRenameInput	Defines settings for a dynamic rename operation.
FieldInput	Represents a single field with its name and data type, typically for defining an output column.
FilterInput	Defines the settings for a filter operation, supporting basic or advanced (expression-based) modes.
FilterOperator	Supported filter comparison operators.
FullJoinKeyResponse	Holds the join key rename responses for both sides of a join.
FunctionInput	Defines a formula to be applied, including the output field information.
FuzzyMatchInput	Data model for fuzzy matching join operations.
FuzzyMatchInputManager	Manager for fuzzy matching join operations.
GraphSolverInput	Defines settings for a graph-solving operation (e.g., finding connected components).
GroupByInput	A data class that represents the input for a group by operation.
JoinInput	Data model for standard SQL-style join operations.
JoinInputManager	Manager for standard SQL-style join operations.
JoinInputs	Data model for join-specific select inputs (extends SelectInputs).
JoinInputsManager	Manager for join-specific operations, extends SelectInputsManager.
JoinKeyRename	Represents the renaming of a join key from its original to a temporary name.
JoinKeyRenameResponse	Contains a list of join key renames for one side of a join.
JoinMap	Defines a single mapping between a left and right column for a join key.
JoinSelectManagerMixin	Mixin providing common methods for join-like operations.
PivotInput	Defines the settings for a pivot (long-to-wide) operation.
PolarsCodeInput	A simple container for a string of user-provided Polars code to be executed.
RecordIdInput	Defines settings for adding a record ID (row number) column to the data.
SelectInput	Defines how a single column should be selected, renamed, or type-cast.
SelectInputs	A container for a list of SelectInput objects (pure data, no logic).
SelectInputsManager	Manager class that provides all query and mutation operations.
SortByInput	Defines a single sort condition on a column, including the direction.
SqlQueryInput	A container for a SQL query to execute against connected data sources.
TextToRowsInput	Defines settings for splitting a text column into multiple rows based on a delimiter.
UnionInput	Defines settings for a union (concatenation) operation.
UniqueInput	Defines settings for a uniqueness operation, specifying columns and which row to keep.
UnpivotInput	Defines settings for an unpivot (wide-to-long) operation.
WindowFunctionInput	A single window-function operation producing one new column.
WindowFunctionsInput	Defines the settings for a window-functions node.

Functions:

Name	Description
construct_join_key_name	Creates a temporary, unique name for a join key column.
get_func_type_mapping	Infers the output data type of common aggregation functions.
get_window_output_type	Infers the output data type of window functions.
is_descending	Whether a sort-direction string means descending.
string_concat	A simple wrapper to concatenate string columns in Polars.

Attributes:

Name	Type	Description
JoinKeyStrategy		Key-based join strategies — every option requires join_mapping
JoinStrategy		Polars join-strategy enum — broad superset retained for backward

JoinKeyStrategy = Literal['inner', 'left', 'right', 'full', 'semi', 'anti', 'outer'] module-attribute

Key-based join strategies — every option requires join_mapping to specify the equality keys. Used by :class:JoinInput.how so the LLM (and the Pydantic validator) cannot pick "cross" on a join node — Cartesian joins are the dedicated cross_join node type's job.

JoinStrategy = Literal['inner', 'left', 'right', 'full', 'semi', 'anti', 'cross', 'outer'] module-attribute

Polars join-strategy enum — broad superset retained for backward compat with code-generator / flow-data-engine plumbing that still threads "cross" through DataFrame.join(how="cross") directly. The join node itself uses :data:JoinKeyStrategy (below) which excludes "cross" so cross/Cartesian joins route through the dedicated cross_join node type — making the choice unambiguous for the AI agent and preventing the join + how="cross" shape that bypasses the dedicated cross_join node.

AggColl pydantic-model

Bases: BaseModel

A data class that represents a single aggregation operation for a group by operation.

Attributes

old_name : str The name of the column in the original DataFrame to be aggregated.

str

The aggregation function to use. This can be a string representing a built-in function or a custom function.

Optional[str]

The name of the resulting aggregated column in the output DataFrame. If not provided, it will default to the old_name appended with the aggregation function.

Optional[str]

The type of the output values of the aggregation. If not provided, it is inferred from the aggregation function using the get_func_type_mapping function.

Example

agg_col = AggColl( old_name='col1', agg='sum', new_name='sum_col1', output_type='float' )

Show JSON schema:

{
  "description": "A data class that represents a single aggregation operation for a group by operation.\n\nAttributes\n----------\nold_name : str\n    The name of the column in the original DataFrame to be aggregated.\n\nagg : str\n    The aggregation function to use. This can be a string representing a built-in function or a custom function.\n\nnew_name : Optional[str]\n    The name of the resulting aggregated column in the output DataFrame. If not provided, it will default to the\n    old_name appended with the aggregation function.\n\noutput_type : Optional[str]\n    The type of the output values of the aggregation. If not provided, it is inferred from the aggregation function\n    using the `get_func_type_mapping` function.\n\nExample\n--------\nagg_col = AggColl(\n    old_name='col1',\n    agg='sum',\n    new_name='sum_col1',\n    output_type='float'\n)",
  "properties": {
    "old_name": {
      "title": "Old Name",
      "type": "string"
    },
    "agg": {
      "title": "Agg",
      "type": "string"
    },
    "new_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "New Name"
    },
    "output_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Type"
    }
  },
  "required": [
    "old_name",
    "agg"
  ],
  "title": "AggColl",
  "type": "object"
}

Fields:

• old_name (str)
• agg (str)
• new_name (str | None)
• output_type (str | None)

Validators:

• set_defaults

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class AggColl(BaseModel):
    """
    A data class that represents a single aggregation operation for a group by operation.

    Attributes
    ----------
    old_name : str
        The name of the column in the original DataFrame to be aggregated.

    agg : str
        The aggregation function to use. This can be a string representing a built-in function or a custom function.

    new_name : Optional[str]
        The name of the resulting aggregated column in the output DataFrame. If not provided, it will default to the
        old_name appended with the aggregation function.

    output_type : Optional[str]
        The type of the output values of the aggregation. If not provided, it is inferred from the aggregation function
        using the `get_func_type_mapping` function.

    Example
    --------
    agg_col = AggColl(
        old_name='col1',
        agg='sum',
        new_name='sum_col1',
        output_type='float'
    )
    """

    old_name: str
    agg: str
    new_name: str | None = None
    output_type: str | None = None

    def __init__(self, old_name: str, agg: str, new_name: str | None = None, output_type: str | None = None):
        data = {"old_name": old_name, "agg": agg}
        if new_name is not None:
            data["new_name"] = new_name
        if output_type is not None:
            data["output_type"] = output_type

        super().__init__(**data)

    @model_validator(mode="after")
    def set_defaults(self):
        """Set default new_name and output_type based on agg function."""
        if self.new_name is None:
            if self.agg != "groupby":
                self.new_name = self.old_name + "_" + self.agg
            else:
                self.new_name = self.old_name

        if self.output_type is None:
            self.output_type = get_func_type_mapping(self.agg)

        self.old_name = str(self.old_name)

        return self

    @property
    def agg_func(self):
        """Returns the corresponding Polars aggregation function from the `agg` string."""
        if self.agg == "groupby":
            return self.agg
        elif self.agg == "concat":
            return string_concat
        else:
            return getattr(pl, self.agg) if isinstance(self.agg, str) else self.agg

agg_func property

Returns the corresponding Polars aggregation function from the agg string.

set_defaults() pydantic-validator

Set default new_name and output_type based on agg function.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="after")
def set_defaults(self):
    """Set default new_name and output_type based on agg function."""
    if self.new_name is None:
        if self.agg != "groupby":
            self.new_name = self.old_name + "_" + self.agg
        else:
            self.new_name = self.old_name

    if self.output_type is None:
        self.output_type = get_func_type_mapping(self.agg)

    self.old_name = str(self.old_name)

    return self

BasicFilter pydantic-model

Bases: BaseModel

Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').

Attributes:

Name	Type	Description
field	str	The column name to filter on.
operator	FilterOperator | str	The comparison operator (FilterOperator enum value or symbol).
value	str	The value to compare against.
value2	str | None	Second value for BETWEEN operator (optional).

Show JSON schema:

{
  "$defs": {
    "FilterOperator": {
      "description": "Supported filter comparison operators.",
      "enum": [
        "equals",
        "not_equals",
        "greater_than",
        "greater_than_or_equals",
        "less_than",
        "less_than_or_equals",
        "contains",
        "not_contains",
        "starts_with",
        "ends_with",
        "is_null",
        "is_not_null",
        "in",
        "not_in",
        "between"
      ],
      "title": "FilterOperator",
      "type": "string"
    }
  },
  "description": "Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').\n\nAttributes:\n    field: The column name to filter on.\n    operator: The comparison operator (FilterOperator enum value or symbol).\n    value: The value to compare against.\n    value2: Second value for BETWEEN operator (optional).",
  "properties": {
    "field": {
      "default": "",
      "title": "Field",
      "type": "string"
    },
    "operator": {
      "anyOf": [
        {
          "$ref": "#/$defs/FilterOperator"
        },
        {
          "type": "string"
        }
      ],
      "default": "equals",
      "title": "Operator"
    },
    "value": {
      "default": "",
      "title": "Value",
      "type": "string"
    },
    "value2": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Value2"
    },
    "filter_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Filter Type"
    },
    "filter_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Filter Value"
    }
  },
  "title": "BasicFilter",
  "type": "object"
}

Fields:

• field (str)
• operator (FilterOperator | str)
• value (str)
• value2 (str | None)
• filter_type (str | None)
• filter_value (str | None)

Validators:

• normalize_operator

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class BasicFilter(BaseModel):
    """Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').

    Attributes:
        field: The column name to filter on.
        operator: The comparison operator (FilterOperator enum value or symbol).
        value: The value to compare against.
        value2: Second value for BETWEEN operator (optional).
    """

    field: str = ""
    operator: FilterOperator | str = FilterOperator.EQUALS
    value: str = ""
    value2: str | None = None  # For BETWEEN operator

    # Keep old field names for backward compatibility
    filter_type: str | None = None
    filter_value: str | None = None

    def __init__(
        self,
        field: str = None,
        operator: FilterOperator | str = None,
        value: str = None,
        value2: str = None,
        # Backward compatibility parameters
        filter_type: str = None,
        filter_value: str = None,
        **data,
    ):
        # Handle backward compatibility
        if filter_type is not None and operator is None:
            data["operator"] = filter_type
        elif operator is not None:
            data["operator"] = operator

        if filter_value is not None and value is None:
            data["value"] = filter_value
        elif value is not None:
            data["value"] = value

        if field is not None:
            data["field"] = field
        if value2 is not None:
            data["value2"] = value2

        super().__init__(**data)

    @model_validator(mode="after")
    def normalize_operator(self):
        """Normalize the operator to FilterOperator enum."""
        if isinstance(self.operator, str):
            try:
                self.operator = FilterOperator.from_symbol(self.operator)
            except ValueError:
                # Keep as string if conversion fails (for backward compat)
                pass
        return self

    def get_operator(self) -> FilterOperator:
        """Get the operator as FilterOperator enum."""
        if isinstance(self.operator, FilterOperator):
            return self.operator
        return FilterOperator.from_symbol(self.operator)

    def to_yaml_dict(self) -> BasicFilterYaml:
        """Serialize for YAML output."""
        result: BasicFilterYaml = {
            "field": self.field,
            "operator": self.operator.value if isinstance(self.operator, FilterOperator) else self.operator,
            "value": self.value,
        }
        if self.value2:
            result["value2"] = self.value2
        return result

    @classmethod
    def from_yaml_dict(cls, data: dict) -> "BasicFilter":
        """Load from YAML format."""
        return cls(
            field=data.get("field", ""),
            operator=data.get("operator", FilterOperator.EQUALS),
            value=data.get("value", ""),
            value2=data.get("value2"),
        )

from_yaml_dict(data) classmethod

Load from YAML format.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_yaml_dict(cls, data: dict) -> "BasicFilter":
    """Load from YAML format."""
    return cls(
        field=data.get("field", ""),
        operator=data.get("operator", FilterOperator.EQUALS),
        value=data.get("value", ""),
        value2=data.get("value2"),
    )

get_operator()

Get the operator as FilterOperator enum.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_operator(self) -> FilterOperator:
    """Get the operator as FilterOperator enum."""
    if isinstance(self.operator, FilterOperator):
        return self.operator
    return FilterOperator.from_symbol(self.operator)

normalize_operator() pydantic-validator

Normalize the operator to FilterOperator enum.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="after")
def normalize_operator(self):
    """Normalize the operator to FilterOperator enum."""
    if isinstance(self.operator, str):
        try:
            self.operator = FilterOperator.from_symbol(self.operator)
        except ValueError:
            # Keep as string if conversion fails (for backward compat)
            pass
    return self

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> BasicFilterYaml:
    """Serialize for YAML output."""
    result: BasicFilterYaml = {
        "field": self.field,
        "operator": self.operator.value if isinstance(self.operator, FilterOperator) else self.operator,
        "value": self.value,
    }
    if self.value2:
        result["value2"] = self.value2
    return result

CrossJoinInput pydantic-model

Bases: BaseModel

Data model for cross join operations.

Show JSON schema:

{
  "$defs": {
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Data model for cross join operations.",
  "properties": {
    "left_select": {
      "$ref": "#/$defs/JoinInputs"
    },
    "right_select": {
      "$ref": "#/$defs/JoinInputs"
    }
  },
  "required": [
    "left_select",
    "right_select"
  ],
  "title": "CrossJoinInput",
  "type": "object"
}

Fields:

• left_select (JoinInputs)
• right_select (JoinInputs)

Validators:

• parse_inputs

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class CrossJoinInput(BaseModel):
    """Data model for cross join operations."""

    left_select: JoinInputs
    right_select: JoinInputs

    @model_validator(mode="before")
    @classmethod
    def parse_inputs(cls, data: Any) -> Any:
        """Parse flexible input formats before validation."""
        if isinstance(data, dict):
            if "join_mapping" in data:
                data["join_mapping"] = cls._parse_join_mapping(data["join_mapping"])

            if "left_select" in data:
                data["left_select"] = cls._parse_select(data["left_select"])

            if "right_select" in data:
                data["right_select"] = cls._parse_select(data["right_select"])

        return data

    @staticmethod
    def _parse_join_mapping(join_mapping: Any) -> list[JoinMap]:
        """Parse various join_mapping formats."""
        if isinstance(join_mapping, list):
            result = []
            for jm in join_mapping:
                if isinstance(jm, JoinMap):
                    result.append(jm)
                elif isinstance(jm, dict):
                    result.append(JoinMap(**jm))
                elif isinstance(jm, tuple | list) and len(jm) == 2:
                    result.append(JoinMap(left_col=jm[0], right_col=jm[1]))
                elif isinstance(jm, str):
                    result.append(JoinMap(left_col=jm, right_col=jm))
                else:
                    raise ValueError(f"Invalid join mapping item: {jm}")
            return result

        if isinstance(join_mapping, JoinMap):
            return [join_mapping]

        # String: same column on both sides
        if isinstance(join_mapping, str):
            return [JoinMap(left_col=join_mapping, right_col=join_mapping)]

        # Tuple: (left, right)
        if isinstance(join_mapping, tuple) and len(join_mapping) == 2:
            return [JoinMap(left_col=join_mapping[0], right_col=join_mapping[1])]

        raise ValueError(f"Invalid join_mapping format: {type(join_mapping)}")

    @staticmethod
    def _parse_select(select: Any) -> JoinInputs:
        """Parse various select input formats."""
        if isinstance(select, JoinInputs):
            return select

        if isinstance(select, list):
            if all(isinstance(s, SelectInput) for s in select):
                return JoinInputs(renames=select)
            elif all(isinstance(s, str) for s in select):
                return JoinInputs(renames=[SelectInput(old_name=s) for s in select])
            elif all(isinstance(s, dict) for s in select):
                return JoinInputs(renames=[SelectInput(**s) for s in select])

        # Dict with 'select' (new YAML) or 'renames' (internal) key
        if isinstance(select, dict):
            if "select" in select:
                return JoinInputs(renames=[SelectInput.from_yaml_dict(s) for s in select["select"]])
            if "renames" in select:
                return JoinInputs(**select)

        raise ValueError(f"Invalid select format: {type(select)}")

    def __init__(
        self,
        left_select: JoinInputs | list[SelectInput] | list[str] = None,
        right_select: JoinInputs | list[SelectInput] | list[str] = None,
        **data,
    ):
        """Custom init for backward compatibility with positional arguments."""
        if left_select is not None:
            data["left_select"] = left_select
        if right_select is not None:
            data["right_select"] = right_select
        super().__init__(**data)

    def to_yaml_dict(self) -> CrossJoinInputYaml:
        """Serialize for YAML output."""
        return {
            "left_select": self.left_select.to_yaml_dict(),
            "right_select": self.right_select.to_yaml_dict(),
        }

    def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
        """Adds a new column to the selection for either the left or right side."""
        target_input = self.right_select if side == "right" else self.left_select
        if select_input.new_name is None:
            select_input.new_name = select_input.old_name
        target_input.renames.append(select_input)

__init__(left_select=None, right_select=None, **data)

Custom init for backward compatibility with positional arguments.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __init__(
    self,
    left_select: JoinInputs | list[SelectInput] | list[str] = None,
    right_select: JoinInputs | list[SelectInput] | list[str] = None,
    **data,
):
    """Custom init for backward compatibility with positional arguments."""
    if left_select is not None:
        data["left_select"] = left_select
    if right_select is not None:
        data["right_select"] = right_select
    super().__init__(**data)

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.right_select if side == "right" else self.left_select
    if select_input.new_name is None:
        select_input.new_name = select_input.old_name
    target_input.renames.append(select_input)

parse_inputs(data) pydantic-validator

Parse flexible input formats before validation.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="before")
@classmethod
def parse_inputs(cls, data: Any) -> Any:
    """Parse flexible input formats before validation."""
    if isinstance(data, dict):
        if "join_mapping" in data:
            data["join_mapping"] = cls._parse_join_mapping(data["join_mapping"])

        if "left_select" in data:
            data["left_select"] = cls._parse_select(data["left_select"])

        if "right_select" in data:
            data["right_select"] = cls._parse_select(data["right_select"])

    return data

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> CrossJoinInputYaml:
    """Serialize for YAML output."""
    return {
        "left_select": self.left_select.to_yaml_dict(),
        "right_select": self.right_select.to_yaml_dict(),
    }

CrossJoinInputManager

Bases: JoinSelectManagerMixin

Manager for cross join operations.

Methods:

Name	Description
add_new_select_column	Adds a new column to the selection for either the left or right side.
auto_generate_new_col_name	Generates a new, non-conflicting column name by adding a suffix if necessary.
auto_rename	Automatically renames columns on the right side to prevent naming conflicts.
create	Factory method to create CrossJoinInput from various input formats.
get_overlapping_columns	Finds column names that would conflict after the join.
get_overlapping_records	Finds column names that would conflict after the join.
parse_select	Parses various input formats into a standardized JoinInputs object.
to_cross_join_input	Creates a new CrossJoinInput instance based on the current manager settings.

Attributes:

Name	Type	Description
left_select	JoinInputsManager	Backward compatibility: Access left_manager as left_select.
overlapping_records	set[str]	Backward compatibility: Returns overlapping column names.
right_select	JoinInputsManager	Backward compatibility: Access right_manager as right_select.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class CrossJoinInputManager(JoinSelectManagerMixin):
    """Manager for cross join operations."""

    def __init__(self, cross_join_input: CrossJoinInput):
        self.input = deepcopy(cross_join_input)
        self.left_manager = JoinInputsManager(self.input.left_select)
        self.right_manager = JoinInputsManager(self.input.right_select)

    @classmethod
    def create(
        cls, left_select: list[SelectInput] | list[str], right_select: list[SelectInput] | list[str]
    ) -> "CrossJoinInputManager":
        """Factory method to create CrossJoinInput from various input formats."""
        left_inputs = cls.parse_select(left_select)
        right_inputs = cls.parse_select(right_select)

        cross_join = CrossJoinInput(left_select=left_inputs, right_select=right_inputs)
        return cls(cross_join)

    def get_overlapping_records(self) -> set[str]:
        """Finds column names that would conflict after the join."""
        return self.get_overlapping_columns()

    def auto_rename(self, rename_mode: Literal["suffix", "prefix"] = "prefix") -> None:
        """Automatically renames columns on the right side to prevent naming conflicts."""
        overlapping_records = self.get_overlapping_records()

        while len(overlapping_records) > 0:
            for right_col in self.input.right_select.renames:
                if right_col.new_name in overlapping_records:
                    if rename_mode == "prefix":
                        right_col.new_name = "right_" + right_col.new_name
                    elif rename_mode == "suffix":
                        right_col.new_name = right_col.new_name + "_right"
                    else:
                        raise ValueError(f"Unknown rename_mode: {rename_mode}")
            overlapping_records = self.get_overlapping_records()

    # === Backward Compatibility Properties ===

    @property
    def left_select(self) -> JoinInputsManager:
        """Backward compatibility: Access left_manager as left_select."""
        return self.left_manager

    @property
    def right_select(self) -> JoinInputsManager:
        """Backward compatibility: Access right_manager as right_select."""
        return self.right_manager

    @property
    def overlapping_records(self) -> set[str]:
        """Backward compatibility: Returns overlapping column names."""
        return self.get_overlapping_records()

    def to_cross_join_input(self) -> CrossJoinInput:
        """Creates a new CrossJoinInput instance based on the current manager settings.

        This is useful when you've modified the manager (e.g., via auto_rename) and
        want to get a fresh CrossJoinInput with all the current settings applied.

        Returns:
            A new CrossJoinInput instance with current settings
        """
        return CrossJoinInput(
            left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
            right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
        )

left_select property

Backward compatibility: Access left_manager as left_select.

overlapping_records property

Backward compatibility: Returns overlapping column names.

right_select property

Backward compatibility: Access right_manager as right_select.

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.input.right_select if side == "right" else self.input.left_select

    select_input.new_name = self.auto_generate_new_col_name(select_input.old_name, side=side)

    target_input.renames.append(select_input)

auto_generate_new_col_name(old_col_name, side)

Generates a new, non-conflicting column name by adding a suffix if necessary.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_generate_new_col_name(self, old_col_name: str, side: str) -> str:
    """Generates a new, non-conflicting column name by adding a suffix if necessary."""
    current_names = self.get_overlapping_columns()
    if old_col_name not in current_names:
        return old_col_name

    new_name = old_col_name
    while new_name in current_names:
        new_name = f"{side}_{new_name}"
    return new_name

auto_rename(rename_mode='prefix')

Automatically renames columns on the right side to prevent naming conflicts.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_rename(self, rename_mode: Literal["suffix", "prefix"] = "prefix") -> None:
    """Automatically renames columns on the right side to prevent naming conflicts."""
    overlapping_records = self.get_overlapping_records()

    while len(overlapping_records) > 0:
        for right_col in self.input.right_select.renames:
            if right_col.new_name in overlapping_records:
                if rename_mode == "prefix":
                    right_col.new_name = "right_" + right_col.new_name
                elif rename_mode == "suffix":
                    right_col.new_name = right_col.new_name + "_right"
                else:
                    raise ValueError(f"Unknown rename_mode: {rename_mode}")
        overlapping_records = self.get_overlapping_records()

create(left_select, right_select) classmethod

Factory method to create CrossJoinInput from various input formats.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create(
    cls, left_select: list[SelectInput] | list[str], right_select: list[SelectInput] | list[str]
) -> "CrossJoinInputManager":
    """Factory method to create CrossJoinInput from various input formats."""
    left_inputs = cls.parse_select(left_select)
    right_inputs = cls.parse_select(right_select)

    cross_join = CrossJoinInput(left_select=left_inputs, right_select=right_inputs)
    return cls(cross_join)

get_overlapping_columns()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_columns(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.left_manager.get_new_cols() & self.right_manager.get_new_cols()

get_overlapping_records()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_records(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.get_overlapping_columns()

parse_select(select) staticmethod

Parses various input formats into a standardized JoinInputs object.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@staticmethod
def parse_select(select: list[SelectInput] | list[str] | list[dict] | dict) -> JoinInputs:
    """Parses various input formats into a standardized `JoinInputs` object."""
    if not select:
        return JoinInputs(renames=[])

    if all(isinstance(c, SelectInput) for c in select):
        return JoinInputs(renames=select)
    elif all(isinstance(c, dict) for c in select):
        return JoinInputs(renames=[SelectInput(**c) for c in select])
    elif isinstance(select, dict):
        renames = select.get("renames")
        if renames:
            return JoinInputs(renames=[SelectInput(**c) for c in renames])
        return JoinInputs(renames=[])
    elif all(isinstance(c, str) for c in select):
        return JoinInputs(renames=[SelectInput(old_name=s, new_name=s) for s in select])

    raise ValueError(f"Unable to parse select input: {type(select)}")

to_cross_join_input()

Creates a new CrossJoinInput instance based on the current manager settings.

This is useful when you've modified the manager (e.g., via auto_rename) and want to get a fresh CrossJoinInput with all the current settings applied.

Returns:

Type	Description
CrossJoinInput	A new CrossJoinInput instance with current settings

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_cross_join_input(self) -> CrossJoinInput:
    """Creates a new CrossJoinInput instance based on the current manager settings.

    This is useful when you've modified the manager (e.g., via auto_rename) and
    want to get a fresh CrossJoinInput with all the current settings applied.

    Returns:
        A new CrossJoinInput instance with current settings
    """
    return CrossJoinInput(
        left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
        right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
    )

DynamicRenameInput pydantic-model

Bases: BaseModel

Defines settings for a dynamic rename operation.

Applies a single rule (prefix / suffix / formula / first_row) to a set of selected columns, rather than requiring the user to rename columns one-by-one.

In formula mode, the flowfile formula syntax is evaluated with [column_name] bound to each target column's current name; for example uppercase([column_name]) or "v2_" + [column_name].

In first_row mode, the first row of the incoming table is promoted to column headers and then dropped from the data. Non-string values are coerced to str; null or empty values raise an error. Selection filters still apply — only selected columns are renamed, but the first row is always dropped.

Show JSON schema:

{
  "description": "Defines settings for a dynamic rename operation.\n\nApplies a single rule (prefix / suffix / formula / first_row) to a set of selected\ncolumns, rather than requiring the user to rename columns one-by-one.\n\nIn formula mode, the flowfile formula syntax is evaluated with `[column_name]`\nbound to each target column's current name; for example `uppercase([column_name])`\nor `\"v2_\" + [column_name]`.\n\nIn first_row mode, the first row of the incoming table is promoted to column\nheaders and then dropped from the data. Non-string values are coerced to `str`;\nnull or empty values raise an error. Selection filters still apply \u2014 only selected\ncolumns are renamed, but the first row is always dropped.",
  "properties": {
    "rename_mode": {
      "default": "prefix",
      "enum": [
        "prefix",
        "suffix",
        "formula",
        "first_row"
      ],
      "title": "Rename Mode",
      "type": "string"
    },
    "prefix": {
      "default": "",
      "title": "Prefix",
      "type": "string"
    },
    "suffix": {
      "default": "",
      "title": "Suffix",
      "type": "string"
    },
    "formula": {
      "default": "",
      "expression": true,
      "title": "Formula",
      "type": "string"
    },
    "selection_mode": {
      "default": "all",
      "enum": [
        "all",
        "list",
        "data_type"
      ],
      "title": "Selection Mode",
      "type": "string"
    },
    "selected_columns": {
      "items": {
        "type": "string"
      },
      "title": "Selected Columns",
      "type": "array"
    },
    "selected_data_type": {
      "anyOf": [
        {
          "enum": [
            "Numeric",
            "String",
            "Date",
            "Other",
            "Boolean",
            "Binary",
            "Complex"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Selected Data Type"
    }
  },
  "title": "DynamicRenameInput",
  "type": "object"
}

Fields:

• rename_mode (RenameMode)
• prefix (str)
• suffix (str)
• formula (str)
• selection_mode (ColumnSelectionMode)
• selected_columns (list[str])
• selected_data_type (ReadableDataTypeGroup | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class DynamicRenameInput(BaseModel):
    """Defines settings for a dynamic rename operation.

    Applies a single rule (prefix / suffix / formula / first_row) to a set of selected
    columns, rather than requiring the user to rename columns one-by-one.

    In formula mode, the flowfile formula syntax is evaluated with `[column_name]`
    bound to each target column's current name; for example `uppercase([column_name])`
    or `"v2_" + [column_name]`.

    In first_row mode, the first row of the incoming table is promoted to column
    headers and then dropped from the data. Non-string values are coerced to `str`;
    null or empty values raise an error. Selection filters still apply — only selected
    columns are renamed, but the first row is always dropped.
    """

    rename_mode: RenameMode = "prefix"
    prefix: str = ""
    suffix: str = ""
    formula: str = Field(default="", json_schema_extra={"expression": True})

    selection_mode: ColumnSelectionMode = "all"
    selected_columns: list[str] = Field(default_factory=list)
    selected_data_type: ReadableDataTypeGroup | None = None

FieldInput pydantic-model

Bases: BaseModel

Represents a single field with its name and data type, typically for defining an output column.

Show JSON schema:

{
  "$defs": {
    "DataType": {
      "description": "Specific data types for fine-grained control.",
      "enum": [
        "Int8",
        "Int16",
        "Int32",
        "Int64",
        "Int128",
        "UInt8",
        "UInt16",
        "UInt32",
        "UInt64",
        "UInt128",
        "Float16",
        "Float32",
        "Float64",
        "Decimal",
        "String",
        "Categorical",
        "Date",
        "Datetime",
        "Time",
        "Duration",
        "Boolean",
        "Binary",
        "List",
        "Struct",
        "Array"
      ],
      "title": "DataType",
      "type": "string"
    }
  },
  "description": "Represents a single field with its name and data type, typically for defining an output column.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "data_type": {
      "anyOf": [
        {
          "$ref": "#/$defs/DataType"
        },
        {
          "const": "Auto",
          "type": "string"
        },
        {
          "enum": [
            "Int8",
            "Int16",
            "Int32",
            "Int64",
            "Int128",
            "UInt8",
            "UInt16",
            "UInt32",
            "UInt64",
            "UInt128",
            "Float16",
            "Float32",
            "Float64",
            "Decimal",
            "String",
            "Date",
            "Datetime",
            "Time",
            "Duration",
            "Boolean",
            "Binary",
            "List",
            "Struct",
            "Array",
            "Integer",
            "Double",
            "Utf8"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "Auto",
      "title": "Data Type"
    }
  },
  "required": [
    "name"
  ],
  "title": "FieldInput",
  "type": "object"
}

Fields:

• name (str)
• data_type (DataType | Literal['Auto'] | DataTypeStr | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FieldInput(BaseModel):
    """Represents a single field with its name and data type, typically for defining an output column."""

    name: str
    data_type: DataType | Literal["Auto"] | DataTypeStr | None = AUTO_DATA_TYPE

FilterInput pydantic-model

Bases: BaseModel

Defines the settings for a filter operation, supporting basic or advanced (expression-based) modes.

Attributes:

Name	Type	Description
mode	FilterModeLiteral	The filter mode - "basic" or "advanced".
basic_filter	BasicFilter | None	The basic filter configuration (used when mode="basic").
advanced_filter	str	The advanced filter expression string (used when mode="advanced").

Show JSON schema:

{
  "$defs": {
    "BasicFilter": {
      "description": "Defines a simple, single-condition filter (e.g., 'column' 'equals' 'value').\n\nAttributes:\n    field: The column name to filter on.\n    operator: The comparison operator (FilterOperator enum value or symbol).\n    value: The value to compare against.\n    value2: Second value for BETWEEN operator (optional).",
      "properties": {
        "field": {
          "default": "",
          "title": "Field",
          "type": "string"
        },
        "operator": {
          "anyOf": [
            {
              "$ref": "#/$defs/FilterOperator"
            },
            {
              "type": "string"
            }
          ],
          "default": "equals",
          "title": "Operator"
        },
        "value": {
          "default": "",
          "title": "Value",
          "type": "string"
        },
        "value2": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Value2"
        },
        "filter_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Filter Type"
        },
        "filter_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Filter Value"
        }
      },
      "title": "BasicFilter",
      "type": "object"
    },
    "FilterOperator": {
      "description": "Supported filter comparison operators.",
      "enum": [
        "equals",
        "not_equals",
        "greater_than",
        "greater_than_or_equals",
        "less_than",
        "less_than_or_equals",
        "contains",
        "not_contains",
        "starts_with",
        "ends_with",
        "is_null",
        "is_not_null",
        "in",
        "not_in",
        "between"
      ],
      "title": "FilterOperator",
      "type": "string"
    }
  },
  "description": "Defines the settings for a filter operation, supporting basic or advanced (expression-based) modes.\n\nAttributes:\n    mode: The filter mode - \"basic\" or \"advanced\".\n    basic_filter: The basic filter configuration (used when mode=\"basic\").\n    advanced_filter: The advanced filter expression string (used when mode=\"advanced\").",
  "properties": {
    "mode": {
      "default": "basic",
      "enum": [
        "basic",
        "advanced"
      ],
      "title": "Mode",
      "type": "string"
    },
    "basic_filter": {
      "anyOf": [
        {
          "$ref": "#/$defs/BasicFilter"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "advanced_filter": {
      "default": "",
      "expression": true,
      "title": "Advanced Filter",
      "type": "string"
    },
    "filter_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Filter Type"
    }
  },
  "title": "FilterInput",
  "type": "object"
}

Fields:

• mode (FilterModeLiteral)
• basic_filter (BasicFilter | None)
• advanced_filter (str)
• filter_type (str | None)

Validators:

• ensure_basic_filter

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FilterInput(BaseModel):
    """Defines the settings for a filter operation, supporting basic or advanced (expression-based) modes.

    Attributes:
        mode: The filter mode - "basic" or "advanced".
        basic_filter: The basic filter configuration (used when mode="basic").
        advanced_filter: The advanced filter expression string (used when mode="advanced").
    """

    mode: FilterModeLiteral = "basic"
    basic_filter: BasicFilter | None = None
    advanced_filter: str = Field(default="", json_schema_extra={"expression": True})

    # Keep old field name for backward compatibility
    filter_type: str | None = None

    def __init__(
        self,
        mode: FilterModeLiteral = None,
        basic_filter: BasicFilter = None,
        advanced_filter: str = None,
        # Backward compatibility
        filter_type: str = None,
        **data,
    ):
        # Handle backward compatibility: filter_type -> mode
        if filter_type is not None and mode is None:
            data["mode"] = filter_type
        elif mode is not None:
            data["mode"] = mode

        if advanced_filter is not None:
            data["advanced_filter"] = advanced_filter
        if basic_filter is not None:
            data["basic_filter"] = basic_filter

        super().__init__(**data)

    @model_validator(mode="after")
    def ensure_basic_filter(self):
        """Ensure basic_filter exists when mode is basic."""
        if self.mode == "basic" and self.basic_filter is None:
            self.basic_filter = BasicFilter()
        return self

    def is_advanced(self) -> bool:
        """Check if filter is in advanced mode."""
        return self.mode == "advanced"

    def to_yaml_dict(self) -> FilterInputYaml:
        """Serialize for YAML output."""
        result: FilterInputYaml = {"mode": self.mode}
        if self.mode == "basic" and self.basic_filter:
            result["basic_filter"] = self.basic_filter.to_yaml_dict()
        elif self.mode == "advanced" and self.advanced_filter:
            result["advanced_filter"] = self.advanced_filter
        return result

    @classmethod
    def from_yaml_dict(cls, data: dict) -> "FilterInput":
        """Load from YAML format."""
        mode = data.get("mode", "basic")
        basic_filter = None
        if "basic_filter" in data:
            basic_filter = BasicFilter.from_yaml_dict(data["basic_filter"])
        return cls(
            mode=mode,
            basic_filter=basic_filter,
            advanced_filter=data.get("advanced_filter", ""),
        )

ensure_basic_filter() pydantic-validator

Ensure basic_filter exists when mode is basic.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="after")
def ensure_basic_filter(self):
    """Ensure basic_filter exists when mode is basic."""
    if self.mode == "basic" and self.basic_filter is None:
        self.basic_filter = BasicFilter()
    return self

from_yaml_dict(data) classmethod

Load from YAML format.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_yaml_dict(cls, data: dict) -> "FilterInput":
    """Load from YAML format."""
    mode = data.get("mode", "basic")
    basic_filter = None
    if "basic_filter" in data:
        basic_filter = BasicFilter.from_yaml_dict(data["basic_filter"])
    return cls(
        mode=mode,
        basic_filter=basic_filter,
        advanced_filter=data.get("advanced_filter", ""),
    )

is_advanced()

Check if filter is in advanced mode.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def is_advanced(self) -> bool:
    """Check if filter is in advanced mode."""
    return self.mode == "advanced"

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> FilterInputYaml:
    """Serialize for YAML output."""
    result: FilterInputYaml = {"mode": self.mode}
    if self.mode == "basic" and self.basic_filter:
        result["basic_filter"] = self.basic_filter.to_yaml_dict()
    elif self.mode == "advanced" and self.advanced_filter:
        result["advanced_filter"] = self.advanced_filter
    return result

FilterOperator

Bases: str, Enum

Supported filter comparison operators.

Methods:

Name	Description
from_symbol	Convert UI symbol to FilterOperator enum.
to_symbol	Convert FilterOperator to UI-friendly symbol.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FilterOperator(str, Enum):
    """Supported filter comparison operators."""

    EQUALS = "equals"
    NOT_EQUALS = "not_equals"
    GREATER_THAN = "greater_than"
    GREATER_THAN_OR_EQUALS = "greater_than_or_equals"
    LESS_THAN = "less_than"
    LESS_THAN_OR_EQUALS = "less_than_or_equals"
    CONTAINS = "contains"
    NOT_CONTAINS = "not_contains"
    STARTS_WITH = "starts_with"
    ENDS_WITH = "ends_with"
    IS_NULL = "is_null"
    IS_NOT_NULL = "is_not_null"
    IN = "in"
    NOT_IN = "not_in"
    BETWEEN = "between"

    def __str__(self) -> str:
        return self.value

    @classmethod
    def from_symbol(cls, symbol: str) -> "FilterOperator":
        """Convert UI symbol to FilterOperator enum."""
        symbol_mapping = {
            "=": cls.EQUALS,
            "==": cls.EQUALS,
            "!=": cls.NOT_EQUALS,
            "<>": cls.NOT_EQUALS,
            ">": cls.GREATER_THAN,
            ">=": cls.GREATER_THAN_OR_EQUALS,
            "<": cls.LESS_THAN,
            "<=": cls.LESS_THAN_OR_EQUALS,
            "contains": cls.CONTAINS,
            "not_contains": cls.NOT_CONTAINS,
            "starts_with": cls.STARTS_WITH,
            "ends_with": cls.ENDS_WITH,
            "is_null": cls.IS_NULL,
            "is_not_null": cls.IS_NOT_NULL,
            "in": cls.IN,
            "not_in": cls.NOT_IN,
            "between": cls.BETWEEN,
        }
        if symbol in symbol_mapping:
            return symbol_mapping[symbol]
        # Try to match by value directly
        try:
            return cls(symbol)
        except ValueError:
            raise ValueError(f"Unknown filter operator symbol: {symbol}") from None

    def to_symbol(self) -> str:
        """Convert FilterOperator to UI-friendly symbol."""
        symbol_mapping = {
            FilterOperator.EQUALS: "=",
            FilterOperator.NOT_EQUALS: "!=",
            FilterOperator.GREATER_THAN: ">",
            FilterOperator.GREATER_THAN_OR_EQUALS: ">=",
            FilterOperator.LESS_THAN: "<",
            FilterOperator.LESS_THAN_OR_EQUALS: "<=",
            FilterOperator.CONTAINS: "contains",
            FilterOperator.NOT_CONTAINS: "not_contains",
            FilterOperator.STARTS_WITH: "starts_with",
            FilterOperator.ENDS_WITH: "ends_with",
            FilterOperator.IS_NULL: "is_null",
            FilterOperator.IS_NOT_NULL: "is_not_null",
            FilterOperator.IN: "in",
            FilterOperator.NOT_IN: "not_in",
            FilterOperator.BETWEEN: "between",
        }
        return symbol_mapping.get(self, self.value)

from_symbol(symbol) classmethod

Convert UI symbol to FilterOperator enum.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_symbol(cls, symbol: str) -> "FilterOperator":
    """Convert UI symbol to FilterOperator enum."""
    symbol_mapping = {
        "=": cls.EQUALS,
        "==": cls.EQUALS,
        "!=": cls.NOT_EQUALS,
        "<>": cls.NOT_EQUALS,
        ">": cls.GREATER_THAN,
        ">=": cls.GREATER_THAN_OR_EQUALS,
        "<": cls.LESS_THAN,
        "<=": cls.LESS_THAN_OR_EQUALS,
        "contains": cls.CONTAINS,
        "not_contains": cls.NOT_CONTAINS,
        "starts_with": cls.STARTS_WITH,
        "ends_with": cls.ENDS_WITH,
        "is_null": cls.IS_NULL,
        "is_not_null": cls.IS_NOT_NULL,
        "in": cls.IN,
        "not_in": cls.NOT_IN,
        "between": cls.BETWEEN,
    }
    if symbol in symbol_mapping:
        return symbol_mapping[symbol]
    # Try to match by value directly
    try:
        return cls(symbol)
    except ValueError:
        raise ValueError(f"Unknown filter operator symbol: {symbol}") from None

to_symbol()

Convert FilterOperator to UI-friendly symbol.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_symbol(self) -> str:
    """Convert FilterOperator to UI-friendly symbol."""
    symbol_mapping = {
        FilterOperator.EQUALS: "=",
        FilterOperator.NOT_EQUALS: "!=",
        FilterOperator.GREATER_THAN: ">",
        FilterOperator.GREATER_THAN_OR_EQUALS: ">=",
        FilterOperator.LESS_THAN: "<",
        FilterOperator.LESS_THAN_OR_EQUALS: "<=",
        FilterOperator.CONTAINS: "contains",
        FilterOperator.NOT_CONTAINS: "not_contains",
        FilterOperator.STARTS_WITH: "starts_with",
        FilterOperator.ENDS_WITH: "ends_with",
        FilterOperator.IS_NULL: "is_null",
        FilterOperator.IS_NOT_NULL: "is_not_null",
        FilterOperator.IN: "in",
        FilterOperator.NOT_IN: "not_in",
        FilterOperator.BETWEEN: "between",
    }
    return symbol_mapping.get(self, self.value)

FullJoinKeyResponse

Bases: NamedTuple

Holds the join key rename responses for both sides of a join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FullJoinKeyResponse(NamedTuple):
    """Holds the join key rename responses for both sides of a join."""

    left: JoinKeyRenameResponse
    right: JoinKeyRenameResponse

FunctionInput pydantic-model

Bases: BaseModel

Defines a formula to be applied, including the output field information.

Show JSON schema:

{
  "$defs": {
    "DataType": {
      "description": "Specific data types for fine-grained control.",
      "enum": [
        "Int8",
        "Int16",
        "Int32",
        "Int64",
        "Int128",
        "UInt8",
        "UInt16",
        "UInt32",
        "UInt64",
        "UInt128",
        "Float16",
        "Float32",
        "Float64",
        "Decimal",
        "String",
        "Categorical",
        "Date",
        "Datetime",
        "Time",
        "Duration",
        "Boolean",
        "Binary",
        "List",
        "Struct",
        "Array"
      ],
      "title": "DataType",
      "type": "string"
    },
    "FieldInput": {
      "description": "Represents a single field with its name and data type, typically for defining an output column.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "anyOf": [
            {
              "$ref": "#/$defs/DataType"
            },
            {
              "const": "Auto",
              "type": "string"
            },
            {
              "enum": [
                "Int8",
                "Int16",
                "Int32",
                "Int64",
                "Int128",
                "UInt8",
                "UInt16",
                "UInt32",
                "UInt64",
                "UInt128",
                "Float16",
                "Float32",
                "Float64",
                "Decimal",
                "String",
                "Date",
                "Datetime",
                "Time",
                "Duration",
                "Boolean",
                "Binary",
                "List",
                "Struct",
                "Array",
                "Integer",
                "Double",
                "Utf8"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "Auto",
          "title": "Data Type"
        }
      },
      "required": [
        "name"
      ],
      "title": "FieldInput",
      "type": "object"
    }
  },
  "description": "Defines a formula to be applied, including the output field information.",
  "properties": {
    "field": {
      "$ref": "#/$defs/FieldInput"
    },
    "function": {
      "expression": true,
      "title": "Function",
      "type": "string"
    }
  },
  "required": [
    "field",
    "function"
  ],
  "title": "FunctionInput",
  "type": "object"
}

Fields:

• field (FieldInput)
• function (str)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FunctionInput(BaseModel):
    """Defines a formula to be applied, including the output field information."""

    field: FieldInput
    function: str = Field(json_schema_extra={"expression": True})

    def __init__(self, field: FieldInput = None, function: str = None, **data):
        if field is not None:
            data["field"] = field
        if function is not None:
            data["function"] = function
        super().__init__(**data)

FuzzyMatchInput pydantic-model

Bases: BaseModel

Data model for fuzzy matching join operations.

Show JSON schema:

{
  "$defs": {
    "FuzzyMapping": {
      "description": "Represents the configuration for a fuzzy string match between two columns.\n\nThis class defines all the necessary parameters to perform a fuzzy join,\nincluding the columns to match, the specific algorithm to use, and the\nsimilarity threshold required to consider two strings a match.\n\nIt generates a default name for the output score column if one is not\nprovided.\n\nAttributes:\n    left_col (str): The name of the column in the left dataframe to join on.\n    right_col (str): The name of the column in the right dataframe to join on.\n    threshold_score (float): The similarity score threshold required for a\n        match, typically on a scale of 0 to 100. Defaults to 80.0.\n    fuzzy_type (FuzzyTypeLiteral): The string-matching algorithm to use.\n        Defaults to \"levenshtein\".\n    perc_unique (float): A parameter that may be used to assess column\n        uniqueness before performing a costly fuzzy match. Defaults to 0.0.\n    output_column_name (str | None): The name for the new column that will\n        contain the calculated fuzzy match score. If None, a name is\n        generated automatically in the format 'fuzzy_score_{left_col}_{right_col}'.\n    valid (bool): A flag to indicate whether this mapping is active and should\n        be used in a join operation. Defaults to True.\n    reversed_threshold_score (float): A property that converts the 0-100\n        threshold score into a 0.0-1.0 distance score, where 0.0 is a\n        perfect match.",
      "properties": {
        "left_col": {
          "title": "Left Col",
          "type": "string"
        },
        "right_col": {
          "title": "Right Col",
          "type": "string"
        },
        "threshold_score": {
          "default": 80.0,
          "title": "Threshold Score",
          "type": "number"
        },
        "fuzzy_type": {
          "default": "levenshtein",
          "enum": [
            "levenshtein",
            "jaro",
            "jaro_winkler",
            "hamming",
            "damerau_levenshtein",
            "indel"
          ],
          "title": "Fuzzy Type",
          "type": "string"
        },
        "perc_unique": {
          "default": 0.0,
          "title": "Perc Unique",
          "type": "number"
        },
        "output_column_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Column Name"
        },
        "valid": {
          "default": true,
          "title": "Valid",
          "type": "boolean"
        }
      },
      "required": [
        "left_col",
        "right_col"
      ],
      "title": "FuzzyMapping",
      "type": "object"
    },
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Data model for fuzzy matching join operations.",
  "properties": {
    "join_mapping": {
      "items": {
        "$ref": "#/$defs/FuzzyMapping"
      },
      "title": "Join Mapping",
      "type": "array"
    },
    "left_select": {
      "$ref": "#/$defs/JoinInputs"
    },
    "right_select": {
      "$ref": "#/$defs/JoinInputs"
    },
    "how": {
      "default": "inner",
      "enum": [
        "inner",
        "left",
        "right",
        "full",
        "semi",
        "anti",
        "cross",
        "outer"
      ],
      "title": "How",
      "type": "string"
    },
    "aggregate_output": {
      "default": false,
      "title": "Aggregate Output",
      "type": "boolean"
    }
  },
  "required": [
    "join_mapping",
    "left_select",
    "right_select"
  ],
  "title": "FuzzyMatchInput",
  "type": "object"
}

Fields:

• join_mapping (list[FuzzyMapping])
• left_select (JoinInputs)
• right_select (JoinInputs)
• how (JoinStrategy)
• aggregate_output (bool)

Validators:

• parse_inputs

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FuzzyMatchInput(BaseModel):
    """Data model for fuzzy matching join operations."""

    join_mapping: list[FuzzyMapping]
    left_select: JoinInputs
    right_select: JoinInputs
    how: JoinStrategy = "inner"
    aggregate_output: bool = False

    def __init__(
        self,
        left_select: JoinInputs | list[SelectInput] | list[str] = None,
        right_select: JoinInputs | list[SelectInput] | list[str] = None,
        **data,
    ):
        """Custom init for backward compatibility with positional arguments."""
        if left_select is not None:
            data["left_select"] = left_select
        if right_select is not None:
            data["right_select"] = right_select

        super().__init__(**data)

    def to_yaml_dict(self) -> FuzzyMatchInputYaml:
        """Serialize for YAML output."""
        return {
            "join_mapping": [asdict(jm) for jm in self.join_mapping],
            "left_select": self.left_select.to_yaml_dict(),
            "right_select": self.right_select.to_yaml_dict(),
            "how": self.how,
            "aggregate_output": self.aggregate_output,
        }

    def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
        """Adds a new column to the selection for either the left or right side."""
        target_input = self.right_select if side == "right" else self.left_select
        if select_input.new_name is None:
            select_input.new_name = select_input.old_name
        target_input.renames.append(select_input)

    @staticmethod
    def _parse_select(select: Any) -> JoinInputs:
        """Parse various select input formats."""
        if isinstance(select, JoinInputs):
            return select

        if isinstance(select, list):
            if all(isinstance(s, SelectInput) for s in select):
                return JoinInputs(renames=select)
            elif all(isinstance(s, str) for s in select):
                return JoinInputs(renames=[SelectInput(old_name=s) for s in select])
            elif all(isinstance(s, dict) for s in select):
                return JoinInputs(renames=[SelectInput(**s) for s in select])

        # Dict with 'select' (new YAML) or 'renames' (internal) key
        if isinstance(select, dict):
            if "select" in select:
                return JoinInputs(renames=[SelectInput.from_yaml_dict(s) for s in select["select"]])
            if "renames" in select:
                return JoinInputs(**select)

        raise ValueError(f"Invalid select format: {type(select)}")

    @model_validator(mode="before")
    @classmethod
    def parse_inputs(cls, data: Any) -> Any:
        """Parse flexible input formats before validation."""
        if isinstance(data, dict):
            if "left_select" in data:
                data["left_select"] = cls._parse_select(data["left_select"])

            if "right_select" in data:
                data["right_select"] = cls._parse_select(data["right_select"])

        return data

__init__(left_select=None, right_select=None, **data)

Custom init for backward compatibility with positional arguments.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __init__(
    self,
    left_select: JoinInputs | list[SelectInput] | list[str] = None,
    right_select: JoinInputs | list[SelectInput] | list[str] = None,
    **data,
):
    """Custom init for backward compatibility with positional arguments."""
    if left_select is not None:
        data["left_select"] = left_select
    if right_select is not None:
        data["right_select"] = right_select

    super().__init__(**data)

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.right_select if side == "right" else self.left_select
    if select_input.new_name is None:
        select_input.new_name = select_input.old_name
    target_input.renames.append(select_input)

parse_inputs(data) pydantic-validator

Parse flexible input formats before validation.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="before")
@classmethod
def parse_inputs(cls, data: Any) -> Any:
    """Parse flexible input formats before validation."""
    if isinstance(data, dict):
        if "left_select" in data:
            data["left_select"] = cls._parse_select(data["left_select"])

        if "right_select" in data:
            data["right_select"] = cls._parse_select(data["right_select"])

    return data

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> FuzzyMatchInputYaml:
    """Serialize for YAML output."""
    return {
        "join_mapping": [asdict(jm) for jm in self.join_mapping],
        "left_select": self.left_select.to_yaml_dict(),
        "right_select": self.right_select.to_yaml_dict(),
        "how": self.how,
        "aggregate_output": self.aggregate_output,
    }

FuzzyMatchInputManager

Bases: JoinInputManager

Manager for fuzzy matching join operations.

Methods:

Name	Description
add_new_select_column	Adds a new column to the selection for either the left or right side.
auto_generate_new_col_name	Generates a new, non-conflicting column name by adding a suffix if necessary.
auto_rename	Automatically renames columns on the right side to prevent naming conflicts.
create	Factory method to create FuzzyMatchInput from various input formats.
get_fuzzy_maps	Returns the final fuzzy mappings after applying all column renames.
get_join_key_renames	Gets the temporary rename mappings for the join keys on both sides.
get_left_join_keys	Returns a set of the left-side join key column names.
get_left_join_keys_list	Returns an ordered list of the left-side join key column names.
get_names_for_table_rename	Gets join mapping with renamed columns applied.
get_overlapping_columns	Finds column names that would conflict after the join.
get_overlapping_records	Finds column names that would conflict after the join.
get_right_join_keys	Returns a set of the right-side join key column names.
get_right_join_keys_list	Returns an ordered list of the right-side join key column names.
get_used_join_mapping	Returns the final join mapping after applying all renames and transformations.
parse_fuzz_mapping	Parses various input formats into a list of FuzzyMapping objects.
parse_select	Parses various input formats into a standardized JoinInputs object.
set_join_keys	Marks the SelectInput objects corresponding to join keys.
to_fuzzy_match_input	Creates a new FuzzyMatchInput instance based on the current manager settings.
to_join_input	Creates a new JoinInput instance based on the current manager settings.

Attributes:

Name	Type	Description
aggregate_output	bool	Backward compatibility: Access aggregate_output setting.
fuzzy_maps	list[FuzzyMapping]	Backward compatibility: Returns fuzzy mappings.
how	JoinStrategy	Backward compatibility: Access join strategy.
join_mapping	list[FuzzyMapping]	Backward compatibility: Access fuzzy join mapping.
left_join_keys	list[str]	Backward compatibility: Returns left join keys list.
left_select	JoinInputsManager	Backward compatibility: Access left_manager as left_select.
overlapping_records	set[str]	Backward compatibility: Returns overlapping column names.
right_join_keys	list[str]	Backward compatibility: Returns right join keys list.
right_select	JoinInputsManager	Backward compatibility: Access right_manager as right_select.
used_join_mapping	list[JoinMap]	Backward compatibility: Returns used join mapping.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class FuzzyMatchInputManager(JoinInputManager):
    """Manager for fuzzy matching join operations."""

    def __init__(self, fuzzy_input: FuzzyMatchInput):
        self.fuzzy_input = deepcopy(fuzzy_input)
        super().__init__(
            JoinInput(
                join_mapping=[
                    JoinMap(left_col=fm.left_col, right_col=fm.right_col) for fm in self.fuzzy_input.join_mapping
                ],
                left_select=self.fuzzy_input.left_select,
                right_select=self.fuzzy_input.right_select,
                how=self.fuzzy_input.how,
            )
        )

    @classmethod
    def create(
        cls,
        join_mapping: list[FuzzyMapping] | tuple[str, str] | str,
        left_select: list[SelectInput] | list[str],
        right_select: list[SelectInput] | list[str],
        aggregate_output: bool = False,
        how: JoinStrategy = "inner",
    ) -> "FuzzyMatchInputManager":
        """Factory method to create FuzzyMatchInput from various input formats."""
        parsed_mapping = cls.parse_fuzz_mapping(join_mapping)
        left_inputs = cls.parse_select(left_select)
        right_inputs = cls.parse_select(right_select)

        fuzzy_input = FuzzyMatchInput(
            join_mapping=parsed_mapping,
            left_select=left_inputs,
            right_select=right_inputs,
            how=how,
            aggregate_output=aggregate_output,
        )

        manager = cls(fuzzy_input)

        right_old_names = {v.old_name for v in fuzzy_input.right_select.renames}
        left_old_names = {v.old_name for v in fuzzy_input.left_select.renames}

        for jm in parsed_mapping:
            if jm.right_col not in right_old_names:
                manager.right_manager.append(SelectInput(old_name=jm.right_col, keep=False, join_key=True))
            if jm.left_col not in left_old_names:
                manager.left_manager.append(SelectInput(old_name=jm.left_col, keep=False, join_key=True))

        manager.set_join_keys()
        return manager

    @staticmethod
    def parse_fuzz_mapping(
        fuzz_mapping: list[FuzzyMapping] | tuple[str, str] | str | FuzzyMapping | list[dict],
    ) -> list[FuzzyMapping]:
        """Parses various input formats into a list of FuzzyMapping objects."""
        if isinstance(fuzz_mapping, tuple | list):
            if len(fuzz_mapping) == 0:
                raise ValueError("Fuzzy mapping cannot be empty")

            if all(isinstance(fm, dict) for fm in fuzz_mapping):
                return [FuzzyMapping(**fm) for fm in fuzz_mapping]

            if all(isinstance(fm, FuzzyMapping) for fm in fuzz_mapping):
                return fuzz_mapping

            if len(fuzz_mapping) <= 2:
                if len(fuzz_mapping) == 2:
                    if isinstance(fuzz_mapping[0], str) and isinstance(fuzz_mapping[1], str):
                        return [FuzzyMapping(left_col=fuzz_mapping[0], right_col=fuzz_mapping[1])]
                elif len(fuzz_mapping) == 1 and isinstance(fuzz_mapping[0], str):
                    return [FuzzyMapping(left_col=fuzz_mapping[0], right_col=fuzz_mapping[0])]

        elif isinstance(fuzz_mapping, str):
            return [FuzzyMapping(left_col=fuzz_mapping, right_col=fuzz_mapping)]

        elif isinstance(fuzz_mapping, FuzzyMapping):
            return [fuzz_mapping]

        raise ValueError(f"No valid fuzzy mapping as input: {type(fuzz_mapping)}")

    def get_fuzzy_maps(self) -> list[FuzzyMapping]:
        """Returns the final fuzzy mappings after applying all column renames."""
        new_mappings = []
        left_rename_table = self.left_manager.get_rename_table()
        right_rename_table = self.right_manager.get_rename_table()

        for org_fuzzy_map in self.fuzzy_input.join_mapping:
            right_col = right_rename_table.get(org_fuzzy_map.right_col, org_fuzzy_map.right_col)
            left_col = left_rename_table.get(org_fuzzy_map.left_col, org_fuzzy_map.left_col)

            if right_col != org_fuzzy_map.right_col or left_col != org_fuzzy_map.left_col:
                new_mapping = deepcopy(org_fuzzy_map)
                new_mapping.left_col = left_col
                new_mapping.right_col = right_col
                new_mappings.append(new_mapping)
            else:
                new_mappings.append(org_fuzzy_map)

        return new_mappings

    # === Backward Compatibility Properties ===

    @property
    def fuzzy_maps(self) -> list[FuzzyMapping]:
        """Backward compatibility: Returns fuzzy mappings."""
        return self.get_fuzzy_maps()

    @property
    def join_mapping(self) -> list[FuzzyMapping]:
        """Backward compatibility: Access fuzzy join mapping."""
        return self.get_fuzzy_maps()

    @property
    def aggregate_output(self) -> bool:
        """Backward compatibility: Access aggregate_output setting."""
        return self.fuzzy_input.aggregate_output

    def to_fuzzy_match_input(self) -> FuzzyMatchInput:
        """Creates a new FuzzyMatchInput instance based on the current manager settings.

        This is useful when you've modified the manager (e.g., via auto_rename) and
        want to get a fresh FuzzyMatchInput with all the current settings applied.

        Returns:
            A new FuzzyMatchInput instance with current settings
        """
        return FuzzyMatchInput(
            join_mapping=self.fuzzy_input.join_mapping,
            left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
            right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
            how=self.fuzzy_input.how,
            aggregate_output=self.fuzzy_input.aggregate_output,
        )

aggregate_output property

Backward compatibility: Access aggregate_output setting.

fuzzy_maps property

Backward compatibility: Returns fuzzy mappings.

how property

Backward compatibility: Access join strategy.

join_mapping property

Backward compatibility: Access fuzzy join mapping.

left_join_keys property

Backward compatibility: Returns left join keys list.

IMPORTANT: Uses the used_join_mapping PROPERTY (not method).

left_select property

Backward compatibility: Access left_manager as left_select.

This returns the MANAGER, not the data model. Usage: manager.left_select.join_key_selects

overlapping_records property

Backward compatibility: Returns overlapping column names.

right_join_keys property

Backward compatibility: Returns right join keys list.

IMPORTANT: Uses the used_join_mapping PROPERTY (not method).

right_select property

Backward compatibility: Access right_manager as right_select.

This returns the MANAGER, not the data model. Usage: manager.right_select.join_key_selects

used_join_mapping property

Backward compatibility: Returns used join mapping.

This property is critical - it's used by left_join_keys and right_join_keys.

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.input.right_select if side == "right" else self.input.left_select

    select_input.new_name = self.auto_generate_new_col_name(select_input.old_name, side=side)

    target_input.renames.append(select_input)

auto_generate_new_col_name(old_col_name, side)

Generates a new, non-conflicting column name by adding a suffix if necessary.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_generate_new_col_name(self, old_col_name: str, side: str) -> str:
    """Generates a new, non-conflicting column name by adding a suffix if necessary."""
    current_names = self.get_overlapping_columns()
    if old_col_name not in current_names:
        return old_col_name

    new_name = old_col_name
    while new_name in current_names:
        new_name = f"{side}_{new_name}"
    return new_name

auto_rename()

Automatically renames columns on the right side to prevent naming conflicts.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_rename(self) -> None:
    """Automatically renames columns on the right side to prevent naming conflicts."""
    self.set_join_keys()
    overlapping_records = self.get_overlapping_records()

    while len(overlapping_records) > 0:
        for right_col in self.input.right_select.renames:
            if right_col.new_name in overlapping_records:
                right_col.new_name = right_col.new_name + "_right"
        overlapping_records = self.get_overlapping_records()

create(join_mapping, left_select, right_select, aggregate_output=False, how='inner') classmethod

Factory method to create FuzzyMatchInput from various input formats.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create(
    cls,
    join_mapping: list[FuzzyMapping] | tuple[str, str] | str,
    left_select: list[SelectInput] | list[str],
    right_select: list[SelectInput] | list[str],
    aggregate_output: bool = False,
    how: JoinStrategy = "inner",
) -> "FuzzyMatchInputManager":
    """Factory method to create FuzzyMatchInput from various input formats."""
    parsed_mapping = cls.parse_fuzz_mapping(join_mapping)
    left_inputs = cls.parse_select(left_select)
    right_inputs = cls.parse_select(right_select)

    fuzzy_input = FuzzyMatchInput(
        join_mapping=parsed_mapping,
        left_select=left_inputs,
        right_select=right_inputs,
        how=how,
        aggregate_output=aggregate_output,
    )

    manager = cls(fuzzy_input)

    right_old_names = {v.old_name for v in fuzzy_input.right_select.renames}
    left_old_names = {v.old_name for v in fuzzy_input.left_select.renames}

    for jm in parsed_mapping:
        if jm.right_col not in right_old_names:
            manager.right_manager.append(SelectInput(old_name=jm.right_col, keep=False, join_key=True))
        if jm.left_col not in left_old_names:
            manager.left_manager.append(SelectInput(old_name=jm.left_col, keep=False, join_key=True))

    manager.set_join_keys()
    return manager

get_fuzzy_maps()

Returns the final fuzzy mappings after applying all column renames.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_fuzzy_maps(self) -> list[FuzzyMapping]:
    """Returns the final fuzzy mappings after applying all column renames."""
    new_mappings = []
    left_rename_table = self.left_manager.get_rename_table()
    right_rename_table = self.right_manager.get_rename_table()

    for org_fuzzy_map in self.fuzzy_input.join_mapping:
        right_col = right_rename_table.get(org_fuzzy_map.right_col, org_fuzzy_map.right_col)
        left_col = left_rename_table.get(org_fuzzy_map.left_col, org_fuzzy_map.left_col)

        if right_col != org_fuzzy_map.right_col or left_col != org_fuzzy_map.left_col:
            new_mapping = deepcopy(org_fuzzy_map)
            new_mapping.left_col = left_col
            new_mapping.right_col = right_col
            new_mappings.append(new_mapping)
        else:
            new_mappings.append(org_fuzzy_map)

    return new_mappings

get_join_key_renames(filter_drop=False)

Gets the temporary rename mappings for the join keys on both sides.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_join_key_renames(self, filter_drop: bool = False) -> FullJoinKeyResponse:
    """Gets the temporary rename mappings for the join keys on both sides."""
    left_renames = self.left_manager.get_join_key_renames(side="left", filter_drop=filter_drop)
    right_renames = self.right_manager.get_join_key_renames(side="right", filter_drop=filter_drop)
    return FullJoinKeyResponse(left_renames, right_renames)

get_left_join_keys()

Returns a set of the left-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_left_join_keys(self) -> set[str]:
    """Returns a set of the left-side join key column names."""
    return self._get_left_join_keys_set()

get_left_join_keys_list()

Returns an ordered list of the left-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_left_join_keys_list(self) -> list[str]:
    """Returns an ordered list of the left-side join key column names."""
    return [jm.left_col for jm in self.used_join_mapping]

get_names_for_table_rename()

Gets join mapping with renamed columns applied.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_names_for_table_rename(self) -> list[JoinMap]:
    """Gets join mapping with renamed columns applied."""
    new_mappings: list[JoinMap] = []
    left_rename_table = self.left_manager.get_rename_table()
    right_rename_table = self.right_manager.get_rename_table()

    for join_map in self.input.join_mapping:
        new_left = left_rename_table.get(join_map.left_col, join_map.left_col)
        new_right = right_rename_table.get(join_map.right_col, join_map.right_col)
        new_mappings.append(JoinMap(left_col=new_left, right_col=new_right))

    return new_mappings

get_overlapping_columns()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_columns(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.left_manager.get_new_cols() & self.right_manager.get_new_cols()

get_overlapping_records()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_records(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.get_overlapping_columns()

get_right_join_keys()

Returns a set of the right-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_right_join_keys(self) -> set[str]:
    """Returns a set of the right-side join key column names."""
    return self._get_right_join_keys_set()

get_right_join_keys_list()

Returns an ordered list of the right-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_right_join_keys_list(self) -> list[str]:
    """Returns an ordered list of the right-side join key column names."""
    return [jm.right_col for jm in self.used_join_mapping]

get_used_join_mapping()

Returns the final join mapping after applying all renames and transformations.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_used_join_mapping(self) -> list[JoinMap]:
    """Returns the final join mapping after applying all renames and transformations."""
    new_mappings: list[JoinMap] = []
    left_rename_table = self.left_manager.get_rename_table()
    right_rename_table = self.right_manager.get_rename_table()
    left_join_rename_mapping = self.left_manager.get_join_key_rename_mapping("left")
    right_join_rename_mapping = self.right_manager.get_join_key_rename_mapping("right")
    for join_map in self.input.join_mapping:
        left_col = left_rename_table.get(join_map.left_col, join_map.left_col)
        right_col = right_rename_table.get(join_map.right_col, join_map.left_col)

        final_left = left_join_rename_mapping.get(left_col, None)
        final_right = right_join_rename_mapping.get(right_col, None)

        new_mappings.append(JoinMap(left_col=final_left, right_col=final_right))

    return new_mappings

parse_fuzz_mapping(fuzz_mapping) staticmethod

Parses various input formats into a list of FuzzyMapping objects.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@staticmethod
def parse_fuzz_mapping(
    fuzz_mapping: list[FuzzyMapping] | tuple[str, str] | str | FuzzyMapping | list[dict],
) -> list[FuzzyMapping]:
    """Parses various input formats into a list of FuzzyMapping objects."""
    if isinstance(fuzz_mapping, tuple | list):
        if len(fuzz_mapping) == 0:
            raise ValueError("Fuzzy mapping cannot be empty")

        if all(isinstance(fm, dict) for fm in fuzz_mapping):
            return [FuzzyMapping(**fm) for fm in fuzz_mapping]

        if all(isinstance(fm, FuzzyMapping) for fm in fuzz_mapping):
            return fuzz_mapping

        if len(fuzz_mapping) <= 2:
            if len(fuzz_mapping) == 2:
                if isinstance(fuzz_mapping[0], str) and isinstance(fuzz_mapping[1], str):
                    return [FuzzyMapping(left_col=fuzz_mapping[0], right_col=fuzz_mapping[1])]
            elif len(fuzz_mapping) == 1 and isinstance(fuzz_mapping[0], str):
                return [FuzzyMapping(left_col=fuzz_mapping[0], right_col=fuzz_mapping[0])]

    elif isinstance(fuzz_mapping, str):
        return [FuzzyMapping(left_col=fuzz_mapping, right_col=fuzz_mapping)]

    elif isinstance(fuzz_mapping, FuzzyMapping):
        return [fuzz_mapping]

    raise ValueError(f"No valid fuzzy mapping as input: {type(fuzz_mapping)}")

parse_select(select) staticmethod

Parses various input formats into a standardized JoinInputs object.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@staticmethod
def parse_select(select: list[SelectInput] | list[str] | list[dict] | dict) -> JoinInputs:
    """Parses various input formats into a standardized `JoinInputs` object."""
    if not select:
        return JoinInputs(renames=[])

    if all(isinstance(c, SelectInput) for c in select):
        return JoinInputs(renames=select)
    elif all(isinstance(c, dict) for c in select):
        return JoinInputs(renames=[SelectInput(**c) for c in select])
    elif isinstance(select, dict):
        renames = select.get("renames")
        if renames:
            return JoinInputs(renames=[SelectInput(**c) for c in renames])
        return JoinInputs(renames=[])
    elif all(isinstance(c, str) for c in select):
        return JoinInputs(renames=[SelectInput(old_name=s, new_name=s) for s in select])

    raise ValueError(f"Unable to parse select input: {type(select)}")

set_join_keys()

Marks the SelectInput objects corresponding to join keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def set_join_keys(self) -> None:
    """Marks the `SelectInput` objects corresponding to join keys."""
    left_join_keys = self._get_left_join_keys_set()
    right_join_keys = self._get_right_join_keys_set()

    for select_input in self.input.left_select.renames:
        select_input.join_key = select_input.old_name in left_join_keys

    for select_input in self.input.right_select.renames:
        select_input.join_key = select_input.old_name in right_join_keys

to_fuzzy_match_input()

Creates a new FuzzyMatchInput instance based on the current manager settings.

This is useful when you've modified the manager (e.g., via auto_rename) and want to get a fresh FuzzyMatchInput with all the current settings applied.

Returns:

Type	Description
FuzzyMatchInput	A new FuzzyMatchInput instance with current settings

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_fuzzy_match_input(self) -> FuzzyMatchInput:
    """Creates a new FuzzyMatchInput instance based on the current manager settings.

    This is useful when you've modified the manager (e.g., via auto_rename) and
    want to get a fresh FuzzyMatchInput with all the current settings applied.

    Returns:
        A new FuzzyMatchInput instance with current settings
    """
    return FuzzyMatchInput(
        join_mapping=self.fuzzy_input.join_mapping,
        left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
        right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
        how=self.fuzzy_input.how,
        aggregate_output=self.fuzzy_input.aggregate_output,
    )

to_join_input()

Creates a new JoinInput instance based on the current manager settings.

This is useful when you've modified the manager (e.g., via auto_rename) and want to get a fresh JoinInput with all the current settings applied.

Returns:

Type	Description
JoinInput	A new JoinInput instance with current settings

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_join_input(self) -> JoinInput:
    """Creates a new JoinInput instance based on the current manager settings.

    This is useful when you've modified the manager (e.g., via auto_rename) and
    want to get a fresh JoinInput with all the current settings applied.

    Returns:
        A new JoinInput instance with current settings
    """
    return JoinInput(
        join_mapping=self.input.join_mapping,
        left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
        right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
        how=self.input.how,
    )

GraphSolverInput pydantic-model

Bases: BaseModel

Defines settings for a graph-solving operation (e.g., finding connected components).

Show JSON schema:

{
  "description": "Defines settings for a graph-solving operation (e.g., finding connected components).",
  "properties": {
    "col_from": {
      "title": "Col From",
      "type": "string"
    },
    "col_to": {
      "title": "Col To",
      "type": "string"
    },
    "output_column_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "graph_group",
      "title": "Output Column Name"
    }
  },
  "required": [
    "col_from",
    "col_to"
  ],
  "title": "GraphSolverInput",
  "type": "object"
}

Fields:

• col_from (str)
• col_to (str)
• output_column_name (str | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class GraphSolverInput(BaseModel):
    """Defines settings for a graph-solving operation (e.g., finding connected components)."""

    col_from: str
    col_to: str
    output_column_name: str | None = "graph_group"

GroupByInput pydantic-model

Bases: BaseModel

A data class that represents the input for a group by operation.

Attributes

agg_cols : List[AggColl] A list of AggColl objects that specify the aggregation operations to perform on the DataFrame columns after grouping. Each AggColl object should specify the column to be aggregated and the aggregation function to use.

Example

group_by_input = GroupByInput( agg_cols=[AggColl(old_name='ix', agg='groupby'), AggColl(old_name='groups', agg='groupby'), AggColl(old_name='col1', agg='sum'), AggColl(old_name='col2', agg='mean')] )

Show JSON schema:

{
  "$defs": {
    "AggColl": {
      "description": "A data class that represents a single aggregation operation for a group by operation.\n\nAttributes\n----------\nold_name : str\n    The name of the column in the original DataFrame to be aggregated.\n\nagg : str\n    The aggregation function to use. This can be a string representing a built-in function or a custom function.\n\nnew_name : Optional[str]\n    The name of the resulting aggregated column in the output DataFrame. If not provided, it will default to the\n    old_name appended with the aggregation function.\n\noutput_type : Optional[str]\n    The type of the output values of the aggregation. If not provided, it is inferred from the aggregation function\n    using the `get_func_type_mapping` function.\n\nExample\n--------\nagg_col = AggColl(\n    old_name='col1',\n    agg='sum',\n    new_name='sum_col1',\n    output_type='float'\n)",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "agg": {
          "title": "Agg",
          "type": "string"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "output_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Type"
        }
      },
      "required": [
        "old_name",
        "agg"
      ],
      "title": "AggColl",
      "type": "object"
    }
  },
  "description": "A data class that represents the input for a group by operation.\n\nAttributes\n----------\nagg_cols : List[AggColl]\n    A list of `AggColl` objects that specify the aggregation operations to perform on the DataFrame columns\n    after grouping. Each `AggColl` object should specify the column to be aggregated and the aggregation\n    function to use.\n\nExample\n--------\ngroup_by_input = GroupByInput(\n    agg_cols=[AggColl(old_name='ix', agg='groupby'), AggColl(old_name='groups', agg='groupby'),\n              AggColl(old_name='col1', agg='sum'), AggColl(old_name='col2', agg='mean')]\n)",
  "properties": {
    "agg_cols": {
      "items": {
        "$ref": "#/$defs/AggColl"
      },
      "title": "Agg Cols",
      "type": "array"
    }
  },
  "required": [
    "agg_cols"
  ],
  "title": "GroupByInput",
  "type": "object"
}

Fields:

• agg_cols (list[AggColl])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class GroupByInput(BaseModel):
    """
    A data class that represents the input for a group by operation.

    Attributes
    ----------
    agg_cols : List[AggColl]
        A list of `AggColl` objects that specify the aggregation operations to perform on the DataFrame columns
        after grouping. Each `AggColl` object should specify the column to be aggregated and the aggregation
        function to use.

    Example
    --------
    group_by_input = GroupByInput(
        agg_cols=[AggColl(old_name='ix', agg='groupby'), AggColl(old_name='groups', agg='groupby'),
                  AggColl(old_name='col1', agg='sum'), AggColl(old_name='col2', agg='mean')]
    )
    """

    agg_cols: list[AggColl]

    def __init__(self, agg_cols: list[AggColl]):
        """Backwards compatibility implementation"""
        super().__init__(agg_cols=agg_cols)

__init__(agg_cols)

Backwards compatibility implementation

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __init__(self, agg_cols: list[AggColl]):
    """Backwards compatibility implementation"""
    super().__init__(agg_cols=agg_cols)

JoinInput pydantic-model

Bases: BaseModel

Data model for standard SQL-style join operations.

Show JSON schema:

{
  "$defs": {
    "JoinInputs": {
      "description": "Data model for join-specific select inputs (extends SelectInputs).",
      "properties": {
        "renames": {
          "items": {
            "$ref": "#/$defs/SelectInput"
          },
          "title": "Renames",
          "type": "array"
        }
      },
      "title": "JoinInputs",
      "type": "object"
    },
    "JoinMap": {
      "description": "Defines a single mapping between a left and right column for a join key.",
      "properties": {
        "left_col": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Left Col"
        },
        "right_col": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Right Col"
        }
      },
      "title": "JoinMap",
      "type": "object"
    },
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Data model for standard SQL-style join operations.",
  "properties": {
    "join_mapping": {
      "items": {
        "$ref": "#/$defs/JoinMap"
      },
      "title": "Join Mapping",
      "type": "array"
    },
    "left_select": {
      "$ref": "#/$defs/JoinInputs"
    },
    "right_select": {
      "$ref": "#/$defs/JoinInputs"
    },
    "how": {
      "default": "inner",
      "enum": [
        "inner",
        "left",
        "right",
        "full",
        "semi",
        "anti",
        "outer"
      ],
      "title": "How",
      "type": "string"
    }
  },
  "required": [
    "join_mapping",
    "left_select",
    "right_select"
  ],
  "title": "JoinInput",
  "type": "object"
}

Fields:

• join_mapping (list[JoinMap])
• left_select (JoinInputs)
• right_select (JoinInputs)
• how (JoinKeyStrategy)

Validators:

• parse_inputs

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinInput(BaseModel):
    """Data model for standard SQL-style join operations."""

    join_mapping: list[JoinMap]
    left_select: JoinInputs
    right_select: JoinInputs
    how: JoinKeyStrategy = "inner"

    @model_validator(mode="before")
    @classmethod
    def parse_inputs(cls, data: Any) -> Any:
        """Parse flexible input formats before validation."""
        if isinstance(data, dict):
            if "join_mapping" in data:
                data["join_mapping"] = cls._parse_join_mapping(data["join_mapping"])

            if "left_select" in data:
                data["left_select"] = cls._parse_select(data["left_select"])

            if "right_select" in data:
                data["right_select"] = cls._parse_select(data["right_select"])

        return data

    @staticmethod
    def _parse_join_mapping(join_mapping: Any) -> list[JoinMap]:
        """Parse various join_mapping formats."""
        if isinstance(join_mapping, list):
            result = []
            for jm in join_mapping:
                if isinstance(jm, JoinMap):
                    result.append(jm)
                elif isinstance(jm, dict):
                    result.append(JoinMap(**jm))
                elif isinstance(jm, tuple | list) and len(jm) == 2:
                    result.append(JoinMap(left_col=jm[0], right_col=jm[1]))
                elif isinstance(jm, str):
                    result.append(JoinMap(left_col=jm, right_col=jm))
                else:
                    raise ValueError(f"Invalid join mapping item: {jm}")
            return result

        if isinstance(join_mapping, JoinMap):
            return [join_mapping]

        # String: same column on both sides
        if isinstance(join_mapping, str):
            return [JoinMap(left_col=join_mapping, right_col=join_mapping)]

        # Tuple: (left, right)
        if isinstance(join_mapping, tuple) and len(join_mapping) == 2:
            return [JoinMap(left_col=join_mapping[0], right_col=join_mapping[1])]

        raise ValueError(f"Invalid join_mapping format: {type(join_mapping)}")

    @staticmethod
    def _parse_select(select: Any) -> JoinInputs:
        """Parse various select input formats."""
        if isinstance(select, JoinInputs):
            return select

        if isinstance(select, list):
            if all(isinstance(s, SelectInput) for s in select):
                return JoinInputs(renames=select)
            elif all(isinstance(s, str) for s in select):
                return JoinInputs(renames=[SelectInput(old_name=s) for s in select])
            elif all(isinstance(s, dict) for s in select):
                return JoinInputs(renames=[SelectInput(**s) for s in select])

        # Dict with 'select' (new YAML) or 'renames' (internal) key
        if isinstance(select, dict):
            if "select" in select:
                return JoinInputs(renames=[SelectInput.from_yaml_dict(s) for s in select["select"]])
            if "renames" in select:
                return JoinInputs(**select)

        raise ValueError(f"Invalid select format: {type(select)}")

    def __init__(
        self,
        join_mapping: list[JoinMap] | JoinMap | tuple[str, str] | str | list[tuple] | list[str] = None,
        left_select: JoinInputs | list[SelectInput] | list[str] = None,
        right_select: JoinInputs | list[SelectInput] | list[str] = None,
        how: JoinKeyStrategy = "inner",
        **data,
    ):
        """Custom init for backward compatibility with positional arguments."""
        if join_mapping is not None:
            data["join_mapping"] = join_mapping
        if left_select is not None:
            data["left_select"] = left_select
        if right_select is not None:
            data["right_select"] = right_select
        if how is not None:
            data["how"] = how

        super().__init__(**data)

    def to_yaml_dict(self) -> JoinInputYaml:
        """Serialize for YAML output."""
        return {
            "join_mapping": [{"left_col": jm.left_col, "right_col": jm.right_col} for jm in self.join_mapping],
            "left_select": self.left_select.to_yaml_dict(),
            "right_select": self.right_select.to_yaml_dict(),
            "how": self.how,
        }

    def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
        """Adds a new column to the selection for either the left or right side."""
        target_input = self.right_select if side == "right" else self.left_select
        if select_input.new_name is None:
            select_input.new_name = select_input.old_name
        target_input.renames.append(select_input)

__init__(join_mapping=None, left_select=None, right_select=None, how='inner', **data)

Custom init for backward compatibility with positional arguments.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __init__(
    self,
    join_mapping: list[JoinMap] | JoinMap | tuple[str, str] | str | list[tuple] | list[str] = None,
    left_select: JoinInputs | list[SelectInput] | list[str] = None,
    right_select: JoinInputs | list[SelectInput] | list[str] = None,
    how: JoinKeyStrategy = "inner",
    **data,
):
    """Custom init for backward compatibility with positional arguments."""
    if join_mapping is not None:
        data["join_mapping"] = join_mapping
    if left_select is not None:
        data["left_select"] = left_select
    if right_select is not None:
        data["right_select"] = right_select
    if how is not None:
        data["how"] = how

    super().__init__(**data)

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.right_select if side == "right" else self.left_select
    if select_input.new_name is None:
        select_input.new_name = select_input.old_name
    target_input.renames.append(select_input)

parse_inputs(data) pydantic-validator

Parse flexible input formats before validation.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="before")
@classmethod
def parse_inputs(cls, data: Any) -> Any:
    """Parse flexible input formats before validation."""
    if isinstance(data, dict):
        if "join_mapping" in data:
            data["join_mapping"] = cls._parse_join_mapping(data["join_mapping"])

        if "left_select" in data:
            data["left_select"] = cls._parse_select(data["left_select"])

        if "right_select" in data:
            data["right_select"] = cls._parse_select(data["right_select"])

    return data

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> JoinInputYaml:
    """Serialize for YAML output."""
    return {
        "join_mapping": [{"left_col": jm.left_col, "right_col": jm.right_col} for jm in self.join_mapping],
        "left_select": self.left_select.to_yaml_dict(),
        "right_select": self.right_select.to_yaml_dict(),
        "how": self.how,
    }

JoinInputManager

Bases: JoinSelectManagerMixin

Manager for standard SQL-style join operations.

Methods:

Name	Description
add_new_select_column	Adds a new column to the selection for either the left or right side.
auto_generate_new_col_name	Generates a new, non-conflicting column name by adding a suffix if necessary.
auto_rename	Automatically renames columns on the right side to prevent naming conflicts.
create	Factory method to create JoinInput from various input formats.
get_join_key_renames	Gets the temporary rename mappings for the join keys on both sides.
get_left_join_keys	Returns a set of the left-side join key column names.
get_left_join_keys_list	Returns an ordered list of the left-side join key column names.
get_names_for_table_rename	Gets join mapping with renamed columns applied.
get_overlapping_columns	Finds column names that would conflict after the join.
get_overlapping_records	Finds column names that would conflict after the join.
get_right_join_keys	Returns a set of the right-side join key column names.
get_right_join_keys_list	Returns an ordered list of the right-side join key column names.
get_used_join_mapping	Returns the final join mapping after applying all renames and transformations.
parse_select	Parses various input formats into a standardized JoinInputs object.
set_join_keys	Marks the SelectInput objects corresponding to join keys.
to_join_input	Creates a new JoinInput instance based on the current manager settings.

Attributes:

Name	Type	Description
how	JoinStrategy	Backward compatibility: Access join strategy.
join_mapping	list[JoinMap]	Backward compatibility: Access join mapping.
left_join_keys	list[str]	Backward compatibility: Returns left join keys list.
left_select	JoinInputsManager	Backward compatibility: Access left_manager as left_select.
overlapping_records	set[str]	Backward compatibility: Returns overlapping column names.
right_join_keys	list[str]	Backward compatibility: Returns right join keys list.
right_select	JoinInputsManager	Backward compatibility: Access right_manager as right_select.
used_join_mapping	list[JoinMap]	Backward compatibility: Returns used join mapping.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinInputManager(JoinSelectManagerMixin):
    """Manager for standard SQL-style join operations."""

    def __init__(self, join_input: JoinInput):
        self.input = deepcopy(join_input)
        self.left_manager = JoinInputsManager(self.input.left_select)
        self.right_manager = JoinInputsManager(self.input.right_select)
        self.set_join_keys()

    @classmethod
    def create(
        cls,
        join_mapping: list[JoinMap] | tuple[str, str] | str,
        left_select: list[SelectInput] | list[str],
        right_select: list[SelectInput] | list[str],
        how: JoinKeyStrategy = "inner",
    ) -> "JoinInputManager":
        """Factory method to create JoinInput from various input formats."""
        join_input = JoinInput(join_mapping=join_mapping, left_select=left_select, right_select=right_select, how=how)

        manager = cls(join_input)
        manager.set_join_keys()
        return manager

    def set_join_keys(self) -> None:
        """Marks the `SelectInput` objects corresponding to join keys."""
        left_join_keys = self._get_left_join_keys_set()
        right_join_keys = self._get_right_join_keys_set()

        for select_input in self.input.left_select.renames:
            select_input.join_key = select_input.old_name in left_join_keys

        for select_input in self.input.right_select.renames:
            select_input.join_key = select_input.old_name in right_join_keys

    def _get_left_join_keys_set(self) -> set[str]:
        """Internal: Returns a set of the left-side join key column names."""
        return {jm.left_col for jm in self.input.join_mapping}

    def _get_right_join_keys_set(self) -> set[str]:
        """Internal: Returns a set of the right-side join key column names."""
        return {jm.right_col for jm in self.input.join_mapping}

    def get_left_join_keys(self) -> set[str]:
        """Returns a set of the left-side join key column names."""
        return self._get_left_join_keys_set()

    def get_right_join_keys(self) -> set[str]:
        """Returns a set of the right-side join key column names."""
        return self._get_right_join_keys_set()

    def get_left_join_keys_list(self) -> list[str]:
        """Returns an ordered list of the left-side join key column names."""
        return [jm.left_col for jm in self.used_join_mapping]

    def get_right_join_keys_list(self) -> list[str]:
        """Returns an ordered list of the right-side join key column names."""
        return [jm.right_col for jm in self.used_join_mapping]

    def get_overlapping_records(self) -> set[str]:
        """Finds column names that would conflict after the join."""
        return self.get_overlapping_columns()

    def auto_rename(self) -> None:
        """Automatically renames columns on the right side to prevent naming conflicts."""
        self.set_join_keys()
        overlapping_records = self.get_overlapping_records()

        while len(overlapping_records) > 0:
            for right_col in self.input.right_select.renames:
                if right_col.new_name in overlapping_records:
                    right_col.new_name = right_col.new_name + "_right"
            overlapping_records = self.get_overlapping_records()

    def get_join_key_renames(self, filter_drop: bool = False) -> FullJoinKeyResponse:
        """Gets the temporary rename mappings for the join keys on both sides."""
        left_renames = self.left_manager.get_join_key_renames(side="left", filter_drop=filter_drop)
        right_renames = self.right_manager.get_join_key_renames(side="right", filter_drop=filter_drop)
        return FullJoinKeyResponse(left_renames, right_renames)

    def get_names_for_table_rename(self) -> list[JoinMap]:
        """Gets join mapping with renamed columns applied."""
        new_mappings: list[JoinMap] = []
        left_rename_table = self.left_manager.get_rename_table()
        right_rename_table = self.right_manager.get_rename_table()

        for join_map in self.input.join_mapping:
            new_left = left_rename_table.get(join_map.left_col, join_map.left_col)
            new_right = right_rename_table.get(join_map.right_col, join_map.right_col)
            new_mappings.append(JoinMap(left_col=new_left, right_col=new_right))

        return new_mappings

    def get_used_join_mapping(self) -> list[JoinMap]:
        """Returns the final join mapping after applying all renames and transformations."""
        new_mappings: list[JoinMap] = []
        left_rename_table = self.left_manager.get_rename_table()
        right_rename_table = self.right_manager.get_rename_table()
        left_join_rename_mapping = self.left_manager.get_join_key_rename_mapping("left")
        right_join_rename_mapping = self.right_manager.get_join_key_rename_mapping("right")
        for join_map in self.input.join_mapping:
            left_col = left_rename_table.get(join_map.left_col, join_map.left_col)
            right_col = right_rename_table.get(join_map.right_col, join_map.left_col)

            final_left = left_join_rename_mapping.get(left_col, None)
            final_right = right_join_rename_mapping.get(right_col, None)

            new_mappings.append(JoinMap(left_col=final_left, right_col=final_right))

        return new_mappings

    def to_join_input(self) -> JoinInput:
        """Creates a new JoinInput instance based on the current manager settings.

        This is useful when you've modified the manager (e.g., via auto_rename) and
        want to get a fresh JoinInput with all the current settings applied.

        Returns:
            A new JoinInput instance with current settings
        """
        return JoinInput(
            join_mapping=self.input.join_mapping,
            left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
            right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
            how=self.input.how,
        )

    @property
    def left_select(self) -> JoinInputsManager:
        """Backward compatibility: Access left_manager as left_select.

        This returns the MANAGER, not the data model.
        Usage: manager.left_select.join_key_selects
        """
        return self.left_manager

    @property
    def right_select(self) -> JoinInputsManager:
        """Backward compatibility: Access right_manager as right_select.

        This returns the MANAGER, not the data model.
        Usage: manager.right_select.join_key_selects
        """
        return self.right_manager

    @property
    def how(self) -> JoinStrategy:
        """Backward compatibility: Access join strategy."""
        return self.input.how

    @property
    def join_mapping(self) -> list[JoinMap]:
        """Backward compatibility: Access join mapping."""
        return self.input.join_mapping

    @property
    def overlapping_records(self) -> set[str]:
        """Backward compatibility: Returns overlapping column names."""
        return self.get_overlapping_records()

    @property
    def used_join_mapping(self) -> list[JoinMap]:
        """Backward compatibility: Returns used join mapping.

        This property is critical - it's used by left_join_keys and right_join_keys.
        """
        return self.get_used_join_mapping()

    @property
    def left_join_keys(self) -> list[str]:
        """Backward compatibility: Returns left join keys list.

        IMPORTANT: Uses the used_join_mapping PROPERTY (not method).
        """
        return [jm.left_col for jm in self.used_join_mapping]

    @property
    def right_join_keys(self) -> list[str]:
        """Backward compatibility: Returns right join keys list.

        IMPORTANT: Uses the used_join_mapping PROPERTY (not method).
        """
        return [jm.right_col for jm in self.used_join_mapping]

    @property
    def _left_join_keys(self) -> set[str]:
        """Backward compatibility: Private property for left join key set."""
        return self._get_left_join_keys_set()

    @property
    def _right_join_keys(self) -> set[str]:
        """Backward compatibility: Private property for right join key set."""
        return self._get_right_join_keys_set()

how property

Backward compatibility: Access join strategy.

join_mapping property

Backward compatibility: Access join mapping.

left_join_keys property

Backward compatibility: Returns left join keys list.

IMPORTANT: Uses the used_join_mapping PROPERTY (not method).

left_select property

Backward compatibility: Access left_manager as left_select.

This returns the MANAGER, not the data model. Usage: manager.left_select.join_key_selects

overlapping_records property

Backward compatibility: Returns overlapping column names.

right_join_keys property

Backward compatibility: Returns right join keys list.

IMPORTANT: Uses the used_join_mapping PROPERTY (not method).

right_select property

Backward compatibility: Access right_manager as right_select.

This returns the MANAGER, not the data model. Usage: manager.right_select.join_key_selects

used_join_mapping property

Backward compatibility: Returns used join mapping.

This property is critical - it's used by left_join_keys and right_join_keys.

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.input.right_select if side == "right" else self.input.left_select

    select_input.new_name = self.auto_generate_new_col_name(select_input.old_name, side=side)

    target_input.renames.append(select_input)

auto_generate_new_col_name(old_col_name, side)

Generates a new, non-conflicting column name by adding a suffix if necessary.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_generate_new_col_name(self, old_col_name: str, side: str) -> str:
    """Generates a new, non-conflicting column name by adding a suffix if necessary."""
    current_names = self.get_overlapping_columns()
    if old_col_name not in current_names:
        return old_col_name

    new_name = old_col_name
    while new_name in current_names:
        new_name = f"{side}_{new_name}"
    return new_name

auto_rename()

Automatically renames columns on the right side to prevent naming conflicts.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_rename(self) -> None:
    """Automatically renames columns on the right side to prevent naming conflicts."""
    self.set_join_keys()
    overlapping_records = self.get_overlapping_records()

    while len(overlapping_records) > 0:
        for right_col in self.input.right_select.renames:
            if right_col.new_name in overlapping_records:
                right_col.new_name = right_col.new_name + "_right"
        overlapping_records = self.get_overlapping_records()

create(join_mapping, left_select, right_select, how='inner') classmethod

Factory method to create JoinInput from various input formats.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create(
    cls,
    join_mapping: list[JoinMap] | tuple[str, str] | str,
    left_select: list[SelectInput] | list[str],
    right_select: list[SelectInput] | list[str],
    how: JoinKeyStrategy = "inner",
) -> "JoinInputManager":
    """Factory method to create JoinInput from various input formats."""
    join_input = JoinInput(join_mapping=join_mapping, left_select=left_select, right_select=right_select, how=how)

    manager = cls(join_input)
    manager.set_join_keys()
    return manager

get_join_key_renames(filter_drop=False)

Gets the temporary rename mappings for the join keys on both sides.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_join_key_renames(self, filter_drop: bool = False) -> FullJoinKeyResponse:
    """Gets the temporary rename mappings for the join keys on both sides."""
    left_renames = self.left_manager.get_join_key_renames(side="left", filter_drop=filter_drop)
    right_renames = self.right_manager.get_join_key_renames(side="right", filter_drop=filter_drop)
    return FullJoinKeyResponse(left_renames, right_renames)

get_left_join_keys()

Returns a set of the left-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_left_join_keys(self) -> set[str]:
    """Returns a set of the left-side join key column names."""
    return self._get_left_join_keys_set()

get_left_join_keys_list()

Returns an ordered list of the left-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_left_join_keys_list(self) -> list[str]:
    """Returns an ordered list of the left-side join key column names."""
    return [jm.left_col for jm in self.used_join_mapping]

get_names_for_table_rename()

Gets join mapping with renamed columns applied.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_names_for_table_rename(self) -> list[JoinMap]:
    """Gets join mapping with renamed columns applied."""
    new_mappings: list[JoinMap] = []
    left_rename_table = self.left_manager.get_rename_table()
    right_rename_table = self.right_manager.get_rename_table()

    for join_map in self.input.join_mapping:
        new_left = left_rename_table.get(join_map.left_col, join_map.left_col)
        new_right = right_rename_table.get(join_map.right_col, join_map.right_col)
        new_mappings.append(JoinMap(left_col=new_left, right_col=new_right))

    return new_mappings

get_overlapping_columns()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_columns(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.left_manager.get_new_cols() & self.right_manager.get_new_cols()

get_overlapping_records()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_records(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.get_overlapping_columns()

get_right_join_keys()

Returns a set of the right-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_right_join_keys(self) -> set[str]:
    """Returns a set of the right-side join key column names."""
    return self._get_right_join_keys_set()

get_right_join_keys_list()

Returns an ordered list of the right-side join key column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_right_join_keys_list(self) -> list[str]:
    """Returns an ordered list of the right-side join key column names."""
    return [jm.right_col for jm in self.used_join_mapping]

get_used_join_mapping()

Returns the final join mapping after applying all renames and transformations.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_used_join_mapping(self) -> list[JoinMap]:
    """Returns the final join mapping after applying all renames and transformations."""
    new_mappings: list[JoinMap] = []
    left_rename_table = self.left_manager.get_rename_table()
    right_rename_table = self.right_manager.get_rename_table()
    left_join_rename_mapping = self.left_manager.get_join_key_rename_mapping("left")
    right_join_rename_mapping = self.right_manager.get_join_key_rename_mapping("right")
    for join_map in self.input.join_mapping:
        left_col = left_rename_table.get(join_map.left_col, join_map.left_col)
        right_col = right_rename_table.get(join_map.right_col, join_map.left_col)

        final_left = left_join_rename_mapping.get(left_col, None)
        final_right = right_join_rename_mapping.get(right_col, None)

        new_mappings.append(JoinMap(left_col=final_left, right_col=final_right))

    return new_mappings

parse_select(select) staticmethod

Parses various input formats into a standardized JoinInputs object.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@staticmethod
def parse_select(select: list[SelectInput] | list[str] | list[dict] | dict) -> JoinInputs:
    """Parses various input formats into a standardized `JoinInputs` object."""
    if not select:
        return JoinInputs(renames=[])

    if all(isinstance(c, SelectInput) for c in select):
        return JoinInputs(renames=select)
    elif all(isinstance(c, dict) for c in select):
        return JoinInputs(renames=[SelectInput(**c) for c in select])
    elif isinstance(select, dict):
        renames = select.get("renames")
        if renames:
            return JoinInputs(renames=[SelectInput(**c) for c in renames])
        return JoinInputs(renames=[])
    elif all(isinstance(c, str) for c in select):
        return JoinInputs(renames=[SelectInput(old_name=s, new_name=s) for s in select])

    raise ValueError(f"Unable to parse select input: {type(select)}")

set_join_keys()

Marks the SelectInput objects corresponding to join keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def set_join_keys(self) -> None:
    """Marks the `SelectInput` objects corresponding to join keys."""
    left_join_keys = self._get_left_join_keys_set()
    right_join_keys = self._get_right_join_keys_set()

    for select_input in self.input.left_select.renames:
        select_input.join_key = select_input.old_name in left_join_keys

    for select_input in self.input.right_select.renames:
        select_input.join_key = select_input.old_name in right_join_keys

to_join_input()

Creates a new JoinInput instance based on the current manager settings.

This is useful when you've modified the manager (e.g., via auto_rename) and want to get a fresh JoinInput with all the current settings applied.

Returns:

Type	Description
JoinInput	A new JoinInput instance with current settings

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_join_input(self) -> JoinInput:
    """Creates a new JoinInput instance based on the current manager settings.

    This is useful when you've modified the manager (e.g., via auto_rename) and
    want to get a fresh JoinInput with all the current settings applied.

    Returns:
        A new JoinInput instance with current settings
    """
    return JoinInput(
        join_mapping=self.input.join_mapping,
        left_select=JoinInputs(renames=self.input.left_select.renames.copy()),
        right_select=JoinInputs(renames=self.input.right_select.renames.copy()),
        how=self.input.how,
    )

JoinInputs pydantic-model

Bases: SelectInputs

Data model for join-specific select inputs (extends SelectInputs).

Show JSON schema:

{
  "$defs": {
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "Data model for join-specific select inputs (extends SelectInputs).",
  "properties": {
    "renames": {
      "items": {
        "$ref": "#/$defs/SelectInput"
      },
      "title": "Renames",
      "type": "array"
    }
  },
  "title": "JoinInputs",
  "type": "object"
}

Fields:

• renames (list[SelectInput])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinInputs(SelectInputs):
    """Data model for join-specific select inputs (extends SelectInputs)."""

    def __init__(self, renames: list[SelectInput] = None, **kwargs):
        if renames is not None:
            kwargs["renames"] = renames
        else:
            kwargs["renames"] = []
        super().__init__(**kwargs)

create_from_list(col_list) classmethod

Creates a SelectInputs object from a simple list of column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create_from_list(cls, col_list: list[str]) -> "SelectInputs":
    """Creates a SelectInputs object from a simple list of column names."""
    return cls(renames=[SelectInput(old_name=c) for c in col_list])

create_from_pl_df(df) classmethod

Creates a SelectInputs object from a Polars DataFrame's columns.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create_from_pl_df(cls, df: pl.DataFrame | pl.LazyFrame) -> "SelectInputs":
    """Creates a SelectInputs object from a Polars DataFrame's columns."""
    return cls(renames=[SelectInput(old_name=c) for c in df.collect_schema().names()])

from_yaml_dict(data) classmethod

Load from slim YAML format. Supports both 'select' (new) and 'renames' (internal).

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_yaml_dict(cls, data: dict) -> "SelectInputs":
    """Load from slim YAML format. Supports both 'select' (new) and 'renames' (internal)."""
    items = data.get("select", data.get("renames", []))
    return cls(renames=[SelectInput.from_yaml_dict(item) for item in items])

remove_select_input(old_key)

Removes a SelectInput from the list based on its original name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def remove_select_input(self, old_key: str) -> None:
    """Removes a SelectInput from the list based on its original name."""
    self.renames = [rename for rename in self.renames if rename.old_name != old_key]

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> JoinInputsYaml:
    """Serialize for YAML output."""
    return {"select": [r.to_yaml_dict() for r in self.renames]}

JoinInputsManager

Bases: SelectInputsManager

Manager for join-specific operations, extends SelectInputsManager.

Methods:

Name	Description
__add__	Backward compatibility: Support += operator for appending.
append	Appends a new SelectInput to the list of renames.
find_by_new_name	Find SelectInput by new column name.
find_by_old_name	Find SelectInput by original column name.
get_drop_columns	Returns a list of SelectInput objects that are marked to be dropped.
get_join_key_rename_mapping	Returns a dictionary mapping original join key names to their temporary names.
get_join_key_renames	Gets the temporary rename mapping for all join keys on one side of a join.
get_join_key_selects	Returns only the SelectInput objects that are marked as join keys.
get_new_cols	Returns a set of new (renamed) column names to be kept in the selection.
get_non_jk_drop_columns	Returns drop columns that are not join keys.
get_old_cols	Returns a set of original column names to be kept in the selection.
get_rename_table	Generates a dictionary for use in Polars' .rename() method.
get_select_cols	Gets a list of original column names to select from the source DataFrame.
get_select_input_on_new_name	Backward compatibility alias: Find SelectInput by new column name.
get_select_input_on_old_name	Backward compatibility alias: Find SelectInput by original column name.
has_drop_cols	Checks if any column is marked to be dropped from the selection.
remove_select_input	Removes a SelectInput from the list based on its original name.
unselect_field	Marks a field to be dropped from the final selection by setting keep to False.

Attributes:

Name	Type	Description
drop_columns	list[SelectInput]	Backward compatibility: Returns list of columns to drop.
join_key_selects	list[SelectInput]	Backward compatibility: Returns join key SelectInputs.
new_cols	set[str]	Backward compatibility: Returns set of new column names.
non_jk_drop_columns	list[SelectInput]	Backward compatibility: Returns non-join-key columns to drop.
old_cols	set[str]	Backward compatibility: Returns set of old column names.
rename_table	dict[str, str]	Backward compatibility: Returns rename table dictionary.
renames	list[SelectInput]	Backward compatibility: Direct access to renames list.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinInputsManager(SelectInputsManager):
    """Manager for join-specific operations, extends SelectInputsManager."""

    def __init__(self, join_inputs: JoinInputs):
        super().__init__(join_inputs)
        self.join_inputs = join_inputs

    # === Query Methods ===

    def get_join_key_selects(self) -> list[SelectInput]:
        """Returns only the `SelectInput` objects that are marked as join keys."""
        return [v for v in self.join_inputs.renames if v.join_key]

    def get_join_key_renames(self, side: SideLit, filter_drop: bool = False) -> JoinKeyRenameResponse:
        """Gets the temporary rename mapping for all join keys on one side of a join."""
        join_key_selects = self.get_join_key_selects()
        join_key_list = [
            JoinKeyRename(jk.new_name, construct_join_key_name(side, jk.new_name))
            for jk in join_key_selects
            if jk.keep or not filter_drop
        ]
        return JoinKeyRenameResponse(side, join_key_list)

    def get_join_key_rename_mapping(self, side: SideLit) -> dict[str, str]:
        """Returns a dictionary mapping original join key names to their temporary names."""
        join_key_response = self.get_join_key_renames(side)
        return {jkr.original_name: jkr.temp_name for jkr in join_key_response.join_key_renames}

    @property
    def join_key_selects(self) -> list[SelectInput]:
        """Backward compatibility: Returns join key SelectInputs."""
        return self.get_join_key_selects()

drop_columns property

Backward compatibility: Returns list of columns to drop.

join_key_selects property

Backward compatibility: Returns join key SelectInputs.

new_cols property

Backward compatibility: Returns set of new column names.

non_jk_drop_columns property

Backward compatibility: Returns non-join-key columns to drop.

old_cols property

Backward compatibility: Returns set of old column names.

rename_table property

Backward compatibility: Returns rename table dictionary.

renames property

Backward compatibility: Direct access to renames list.

__add__(other)

Backward compatibility: Support += operator for appending.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __add__(self, other: SelectInput) -> "SelectInputsManager":
    """Backward compatibility: Support += operator for appending."""
    self.append(other)
    return self

append(other)

Appends a new SelectInput to the list of renames.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def append(self, other: SelectInput) -> None:
    """Appends a new SelectInput to the list of renames."""
    self.select_inputs.renames.append(other)

find_by_new_name(new_name)

Find SelectInput by new column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def find_by_new_name(self, new_name: str) -> SelectInput | None:
    """Find SelectInput by new column name."""
    return next((v for v in self.select_inputs.renames if v.new_name == new_name), None)

find_by_old_name(old_name)

Find SelectInput by original column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def find_by_old_name(self, old_name: str) -> SelectInput | None:
    """Find SelectInput by original column name."""
    return next((v for v in self.select_inputs.renames if v.old_name == old_name), None)

get_drop_columns()

Returns a list of SelectInput objects that are marked to be dropped.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_drop_columns(self) -> list[SelectInput]:
    """Returns a list of SelectInput objects that are marked to be dropped."""
    return [v for v in self.select_inputs.renames if not v.keep and v.is_available]

get_join_key_rename_mapping(side)

Returns a dictionary mapping original join key names to their temporary names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_join_key_rename_mapping(self, side: SideLit) -> dict[str, str]:
    """Returns a dictionary mapping original join key names to their temporary names."""
    join_key_response = self.get_join_key_renames(side)
    return {jkr.original_name: jkr.temp_name for jkr in join_key_response.join_key_renames}

get_join_key_renames(side, filter_drop=False)

Gets the temporary rename mapping for all join keys on one side of a join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_join_key_renames(self, side: SideLit, filter_drop: bool = False) -> JoinKeyRenameResponse:
    """Gets the temporary rename mapping for all join keys on one side of a join."""
    join_key_selects = self.get_join_key_selects()
    join_key_list = [
        JoinKeyRename(jk.new_name, construct_join_key_name(side, jk.new_name))
        for jk in join_key_selects
        if jk.keep or not filter_drop
    ]
    return JoinKeyRenameResponse(side, join_key_list)

get_join_key_selects()

Returns only the SelectInput objects that are marked as join keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_join_key_selects(self) -> list[SelectInput]:
    """Returns only the `SelectInput` objects that are marked as join keys."""
    return [v for v in self.join_inputs.renames if v.join_key]

get_new_cols()

Returns a set of new (renamed) column names to be kept in the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_new_cols(self) -> set[str]:
    """Returns a set of new (renamed) column names to be kept in the selection."""
    return set(v.new_name for v in self.select_inputs.renames if v.keep)

get_non_jk_drop_columns()

Returns drop columns that are not join keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_non_jk_drop_columns(self) -> list[SelectInput]:
    """Returns drop columns that are not join keys."""
    return [v for v in self.select_inputs.renames if not v.keep and v.is_available and not v.join_key]

get_old_cols()

Returns a set of original column names to be kept in the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_old_cols(self) -> set[str]:
    """Returns a set of original column names to be kept in the selection."""
    return set(v.old_name for v in self.select_inputs.renames if v.keep)

get_rename_table()

Generates a dictionary for use in Polars' .rename() method.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_rename_table(self) -> dict[str, str]:
    """Generates a dictionary for use in Polars' `.rename()` method."""
    return {v.old_name: v.new_name for v in self.select_inputs.renames if v.is_available and (v.keep or v.join_key)}

get_select_cols(include_join_key=True)

Gets a list of original column names to select from the source DataFrame.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_cols(self, include_join_key: bool = True) -> list[str]:
    """Gets a list of original column names to select from the source DataFrame."""
    return [
        v.old_name
        for v in self.select_inputs.renames
        if v.is_available and (v.keep or (v.join_key and include_join_key))
    ]

get_select_input_on_new_name(new_name)

Backward compatibility alias: Find SelectInput by new column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_input_on_new_name(self, new_name: str) -> SelectInput | None:
    """Backward compatibility alias: Find SelectInput by new column name."""
    return self.find_by_new_name(new_name)

get_select_input_on_old_name(old_name)

Backward compatibility alias: Find SelectInput by original column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_input_on_old_name(self, old_name: str) -> SelectInput | None:
    """Backward compatibility alias: Find SelectInput by original column name."""
    return self.find_by_old_name(old_name)

has_drop_cols()

Checks if any column is marked to be dropped from the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def has_drop_cols(self) -> bool:
    """Checks if any column is marked to be dropped from the selection."""
    return any(not v.keep for v in self.select_inputs.renames)

remove_select_input(old_key)

Removes a SelectInput from the list based on its original name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def remove_select_input(self, old_key: str) -> None:
    """Removes a SelectInput from the list based on its original name."""
    self.select_inputs.renames = [rename for rename in self.select_inputs.renames if rename.old_name != old_key]

unselect_field(old_key)

Marks a field to be dropped from the final selection by setting keep to False.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def unselect_field(self, old_key: str) -> None:
    """Marks a field to be dropped from the final selection by setting `keep` to False."""
    for rename in self.select_inputs.renames:
        if old_key == rename.old_name:
            rename.keep = False

JoinKeyRename

Bases: NamedTuple

Represents the renaming of a join key from its original to a temporary name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinKeyRename(NamedTuple):
    """Represents the renaming of a join key from its original to a temporary name."""

    original_name: str
    temp_name: str

JoinKeyRenameResponse

Bases: NamedTuple

Contains a list of join key renames for one side of a join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinKeyRenameResponse(NamedTuple):
    """Contains a list of join key renames for one side of a join."""

    side: SideLit
    join_key_renames: list[JoinKeyRename]

JoinMap pydantic-model

Bases: BaseModel

Defines a single mapping between a left and right column for a join key.

Show JSON schema:

{
  "description": "Defines a single mapping between a left and right column for a join key.",
  "properties": {
    "left_col": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Left Col"
    },
    "right_col": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Right Col"
    }
  },
  "title": "JoinMap",
  "type": "object"
}

Fields:

• left_col (str | None)
• right_col (str | None)

Validators:

• set_default_right_col

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinMap(BaseModel):
    """Defines a single mapping between a left and right column for a join key."""

    left_col: str | None = None
    right_col: str | None = None

    def __init__(self, left_col: str = None, right_col: str = None, **data):
        if left_col is not None:
            data["left_col"] = left_col
        if right_col is not None:
            data["right_col"] = right_col
        super().__init__(**data)

    @model_validator(mode="after")
    def set_default_right_col(self):
        """If right_col is None, default it to left_col."""
        if self.right_col is None:
            self.right_col = self.left_col
        return self

set_default_right_col() pydantic-validator

If right_col is None, default it to left_col.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="after")
def set_default_right_col(self):
    """If right_col is None, default it to left_col."""
    if self.right_col is None:
        self.right_col = self.left_col
    return self

JoinSelectManagerMixin

Mixin providing common methods for join-like operations.

Methods:

Name	Description
add_new_select_column	Adds a new column to the selection for either the left or right side.
auto_generate_new_col_name	Generates a new, non-conflicting column name by adding a suffix if necessary.
get_overlapping_columns	Finds column names that would conflict after the join.
parse_select	Parses various input formats into a standardized JoinInputs object.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class JoinSelectManagerMixin:
    """Mixin providing common methods for join-like operations."""

    left_manager: JoinInputsManager
    right_manager: JoinInputsManager
    input: CrossJoinInput | JoinInput | FuzzyMatchInput

    @staticmethod
    def parse_select(select: list[SelectInput] | list[str] | list[dict] | dict) -> JoinInputs:
        """Parses various input formats into a standardized `JoinInputs` object."""
        if not select:
            return JoinInputs(renames=[])

        if all(isinstance(c, SelectInput) for c in select):
            return JoinInputs(renames=select)
        elif all(isinstance(c, dict) for c in select):
            return JoinInputs(renames=[SelectInput(**c) for c in select])
        elif isinstance(select, dict):
            renames = select.get("renames")
            if renames:
                return JoinInputs(renames=[SelectInput(**c) for c in renames])
            return JoinInputs(renames=[])
        elif all(isinstance(c, str) for c in select):
            return JoinInputs(renames=[SelectInput(old_name=s, new_name=s) for s in select])

        raise ValueError(f"Unable to parse select input: {type(select)}")

    def get_overlapping_columns(self) -> set[str]:
        """Finds column names that would conflict after the join."""
        return self.left_manager.get_new_cols() & self.right_manager.get_new_cols()

    def auto_generate_new_col_name(self, old_col_name: str, side: str) -> str:
        """Generates a new, non-conflicting column name by adding a suffix if necessary."""
        current_names = self.get_overlapping_columns()
        if old_col_name not in current_names:
            return old_col_name

        new_name = old_col_name
        while new_name in current_names:
            new_name = f"{side}_{new_name}"
        return new_name

    def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
        """Adds a new column to the selection for either the left or right side."""
        target_input = self.input.right_select if side == "right" else self.input.left_select

        select_input.new_name = self.auto_generate_new_col_name(select_input.old_name, side=side)

        target_input.renames.append(select_input)

add_new_select_column(select_input, side)

Adds a new column to the selection for either the left or right side.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def add_new_select_column(self, select_input: SelectInput, side: str) -> None:
    """Adds a new column to the selection for either the left or right side."""
    target_input = self.input.right_select if side == "right" else self.input.left_select

    select_input.new_name = self.auto_generate_new_col_name(select_input.old_name, side=side)

    target_input.renames.append(select_input)

auto_generate_new_col_name(old_col_name, side)

Generates a new, non-conflicting column name by adding a suffix if necessary.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def auto_generate_new_col_name(self, old_col_name: str, side: str) -> str:
    """Generates a new, non-conflicting column name by adding a suffix if necessary."""
    current_names = self.get_overlapping_columns()
    if old_col_name not in current_names:
        return old_col_name

    new_name = old_col_name
    while new_name in current_names:
        new_name = f"{side}_{new_name}"
    return new_name

get_overlapping_columns()

Finds column names that would conflict after the join.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_overlapping_columns(self) -> set[str]:
    """Finds column names that would conflict after the join."""
    return self.left_manager.get_new_cols() & self.right_manager.get_new_cols()

parse_select(select) staticmethod

Parses various input formats into a standardized JoinInputs object.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@staticmethod
def parse_select(select: list[SelectInput] | list[str] | list[dict] | dict) -> JoinInputs:
    """Parses various input formats into a standardized `JoinInputs` object."""
    if not select:
        return JoinInputs(renames=[])

    if all(isinstance(c, SelectInput) for c in select):
        return JoinInputs(renames=select)
    elif all(isinstance(c, dict) for c in select):
        return JoinInputs(renames=[SelectInput(**c) for c in select])
    elif isinstance(select, dict):
        renames = select.get("renames")
        if renames:
            return JoinInputs(renames=[SelectInput(**c) for c in renames])
        return JoinInputs(renames=[])
    elif all(isinstance(c, str) for c in select):
        return JoinInputs(renames=[SelectInput(old_name=s, new_name=s) for s in select])

    raise ValueError(f"Unable to parse select input: {type(select)}")

PivotInput pydantic-model

Bases: BaseModel

Defines the settings for a pivot (long-to-wide) operation.

Show JSON schema:

{
  "description": "Defines the settings for a pivot (long-to-wide) operation.",
  "properties": {
    "index_columns": {
      "items": {
        "type": "string"
      },
      "title": "Index Columns",
      "type": "array"
    },
    "pivot_column": {
      "title": "Pivot Column",
      "type": "string"
    },
    "value_col": {
      "title": "Value Col",
      "type": "string"
    },
    "aggregations": {
      "items": {
        "type": "string"
      },
      "title": "Aggregations",
      "type": "array"
    }
  },
  "required": [
    "index_columns",
    "pivot_column",
    "value_col",
    "aggregations"
  ],
  "title": "PivotInput",
  "type": "object"
}

Fields:

• index_columns (list[str])
• pivot_column (str)
• value_col (str)
• aggregations (list[str])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class PivotInput(BaseModel):
    """Defines the settings for a pivot (long-to-wide) operation."""

    index_columns: list[str]
    pivot_column: str
    value_col: str
    aggregations: list[str]

    @property
    def grouped_columns(self) -> list[str]:
        """Returns the list of columns to be used for the initial grouping stage of the pivot."""
        return self.index_columns + [self.pivot_column]

    def get_group_by_input(self) -> GroupByInput:
        """Constructs the `GroupByInput` needed for the pre-aggregation step of the pivot."""
        group_by_cols = [AggColl(old_name=c, agg="groupby") for c in self.grouped_columns]
        agg_cols = [
            AggColl(old_name=self.value_col, agg=aggregation, new_name=aggregation) for aggregation in self.aggregations
        ]
        return GroupByInput(agg_cols=group_by_cols + agg_cols)

    def get_index_columns(self) -> list[pl.col]:
        """Returns the index columns as Polars column expressions."""
        return [pl.col(c) for c in self.index_columns]

    def get_pivot_column(self) -> pl.Expr:
        """Returns the pivot column as a Polars column expression."""
        return pl.col(self.pivot_column)

    def get_values_expr(self) -> pl.Expr:
        """Creates the struct expression used to gather the values for pivoting."""
        return pl.struct([pl.col(c) for c in self.aggregations]).alias("vals")

grouped_columns property

Returns the list of columns to be used for the initial grouping stage of the pivot.

get_group_by_input()

Constructs the GroupByInput needed for the pre-aggregation step of the pivot.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_group_by_input(self) -> GroupByInput:
    """Constructs the `GroupByInput` needed for the pre-aggregation step of the pivot."""
    group_by_cols = [AggColl(old_name=c, agg="groupby") for c in self.grouped_columns]
    agg_cols = [
        AggColl(old_name=self.value_col, agg=aggregation, new_name=aggregation) for aggregation in self.aggregations
    ]
    return GroupByInput(agg_cols=group_by_cols + agg_cols)

get_index_columns()

Returns the index columns as Polars column expressions.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_index_columns(self) -> list[pl.col]:
    """Returns the index columns as Polars column expressions."""
    return [pl.col(c) for c in self.index_columns]

get_pivot_column()

Returns the pivot column as a Polars column expression.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_pivot_column(self) -> pl.Expr:
    """Returns the pivot column as a Polars column expression."""
    return pl.col(self.pivot_column)

get_values_expr()

Creates the struct expression used to gather the values for pivoting.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_values_expr(self) -> pl.Expr:
    """Creates the struct expression used to gather the values for pivoting."""
    return pl.struct([pl.col(c) for c in self.aggregations]).alias("vals")

PolarsCodeInput pydantic-model

Bases: BaseModel

A simple container for a string of user-provided Polars code to be executed.

Show JSON schema:

{
  "description": "A simple container for a string of user-provided Polars code to be executed.",
  "properties": {
    "polars_code": {
      "title": "Polars Code",
      "type": "string"
    }
  },
  "required": [
    "polars_code"
  ],
  "title": "PolarsCodeInput",
  "type": "object"
}

Fields:

• polars_code (str)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class PolarsCodeInput(BaseModel):
    """A simple container for a string of user-provided Polars code to be executed."""

    polars_code: str

RecordIdInput pydantic-model

Bases: BaseModel

Defines settings for adding a record ID (row number) column to the data.

Show JSON schema:

{
  "description": "Defines settings for adding a record ID (row number) column to the data.",
  "properties": {
    "output_column_name": {
      "default": "record_id",
      "title": "Output Column Name",
      "type": "string"
    },
    "offset": {
      "default": 1,
      "title": "Offset",
      "type": "integer"
    },
    "group_by": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": false,
      "title": "Group By"
    },
    "group_by_columns": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "title": "Group By Columns"
    }
  },
  "title": "RecordIdInput",
  "type": "object"
}

Fields:

• output_column_name (str)
• offset (int)
• group_by (bool | None)
• group_by_columns (list[str] | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class RecordIdInput(BaseModel):
    """Defines settings for adding a record ID (row number) column to the data."""

    output_column_name: str = "record_id"
    offset: int = 1
    group_by: bool | None = False
    group_by_columns: list[str] | None = Field(default_factory=list)

SelectInput pydantic-model

Bases: BaseModel

Defines how a single column should be selected, renamed, or type-cast.

This is a core building block for any operation that involves column manipulation. It holds all the configuration for a single field in a selection operation.

Show JSON schema:

{
  "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
  "properties": {
    "old_name": {
      "title": "Old Name",
      "type": "string"
    },
    "original_position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Original Position"
    },
    "new_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "New Name"
    },
    "data_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Data Type"
    },
    "data_type_change": {
      "default": false,
      "title": "Data Type Change",
      "type": "boolean"
    },
    "join_key": {
      "default": false,
      "title": "Join Key",
      "type": "boolean"
    },
    "is_altered": {
      "default": false,
      "title": "Is Altered",
      "type": "boolean"
    },
    "position": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Position"
    },
    "is_available": {
      "default": true,
      "title": "Is Available",
      "type": "boolean"
    },
    "keep": {
      "default": true,
      "title": "Keep",
      "type": "boolean"
    }
  },
  "required": [
    "old_name"
  ],
  "title": "SelectInput",
  "type": "object"
}

Config:

• frozen: False

Fields:

• old_name (str)
• original_position (int | None)
• new_name (str | None)
• data_type (str | None)
• data_type_change (bool)
• join_key (bool)
• is_altered (bool)
• position (int | None)
• is_available (bool)
• keep (bool)

Validators:

• infer_data_type_change
• set_default_new_name

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class SelectInput(BaseModel):
    """Defines how a single column should be selected, renamed, or type-cast.

    This is a core building block for any operation that involves column manipulation.
    It holds all the configuration for a single field in a selection operation.
    """

    model_config = ConfigDict(frozen=False)

    old_name: str
    original_position: int | None = None
    new_name: str | None = None
    data_type: str | None = None
    data_type_change: bool = False
    join_key: bool = False
    is_altered: bool = False
    position: int | None = None
    is_available: bool = True
    keep: bool = True

    def __init__(self, old_name: str = None, new_name: str = None, **data):
        if old_name is not None:
            data["old_name"] = old_name
        if new_name is not None:
            data["new_name"] = new_name
        super().__init__(**data)

    def to_yaml_dict(self) -> SelectInputYaml:
        """Serialize for YAML output - only user-relevant fields."""
        result: SelectInputYaml = {"old_name": self.old_name}
        if self.new_name != self.old_name:
            result["new_name"] = self.new_name
        if not self.keep:
            result["keep"] = self.keep
        # Always include data_type if it's set, not just when data_type_change is True
        # This ensures undo/redo snapshots preserve the data_type field
        if self.data_type:
            result["data_type"] = self.data_type
        return result

    @classmethod
    def from_yaml_dict(cls, data: dict) -> "SelectInput":
        """Load from slim YAML format."""
        old_name = data["old_name"]
        new_name = data.get("new_name", old_name)
        data_type = data.get("data_type")
        # is_altered should be True if either name was changed OR data_type was explicitly set
        # This ensures updateNodeSelect in the frontend won't overwrite user-specified data_type
        is_altered = (old_name != new_name) or (data_type is not None)
        return cls(
            old_name=old_name,
            new_name=new_name,
            keep=data.get("keep", True),
            data_type=data_type,
            data_type_change=data_type is not None,
            is_altered=is_altered,
        )

    @model_validator(mode="before")
    @classmethod
    def infer_data_type_change(cls, data):
        """Infer data_type_change when loading from YAML.

        When data_type is present but data_type_change is not explicitly set,
        infer that the user explicitly set the data_type (e.g., when loading from YAML).
        This ensures is_altered will be set correctly in the after validator.
        """
        if isinstance(data, dict):
            if data.get("data_type") is not None and "data_type_change" not in data:
                data["data_type_change"] = True
        return data

    @model_validator(mode="after")
    def set_default_new_name(self):
        """If new_name is None, default it to old_name. Also set is_altered if needed."""
        if self.new_name is None:
            self.new_name = self.old_name
        if self.old_name != self.new_name:
            self.is_altered = True
        if self.data_type_change:
            self.is_altered = True
        return self

    def __hash__(self):
        """Allow SelectInput to be used in sets and as dict keys."""
        return hash(self.old_name)

    def __eq__(self, other):
        """Required when implementing __hash__."""
        if not isinstance(other, SelectInput):
            return False
        return self.old_name == other.old_name

    @property
    def polars_type(self) -> str:
        """Translates a user-friendly type name to a Polars data type string."""
        data_type_lower = self.data_type.lower()
        if data_type_lower == "string":
            return "Utf8"
        elif data_type_lower == "integer":
            return "Int64"
        elif data_type_lower == "double":
            return "Float64"
        return self.data_type

polars_type property

Translates a user-friendly type name to a Polars data type string.

__eq__(other)

Required when implementing hash.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __eq__(self, other):
    """Required when implementing __hash__."""
    if not isinstance(other, SelectInput):
        return False
    return self.old_name == other.old_name

__hash__()

Allow SelectInput to be used in sets and as dict keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __hash__(self):
    """Allow SelectInput to be used in sets and as dict keys."""
    return hash(self.old_name)

from_yaml_dict(data) classmethod

Load from slim YAML format.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_yaml_dict(cls, data: dict) -> "SelectInput":
    """Load from slim YAML format."""
    old_name = data["old_name"]
    new_name = data.get("new_name", old_name)
    data_type = data.get("data_type")
    # is_altered should be True if either name was changed OR data_type was explicitly set
    # This ensures updateNodeSelect in the frontend won't overwrite user-specified data_type
    is_altered = (old_name != new_name) or (data_type is not None)
    return cls(
        old_name=old_name,
        new_name=new_name,
        keep=data.get("keep", True),
        data_type=data_type,
        data_type_change=data_type is not None,
        is_altered=is_altered,
    )

infer_data_type_change(data) pydantic-validator

Infer data_type_change when loading from YAML.

When data_type is present but data_type_change is not explicitly set, infer that the user explicitly set the data_type (e.g., when loading from YAML). This ensures is_altered will be set correctly in the after validator.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="before")
@classmethod
def infer_data_type_change(cls, data):
    """Infer data_type_change when loading from YAML.

    When data_type is present but data_type_change is not explicitly set,
    infer that the user explicitly set the data_type (e.g., when loading from YAML).
    This ensures is_altered will be set correctly in the after validator.
    """
    if isinstance(data, dict):
        if data.get("data_type") is not None and "data_type_change" not in data:
            data["data_type_change"] = True
    return data

set_default_new_name() pydantic-validator

If new_name is None, default it to old_name. Also set is_altered if needed.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@model_validator(mode="after")
def set_default_new_name(self):
    """If new_name is None, default it to old_name. Also set is_altered if needed."""
    if self.new_name is None:
        self.new_name = self.old_name
    if self.old_name != self.new_name:
        self.is_altered = True
    if self.data_type_change:
        self.is_altered = True
    return self

to_yaml_dict()

Serialize for YAML output - only user-relevant fields.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> SelectInputYaml:
    """Serialize for YAML output - only user-relevant fields."""
    result: SelectInputYaml = {"old_name": self.old_name}
    if self.new_name != self.old_name:
        result["new_name"] = self.new_name
    if not self.keep:
        result["keep"] = self.keep
    # Always include data_type if it's set, not just when data_type_change is True
    # This ensures undo/redo snapshots preserve the data_type field
    if self.data_type:
        result["data_type"] = self.data_type
    return result

SelectInputs pydantic-model

Bases: BaseModel

A container for a list of SelectInput objects (pure data, no logic).

Show JSON schema:

{
  "$defs": {
    "SelectInput": {
      "description": "Defines how a single column should be selected, renamed, or type-cast.\n\nThis is a core building block for any operation that involves column manipulation.\nIt holds all the configuration for a single field in a selection operation.",
      "properties": {
        "old_name": {
          "title": "Old Name",
          "type": "string"
        },
        "original_position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Original Position"
        },
        "new_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "New Name"
        },
        "data_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data Type"
        },
        "data_type_change": {
          "default": false,
          "title": "Data Type Change",
          "type": "boolean"
        },
        "join_key": {
          "default": false,
          "title": "Join Key",
          "type": "boolean"
        },
        "is_altered": {
          "default": false,
          "title": "Is Altered",
          "type": "boolean"
        },
        "position": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Position"
        },
        "is_available": {
          "default": true,
          "title": "Is Available",
          "type": "boolean"
        },
        "keep": {
          "default": true,
          "title": "Keep",
          "type": "boolean"
        }
      },
      "required": [
        "old_name"
      ],
      "title": "SelectInput",
      "type": "object"
    }
  },
  "description": "A container for a list of `SelectInput` objects (pure data, no logic).",
  "properties": {
    "renames": {
      "items": {
        "$ref": "#/$defs/SelectInput"
      },
      "title": "Renames",
      "type": "array"
    }
  },
  "title": "SelectInputs",
  "type": "object"
}

Fields:

• renames (list[SelectInput])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class SelectInputs(BaseModel):
    """A container for a list of `SelectInput` objects (pure data, no logic)."""

    renames: list[SelectInput] = Field(default_factory=list)

    def __init__(self, renames: list[SelectInput] = None, **kwargs):
        if renames is not None:
            kwargs["renames"] = renames
        else:
            kwargs["renames"] = []
        super().__init__(**kwargs)

    def to_yaml_dict(self) -> JoinInputsYaml:
        """Serialize for YAML output."""
        return {"select": [r.to_yaml_dict() for r in self.renames]}

    @classmethod
    def from_yaml_dict(cls, data: dict) -> "SelectInputs":
        """Load from slim YAML format. Supports both 'select' (new) and 'renames' (internal)."""
        items = data.get("select", data.get("renames", []))
        return cls(renames=[SelectInput.from_yaml_dict(item) for item in items])

    @classmethod
    def create_from_list(cls, col_list: list[str]) -> "SelectInputs":
        """Creates a SelectInputs object from a simple list of column names."""
        return cls(renames=[SelectInput(old_name=c) for c in col_list])

    @classmethod
    def create_from_pl_df(cls, df: pl.DataFrame | pl.LazyFrame) -> "SelectInputs":
        """Creates a SelectInputs object from a Polars DataFrame's columns."""
        return cls(renames=[SelectInput(old_name=c) for c in df.collect_schema().names()])

    def remove_select_input(self, old_key: str) -> None:
        """Removes a SelectInput from the list based on its original name."""
        self.renames = [rename for rename in self.renames if rename.old_name != old_key]

create_from_list(col_list) classmethod

Creates a SelectInputs object from a simple list of column names.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create_from_list(cls, col_list: list[str]) -> "SelectInputs":
    """Creates a SelectInputs object from a simple list of column names."""
    return cls(renames=[SelectInput(old_name=c) for c in col_list])

create_from_pl_df(df) classmethod

Creates a SelectInputs object from a Polars DataFrame's columns.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def create_from_pl_df(cls, df: pl.DataFrame | pl.LazyFrame) -> "SelectInputs":
    """Creates a SelectInputs object from a Polars DataFrame's columns."""
    return cls(renames=[SelectInput(old_name=c) for c in df.collect_schema().names()])

from_yaml_dict(data) classmethod

Load from slim YAML format. Supports both 'select' (new) and 'renames' (internal).

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

@classmethod
def from_yaml_dict(cls, data: dict) -> "SelectInputs":
    """Load from slim YAML format. Supports both 'select' (new) and 'renames' (internal)."""
    items = data.get("select", data.get("renames", []))
    return cls(renames=[SelectInput.from_yaml_dict(item) for item in items])

remove_select_input(old_key)

Removes a SelectInput from the list based on its original name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def remove_select_input(self, old_key: str) -> None:
    """Removes a SelectInput from the list based on its original name."""
    self.renames = [rename for rename in self.renames if rename.old_name != old_key]

to_yaml_dict()

Serialize for YAML output.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def to_yaml_dict(self) -> JoinInputsYaml:
    """Serialize for YAML output."""
    return {"select": [r.to_yaml_dict() for r in self.renames]}

SelectInputsManager

Manager class that provides all query and mutation operations.

Methods:

Name	Description
__add__	Backward compatibility: Support += operator for appending.
append	Appends a new SelectInput to the list of renames.
find_by_new_name	Find SelectInput by new column name.
find_by_old_name	Find SelectInput by original column name.
get_drop_columns	Returns a list of SelectInput objects that are marked to be dropped.
get_new_cols	Returns a set of new (renamed) column names to be kept in the selection.
get_non_jk_drop_columns	Returns drop columns that are not join keys.
get_old_cols	Returns a set of original column names to be kept in the selection.
get_rename_table	Generates a dictionary for use in Polars' .rename() method.
get_select_cols	Gets a list of original column names to select from the source DataFrame.
get_select_input_on_new_name	Backward compatibility alias: Find SelectInput by new column name.
get_select_input_on_old_name	Backward compatibility alias: Find SelectInput by original column name.
has_drop_cols	Checks if any column is marked to be dropped from the selection.
remove_select_input	Removes a SelectInput from the list based on its original name.
unselect_field	Marks a field to be dropped from the final selection by setting keep to False.

Attributes:

Name	Type	Description
drop_columns	list[SelectInput]	Backward compatibility: Returns list of columns to drop.
new_cols	set[str]	Backward compatibility: Returns set of new column names.
non_jk_drop_columns	list[SelectInput]	Backward compatibility: Returns non-join-key columns to drop.
old_cols	set[str]	Backward compatibility: Returns set of old column names.
rename_table	dict[str, str]	Backward compatibility: Returns rename table dictionary.
renames	list[SelectInput]	Backward compatibility: Direct access to renames list.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class SelectInputsManager:
    """Manager class that provides all query and mutation operations."""

    def __init__(self, select_inputs: SelectInputs):
        self.select_inputs = select_inputs

    # === Query Methods (read-only) ===

    def get_old_cols(self) -> set[str]:
        """Returns a set of original column names to be kept in the selection."""
        return set(v.old_name for v in self.select_inputs.renames if v.keep)

    def get_new_cols(self) -> set[str]:
        """Returns a set of new (renamed) column names to be kept in the selection."""
        return set(v.new_name for v in self.select_inputs.renames if v.keep)

    def get_rename_table(self) -> dict[str, str]:
        """Generates a dictionary for use in Polars' `.rename()` method."""
        return {v.old_name: v.new_name for v in self.select_inputs.renames if v.is_available and (v.keep or v.join_key)}

    def get_select_cols(self, include_join_key: bool = True) -> list[str]:
        """Gets a list of original column names to select from the source DataFrame."""
        return [
            v.old_name
            for v in self.select_inputs.renames
            if v.is_available and (v.keep or (v.join_key and include_join_key))
        ]

    def has_drop_cols(self) -> bool:
        """Checks if any column is marked to be dropped from the selection."""
        return any(not v.keep for v in self.select_inputs.renames)

    def get_drop_columns(self) -> list[SelectInput]:
        """Returns a list of SelectInput objects that are marked to be dropped."""
        return [v for v in self.select_inputs.renames if not v.keep and v.is_available]

    def get_non_jk_drop_columns(self) -> list[SelectInput]:
        """Returns drop columns that are not join keys."""
        return [v for v in self.select_inputs.renames if not v.keep and v.is_available and not v.join_key]

    def find_by_old_name(self, old_name: str) -> SelectInput | None:
        """Find SelectInput by original column name."""
        return next((v for v in self.select_inputs.renames if v.old_name == old_name), None)

    def find_by_new_name(self, new_name: str) -> SelectInput | None:
        """Find SelectInput by new column name."""
        return next((v for v in self.select_inputs.renames if v.new_name == new_name), None)

    # === Mutation Methods ===

    def append(self, other: SelectInput) -> None:
        """Appends a new SelectInput to the list of renames."""
        self.select_inputs.renames.append(other)

    def remove_select_input(self, old_key: str) -> None:
        """Removes a SelectInput from the list based on its original name."""
        self.select_inputs.renames = [rename for rename in self.select_inputs.renames if rename.old_name != old_key]

    def unselect_field(self, old_key: str) -> None:
        """Marks a field to be dropped from the final selection by setting `keep` to False."""
        for rename in self.select_inputs.renames:
            if old_key == rename.old_name:
                rename.keep = False

    # === Backward Compatibility Properties ===

    @property
    def old_cols(self) -> set[str]:
        """Backward compatibility: Returns set of old column names."""
        return self.get_old_cols()

    @property
    def new_cols(self) -> set[str]:
        """Backward compatibility: Returns set of new column names."""
        return self.get_new_cols()

    @property
    def rename_table(self) -> dict[str, str]:
        """Backward compatibility: Returns rename table dictionary."""
        return self.get_rename_table()

    @property
    def drop_columns(self) -> list[SelectInput]:
        """Backward compatibility: Returns list of columns to drop."""
        return self.get_drop_columns()

    @property
    def non_jk_drop_columns(self) -> list[SelectInput]:
        """Backward compatibility: Returns non-join-key columns to drop."""
        return self.get_non_jk_drop_columns()

    @property
    def renames(self) -> list[SelectInput]:
        """Backward compatibility: Direct access to renames list."""
        return self.select_inputs.renames

    def get_select_input_on_old_name(self, old_name: str) -> SelectInput | None:
        """Backward compatibility alias: Find SelectInput by original column name."""
        return self.find_by_old_name(old_name)

    def get_select_input_on_new_name(self, new_name: str) -> SelectInput | None:
        """Backward compatibility alias: Find SelectInput by new column name."""
        return self.find_by_new_name(new_name)

    def __add__(self, other: SelectInput) -> "SelectInputsManager":
        """Backward compatibility: Support += operator for appending."""
        self.append(other)
        return self

drop_columns property

Backward compatibility: Returns list of columns to drop.

new_cols property

Backward compatibility: Returns set of new column names.

non_jk_drop_columns property

Backward compatibility: Returns non-join-key columns to drop.

old_cols property

Backward compatibility: Returns set of old column names.

rename_table property

Backward compatibility: Returns rename table dictionary.

renames property

Backward compatibility: Direct access to renames list.

__add__(other)

Backward compatibility: Support += operator for appending.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def __add__(self, other: SelectInput) -> "SelectInputsManager":
    """Backward compatibility: Support += operator for appending."""
    self.append(other)
    return self

append(other)

Appends a new SelectInput to the list of renames.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def append(self, other: SelectInput) -> None:
    """Appends a new SelectInput to the list of renames."""
    self.select_inputs.renames.append(other)

find_by_new_name(new_name)

Find SelectInput by new column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def find_by_new_name(self, new_name: str) -> SelectInput | None:
    """Find SelectInput by new column name."""
    return next((v for v in self.select_inputs.renames if v.new_name == new_name), None)

find_by_old_name(old_name)

Find SelectInput by original column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def find_by_old_name(self, old_name: str) -> SelectInput | None:
    """Find SelectInput by original column name."""
    return next((v for v in self.select_inputs.renames if v.old_name == old_name), None)

get_drop_columns()

Returns a list of SelectInput objects that are marked to be dropped.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_drop_columns(self) -> list[SelectInput]:
    """Returns a list of SelectInput objects that are marked to be dropped."""
    return [v for v in self.select_inputs.renames if not v.keep and v.is_available]

get_new_cols()

Returns a set of new (renamed) column names to be kept in the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_new_cols(self) -> set[str]:
    """Returns a set of new (renamed) column names to be kept in the selection."""
    return set(v.new_name for v in self.select_inputs.renames if v.keep)

get_non_jk_drop_columns()

Returns drop columns that are not join keys.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_non_jk_drop_columns(self) -> list[SelectInput]:
    """Returns drop columns that are not join keys."""
    return [v for v in self.select_inputs.renames if not v.keep and v.is_available and not v.join_key]

get_old_cols()

Returns a set of original column names to be kept in the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_old_cols(self) -> set[str]:
    """Returns a set of original column names to be kept in the selection."""
    return set(v.old_name for v in self.select_inputs.renames if v.keep)

get_rename_table()

Generates a dictionary for use in Polars' .rename() method.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_rename_table(self) -> dict[str, str]:
    """Generates a dictionary for use in Polars' `.rename()` method."""
    return {v.old_name: v.new_name for v in self.select_inputs.renames if v.is_available and (v.keep or v.join_key)}

get_select_cols(include_join_key=True)

Gets a list of original column names to select from the source DataFrame.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_cols(self, include_join_key: bool = True) -> list[str]:
    """Gets a list of original column names to select from the source DataFrame."""
    return [
        v.old_name
        for v in self.select_inputs.renames
        if v.is_available and (v.keep or (v.join_key and include_join_key))
    ]

get_select_input_on_new_name(new_name)

Backward compatibility alias: Find SelectInput by new column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_input_on_new_name(self, new_name: str) -> SelectInput | None:
    """Backward compatibility alias: Find SelectInput by new column name."""
    return self.find_by_new_name(new_name)

get_select_input_on_old_name(old_name)

Backward compatibility alias: Find SelectInput by original column name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_select_input_on_old_name(self, old_name: str) -> SelectInput | None:
    """Backward compatibility alias: Find SelectInput by original column name."""
    return self.find_by_old_name(old_name)

has_drop_cols()

Checks if any column is marked to be dropped from the selection.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def has_drop_cols(self) -> bool:
    """Checks if any column is marked to be dropped from the selection."""
    return any(not v.keep for v in self.select_inputs.renames)

remove_select_input(old_key)

Removes a SelectInput from the list based on its original name.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def remove_select_input(self, old_key: str) -> None:
    """Removes a SelectInput from the list based on its original name."""
    self.select_inputs.renames = [rename for rename in self.select_inputs.renames if rename.old_name != old_key]

unselect_field(old_key)

Marks a field to be dropped from the final selection by setting keep to False.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def unselect_field(self, old_key: str) -> None:
    """Marks a field to be dropped from the final selection by setting `keep` to False."""
    for rename in self.select_inputs.renames:
        if old_key == rename.old_name:
            rename.keep = False

SortByInput pydantic-model

Bases: BaseModel

Defines a single sort condition on a column, including the direction.

Show JSON schema:

{
  "description": "Defines a single sort condition on a column, including the direction.",
  "properties": {
    "column": {
      "title": "Column",
      "type": "string"
    },
    "how": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "asc",
      "title": "How"
    }
  },
  "required": [
    "column"
  ],
  "title": "SortByInput",
  "type": "object"
}

Fields:

• column (str)
• how (str | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class SortByInput(BaseModel):
    """Defines a single sort condition on a column, including the direction."""

    column: str
    how: str | None = "asc"

    @property
    def descending(self) -> bool:
        """Resolve ``how`` to the boolean Polars expects, accepting both conventions."""
        return is_descending(self.how)

descending property

Resolve how to the boolean Polars expects, accepting both conventions.

SqlQueryInput pydantic-model

Bases: BaseModel

A container for a SQL query to execute against connected data sources.

Note: sql_code is not validated at schema-construction time. Construction is a passive shape-check; the unsafe-SQL gate lives at the executor seam in execute_sql_query (and is also enforced by the underlying validate_sql_query utility callers can use directly). Validating here too would block legitimate non-AI callers from drafting/testing SQL before execution.

Show JSON schema:

{
  "description": "A container for a SQL query to execute against connected data sources.\n\nNote: ``sql_code`` is *not* validated at schema-construction time. Construction\nis a passive shape-check; the unsafe-SQL gate lives at the executor seam in\n``execute_sql_query`` (and is also enforced by the underlying\n``validate_sql_query`` utility callers can use directly). Validating here too\nwould block legitimate non-AI callers from drafting/testing SQL before\nexecution.",
  "properties": {
    "sql_code": {
      "title": "Sql Code",
      "type": "string"
    }
  },
  "required": [
    "sql_code"
  ],
  "title": "SqlQueryInput",
  "type": "object"
}

Fields:

• sql_code (str)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class SqlQueryInput(BaseModel):
    """A container for a SQL query to execute against connected data sources.

    Note: ``sql_code`` is *not* validated at schema-construction time. Construction
    is a passive shape-check; the unsafe-SQL gate lives at the executor seam in
    ``execute_sql_query`` (and is also enforced by the underlying
    ``validate_sql_query`` utility callers can use directly). Validating here too
    would block legitimate non-AI callers from drafting/testing SQL before
    execution.
    """

    sql_code: str

TextToRowsInput pydantic-model

Bases: BaseModel

Defines settings for splitting a text column into multiple rows based on a delimiter.

Show JSON schema:

{
  "description": "Defines settings for splitting a text column into multiple rows based on a delimiter.",
  "properties": {
    "column_to_split": {
      "title": "Column To Split",
      "type": "string"
    },
    "output_column_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Column Name"
    },
    "split_by_fixed_value": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Split By Fixed Value"
    },
    "split_fixed_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": ",",
      "title": "Split Fixed Value"
    },
    "split_by_column": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Split By Column"
    }
  },
  "required": [
    "column_to_split"
  ],
  "title": "TextToRowsInput",
  "type": "object"
}

Fields:

• column_to_split (str)
• output_column_name (str | None)
• split_by_fixed_value (bool | None)
• split_fixed_value (str | None)
• split_by_column (str | None)

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class TextToRowsInput(BaseModel):
    """Defines settings for splitting a text column into multiple rows based on a delimiter."""

    column_to_split: str
    output_column_name: str | None = None
    split_by_fixed_value: bool | None = True
    split_fixed_value: str | None = ","
    split_by_column: str | None = None

UnionInput pydantic-model

Bases: BaseModel

Defines settings for a union (concatenation) operation.

Show JSON schema:

{
  "description": "Defines settings for a union (concatenation) operation.",
  "properties": {
    "mode": {
      "default": "relaxed",
      "enum": [
        "selective",
        "relaxed"
      ],
      "title": "Mode",
      "type": "string"
    }
  },
  "title": "UnionInput",
  "type": "object"
}

Fields:

• mode (Literal['selective', 'relaxed'])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class UnionInput(BaseModel):
    """Defines settings for a union (concatenation) operation."""

    mode: Literal["selective", "relaxed"] = "relaxed"

UniqueInput pydantic-model

Bases: BaseModel

Defines settings for a uniqueness operation, specifying columns and which row to keep.

Show JSON schema:

{
  "description": "Defines settings for a uniqueness operation, specifying columns and which row to keep.",
  "properties": {
    "columns": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Columns"
    },
    "strategy": {
      "default": "any",
      "enum": [
        "first",
        "last",
        "any",
        "none"
      ],
      "title": "Strategy",
      "type": "string"
    }
  },
  "title": "UniqueInput",
  "type": "object"
}

Fields:

• columns (list[str] | None)
• strategy (Literal['first', 'last', 'any', 'none'])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class UniqueInput(BaseModel):
    """Defines settings for a uniqueness operation, specifying columns and which row to keep."""

    columns: list[str] | None = None
    strategy: Literal["first", "last", "any", "none"] = "any"

UnpivotInput pydantic-model

Bases: BaseModel

Defines settings for an unpivot (wide-to-long) operation.

Show JSON schema:

{
  "description": "Defines settings for an unpivot (wide-to-long) operation.",
  "properties": {
    "index_columns": {
      "items": {
        "type": "string"
      },
      "title": "Index Columns",
      "type": "array"
    },
    "value_columns": {
      "items": {
        "type": "string"
      },
      "title": "Value Columns",
      "type": "array"
    },
    "data_type_selector": {
      "anyOf": [
        {
          "enum": [
            "float",
            "all",
            "date",
            "numeric",
            "string"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Data Type Selector"
    },
    "data_type_selector_mode": {
      "default": "column",
      "enum": [
        "data_type",
        "column"
      ],
      "title": "Data Type Selector Mode",
      "type": "string"
    }
  },
  "title": "UnpivotInput",
  "type": "object"
}

Config:

• arbitrary_types_allowed: True

Fields:

• index_columns (list[str])
• value_columns (list[str])
• data_type_selector (Literal['float', 'all', 'date', 'numeric', 'string'] | None)
• data_type_selector_mode (Literal['data_type', 'column'])

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class UnpivotInput(BaseModel):
    """Defines settings for an unpivot (wide-to-long) operation."""

    model_config = ConfigDict(arbitrary_types_allowed=True)

    index_columns: list[str] = Field(default_factory=list)
    value_columns: list[str] = Field(default_factory=list)
    data_type_selector: Literal["float", "all", "date", "numeric", "string"] | None = None
    data_type_selector_mode: Literal["data_type", "column"] = "column"

    @property
    def data_type_selector_expr(self) -> Callable | None:
        """Returns a Polars selector function based on the `data_type_selector` string."""
        if self.data_type_selector_mode == "data_type":
            if self.data_type_selector is not None:
                try:
                    return getattr(selectors, self.data_type_selector)
                except Exception:
                    print(f"Could not find the selector: {self.data_type_selector}")
                    return selectors.all
            return selectors.all
        return None

data_type_selector_expr property

Returns a Polars selector function based on the data_type_selector string.

WindowFunctionInput pydantic-model

Bases: BaseModel

A single window-function operation producing one new column.

column is the source column for rolling, cumulative and rank functions. For tile, column is ignored (ordering comes from the outer WindowFunctionsInput.order_by).

Show JSON schema:

{
  "description": "A single window-function operation producing one new column.\n\n`column` is the source column for rolling, cumulative and rank functions.\nFor `tile`, `column` is ignored (ordering comes from the outer\n``WindowFunctionsInput.order_by``).",
  "properties": {
    "column": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Column"
    },
    "function": {
      "enum": [
        "rolling_sum",
        "rolling_mean",
        "rolling_min",
        "rolling_max",
        "rolling_std",
        "cum_sum",
        "cum_count",
        "cum_min",
        "cum_max",
        "rank",
        "tile"
      ],
      "title": "Function",
      "type": "string"
    },
    "new_column_name": {
      "title": "New Column Name",
      "type": "string"
    },
    "window_size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Window Size"
    },
    "min_periods": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Min Periods"
    },
    "edge_behavior": {
      "anyOf": [
        {
          "enum": [
            "require_full",
            "partial",
            "fill_zero"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "require_full",
      "title": "Edge Behavior"
    },
    "number_of_groups": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Number Of Groups"
    },
    "rank_method": {
      "anyOf": [
        {
          "enum": [
            "ordinal",
            "dense",
            "min",
            "max",
            "average"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "ordinal",
      "title": "Rank Method"
    },
    "output_type": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Output Type"
    }
  },
  "required": [
    "function",
    "new_column_name"
  ],
  "title": "WindowFunctionInput",
  "type": "object"
}

Fields:

• column (str | None)
• function (WindowFunctionName)
• new_column_name (str)
• window_size (int | None)
• min_periods (int | None)
• edge_behavior (RollingEdgeBehavior | None)
• number_of_groups (int | None)
• rank_method (RankMethod | None)
• output_type (str | None)

Validators:

• _validate

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class WindowFunctionInput(BaseModel):
    """A single window-function operation producing one new column.

    `column` is the source column for rolling, cumulative and rank functions.
    For `tile`, `column` is ignored (ordering comes from the outer
    ``WindowFunctionsInput.order_by``).
    """

    column: str | None = None
    function: WindowFunctionName
    new_column_name: str
    window_size: int | None = None
    min_periods: int | None = None
    edge_behavior: RollingEdgeBehavior | None = "require_full"
    number_of_groups: int | None = None
    rank_method: RankMethod | None = "ordinal"
    output_type: str | None = None

    @model_validator(mode="after")
    def _validate(self) -> "WindowFunctionInput":
        if _is_rolling(self.function):
            if self.window_size is None or self.window_size < 1:
                raise ValueError(f"{self.function!r} requires a positive window_size")
            if self.column is None:
                raise ValueError(f"{self.function!r} requires a source column")
        elif _is_cumulative(self.function) or self.function == "rank":
            if self.column is None:
                raise ValueError(f"{self.function!r} requires a source column")
        elif self.function == "tile":
            if self.number_of_groups is None or self.number_of_groups < 1:
                raise ValueError("'tile' requires a positive number_of_groups")
        if self.output_type is None:
            self.output_type = get_window_output_type(self.function)
        return self

WindowFunctionsInput pydantic-model

Bases: BaseModel

Defines the settings for a window-functions node.

Attributes

partition_by : list[str] Optional list of columns to partition by (equivalent to .over(...)). order_by : list[SortByInput] Ordering within each partition. Required for rolling and tile functions; optional (but usually wanted) for cumulative functions. window_functions : list[WindowFunctionInput] Ordered list of per-column window operations to apply. Each produces one new column; all are applied in a single with_columns call.

Show JSON schema:

{
  "$defs": {
    "SortByInput": {
      "description": "Defines a single sort condition on a column, including the direction.",
      "properties": {
        "column": {
          "title": "Column",
          "type": "string"
        },
        "how": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "asc",
          "title": "How"
        }
      },
      "required": [
        "column"
      ],
      "title": "SortByInput",
      "type": "object"
    },
    "WindowFunctionInput": {
      "description": "A single window-function operation producing one new column.\n\n`column` is the source column for rolling, cumulative and rank functions.\nFor `tile`, `column` is ignored (ordering comes from the outer\n``WindowFunctionsInput.order_by``).",
      "properties": {
        "column": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Column"
        },
        "function": {
          "enum": [
            "rolling_sum",
            "rolling_mean",
            "rolling_min",
            "rolling_max",
            "rolling_std",
            "cum_sum",
            "cum_count",
            "cum_min",
            "cum_max",
            "rank",
            "tile"
          ],
          "title": "Function",
          "type": "string"
        },
        "new_column_name": {
          "title": "New Column Name",
          "type": "string"
        },
        "window_size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Window Size"
        },
        "min_periods": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Min Periods"
        },
        "edge_behavior": {
          "anyOf": [
            {
              "enum": [
                "require_full",
                "partial",
                "fill_zero"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "require_full",
          "title": "Edge Behavior"
        },
        "number_of_groups": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Groups"
        },
        "rank_method": {
          "anyOf": [
            {
              "enum": [
                "ordinal",
                "dense",
                "min",
                "max",
                "average"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "ordinal",
          "title": "Rank Method"
        },
        "output_type": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Output Type"
        }
      },
      "required": [
        "function",
        "new_column_name"
      ],
      "title": "WindowFunctionInput",
      "type": "object"
    }
  },
  "description": "Defines the settings for a window-functions node.\n\nAttributes\n----------\npartition_by : list[str]\n    Optional list of columns to partition by (equivalent to ``.over(...)``).\norder_by : list[SortByInput]\n    Ordering within each partition. Required for rolling and tile\n    functions; optional (but usually wanted) for cumulative functions.\nwindow_functions : list[WindowFunctionInput]\n    Ordered list of per-column window operations to apply. Each produces\n    one new column; all are applied in a single ``with_columns`` call.",
  "properties": {
    "partition_by": {
      "items": {
        "type": "string"
      },
      "title": "Partition By",
      "type": "array"
    },
    "order_by": {
      "items": {
        "$ref": "#/$defs/SortByInput"
      },
      "title": "Order By",
      "type": "array"
    },
    "window_functions": {
      "items": {
        "$ref": "#/$defs/WindowFunctionInput"
      },
      "title": "Window Functions",
      "type": "array"
    }
  },
  "title": "WindowFunctionsInput",
  "type": "object"
}

Fields:

• partition_by (list[str])
• order_by (list[SortByInput])
• window_functions (list[WindowFunctionInput])

Validators:

• _validate

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

class WindowFunctionsInput(BaseModel):
    """Defines the settings for a window-functions node.

    Attributes
    ----------
    partition_by : list[str]
        Optional list of columns to partition by (equivalent to ``.over(...)``).
    order_by : list[SortByInput]
        Ordering within each partition. Required for rolling and tile
        functions; optional (but usually wanted) for cumulative functions.
    window_functions : list[WindowFunctionInput]
        Ordered list of per-column window operations to apply. Each produces
        one new column; all are applied in a single ``with_columns`` call.
    """

    partition_by: list[str] = Field(default_factory=list)
    order_by: list[SortByInput] = Field(default_factory=list)
    window_functions: list[WindowFunctionInput] = Field(default_factory=list)

    @model_validator(mode="after")
    def _validate(self) -> "WindowFunctionsInput":
        needs_order = any(_is_rolling(w.function) or w.function == "tile" for w in self.window_functions)
        if needs_order and not self.order_by:
            raise ValueError("Rolling and tile functions require at least one order_by column")
        seen: set[str] = set()
        for w in self.window_functions:
            if w.new_column_name in seen:
                raise ValueError(f"Duplicate new_column_name: {w.new_column_name!r}")
            seen.add(w.new_column_name)
        return self

construct_join_key_name(side, column_name)

Creates a temporary, unique name for a join key column.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def construct_join_key_name(side: SideLit, column_name: str) -> str:
    """Creates a temporary, unique name for a join key column."""
    return "_FLOWFILE_JOIN_KEY_" + side.upper() + "_" + column_name

get_func_type_mapping(func)

Infers the output data type of common aggregation functions.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_func_type_mapping(func: str):
    """Infers the output data type of common aggregation functions."""
    if func in ["mean", "avg", "median", "std", "var"]:
        return "Float64"
    elif func in ["min", "max", "first", "last", "cumsum", "sum"]:
        return None
    elif func in ["count", "n_unique"]:
        return "Int64"
    elif func in ["concat"]:
        return "Utf8"

get_window_output_type(func, input_type=None)

Infers the output data type of window functions.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def get_window_output_type(func: str, input_type: str | None = None) -> str | None:
    """Infers the output data type of window functions."""
    if func in {"rolling_mean", "rolling_std"}:
        return "Float64"
    if func in {"rolling_sum", "rolling_min", "rolling_max", "cum_sum", "cum_min", "cum_max"}:
        return input_type
    if func in {"cum_count", "tile"}:
        return "Int64"
    if func == "rank":
        return "UInt32"
    return input_type

is_descending(how)

Whether a sort-direction string means descending.

Accepts both the programmatic "asc"/"desc" form (flowfile_frame) and the visual editor's "Ascending"/"Descending" form (case-insensitive).

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def is_descending(how: str | None) -> bool:
    """Whether a sort-direction string means descending.

    Accepts both the programmatic ``"asc"``/``"desc"`` form (flowfile_frame) and the
    visual editor's ``"Ascending"``/``"Descending"`` form (case-insensitive).
    """
    return (how or "").lower() in ("desc", "descending")

string_concat(*column)

A simple wrapper to concatenate string columns in Polars.

Source code in flowfile_core/flowfile_core/schemas/transform_schema.py

def string_concat(*column: str):
    """A simple wrapper to concatenate string columns in Polars."""
    return pl.col(column).cast(pl.Utf8).str.concat(delimiter=",")

cloud_storage_schemas

flowfile_core.schemas.cloud_storage_schemas

Cloud storage connection schemas for S3, ADLS, and other cloud providers.

Classes:

Name	Description
AuthSettingsInput	The information needed for the user to provide the details that are needed to provide how to connect to the
CloudStorageReadSettings	Settings for reading from cloud storage
CloudStorageSettings	Settings for cloud storage nodes in the visual designer
CloudStorageWriteSettings	Settings for writing to cloud storage
CloudStorageWriteSettingsWorkerInterface	Settings for writing to cloud storage in worker context
FullCloudStorageConnection	Internal model with decrypted secrets
FullCloudStorageConnectionInterface	API response model - no secrets exposed
FullCloudStorageConnectionWorkerInterface	Internal model with decrypted secrets
WriteSettingsWorkerInterface	Settings for writing to cloud storage

Functions:

Name	Description
encrypt_for_worker	Encrypts a secret value for use in worker contexts using per-user key derivation.
get_cloud_storage_write_settings_worker_interface	Convert to a worker interface model with encrypted secrets.

AuthSettingsInput pydantic-model

Bases: BaseModel

The information needed for the user to provide the details that are needed to provide how to connect to the Cloud provider

Show JSON schema:

{
  "description": "The information needed for the user to provide the details that are needed to provide how to connect to the\n Cloud provider",
  "properties": {
    "storage_type": {
      "enum": [
        "s3",
        "adls",
        "gcs"
      ],
      "title": "Storage Type",
      "type": "string"
    },
    "auth_method": {
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account"
      ],
      "title": "Auth Method",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "None",
      "title": "Connection Name"
    }
  },
  "required": [
    "storage_type",
    "auth_method"
  ],
  "title": "AuthSettingsInput",
  "type": "object"
}

Fields:

• storage_type (CloudStorageType)
• auth_method (AuthMethod)
• connection_name (str | None)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class AuthSettingsInput(BaseModel):
    """
    The information needed for the user to provide the details that are needed to provide how to connect to the
     Cloud provider
    """

    storage_type: CloudStorageType
    auth_method: AuthMethod
    connection_name: str | None = "None"  # This is the reference to the item we will fetch that contains the data

CloudStorageReadSettings pydantic-model

Bases: CloudStorageSettings

Settings for reading from cloud storage

Show JSON schema:

{
  "description": "Settings for reading from cloud storage",
  "properties": {
    "auth_mode": {
      "default": "auto",
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account",
        "auto"
      ],
      "title": "Auth Mode",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Connection Name"
    },
    "resource_path": {
      "title": "Resource Path",
      "type": "string"
    },
    "scan_mode": {
      "default": "single_file",
      "enum": [
        "single_file",
        "directory"
      ],
      "title": "Scan Mode",
      "type": "string"
    },
    "file_format": {
      "default": "parquet",
      "enum": [
        "csv",
        "parquet",
        "json",
        "delta",
        "iceberg"
      ],
      "title": "File Format",
      "type": "string"
    },
    "csv_has_header": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": true,
      "title": "Csv Has Header"
    },
    "csv_delimiter": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": ",",
      "title": "Csv Delimiter"
    },
    "csv_encoding": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "utf8",
      "title": "Csv Encoding"
    },
    "delta_version": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Delta Version"
    }
  },
  "required": [
    "resource_path"
  ],
  "title": "CloudStorageReadSettings",
  "type": "object"
}

Fields:

• auth_mode (CloudStorageAuthMode)
• connection_name (str | None)
• resource_path (str)
• scan_mode (Literal['single_file', 'directory'])
• file_format (Literal['csv', 'parquet', 'json', 'delta', 'iceberg'])
• csv_has_header (bool | None)
• csv_delimiter (str | None)
• csv_encoding (str | None)
• delta_version (int | None)

Validators:

• validate_auth_requirements → auth_mode

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class CloudStorageReadSettings(CloudStorageSettings):
    """Settings for reading from cloud storage"""

    scan_mode: Literal["single_file", "directory"] = "single_file"
    file_format: Literal["csv", "parquet", "json", "delta", "iceberg"] = "parquet"
    csv_has_header: bool | None = True
    csv_delimiter: str | None = ","
    csv_encoding: str | None = "utf8"
    delta_version: int | None = None

CloudStorageSettings pydantic-model

Bases: BaseModel

Settings for cloud storage nodes in the visual designer

Show JSON schema:

{
  "description": "Settings for cloud storage nodes in the visual designer",
  "properties": {
    "auth_mode": {
      "default": "auto",
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account",
        "auto"
      ],
      "title": "Auth Mode",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Connection Name"
    },
    "resource_path": {
      "title": "Resource Path",
      "type": "string"
    }
  },
  "required": [
    "resource_path"
  ],
  "title": "CloudStorageSettings",
  "type": "object"
}

Fields:

• auth_mode (CloudStorageAuthMode)
• connection_name (str | None)
• resource_path (str)

Validators:

• validate_auth_requirements → auth_mode

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class CloudStorageSettings(BaseModel):
    """Settings for cloud storage nodes in the visual designer"""

    auth_mode: CloudStorageAuthMode = "auto"
    connection_name: str | None = None  # Required only for 'reference' mode
    resource_path: str  # s3://bucket/path/to/file.csv

    @field_validator("auth_mode", mode="after")
    def validate_auth_requirements(cls, v, values):
        data = values.data
        if v == "reference" and not data.get("connection_name"):
            raise ValueError("connection_name required when using reference mode")
        return v

CloudStorageWriteSettings pydantic-model

Bases: CloudStorageSettings, WriteSettingsWorkerInterface

Settings for writing to cloud storage

Show JSON schema:

{
  "description": "Settings for writing to cloud storage",
  "properties": {
    "resource_path": {
      "title": "Resource Path",
      "type": "string"
    },
    "write_mode": {
      "default": "overwrite",
      "enum": [
        "overwrite",
        "append"
      ],
      "title": "Write Mode",
      "type": "string"
    },
    "file_format": {
      "default": "parquet",
      "enum": [
        "csv",
        "parquet",
        "json",
        "delta"
      ],
      "title": "File Format",
      "type": "string"
    },
    "parquet_compression": {
      "default": "snappy",
      "enum": [
        "snappy",
        "gzip",
        "brotli",
        "lz4",
        "zstd"
      ],
      "title": "Parquet Compression",
      "type": "string"
    },
    "csv_delimiter": {
      "default": ",",
      "title": "Csv Delimiter",
      "type": "string"
    },
    "csv_encoding": {
      "default": "utf8",
      "title": "Csv Encoding",
      "type": "string"
    },
    "partition_by": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Partition By"
    },
    "auth_mode": {
      "default": "auto",
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account",
        "auto"
      ],
      "title": "Auth Mode",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Connection Name"
    }
  },
  "required": [
    "resource_path"
  ],
  "title": "CloudStorageWriteSettings",
  "type": "object"
}

Fields:

• resource_path (str)
• write_mode (Literal['overwrite', 'append'])
• file_format (Literal['csv', 'parquet', 'json', 'delta'])
• parquet_compression (Literal['snappy', 'gzip', 'brotli', 'lz4', 'zstd'])
• csv_delimiter (str)
• csv_encoding (str)
• partition_by (list[str] | None)
• auth_mode (CloudStorageAuthMode)
• connection_name (str | None)

Validators:

• validate_auth_requirements → auth_mode

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class CloudStorageWriteSettings(CloudStorageSettings, WriteSettingsWorkerInterface):
    """Settings for writing to cloud storage"""

    pass

    def get_write_setting_worker_interface(self) -> WriteSettingsWorkerInterface:
        """
        Convert to a worker interface model without secrets.
        """
        return WriteSettingsWorkerInterface(
            resource_path=self.resource_path,
            write_mode=self.write_mode,
            file_format=self.file_format,
            parquet_compression=self.parquet_compression,
            csv_delimiter=self.csv_delimiter,
            csv_encoding=self.csv_encoding,
            partition_by=self.partition_by,
        )

get_write_setting_worker_interface()

Convert to a worker interface model without secrets.

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

def get_write_setting_worker_interface(self) -> WriteSettingsWorkerInterface:
    """
    Convert to a worker interface model without secrets.
    """
    return WriteSettingsWorkerInterface(
        resource_path=self.resource_path,
        write_mode=self.write_mode,
        file_format=self.file_format,
        parquet_compression=self.parquet_compression,
        csv_delimiter=self.csv_delimiter,
        csv_encoding=self.csv_encoding,
        partition_by=self.partition_by,
    )

CloudStorageWriteSettingsWorkerInterface pydantic-model

Bases: BaseModel

Settings for writing to cloud storage in worker context

Show JSON schema:

{
  "$defs": {
    "FullCloudStorageConnectionWorkerInterface": {
      "description": "Internal model with decrypted secrets",
      "properties": {
        "storage_type": {
          "enum": [
            "s3",
            "adls",
            "gcs"
          ],
          "title": "Storage Type",
          "type": "string"
        },
        "auth_method": {
          "enum": [
            "access_key",
            "iam_role",
            "service_principal",
            "managed_identity",
            "sas_token",
            "aws-cli",
            "env_vars",
            "service_account"
          ],
          "title": "Auth Method",
          "type": "string"
        },
        "connection_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": "None",
          "title": "Connection Name"
        },
        "aws_region": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Region"
        },
        "aws_access_key_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Access Key Id"
        },
        "aws_secret_access_key": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Secret Access Key"
        },
        "aws_role_arn": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Role Arn"
        },
        "aws_allow_unsafe_html": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Allow Unsafe Html"
        },
        "aws_session_token": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aws Session Token"
        },
        "azure_account_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Account Name"
        },
        "azure_account_key": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Account Key"
        },
        "azure_tenant_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Tenant Id"
        },
        "azure_client_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Client Id"
        },
        "azure_client_secret": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Client Secret"
        },
        "azure_sas_token": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Azure Sas Token"
        },
        "gcs_service_account_key": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Gcs Service Account Key"
        },
        "gcs_project_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Gcs Project Id"
        },
        "endpoint_url": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Endpoint Url"
        },
        "verify_ssl": {
          "default": true,
          "title": "Verify Ssl",
          "type": "boolean"
        }
      },
      "required": [
        "storage_type",
        "auth_method"
      ],
      "title": "FullCloudStorageConnectionWorkerInterface",
      "type": "object"
    },
    "WriteSettingsWorkerInterface": {
      "description": "Settings for writing to cloud storage",
      "properties": {
        "resource_path": {
          "title": "Resource Path",
          "type": "string"
        },
        "write_mode": {
          "default": "overwrite",
          "enum": [
            "overwrite",
            "append"
          ],
          "title": "Write Mode",
          "type": "string"
        },
        "file_format": {
          "default": "parquet",
          "enum": [
            "csv",
            "parquet",
            "json",
            "delta"
          ],
          "title": "File Format",
          "type": "string"
        },
        "parquet_compression": {
          "default": "snappy",
          "enum": [
            "snappy",
            "gzip",
            "brotli",
            "lz4",
            "zstd"
          ],
          "title": "Parquet Compression",
          "type": "string"
        },
        "csv_delimiter": {
          "default": ",",
          "title": "Csv Delimiter",
          "type": "string"
        },
        "csv_encoding": {
          "default": "utf8",
          "title": "Csv Encoding",
          "type": "string"
        },
        "partition_by": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Partition By"
        }
      },
      "required": [
        "resource_path"
      ],
      "title": "WriteSettingsWorkerInterface",
      "type": "object"
    }
  },
  "description": "Settings for writing to cloud storage in worker context",
  "properties": {
    "operation": {
      "title": "Operation",
      "type": "string"
    },
    "write_settings": {
      "$ref": "#/$defs/WriteSettingsWorkerInterface"
    },
    "connection": {
      "$ref": "#/$defs/FullCloudStorageConnectionWorkerInterface"
    },
    "flowfile_flow_id": {
      "default": 1,
      "title": "Flowfile Flow Id",
      "type": "integer"
    },
    "flowfile_node_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "string"
        }
      ],
      "default": -1,
      "title": "Flowfile Node Id"
    }
  },
  "required": [
    "operation",
    "write_settings",
    "connection"
  ],
  "title": "CloudStorageWriteSettingsWorkerInterface",
  "type": "object"
}

Fields:

• operation (str)
• write_settings (WriteSettingsWorkerInterface)
• connection (FullCloudStorageConnectionWorkerInterface)
• flowfile_flow_id (int)
• flowfile_node_id (int | str)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class CloudStorageWriteSettingsWorkerInterface(BaseModel):
    """Settings for writing to cloud storage in worker context"""

    operation: str
    write_settings: WriteSettingsWorkerInterface
    connection: FullCloudStorageConnectionWorkerInterface
    flowfile_flow_id: int = 1
    flowfile_node_id: int | str = -1

FullCloudStorageConnection pydantic-model

Bases: AuthSettingsInput

Internal model with decrypted secrets

Show JSON schema:

{
  "description": "Internal model with decrypted secrets",
  "properties": {
    "storage_type": {
      "enum": [
        "s3",
        "adls",
        "gcs"
      ],
      "title": "Storage Type",
      "type": "string"
    },
    "auth_method": {
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account"
      ],
      "title": "Auth Method",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "None",
      "title": "Connection Name"
    },
    "aws_region": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Region"
    },
    "aws_access_key_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Access Key Id"
    },
    "aws_secret_access_key": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Secret Access Key"
    },
    "aws_role_arn": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Role Arn"
    },
    "aws_allow_unsafe_html": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Allow Unsafe Html"
    },
    "aws_session_token": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Session Token"
    },
    "azure_account_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Account Name"
    },
    "azure_account_key": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Account Key"
    },
    "azure_tenant_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Tenant Id"
    },
    "azure_client_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Client Id"
    },
    "azure_client_secret": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Client Secret"
    },
    "azure_sas_token": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Sas Token"
    },
    "gcs_service_account_key": {
      "anyOf": [
        {
          "format": "password",
          "type": "string",
          "writeOnly": true
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gcs Service Account Key"
    },
    "gcs_project_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gcs Project Id"
    },
    "endpoint_url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Endpoint Url"
    },
    "verify_ssl": {
      "default": true,
      "title": "Verify Ssl",
      "type": "boolean"
    }
  },
  "required": [
    "storage_type",
    "auth_method"
  ],
  "title": "FullCloudStorageConnection",
  "type": "object"
}

Fields:

• storage_type (CloudStorageType)
• auth_method (AuthMethod)
• connection_name (str | None)
• aws_region (str | None)
• aws_access_key_id (str | None)
• aws_secret_access_key (SecretStr | None)
• aws_role_arn (str | None)
• aws_allow_unsafe_html (bool | None)
• aws_session_token (SecretStr | None)
• azure_account_name (str | None)
• azure_account_key (SecretStr | None)
• azure_tenant_id (str | None)
• azure_client_id (str | None)
• azure_client_secret (SecretStr | None)
• azure_sas_token (SecretStr | None)
• gcs_service_account_key (SecretStr | None)
• gcs_project_id (str | None)
• endpoint_url (str | None)
• verify_ssl (bool)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class FullCloudStorageConnection(AuthSettingsInput):
    """Internal model with decrypted secrets"""

    # AWS S3
    aws_region: str | None = None
    aws_access_key_id: str | None = None
    aws_secret_access_key: SecretStr | None = None
    aws_role_arn: str | None = None
    aws_allow_unsafe_html: bool | None = None
    aws_session_token: SecretStr | None = None

    # Azure ADLS
    azure_account_name: str | None = None
    azure_account_key: SecretStr | None = None
    azure_tenant_id: str | None = None
    azure_client_id: str | None = None
    azure_client_secret: SecretStr | None = None
    azure_sas_token: SecretStr | None = None

    # Google Cloud Storage
    gcs_service_account_key: SecretStr | None = None
    gcs_project_id: str | None = None

    # Common
    endpoint_url: str | None = None
    verify_ssl: bool = True

    def get_worker_interface(self, user_id: int) -> "FullCloudStorageConnectionWorkerInterface":
        """
        Convert to a worker interface model with encrypted secrets.

        Args:
            user_id: The user ID for per-user key derivation

        Returns:
            FullCloudStorageConnectionWorkerInterface with encrypted secrets
        """
        return FullCloudStorageConnectionWorkerInterface(
            storage_type=self.storage_type,
            auth_method=self.auth_method,
            connection_name=self.connection_name,
            aws_allow_unsafe_html=self.aws_allow_unsafe_html,
            aws_secret_access_key=encrypt_for_worker(self.aws_secret_access_key, user_id),
            aws_region=self.aws_region,
            aws_access_key_id=self.aws_access_key_id,
            aws_role_arn=self.aws_role_arn,
            aws_session_token=encrypt_for_worker(self.aws_session_token, user_id),
            azure_account_name=self.azure_account_name,
            azure_tenant_id=self.azure_tenant_id,
            azure_account_key=encrypt_for_worker(self.azure_account_key, user_id),
            azure_client_id=self.azure_client_id,
            azure_client_secret=encrypt_for_worker(self.azure_client_secret, user_id),
            azure_sas_token=encrypt_for_worker(self.azure_sas_token, user_id),
            gcs_service_account_key=encrypt_for_worker(self.gcs_service_account_key, user_id),
            gcs_project_id=self.gcs_project_id,
            endpoint_url=self.endpoint_url,
            verify_ssl=self.verify_ssl,
        )

get_worker_interface(user_id)

Convert to a worker interface model with encrypted secrets.

Parameters:

Name	Type	Description	Default
user_id	int	The user ID for per-user key derivation	required

Returns:

Type	Description
FullCloudStorageConnectionWorkerInterface	FullCloudStorageConnectionWorkerInterface with encrypted secrets

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

def get_worker_interface(self, user_id: int) -> "FullCloudStorageConnectionWorkerInterface":
    """
    Convert to a worker interface model with encrypted secrets.

    Args:
        user_id: The user ID for per-user key derivation

    Returns:
        FullCloudStorageConnectionWorkerInterface with encrypted secrets
    """
    return FullCloudStorageConnectionWorkerInterface(
        storage_type=self.storage_type,
        auth_method=self.auth_method,
        connection_name=self.connection_name,
        aws_allow_unsafe_html=self.aws_allow_unsafe_html,
        aws_secret_access_key=encrypt_for_worker(self.aws_secret_access_key, user_id),
        aws_region=self.aws_region,
        aws_access_key_id=self.aws_access_key_id,
        aws_role_arn=self.aws_role_arn,
        aws_session_token=encrypt_for_worker(self.aws_session_token, user_id),
        azure_account_name=self.azure_account_name,
        azure_tenant_id=self.azure_tenant_id,
        azure_account_key=encrypt_for_worker(self.azure_account_key, user_id),
        azure_client_id=self.azure_client_id,
        azure_client_secret=encrypt_for_worker(self.azure_client_secret, user_id),
        azure_sas_token=encrypt_for_worker(self.azure_sas_token, user_id),
        gcs_service_account_key=encrypt_for_worker(self.gcs_service_account_key, user_id),
        gcs_project_id=self.gcs_project_id,
        endpoint_url=self.endpoint_url,
        verify_ssl=self.verify_ssl,
    )

FullCloudStorageConnectionInterface pydantic-model

Bases: AuthSettingsInput

API response model - no secrets exposed

Show JSON schema:

{
  "$defs": {
    "AccessInfo": {
      "description": "How the requesting user can access a resource; attached to list/detail responses.",
      "properties": {
        "is_owner": {
          "title": "Is Owner",
          "type": "boolean"
        },
        "access_level": {
          "enum": [
            "owner",
            "manage",
            "use"
          ],
          "title": "Access Level",
          "type": "string"
        },
        "shared_by": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Shared By"
        }
      },
      "required": [
        "is_owner",
        "access_level"
      ],
      "title": "AccessInfo",
      "type": "object"
    }
  },
  "description": "API response model - no secrets exposed",
  "properties": {
    "storage_type": {
      "enum": [
        "s3",
        "adls",
        "gcs"
      ],
      "title": "Storage Type",
      "type": "string"
    },
    "auth_method": {
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account"
      ],
      "title": "Auth Method",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "None",
      "title": "Connection Name"
    },
    "aws_allow_unsafe_html": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Allow Unsafe Html"
    },
    "aws_region": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Region"
    },
    "aws_access_key_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Access Key Id"
    },
    "aws_role_arn": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Role Arn"
    },
    "azure_account_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Account Name"
    },
    "azure_tenant_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Tenant Id"
    },
    "azure_client_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Client Id"
    },
    "gcs_project_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gcs Project Id"
    },
    "endpoint_url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Endpoint Url"
    },
    "verify_ssl": {
      "default": true,
      "title": "Verify Ssl",
      "type": "boolean"
    },
    "id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Id"
    },
    "access": {
      "anyOf": [
        {
          "$ref": "#/$defs/AccessInfo"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "storage_type",
    "auth_method"
  ],
  "title": "FullCloudStorageConnectionInterface",
  "type": "object"
}

Fields:

• storage_type (CloudStorageType)
• auth_method (AuthMethod)
• connection_name (str | None)
• aws_allow_unsafe_html (bool | None)
• aws_region (str | None)
• aws_access_key_id (str | None)
• aws_role_arn (str | None)
• azure_account_name (str | None)
• azure_tenant_id (str | None)
• azure_client_id (str | None)
• gcs_project_id (str | None)
• endpoint_url (str | None)
• verify_ssl (bool)
• id (int | None)
• access (AccessInfo | None)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class FullCloudStorageConnectionInterface(AuthSettingsInput):
    """API response model - no secrets exposed"""

    # Public fields only
    aws_allow_unsafe_html: bool | None = None
    aws_region: str | None = None
    aws_access_key_id: str | None = None
    aws_role_arn: str | None = None
    azure_account_name: str | None = None
    azure_tenant_id: str | None = None
    azure_client_id: str | None = None
    gcs_project_id: str | None = None
    endpoint_url: str | None = None
    verify_ssl: bool = True
    id: int | None = None
    access: AccessInfo | None = None

FullCloudStorageConnectionWorkerInterface pydantic-model

Bases: AuthSettingsInput

Internal model with decrypted secrets

Show JSON schema:

{
  "description": "Internal model with decrypted secrets",
  "properties": {
    "storage_type": {
      "enum": [
        "s3",
        "adls",
        "gcs"
      ],
      "title": "Storage Type",
      "type": "string"
    },
    "auth_method": {
      "enum": [
        "access_key",
        "iam_role",
        "service_principal",
        "managed_identity",
        "sas_token",
        "aws-cli",
        "env_vars",
        "service_account"
      ],
      "title": "Auth Method",
      "type": "string"
    },
    "connection_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "None",
      "title": "Connection Name"
    },
    "aws_region": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Region"
    },
    "aws_access_key_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Access Key Id"
    },
    "aws_secret_access_key": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Secret Access Key"
    },
    "aws_role_arn": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Role Arn"
    },
    "aws_allow_unsafe_html": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Allow Unsafe Html"
    },
    "aws_session_token": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Aws Session Token"
    },
    "azure_account_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Account Name"
    },
    "azure_account_key": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Account Key"
    },
    "azure_tenant_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Tenant Id"
    },
    "azure_client_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Client Id"
    },
    "azure_client_secret": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Client Secret"
    },
    "azure_sas_token": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Azure Sas Token"
    },
    "gcs_service_account_key": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gcs Service Account Key"
    },
    "gcs_project_id": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Gcs Project Id"
    },
    "endpoint_url": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Endpoint Url"
    },
    "verify_ssl": {
      "default": true,
      "title": "Verify Ssl",
      "type": "boolean"
    }
  },
  "required": [
    "storage_type",
    "auth_method"
  ],
  "title": "FullCloudStorageConnectionWorkerInterface",
  "type": "object"
}

Fields:

• storage_type (CloudStorageType)
• auth_method (AuthMethod)
• connection_name (str | None)
• aws_region (str | None)
• aws_access_key_id (str | None)
• aws_secret_access_key (str | None)
• aws_role_arn (str | None)
• aws_allow_unsafe_html (bool | None)
• aws_session_token (str | None)
• azure_account_name (str | None)
• azure_account_key (str | None)
• azure_tenant_id (str | None)
• azure_client_id (str | None)
• azure_client_secret (str | None)
• azure_sas_token (str | None)
• gcs_service_account_key (str | None)
• gcs_project_id (str | None)
• endpoint_url (str | None)
• verify_ssl (bool)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class FullCloudStorageConnectionWorkerInterface(AuthSettingsInput):
    """Internal model with decrypted secrets"""

    # AWS S3
    aws_region: str | None = None
    aws_access_key_id: str | None = None
    aws_secret_access_key: str | None = None
    aws_role_arn: str | None = None
    aws_allow_unsafe_html: bool | None = None
    aws_session_token: str | None = None

    # Azure ADLS
    azure_account_name: str | None = None
    azure_account_key: str | None = None
    azure_tenant_id: str | None = None
    azure_client_id: str | None = None
    azure_client_secret: str | None = None
    azure_sas_token: str | None = None

    # Google Cloud Storage
    gcs_service_account_key: str | None = None
    gcs_project_id: str | None = None

    # Common
    endpoint_url: str | None = None
    verify_ssl: bool = True

WriteSettingsWorkerInterface pydantic-model

Bases: BaseModel

Settings for writing to cloud storage

Show JSON schema:

{
  "description": "Settings for writing to cloud storage",
  "properties": {
    "resource_path": {
      "title": "Resource Path",
      "type": "string"
    },
    "write_mode": {
      "default": "overwrite",
      "enum": [
        "overwrite",
        "append"
      ],
      "title": "Write Mode",
      "type": "string"
    },
    "file_format": {
      "default": "parquet",
      "enum": [
        "csv",
        "parquet",
        "json",
        "delta"
      ],
      "title": "File Format",
      "type": "string"
    },
    "parquet_compression": {
      "default": "snappy",
      "enum": [
        "snappy",
        "gzip",
        "brotli",
        "lz4",
        "zstd"
      ],
      "title": "Parquet Compression",
      "type": "string"
    },
    "csv_delimiter": {
      "default": ",",
      "title": "Csv Delimiter",
      "type": "string"
    },
    "csv_encoding": {
      "default": "utf8",
      "title": "Csv Encoding",
      "type": "string"
    },
    "partition_by": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Partition By"
    }
  },
  "required": [
    "resource_path"
  ],
  "title": "WriteSettingsWorkerInterface",
  "type": "object"
}

Fields:

• resource_path (str)
• write_mode (Literal['overwrite', 'append'])
• file_format (Literal['csv', 'parquet', 'json', 'delta'])
• parquet_compression (Literal['snappy', 'gzip', 'brotli', 'lz4', 'zstd'])
• csv_delimiter (str)
• csv_encoding (str)
• partition_by (list[str] | None)

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

class WriteSettingsWorkerInterface(BaseModel):
    """Settings for writing to cloud storage"""

    resource_path: str  # s3://bucket/path/to/file.csv

    write_mode: Literal["overwrite", "append"] = "overwrite"
    file_format: Literal["csv", "parquet", "json", "delta"] = "parquet"

    parquet_compression: Literal["snappy", "gzip", "brotli", "lz4", "zstd"] = "snappy"

    csv_delimiter: str = ","
    csv_encoding: str = "utf8"

    # Delta only: partition columns, applied at table creation
    partition_by: list[str] | None = None

encrypt_for_worker(secret_value, user_id)

Encrypts a secret value for use in worker contexts using per-user key derivation.

Parameters:

Name	Type	Description	Default
secret_value	SecretStr | None	The secret value to encrypt	required
user_id	int	The user ID for key derivation	required

Returns:

Type	Description
str | None	Encrypted secret with embedded user_id, or None if secret_value is None

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

def encrypt_for_worker(secret_value: SecretStr | None, user_id: int) -> str | None:
    """
    Encrypts a secret value for use in worker contexts using per-user key derivation.

    Args:
        secret_value: The secret value to encrypt
        user_id: The user ID for key derivation

    Returns:
        Encrypted secret with embedded user_id, or None if secret_value is None
    """
    if secret_value is not None:
        return encrypt_secret(secret_value.get_secret_value(), user_id)
    return None

get_cloud_storage_write_settings_worker_interface(write_settings, connection, lf, user_id, flowfile_flow_id=1, flowfile_node_id=-1)

Convert to a worker interface model with encrypted secrets.

Parameters:

Name	Type	Description	Default
write_settings	CloudStorageWriteSettings	Cloud storage write settings	required
connection	FullCloudStorageConnection	Full cloud storage connection with secrets	required
lf	LazyFrame	LazyFrame to serialize	required
user_id	int	User ID for per-user key derivation	required
flowfile_flow_id	int	Flow ID for tracking	1
flowfile_node_id	int | str	Node ID for tracking	-1

Returns:

Type	Description
CloudStorageWriteSettingsWorkerInterface	CloudStorageWriteSettingsWorkerInterface ready for worker

Source code in flowfile_core/flowfile_core/schemas/cloud_storage_schemas.py

def get_cloud_storage_write_settings_worker_interface(
    write_settings: CloudStorageWriteSettings,
    connection: FullCloudStorageConnection,
    lf: pl.LazyFrame,
    user_id: int,
    flowfile_flow_id: int = 1,
    flowfile_node_id: int | str = -1,
) -> CloudStorageWriteSettingsWorkerInterface:
    """
    Convert to a worker interface model with encrypted secrets.

    Args:
        write_settings: Cloud storage write settings
        connection: Full cloud storage connection with secrets
        lf: LazyFrame to serialize
        user_id: User ID for per-user key derivation
        flowfile_flow_id: Flow ID for tracking
        flowfile_node_id: Node ID for tracking

    Returns:
        CloudStorageWriteSettingsWorkerInterface ready for worker
    """
    operation = base64.b64encode(lf.serialize()).decode()

    return CloudStorageWriteSettingsWorkerInterface(
        operation=operation,
        write_settings=write_settings.get_write_setting_worker_interface(),
        connection=connection.get_worker_interface(user_id),
        flowfile_flow_id=flowfile_flow_id,
        flowfile_node_id=flowfile_node_id,
    )

output_model

flowfile_core.schemas.output_model

Classes:

Name	Description
BaseItem	A base model for any item in a file system, like a file or directory.
ExpressionRef	A reference to a single Polars expression, including its name and docstring.
ExpressionsOverview	Represents a categorized list of available Polars expressions.
FileColumn	Represents detailed schema and statistics for a single column (field).
InstantFuncResult	Represents the result of a function that is expected to execute instantly.
ItemInfo	Provides detailed information about a single item in an output directory.
NodeData	A comprehensive model holding the complete state and data for a single node.
NodeDescriptionResponse	Response model for the node description endpoint.
NodeInputNameInfo	Describes a named input available for a kernel node.
NodeResult	Represents the execution result of a single node in a FlowGraph run.
OutputDir	Represents the contents of a single output directory.
OutputFile	Represents a single file in an output directory, extending BaseItem.
OutputFiles	Represents a collection of files, typically within a directory.
OutputTree	Represents a directory tree, including subdirectories.
ProjectExportFile	A single file in a project export (path relative to the project root).
ProjectExportManifest	The full file manifest of a flow exported as a Python project.
ProjectSaveRequest	Request to write a project export to a directory on the server.
ProjectSaveResponse	Result of writing a project export to disk.
RunInformation	Contains summary information about a complete FlowGraph execution.
TableExample	Represents a preview of a table, including schema and sample data.

BaseItem pydantic-model

Bases: BaseModel

A base model for any item in a file system, like a file or directory.

Show JSON schema:

{
  "description": "A base model for any item in a file system, like a file or directory.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    }
  },
  "required": [
    "name",
    "path"
  ],
  "title": "BaseItem",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class BaseItem(BaseModel):
    """A base model for any item in a file system, like a file or directory."""

    name: str
    path: str
    size: int | None = None
    creation_date: datetime | None = None
    access_date: datetime | None = None
    modification_date: datetime | None = None
    source_path: str | None = None
    number_of_items: int = -1

ExpressionRef pydantic-model

Bases: BaseModel

A reference to a single Polars expression, including its name and docstring.

Show JSON schema:

{
  "description": "A reference to a single Polars expression, including its name and docstring.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "doc": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "title": "Doc"
    }
  },
  "required": [
    "name",
    "doc"
  ],
  "title": "ExpressionRef",
  "type": "object"
}

Fields:

• name (str)
• doc (str | None)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ExpressionRef(BaseModel):
    """A reference to a single Polars expression, including its name and docstring."""

    name: str
    doc: str | None

ExpressionsOverview pydantic-model

Bases: BaseModel

Represents a categorized list of available Polars expressions.

Show JSON schema:

{
  "$defs": {
    "ExpressionRef": {
      "description": "A reference to a single Polars expression, including its name and docstring.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "doc": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "title": "Doc"
        }
      },
      "required": [
        "name",
        "doc"
      ],
      "title": "ExpressionRef",
      "type": "object"
    }
  },
  "description": "Represents a categorized list of available Polars expressions.",
  "properties": {
    "expression_type": {
      "title": "Expression Type",
      "type": "string"
    },
    "expressions": {
      "items": {
        "$ref": "#/$defs/ExpressionRef"
      },
      "title": "Expressions",
      "type": "array"
    }
  },
  "required": [
    "expression_type",
    "expressions"
  ],
  "title": "ExpressionsOverview",
  "type": "object"
}

Fields:

• expression_type (str)
• expressions (list[ExpressionRef])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ExpressionsOverview(BaseModel):
    """Represents a categorized list of available Polars expressions."""

    expression_type: str
    expressions: list[ExpressionRef]

FileColumn pydantic-model

Bases: BaseModel

Represents detailed schema and statistics for a single column (field).

The statistics fields are None until they are actually computed — either never (plain schema previews) or exactly, on demand, via the column-stats endpoint writing into the node's FlowfileColumn.

Show JSON schema:

{
  "description": "Represents detailed schema and statistics for a single column (field).\n\nThe statistics fields are None until they are actually computed \u2014 either\nnever (plain schema previews) or exactly, on demand, via the column-stats\nendpoint writing into the node's ``FlowfileColumn``.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "data_type": {
      "title": "Data Type",
      "type": "string"
    },
    "data_type_group": {
      "default": "Other",
      "enum": [
        "Numeric",
        "String",
        "Date",
        "Other",
        "Boolean",
        "Binary",
        "Complex"
      ],
      "title": "Data Type Group",
      "type": "string"
    },
    "is_unique": {
      "default": false,
      "title": "Is Unique",
      "type": "boolean"
    },
    "max_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Max Value"
    },
    "min_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Min Value"
    },
    "average_value": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Average Value"
    },
    "number_of_empty_values": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Number Of Empty Values"
    },
    "number_of_filled_values": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Number Of Filled Values"
    },
    "number_of_unique_values": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Number Of Unique Values"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    }
  },
  "required": [
    "name",
    "data_type"
  ],
  "title": "FileColumn",
  "type": "object"
}

Fields:

• name (str)
• data_type (str)
• data_type_group (ReadableDataTypeGroup)
• is_unique (bool)
• max_value (str | None)
• min_value (str | None)
• average_value (str | None)
• number_of_empty_values (int | None)
• number_of_filled_values (int | None)
• number_of_unique_values (int | None)
• size (int | None)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class FileColumn(BaseModel):
    """Represents detailed schema and statistics for a single column (field).

    The statistics fields are None until they are actually computed — either
    never (plain schema previews) or exactly, on demand, via the column-stats
    endpoint writing into the node's ``FlowfileColumn``.
    """

    name: str
    data_type: str
    data_type_group: ReadableDataTypeGroup = "Other"
    is_unique: bool = False
    max_value: str | None = None
    min_value: str | None = None
    average_value: str | None = None
    number_of_empty_values: int | None = None
    number_of_filled_values: int | None = None
    number_of_unique_values: int | None = None
    size: int | None = None

InstantFuncResult pydantic-model

Bases: BaseModel

Represents the result of a function that is expected to execute instantly.

Show JSON schema:

{
  "description": "Represents the result of a function that is expected to execute instantly.",
  "properties": {
    "success": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Success"
    },
    "result": {
      "title": "Result",
      "type": "string"
    }
  },
  "required": [
    "result"
  ],
  "title": "InstantFuncResult",
  "type": "object"
}

Fields:

• success (bool | None)
• result (str)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class InstantFuncResult(BaseModel):
    """Represents the result of a function that is expected to execute instantly."""

    success: bool | None = None
    result: str

ItemInfo pydantic-model

Bases: OutputFile

Provides detailed information about a single item in an output directory.

Show JSON schema:

{
  "description": "Provides detailed information about a single item in an output directory.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    },
    "ext": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Ext"
    },
    "mimetype": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Mimetype"
    },
    "id": {
      "default": -1,
      "title": "Id",
      "type": "integer"
    },
    "type": {
      "title": "Type",
      "type": "string"
    },
    "analysis_file_available": {
      "default": false,
      "title": "Analysis File Available",
      "type": "boolean"
    },
    "analysis_file_location": {
      "default": null,
      "title": "Analysis File Location",
      "type": "string"
    },
    "analysis_file_error": {
      "default": null,
      "title": "Analysis File Error",
      "type": "string"
    }
  },
  "required": [
    "name",
    "path",
    "type"
  ],
  "title": "ItemInfo",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)
• ext (str | None)
• mimetype (str | None)
• id (int)
• type (str)
• analysis_file_available (bool)
• analysis_file_location (str)
• analysis_file_error (str)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ItemInfo(OutputFile):
    """Provides detailed information about a single item in an output directory."""

    id: int = -1
    type: str
    analysis_file_available: bool = False
    analysis_file_location: str = None
    analysis_file_error: str = None

NodeData pydantic-model

Bases: BaseModel

A comprehensive model holding the complete state and data for a single node.

This includes its input/output data previews, settings, and run status.

Show JSON schema:

{
  "$defs": {
    "FileColumn": {
      "description": "Represents detailed schema and statistics for a single column (field).\n\nThe statistics fields are None until they are actually computed \u2014 either\nnever (plain schema previews) or exactly, on demand, via the column-stats\nendpoint writing into the node's ``FlowfileColumn``.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "title": "Data Type",
          "type": "string"
        },
        "data_type_group": {
          "default": "Other",
          "enum": [
            "Numeric",
            "String",
            "Date",
            "Other",
            "Boolean",
            "Binary",
            "Complex"
          ],
          "title": "Data Type Group",
          "type": "string"
        },
        "is_unique": {
          "default": false,
          "title": "Is Unique",
          "type": "boolean"
        },
        "max_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Max Value"
        },
        "min_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Min Value"
        },
        "average_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Average Value"
        },
        "number_of_empty_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Empty Values"
        },
        "number_of_filled_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Filled Values"
        },
        "number_of_unique_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Unique Values"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        }
      },
      "required": [
        "name",
        "data_type"
      ],
      "title": "FileColumn",
      "type": "object"
    },
    "TableExample": {
      "description": "Represents a preview of a table, including schema and sample data.\n\n``number_of_records`` is None when the total is unknown (e.g. a lazy result\nwhose count was never computed); 0 always means a genuinely empty result.",
      "properties": {
        "node_id": {
          "title": "Node Id",
          "type": "integer"
        },
        "number_of_records": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Records"
        },
        "number_of_columns": {
          "title": "Number Of Columns",
          "type": "integer"
        },
        "name": {
          "title": "Name",
          "type": "string"
        },
        "table_schema": {
          "items": {
            "$ref": "#/$defs/FileColumn"
          },
          "title": "Table Schema",
          "type": "array"
        },
        "columns": {
          "items": {
            "type": "string"
          },
          "title": "Columns",
          "type": "array"
        },
        "data": {
          "anyOf": [
            {
              "items": {
                "additionalProperties": true,
                "type": "object"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Data"
        },
        "has_example_data": {
          "default": false,
          "title": "Has Example Data",
          "type": "boolean"
        },
        "has_run_with_current_setup": {
          "default": false,
          "title": "Has Run With Current Setup",
          "type": "boolean"
        }
      },
      "required": [
        "node_id",
        "number_of_columns",
        "name",
        "table_schema",
        "columns"
      ],
      "title": "TableExample",
      "type": "object"
    }
  },
  "description": "A comprehensive model holding the complete state and data for a single node.\n\nThis includes its input/output data previews, settings, and run status.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "flow_type": {
      "title": "Flow Type",
      "type": "string"
    },
    "left_input": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "right_input": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "main_input": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "main_output": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "left_output": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "right_output": {
      "anyOf": [
        {
          "$ref": "#/$defs/TableExample"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    },
    "has_run": {
      "default": false,
      "title": "Has Run",
      "type": "boolean"
    },
    "is_cached": {
      "default": false,
      "title": "Is Cached",
      "type": "boolean"
    },
    "setting_input": {
      "default": null,
      "title": "Setting Input"
    },
    "prediction_warning": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Prediction Warning"
    }
  },
  "required": [
    "flow_id",
    "node_id",
    "flow_type"
  ],
  "title": "NodeData",
  "type": "object"
}

Fields:

• flow_id (int)
• node_id (int)
• flow_type (str)
• left_input (TableExample | None)
• right_input (TableExample | None)
• main_input (TableExample | None)
• main_output (TableExample | None)
• left_output (TableExample | None)
• right_output (TableExample | None)
• has_run (bool)
• is_cached (bool)
• setting_input (Any)
• prediction_warning (str | None)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class NodeData(BaseModel):
    """A comprehensive model holding the complete state and data for a single node.

    This includes its input/output data previews, settings, and run status.
    """

    flow_id: int
    node_id: int
    flow_type: str
    left_input: TableExample | None = None
    right_input: TableExample | None = None
    main_input: TableExample | None = None
    main_output: TableExample | None = None
    left_output: TableExample | None = None
    right_output: TableExample | None = None
    has_run: bool = False
    is_cached: bool = False
    setting_input: Any = None
    # Set when column prediction for this node (or one of its inputs) would
    # require executing an un-run kernel node — the user-facing warning text.
    prediction_warning: str | None = None

NodeDescriptionResponse pydantic-model

Bases: BaseModel

Response model for the node description endpoint.

Show JSON schema:

{
  "description": "Response model for the node description endpoint.",
  "properties": {
    "description": {
      "default": "",
      "title": "Description",
      "type": "string"
    },
    "is_auto_generated": {
      "default": false,
      "title": "Is Auto Generated",
      "type": "boolean"
    }
  },
  "title": "NodeDescriptionResponse",
  "type": "object"
}

Fields:

• description (str)
• is_auto_generated (bool)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class NodeDescriptionResponse(BaseModel):
    """Response model for the node description endpoint."""

    description: str = ""
    is_auto_generated: bool = False

NodeInputNameInfo pydantic-model

Bases: BaseModel

Describes a named input available for a kernel node.

Show JSON schema:

{
  "description": "Describes a named input available for a kernel node.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "source_node_id": {
      "title": "Source Node Id",
      "type": "integer"
    },
    "source_node_type": {
      "title": "Source Node Type",
      "type": "string"
    }
  },
  "required": [
    "name",
    "source_node_id",
    "source_node_type"
  ],
  "title": "NodeInputNameInfo",
  "type": "object"
}

Fields:

• name (str)
• source_node_id (int)
• source_node_type (str)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class NodeInputNameInfo(BaseModel):
    """Describes a named input available for a kernel node."""

    name: str
    source_node_id: int
    source_node_type: str

NodeResult pydantic-model

Bases: BaseModel

Represents the execution result of a single node in a FlowGraph run.

Show JSON schema:

{
  "description": "Represents the execution result of a single node in a FlowGraph run.",
  "properties": {
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "node_name": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Node Name"
    },
    "description": {
      "default": "",
      "title": "Description",
      "type": "string"
    },
    "start_timestamp": {
      "title": "Start Timestamp",
      "type": "number"
    },
    "end_timestamp": {
      "default": 0,
      "title": "End Timestamp",
      "type": "number"
    },
    "success": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Success"
    },
    "error": {
      "default": "",
      "title": "Error",
      "type": "string"
    },
    "run_time_ms": {
      "default": -1,
      "description": "Run time in milliseconds",
      "title": "Run Time Ms",
      "type": "integer"
    },
    "is_running": {
      "default": true,
      "title": "Is Running",
      "type": "boolean"
    }
  },
  "required": [
    "node_id"
  ],
  "title": "NodeResult",
  "type": "object"
}

Fields:

• node_id (int)
• node_name (str | None)
• description (str)
• start_timestamp (float)
• end_timestamp (float)
• success (bool | None)
• error (str)
• run_time_ms (int)
• is_running (bool)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class NodeResult(BaseModel):
    """Represents the execution result of a single node in a FlowGraph run."""

    node_id: int
    node_name: str | None = None
    description: str = ""
    start_timestamp: float = Field(default_factory=time.time)
    end_timestamp: float = 0
    success: bool | None = None
    error: str = ""
    run_time_ms: int = Field(
        default=-1,
        description="Run time in milliseconds",
        validation_alias=AliasChoices("run_time_ms", "run_time"),
    )
    is_running: bool = True

run_time_ms = -1 pydantic-field

Run time in milliseconds

OutputDir pydantic-model

Bases: BaseItem

Represents the contents of a single output directory.

Show JSON schema:

{
  "$defs": {
    "ItemInfo": {
      "description": "Provides detailed information about a single item in an output directory.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "path": {
          "title": "Path",
          "type": "string"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        },
        "creation_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Creation Date"
        },
        "access_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Access Date"
        },
        "modification_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Modification Date"
        },
        "source_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Source Path"
        },
        "number_of_items": {
          "default": -1,
          "title": "Number Of Items",
          "type": "integer"
        },
        "ext": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Ext"
        },
        "mimetype": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Mimetype"
        },
        "id": {
          "default": -1,
          "title": "Id",
          "type": "integer"
        },
        "type": {
          "title": "Type",
          "type": "string"
        },
        "analysis_file_available": {
          "default": false,
          "title": "Analysis File Available",
          "type": "boolean"
        },
        "analysis_file_location": {
          "default": null,
          "title": "Analysis File Location",
          "type": "string"
        },
        "analysis_file_error": {
          "default": null,
          "title": "Analysis File Error",
          "type": "string"
        }
      },
      "required": [
        "name",
        "path",
        "type"
      ],
      "title": "ItemInfo",
      "type": "object"
    }
  },
  "description": "Represents the contents of a single output directory.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    },
    "all_items": {
      "items": {
        "type": "string"
      },
      "title": "All Items",
      "type": "array"
    },
    "items": {
      "items": {
        "$ref": "#/$defs/ItemInfo"
      },
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "name",
    "path",
    "all_items",
    "items"
  ],
  "title": "OutputDir",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)
• all_items (list[str])
• items (list[ItemInfo])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class OutputDir(BaseItem):
    """Represents the contents of a single output directory."""

    all_items: list[str]
    items: list[ItemInfo]

OutputFile pydantic-model

Bases: BaseItem

Represents a single file in an output directory, extending BaseItem.

Show JSON schema:

{
  "description": "Represents a single file in an output directory, extending BaseItem.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    },
    "ext": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Ext"
    },
    "mimetype": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Mimetype"
    }
  },
  "required": [
    "name",
    "path"
  ],
  "title": "OutputFile",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)
• ext (str | None)
• mimetype (str | None)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class OutputFile(BaseItem):
    """Represents a single file in an output directory, extending BaseItem."""

    ext: str | None = None
    mimetype: str | None = None

OutputFiles pydantic-model

Bases: BaseItem

Represents a collection of files, typically within a directory.

Show JSON schema:

{
  "$defs": {
    "OutputFile": {
      "description": "Represents a single file in an output directory, extending BaseItem.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "path": {
          "title": "Path",
          "type": "string"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        },
        "creation_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Creation Date"
        },
        "access_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Access Date"
        },
        "modification_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Modification Date"
        },
        "source_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Source Path"
        },
        "number_of_items": {
          "default": -1,
          "title": "Number Of Items",
          "type": "integer"
        },
        "ext": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Ext"
        },
        "mimetype": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Mimetype"
        }
      },
      "required": [
        "name",
        "path"
      ],
      "title": "OutputFile",
      "type": "object"
    }
  },
  "description": "Represents a collection of files, typically within a directory.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    },
    "files": {
      "items": {
        "$ref": "#/$defs/OutputFile"
      },
      "title": "Files",
      "type": "array"
    }
  },
  "required": [
    "name",
    "path"
  ],
  "title": "OutputFiles",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)
• files (list[OutputFile])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class OutputFiles(BaseItem):
    """Represents a collection of files, typically within a directory."""

    files: list[OutputFile] = Field(default_factory=list)

OutputTree pydantic-model

Bases: OutputFiles

Represents a directory tree, including subdirectories.

Show JSON schema:

{
  "$defs": {
    "OutputFile": {
      "description": "Represents a single file in an output directory, extending BaseItem.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "path": {
          "title": "Path",
          "type": "string"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        },
        "creation_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Creation Date"
        },
        "access_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Access Date"
        },
        "modification_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Modification Date"
        },
        "source_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Source Path"
        },
        "number_of_items": {
          "default": -1,
          "title": "Number Of Items",
          "type": "integer"
        },
        "ext": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Ext"
        },
        "mimetype": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Mimetype"
        }
      },
      "required": [
        "name",
        "path"
      ],
      "title": "OutputFile",
      "type": "object"
    },
    "OutputFiles": {
      "description": "Represents a collection of files, typically within a directory.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "path": {
          "title": "Path",
          "type": "string"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        },
        "creation_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Creation Date"
        },
        "access_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Access Date"
        },
        "modification_date": {
          "anyOf": [
            {
              "format": "date-time",
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Modification Date"
        },
        "source_path": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Source Path"
        },
        "number_of_items": {
          "default": -1,
          "title": "Number Of Items",
          "type": "integer"
        },
        "files": {
          "items": {
            "$ref": "#/$defs/OutputFile"
          },
          "title": "Files",
          "type": "array"
        }
      },
      "required": [
        "name",
        "path"
      ],
      "title": "OutputFiles",
      "type": "object"
    }
  },
  "description": "Represents a directory tree, including subdirectories.",
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "path": {
      "title": "Path",
      "type": "string"
    },
    "size": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Size"
    },
    "creation_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Creation Date"
    },
    "access_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Access Date"
    },
    "modification_date": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Modification Date"
    },
    "source_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Source Path"
    },
    "number_of_items": {
      "default": -1,
      "title": "Number Of Items",
      "type": "integer"
    },
    "files": {
      "items": {
        "$ref": "#/$defs/OutputFile"
      },
      "title": "Files",
      "type": "array"
    },
    "directories": {
      "items": {
        "$ref": "#/$defs/OutputFiles"
      },
      "title": "Directories",
      "type": "array"
    }
  },
  "required": [
    "name",
    "path"
  ],
  "title": "OutputTree",
  "type": "object"
}

Fields:

• name (str)
• path (str)
• size (int | None)
• creation_date (datetime | None)
• access_date (datetime | None)
• modification_date (datetime | None)
• source_path (str | None)
• number_of_items (int)
• files (list[OutputFile])
• directories (list[OutputFiles])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class OutputTree(OutputFiles):
    """Represents a directory tree, including subdirectories."""

    directories: list[OutputFiles] = Field(default_factory=list)

ProjectExportFile pydantic-model

Bases: BaseModel

A single file in a project export (path relative to the project root).

Show JSON schema:

{
  "description": "A single file in a project export (path relative to the project root).",
  "properties": {
    "path": {
      "title": "Path",
      "type": "string"
    },
    "content": {
      "title": "Content",
      "type": "string"
    }
  },
  "required": [
    "path",
    "content"
  ],
  "title": "ProjectExportFile",
  "type": "object"
}

Fields:

• path (str)
• content (str)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ProjectExportFile(BaseModel):
    """A single file in a project export (path relative to the project root)."""

    path: str
    content: str

ProjectExportManifest pydantic-model

Bases: BaseModel

The full file manifest of a flow exported as a Python project.

Show JSON schema:

{
  "$defs": {
    "ProjectExportFile": {
      "description": "A single file in a project export (path relative to the project root).",
      "properties": {
        "path": {
          "title": "Path",
          "type": "string"
        },
        "content": {
          "title": "Content",
          "type": "string"
        }
      },
      "required": [
        "path",
        "content"
      ],
      "title": "ProjectExportFile",
      "type": "object"
    }
  },
  "description": "The full file manifest of a flow exported as a Python project.",
  "properties": {
    "project_name": {
      "title": "Project Name",
      "type": "string"
    },
    "files": {
      "items": {
        "$ref": "#/$defs/ProjectExportFile"
      },
      "title": "Files",
      "type": "array"
    },
    "warnings": {
      "items": {
        "type": "string"
      },
      "title": "Warnings",
      "type": "array"
    }
  },
  "required": [
    "project_name",
    "files"
  ],
  "title": "ProjectExportManifest",
  "type": "object"
}

Fields:

• project_name (str)
• files (list[ProjectExportFile])
• warnings (list[str])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ProjectExportManifest(BaseModel):
    """The full file manifest of a flow exported as a Python project."""

    project_name: str
    files: list[ProjectExportFile]
    warnings: list[str] = Field(default_factory=list)

ProjectSaveRequest pydantic-model

Bases: BaseModel

Request to write a project export to a directory on the server.

Show JSON schema:

{
  "description": "Request to write a project export to a directory on the server.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "target_directory": {
      "title": "Target Directory",
      "type": "string"
    },
    "overwrite": {
      "default": false,
      "title": "Overwrite",
      "type": "boolean"
    }
  },
  "required": [
    "flow_id",
    "target_directory"
  ],
  "title": "ProjectSaveRequest",
  "type": "object"
}

Fields:

• flow_id (int)
• target_directory (str)
• overwrite (bool)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ProjectSaveRequest(BaseModel):
    """Request to write a project export to a directory on the server."""

    flow_id: int
    target_directory: str
    overwrite: bool = False

ProjectSaveResponse pydantic-model

Bases: BaseModel

Result of writing a project export to disk.

Show JSON schema:

{
  "description": "Result of writing a project export to disk.",
  "properties": {
    "saved_to": {
      "title": "Saved To",
      "type": "string"
    },
    "file_count": {
      "title": "File Count",
      "type": "integer"
    }
  },
  "required": [
    "saved_to",
    "file_count"
  ],
  "title": "ProjectSaveResponse",
  "type": "object"
}

Fields:

• saved_to (str)
• file_count (int)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class ProjectSaveResponse(BaseModel):
    """Result of writing a project export to disk."""

    saved_to: str
    file_count: int

RunInformation pydantic-model

Bases: BaseModel

Contains summary information about a complete FlowGraph execution.

Show JSON schema:

{
  "$defs": {
    "NodeResult": {
      "description": "Represents the execution result of a single node in a FlowGraph run.",
      "properties": {
        "node_id": {
          "title": "Node Id",
          "type": "integer"
        },
        "node_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Node Name"
        },
        "description": {
          "default": "",
          "title": "Description",
          "type": "string"
        },
        "start_timestamp": {
          "title": "Start Timestamp",
          "type": "number"
        },
        "end_timestamp": {
          "default": 0,
          "title": "End Timestamp",
          "type": "number"
        },
        "success": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Success"
        },
        "error": {
          "default": "",
          "title": "Error",
          "type": "string"
        },
        "run_time_ms": {
          "default": -1,
          "description": "Run time in milliseconds",
          "title": "Run Time Ms",
          "type": "integer"
        },
        "is_running": {
          "default": true,
          "title": "Is Running",
          "type": "boolean"
        }
      },
      "required": [
        "node_id"
      ],
      "title": "NodeResult",
      "type": "object"
    }
  },
  "description": "Contains summary information about a complete FlowGraph execution.",
  "properties": {
    "flow_id": {
      "title": "Flow Id",
      "type": "integer"
    },
    "start_time": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "title": "Start Time"
    },
    "end_time": {
      "anyOf": [
        {
          "format": "date-time",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "End Time"
    },
    "success": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Success"
    },
    "is_running": {
      "default": false,
      "title": "Is Running",
      "type": "boolean"
    },
    "execution_mode": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Execution Mode"
    },
    "nodes_completed": {
      "default": 0,
      "title": "Nodes Completed",
      "type": "integer"
    },
    "number_of_nodes": {
      "default": 0,
      "title": "Number Of Nodes",
      "type": "integer"
    },
    "node_step_result": {
      "items": {
        "$ref": "#/$defs/NodeResult"
      },
      "title": "Node Step Result",
      "type": "array"
    },
    "run_type": {
      "enum": [
        "fetch_one",
        "full_run",
        "init"
      ],
      "title": "Run Type",
      "type": "string"
    }
  },
  "required": [
    "flow_id",
    "node_step_result",
    "run_type"
  ],
  "title": "RunInformation",
  "type": "object"
}

Fields:

• flow_id (int)
• start_time (datetime | None)
• end_time (datetime | None)
• success (bool | None)
• is_running (bool)
• execution_mode (str | None)
• nodes_completed (int)
• number_of_nodes (int)
• node_step_result (list[NodeResult])
• run_type (Literal['fetch_one', 'full_run', 'init'])

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class RunInformation(BaseModel):
    """Contains summary information about a complete FlowGraph execution."""

    flow_id: int
    start_time: datetime | None = Field(default_factory=datetime.now)
    end_time: datetime | None = None
    success: bool | None = None
    is_running: bool = False
    execution_mode: str | None = None
    nodes_completed: int = 0
    number_of_nodes: int = 0
    node_step_result: list[NodeResult]
    run_type: Literal["fetch_one", "full_run", "init"]

TableExample pydantic-model

Bases: BaseModel

Represents a preview of a table, including schema and sample data.

number_of_records is None when the total is unknown (e.g. a lazy result whose count was never computed); 0 always means a genuinely empty result.

Show JSON schema:

{
  "$defs": {
    "FileColumn": {
      "description": "Represents detailed schema and statistics for a single column (field).\n\nThe statistics fields are None until they are actually computed \u2014 either\nnever (plain schema previews) or exactly, on demand, via the column-stats\nendpoint writing into the node's ``FlowfileColumn``.",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "title": "Data Type",
          "type": "string"
        },
        "data_type_group": {
          "default": "Other",
          "enum": [
            "Numeric",
            "String",
            "Date",
            "Other",
            "Boolean",
            "Binary",
            "Complex"
          ],
          "title": "Data Type Group",
          "type": "string"
        },
        "is_unique": {
          "default": false,
          "title": "Is Unique",
          "type": "boolean"
        },
        "max_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Max Value"
        },
        "min_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Min Value"
        },
        "average_value": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Average Value"
        },
        "number_of_empty_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Empty Values"
        },
        "number_of_filled_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Filled Values"
        },
        "number_of_unique_values": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Number Of Unique Values"
        },
        "size": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Size"
        }
      },
      "required": [
        "name",
        "data_type"
      ],
      "title": "FileColumn",
      "type": "object"
    }
  },
  "description": "Represents a preview of a table, including schema and sample data.\n\n``number_of_records`` is None when the total is unknown (e.g. a lazy result\nwhose count was never computed); 0 always means a genuinely empty result.",
  "properties": {
    "node_id": {
      "title": "Node Id",
      "type": "integer"
    },
    "number_of_records": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Number Of Records"
    },
    "number_of_columns": {
      "title": "Number Of Columns",
      "type": "integer"
    },
    "name": {
      "title": "Name",
      "type": "string"
    },
    "table_schema": {
      "items": {
        "$ref": "#/$defs/FileColumn"
      },
      "title": "Table Schema",
      "type": "array"
    },
    "columns": {
      "items": {
        "type": "string"
      },
      "title": "Columns",
      "type": "array"
    },
    "data": {
      "anyOf": [
        {
          "items": {
            "additionalProperties": true,
            "type": "object"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Data"
    },
    "has_example_data": {
      "default": false,
      "title": "Has Example Data",
      "type": "boolean"
    },
    "has_run_with_current_setup": {
      "default": false,
      "title": "Has Run With Current Setup",
      "type": "boolean"
    }
  },
  "required": [
    "node_id",
    "number_of_columns",
    "name",
    "table_schema",
    "columns"
  ],
  "title": "TableExample",
  "type": "object"
}

Fields:

• node_id (int)
• number_of_records (int | None)
• number_of_columns (int)
• name (str)
• table_schema (list[FileColumn])
• columns (list[str])
• data (list[dict] | None)
• has_example_data (bool)
• has_run_with_current_setup (bool)

Source code in flowfile_core/flowfile_core/schemas/output_model.py

class TableExample(BaseModel):
    """Represents a preview of a table, including schema and sample data.

    ``number_of_records`` is None when the total is unknown (e.g. a lazy result
    whose count was never computed); 0 always means a genuinely empty result.
    """

    node_id: int
    number_of_records: int | None = None
    number_of_columns: int
    name: str
    table_schema: list[FileColumn]
    columns: list[str]
    data: list[dict] | None = None
    has_example_data: bool = False
    has_run_with_current_setup: bool = False

---

Web API

This section documents the FastAPI routes that expose flowfile-core's functionality over HTTP.

routes

flowfile_core.routes.routes

Main API router and endpoint definitions for the Flowfile application.

This module sets up the FastAPI router, defines all the API endpoints for interacting with flows, nodes, files, and other core components of the application. It handles the logic for creating, reading, updating, and deleting these resources.

Classes:

Name	Description
DynamicRenamePreviewRequest	Request body for /dynamic_rename/preview.
DynamicRenamePreviewResponse	Response body for /dynamic_rename/preview.
GroupOperationResponse	OperationResponse that also returns the affected group (for server-assigned ids).
RestApiSampleResponse	Inferred output columns from a REST API sample fetch.

Functions:

Name	Description
add_generic_settings	A generic endpoint to update the settings of any node.
add_node	Adds a new, unconfigured node (a "promise") to the flow graph.
add_nodes_to_group	Add nodes to an existing group.
cancel_flow	Cancels a currently running flow execution.
check_flow_laziness	Check whether a flow supports fully lazy execution for virtual tables.
clear_history	Clear all history for a flow.
close_flow	Closes an active flow session for the current user. Idempotent: closing a flow that isn't in
compute_node_visualization	Compute Graphic Walker chart rows for an Explore Data node.
connect_node	Creates a connection (edge) between two nodes in the flow graph.
copy_node	Copies an existing node's settings to a new node promise.
create_db_connection	Creates and securely stores a new database connection.
create_directory	Creates a new directory at the specified path.
create_flow	Creates a new, empty flow file at the specified path and registers a session for it.
create_from_template	Instantiates a template as a new flow session.
create_group	Create a visual group around a set of nodes. Returns the new server-assigned group.
delete_db_connection	Deletes a stored database connection (own, or group-shared with manage access).
delete_group	Delete a group box (ungroup). Member nodes are kept.
delete_node	Deletes a node from the flow graph.
delete_node_connection	Deletes a connection (edge) between two nodes.
download_generated_project	Generates the project export and returns it as a zip archive.
ensure_templates_available	Downloads template flow YAMLs from GitHub if not already cached locally.
fetch_rest_api_sample	Fetch a small sample from the configured REST API and infer its schema.
get_active_flow_file_sessions	Retrieves a list of all currently active flow sessions for the current user.
get_catalog_flows_directory	Returns the managed flows directory used for catalog-tab saves.
get_db_connections	Retrieves all stored database connections for the current user (without passwords).
get_db_dialects	Returns the supported database dialects (drives the frontend's dialect dropdowns).
get_db_schemas	Returns available schema names for the given database connection.
get_db_tables	Returns available table names for the given database connection and optional schema.
get_default_path	Returns the default starting path for the file browser (user data directory).
get_description_node	Retrieves the description text for a specific node.
get_directory_contents	Gets the contents of a directory path.
get_downstream_node_ids	Gets a list of all node IDs that are downstream dependencies of a given node.
get_excel_sheet_names	Retrieves the sheet names from an Excel file.
get_expression_doc	Retrieves documentation for available Polars expressions.
get_expressions	Retrieves a list of all available Flowfile expression names.
get_flow	Retrieves the settings for a specific flow (including runtime dirty state).
get_flow_artifacts	Returns artifact visualization data for the canvas.
get_flow_frontend_data	Retrieves the data needed to render the flow graph in the frontend.
get_flow_settings	Retrieves the main settings for a flow (including dirty-state info).
get_flow_settings_validation	Conservative static check: node settings that reference missing input columns.
get_generated_code	Generates and returns a Python script with Polars code representing the flow.
get_generated_flowframe_code	Generates and returns a Python script with FlowFrame code representing the flow.
get_generated_project	Generates a multi-file Python project (FlowFrame code) representing the flow.
get_graphic_walker_input	Gets the saved chart specs and field schema for the Graphic Walker explorer.
get_history_status	Get the current state of the history system for a flow.
get_instant_function_result	Executes a simple, instant function on a node's data and returns the result.
get_list_of_saved_flows	Scans a directory for saved flow files (.flowfile).
get_local_files	Retrieves a list of files from a specified local directory.
get_node	Retrieves the complete state and data preview for a single node.
get_node_available_artifacts	Return available artifact metadata for a node.
get_node_column_stats	Computes on-demand statistics for one column of a node's cached result.
get_node_input_names	Returns the named inputs available for a kernel node.
get_node_list	Retrieves the list of all available node types and their templates.
get_node_model	(Internal) Retrieves a node's Pydantic model from the input_schema module by its name.
get_node_upstream_ids	Return the transitive upstream node IDs for a given node.
get_node_visualization_fields	Return the Graphic Walker field schema for an Explore Data node's result.
get_reference_node	Retrieves the reference identifier for a specific node.
get_run_status	Retrieves the run status information for a specific flow.
get_table_example	Retrieves a data preview (schema and sample rows) for a node's output.
get_vue_flow_data	Retrieves the flow data formatted for the Vue-based frontend.
import_saved_flow	Imports a flow from a saved .yaml and registers it as a new session for the current user.
list_templates	Returns metadata for all available flow templates.
overwrite_flow_in_catalog	Overwrite an existing catalog flow's YAML with the contents of another flow.
preview_dynamic_rename	Resolves a dynamic-rename rule against a given schema without mutating any flow.
redo_action	Redo the last undone action on the flow graph.
register_flow	Registers a new flow session with the application for the current user.
remove_nodes_from_group	Remove nodes from their group; a group emptied this way is pruned.
rename_flow	Renames a flow's display name: the catalog registration (when one exists) plus the
run_flow	Executes a flow in a background task.
save_flow	Deprecated GET variant of /save_flow. Prefer POST.
save_flow_post	Saves the current state of a flow to a .yaml.
save_flow_to_catalog	Save a flow into the managed catalog flows directory with a collision-free filename.
save_generated_project	Generates the project export and writes it into a directory on the server.
trigger_fetch_node_data	Fetches and refreshes the data for a specific node.
undo_action	Undo the last action on the flow graph.
update_db_connection	Updates an existing database connection (own, or group-shared with manage access).
update_description_node	Updates the description text for a specific node.
update_flow_settings	Updates the main settings for a flow.
update_group	Rename / recolor / move / resize / collapse a group box.
update_layout	Persist dragged node positions and/or group bounds (one drag-end -> one call).
update_reference_node	Updates the reference identifier for a specific node.
validate_db_settings	Validates that a connection can be made to a database with the given settings.
validate_node_reference	Validates if a reference is valid and unique for a node.

DynamicRenamePreviewRequest pydantic-model

Bases: BaseModel

Request body for /dynamic_rename/preview.

Show JSON schema:

{
  "$defs": {
    "DynamicRenameInput": {
      "description": "Defines settings for a dynamic rename operation.\n\nApplies a single rule (prefix / suffix / formula / first_row) to a set of selected\ncolumns, rather than requiring the user to rename columns one-by-one.\n\nIn formula mode, the flowfile formula syntax is evaluated with `[column_name]`\nbound to each target column's current name; for example `uppercase([column_name])`\nor `\"v2_\" + [column_name]`.\n\nIn first_row mode, the first row of the incoming table is promoted to column\nheaders and then dropped from the data. Non-string values are coerced to `str`;\nnull or empty values raise an error. Selection filters still apply \u2014 only selected\ncolumns are renamed, but the first row is always dropped.",
      "properties": {
        "rename_mode": {
          "default": "prefix",
          "enum": [
            "prefix",
            "suffix",
            "formula",
            "first_row"
          ],
          "title": "Rename Mode",
          "type": "string"
        },
        "prefix": {
          "default": "",
          "title": "Prefix",
          "type": "string"
        },
        "suffix": {
          "default": "",
          "title": "Suffix",
          "type": "string"
        },
        "formula": {
          "default": "",
          "expression": true,
          "title": "Formula",
          "type": "string"
        },
        "selection_mode": {
          "default": "all",
          "enum": [
            "all",
            "list",
            "data_type"
          ],
          "title": "Selection Mode",
          "type": "string"
        },
        "selected_columns": {
          "items": {
            "type": "string"
          },
          "title": "Selected Columns",
          "type": "array"
        },
        "selected_data_type": {
          "anyOf": [
            {
              "enum": [
                "Numeric",
                "String",
                "Date",
                "Other",
                "Boolean",
                "Binary",
                "Complex"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Selected Data Type"
        }
      },
      "title": "DynamicRenameInput",
      "type": "object"
    },
    "_DynamicRenameColumn": {
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type_group": {
          "default": "",
          "title": "Data Type Group",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "_DynamicRenameColumn",
      "type": "object"
    }
  },
  "description": "Request body for `/dynamic_rename/preview`.",
  "properties": {
    "settings": {
      "$ref": "#/$defs/DynamicRenameInput"
    },
    "incoming_columns": {
      "items": {
        "$ref": "#/$defs/_DynamicRenameColumn"
      },
      "title": "Incoming Columns",
      "type": "array"
    }
  },
  "required": [
    "settings"
  ],
  "title": "DynamicRenamePreviewRequest",
  "type": "object"
}

Fields:

• settings (DynamicRenameInput)
• incoming_columns (list[_DynamicRenameColumn])

Source code in flowfile_core/flowfile_core/routes/routes.py

class DynamicRenamePreviewRequest(BaseModel):
    """Request body for `/dynamic_rename/preview`."""

    settings: transform_schema.DynamicRenameInput
    incoming_columns: list[_DynamicRenameColumn] = Field(default_factory=list)

DynamicRenamePreviewResponse pydantic-model

Bases: BaseModel

Response body for /dynamic_rename/preview.

Show JSON schema:

{
  "description": "Response body for `/dynamic_rename/preview`.",
  "properties": {
    "rename_map": {
      "additionalProperties": {
        "type": "string"
      },
      "title": "Rename Map",
      "type": "object"
    },
    "error": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "title": "Error"
    }
  },
  "required": [
    "rename_map"
  ],
  "title": "DynamicRenamePreviewResponse",
  "type": "object"
}

Fields:

• rename_map (dict[str, str])
• error (str | None)

Source code in flowfile_core/flowfile_core/routes/routes.py

class DynamicRenamePreviewResponse(BaseModel):
    """Response body for `/dynamic_rename/preview`."""

    rename_map: dict[str, str]
    error: str | None = None

GroupOperationResponse pydantic-model

Bases: OperationResponse

OperationResponse that also returns the affected group (for server-assigned ids).

Show JSON schema:

{
  "$defs": {
    "FlowfileGroup": {
      "description": "Serialized representation of a visual node group (YAML/JSON).",
      "properties": {
        "id": {
          "title": "Id",
          "type": "integer"
        },
        "name": {
          "default": "Group",
          "title": "Name",
          "type": "string"
        },
        "color": {
          "anyOf": [
            {
              "enum": [
                "slate",
                "blue",
                "green",
                "amber",
                "rose",
                "violet",
                "cyan"
              ],
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Color"
        },
        "x_position": {
          "default": 0.0,
          "title": "X Position",
          "type": "number"
        },
        "y_position": {
          "default": 0.0,
          "title": "Y Position",
          "type": "number"
        },
        "width": {
          "default": 400.0,
          "title": "Width",
          "type": "number"
        },
        "height": {
          "default": 250.0,
          "title": "Height",
          "type": "number"
        },
        "collapsed": {
          "default": false,
          "title": "Collapsed",
          "type": "boolean"
        },
        "parent_group_id": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Parent Group Id"
        }
      },
      "required": [
        "id"
      ],
      "title": "FlowfileGroup",
      "type": "object"
    },
    "HistoryState": {
      "description": "Current state of the history system.\n\nProvides information about what undo/redo operations are available.",
      "properties": {
        "can_undo": {
          "default": false,
          "description": "Whether undo is available",
          "title": "Can Undo",
          "type": "boolean"
        },
        "can_redo": {
          "default": false,
          "description": "Whether redo is available",
          "title": "Can Redo",
          "type": "boolean"
        },
        "undo_description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Description of the action that would be undone",
          "title": "Undo Description"
        },
        "redo_description": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Description of the action that would be redone",
          "title": "Redo Description"
        },
        "undo_count": {
          "default": 0,
          "description": "Number of available undo steps",
          "title": "Undo Count",
          "type": "integer"
        },
        "redo_count": {
          "default": 0,
          "description": "Number of available redo steps",
          "title": "Redo Count",
          "type": "integer"
        }
      },
      "title": "HistoryState",
      "type": "object"
    }
  },
  "description": "OperationResponse that also returns the affected group (for server-assigned ids).",
  "properties": {
    "success": {
      "default": true,
      "description": "Whether the operation succeeded",
      "title": "Success",
      "type": "boolean"
    },
    "message": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Optional message",
      "title": "Message"
    },
    "history": {
      "$ref": "#/$defs/HistoryState",
      "description": "Current history state after the operation"
    },
    "group": {
      "anyOf": [
        {
          "$ref": "#/$defs/FlowfileGroup"
        },
        {
          "type": "null"
        }
      ],
      "default": null
    }
  },
  "required": [
    "history"
  ],
  "title": "GroupOperationResponse",
  "type": "object"
}

Fields:

• success (bool)
• message (str | None)
• history (HistoryState)
• group (FlowfileGroup | None)

Source code in flowfile_core/flowfile_core/routes/routes.py

class GroupOperationResponse(OperationResponse):
    """OperationResponse that also returns the affected group (for server-assigned ids)."""

    group: schemas.FlowfileGroup | None = None

history pydantic-field

Current history state after the operation

message = None pydantic-field

Optional message

success = True pydantic-field

Whether the operation succeeded

RestApiSampleResponse pydantic-model

Bases: BaseModel

Inferred output columns from a REST API sample fetch.

Show JSON schema:

{
  "$defs": {
    "MinimalFieldInfo": {
      "description": "Represents the most basic information about a data field (column).",
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        },
        "data_type": {
          "default": "String",
          "title": "Data Type",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "MinimalFieldInfo",
      "type": "object"
    }
  },
  "description": "Inferred output columns from a REST API sample fetch.",
  "properties": {
    "fields": {
      "items": {
        "$ref": "#/$defs/MinimalFieldInfo"
      },
      "title": "Fields",
      "type": "array"
    }
  },
  "required": [
    "fields"
  ],
  "title": "RestApiSampleResponse",
  "type": "object"
}

Fields:

• fields (list[MinimalFieldInfo])

Source code in flowfile_core/flowfile_core/routes/routes.py

class RestApiSampleResponse(BaseModel):
    """Inferred output columns from a REST API sample fetch."""

    fields: list[input_schema.MinimalFieldInfo]

add_generic_settings(input_data, node_type, current_user=Depends(get_current_active_user))

A generic endpoint to update the settings of any node.

This endpoint dynamically determines the correct Pydantic model and update function based on the node_type parameter.

Returns:

Type	Description
OperationResponse	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/update_settings/", tags=["transform"], response_model=OperationResponse)
def add_generic_settings(
    input_data: dict[str, Any], node_type: str, current_user=Depends(get_current_active_user)
) -> OperationResponse:
    """A generic endpoint to update the settings of any node.

    This endpoint dynamically determines the correct Pydantic model and update
    function based on the `node_type` parameter.

    Returns:
        OperationResponse with current history state.
    """
    input_data["user_id"] = current_user.id
    node_type = camel_case_to_snake_case(node_type)
    flow_id = int(input_data.get("flow_id"))
    node_id = int(input_data.get("node_id"))
    logger.info(f"Updating the data for flow: {flow_id}, node {node_id}")
    flow = flow_file_handler.get_flow(flow_id)
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    add_func = getattr(flow, "add_" + node_type)
    parsed_input = None
    setting_name_ref = "node" + node_type.replace("_", "")

    if add_func is None:
        raise HTTPException(404, "could not find the function")
    try:
        ref = get_node_model(setting_name_ref)
        if ref:
            parsed_input = ref(**input_data)
    except ValidationError as e:
        raise HTTPException(422, _format_validation_error(e)) from e
    except Exception as e:
        raise HTTPException(422, str(e)) from e
    if parsed_input is None:
        raise HTTPException(404, "could not find the interface")
    if node_type == "catalog_writer":
        _validate_catalog_writer_target(parsed_input, current_user, flow)
    elif node_type == "train_model":
        _validate_train_model_target(parsed_input, current_user, flow)
    try:
        # History capture is handled by the decorator on each add_* method
        add_func(parsed_input)
    except Exception as e:
        logger.error(e)
        raise HTTPException(419, str(f"error: {e}")) from e

    return OperationResponse(success=True, history=flow.get_history_state())

add_node(flow_id, node_id, node_type, pos_x=0, pos_y=0)

Adds a new, unconfigured node (a "promise") to the flow graph.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to add the node to.	required
node_id	int	The client-generated ID for the new node.	required
node_type	str	The type of the node to add (e.g., 'filter', 'join').	required
pos_x	int | float	The X coordinate for the node's position in the UI.	0
pos_y	int | float	The Y coordinate for the node's position in the UI.	0

Returns:

Type	Description
OperationResponse | None	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/add_node/", tags=["editor"], response_model=OperationResponse)
def add_node(
    flow_id: int, node_id: int, node_type: str, pos_x: int | float = 0, pos_y: int | float = 0
) -> OperationResponse | None:
    """Adds a new, unconfigured node (a "promise") to the flow graph.

    Args:
        flow_id: The ID of the flow to add the node to.
        node_id: The client-generated ID for the new node.
        node_type: The type of the node to add (e.g., 'filter', 'join').
        pos_x: The X coordinate for the node's position in the UI.
        pos_y: The Y coordinate for the node's position in the UI.

    Returns:
        OperationResponse with current history state.
    """
    if isinstance(pos_x, float):
        pos_x = int(pos_x)
    if isinstance(pos_y, float):
        pos_y = int(pos_y)
    flow = flow_file_handler.get_flow(flow_id)
    logger.info(f"Adding a promise for {node_type}")
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")

    node = flow.get_node(node_id)
    if node is not None:
        flow.delete_node(node_id)
    node_promise = input_schema.NodePromise(
        flow_id=flow_id, node_id=node_id, cache_results=False, pos_x=pos_x, pos_y=pos_y, node_type=node_type
    )
    if node_type == "explore_data":
        flow.add_initial_node_analysis(node_promise)
    else:
        pre_snapshot = flow.get_flowfile_data() if flow.flow_settings.track_history else None

        logger.info("Adding node")
        try:
            flow.add_node_promise(node_promise, track_history=False)
        except ValueError as e:
            raise HTTPException(422, str(e)) from e

        is_subflow_port = node_type in ("flow_input", "flow_output")
        if check_if_has_default_setting(node_type) or is_subflow_port:
            logger.info(f"Found standard settings for {node_type}, trying to upload them")
            setting_name_ref = "node" + node_type.replace("_", "")
            node_model = get_node_model(setting_name_ref)

            # Temporarily disable history tracking for initial settings
            original_track_history = flow.flow_settings.track_history
            flow.flow_settings.track_history = False
            try:
                add_func = getattr(flow, "add_" + node_type)
                initial_settings = node_model(
                    flow_id=flow_id, node_id=node_id, cache_results=False, pos_x=pos_x, pos_y=pos_y, node_type=node_type
                )
                if is_subflow_port:
                    # A second subflow port must not collide on the default name ('output'/'input').
                    name_attr = "output_name" if node_type == "flow_output" else "input_name"
                    setattr(
                        initial_settings,
                        name_attr,
                        flow._unique_subflow_port_name(getattr(initial_settings, name_attr), node_type, node_id),
                    )
                add_func(initial_settings)
            finally:
                flow.flow_settings.track_history = original_track_history

        if pre_snapshot is not None and flow.flow_settings.track_history:
            flow._history_manager.capture_if_changed(
                flow,
                pre_snapshot,
                HistoryActionType.ADD_NODE,
                f"Add {node_type} node",
                node_id,
            )
            logger.info(f"History: Captured batched 'Add {node_type} node' entry")

    logger.info(f"History state after add_node: {flow.get_history_state()}")
    return OperationResponse(success=True, history=flow.get_history_state())

add_nodes_to_group(flow_id, group_id, request)

Add nodes to an existing group.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/group/add_nodes/", tags=["editor"], response_model=GroupOperationResponse)
def add_nodes_to_group(flow_id: int, group_id: int, request: schemas.GroupMembershipRequest) -> GroupOperationResponse:
    """Add nodes to an existing group."""
    flow = _get_running_flow(flow_id)
    try:
        group = flow.add_nodes_to_group(group_id, request.node_ids)
    except ValueError as exc:
        raise HTTPException(404, str(exc)) from exc
    return GroupOperationResponse(success=True, history=flow.get_history_state(), group=_group_to_schema(group))

cancel_flow(flow_id)

Cancels a currently running flow execution.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/flow/cancel/", tags=["editor"])
def cancel_flow(flow_id: int):
    """Cancels a currently running flow execution."""
    flow = flow_file_handler.get_flow(flow_id)
    if not flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is not running")
    flow.cancel()

check_flow_laziness(flow_id)

Check whether a flow supports fully lazy execution for virtual tables.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/laziness_check", tags=["editor"])
def check_flow_laziness(flow_id: int):
    """Check whether a flow supports fully lazy execution for virtual tables."""
    flow = flow_file_handler.get_flow(int(flow_id))
    if flow is None:
        raise HTTPException(404, "Flow not found")
    is_lazy, reasons = flow.check_flow_laziness()
    return {"is_optimizable": is_lazy, "blockers": reasons}

clear_history(flow_id)

Clear all history for a flow.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to clear history for.	required

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/history_clear/", tags=["editor"])
def clear_history(flow_id: int):
    """Clear all history for a flow.

    Args:
        flow_id: The ID of the flow to clear history for.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    flow._history_manager.clear()
    return {"message": "History cleared successfully"}

close_flow(flow_id, current_user=Depends(get_current_active_user))

Closes an active flow session for the current user. Idempotent: closing a flow that isn't in the session (e.g. a stale tab after a restore pruned it) is a no-op, not a 500.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/close_flow/", tags=["editor"])
def close_flow(flow_id: int, current_user=Depends(get_current_active_user)) -> None:
    """Closes an active flow session for the current user. Idempotent: closing a flow that isn't in
    the session (e.g. a stale tab after a restore pruned it) is a no-op, not a 500."""
    user_id = current_user.id if current_user else None
    if not flow_file_handler.user_has_flow(user_id, flow_id):
        return
    flow_file_handler.delete_flow(flow_id, user_id=user_id)

compute_node_visualization(body, current_user=Depends(get_current_active_user))

Compute Graphic Walker chart rows for an Explore Data node.

GW's computation callback posts its IDataQueryPayload here on every aggregation; the worker's session cache keeps the node's lazy frame warm so successive calls skip the load.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/analysis_data/compute", tags=["analysis"], response_model=VisualizationComputeResponse)
def compute_node_visualization(
    body: gs_schemas.NodeVisualizationComputeRequest,
    current_user=Depends(get_current_active_user),
):
    """Compute Graphic Walker chart rows for an Explore Data node.

    GW's ``computation`` callback posts its IDataQueryPayload here on every
    aggregation; the worker's session cache keeps the node's lazy frame warm so
    successive calls skip the load.
    """
    flow, node = _get_analysis_node(body.flow_id, body.node_id, current_user.id if current_user else None)
    try:
        return node_viz.compute_node_rows(flow, node, body.payload, body.max_rows)
    except node_viz.NodeNotRunError as exc:
        raise HTTPException(422, str(exc)) from exc
    except node_viz.CloudPlanNotVisualizableError as exc:
        raise HTTPException(400, str(exc)) from exc
    except RuntimeError as exc:
        raise HTTPException(502, str(exc)) from exc

connect_node(flow_id, node_connection)

Creates a connection (edge) between two nodes in the flow graph.

Returns:

Type	Description
OperationResponse	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/connect_node/", tags=["editor"], response_model=OperationResponse)
def connect_node(flow_id: int, node_connection: input_schema.NodeConnection) -> OperationResponse:
    """Creates a connection (edge) between two nodes in the flow graph.

    Returns:
        OperationResponse with current history state.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        logger.info("could not find the flow")
        raise HTTPException(404, "could not find the flow")
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")

    from_id = node_connection.output_connection.node_id
    to_id = node_connection.input_connection.node_id
    flow.capture_history_snapshot(HistoryActionType.ADD_CONNECTION, f"Connect {from_id} -> {to_id}")

    add_connection(flow, node_connection)

    return OperationResponse(success=True, history=flow.get_history_state())

copy_node(node_id_to_copy_from, flow_id_to_copy_from, node_promise)

Copies an existing node's settings to a new node promise.

Parameters:

Name	Type	Description	Default
node_id_to_copy_from	int	The ID of the node to copy the settings from.	required
flow_id_to_copy_from	int	The ID of the flow containing the source node.	required
node_promise	NodePromise	A NodePromise representing the new node to be created.	required

Returns:

Type	Description
OperationResponse	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/copy_node", tags=["editor"], response_model=OperationResponse)
def copy_node(
    node_id_to_copy_from: int, flow_id_to_copy_from: int, node_promise: input_schema.NodePromise
) -> OperationResponse:
    """Copies an existing node's settings to a new node promise.

    Args:
        node_id_to_copy_from: The ID of the node to copy the settings from.
        flow_id_to_copy_from: The ID of the flow containing the source node.
        node_promise: A `NodePromise` representing the new node to be created.

    Returns:
        OperationResponse with current history state.
    """
    try:
        flow_to_copy_from = flow_file_handler.get_flow(flow_id_to_copy_from)
        flow = (
            flow_to_copy_from
            if flow_id_to_copy_from == node_promise.flow_id
            else flow_file_handler.get_flow(node_promise.flow_id)
        )
        node_to_copy = flow_to_copy_from.get_node(node_id_to_copy_from)
        logger.info(f"Copying data {node_promise.node_type}")

        if flow.flow_settings.is_running:
            raise HTTPException(422, "Flow is running")

        flow.capture_history_snapshot(
            HistoryActionType.COPY_NODE, f"Copy {node_promise.node_type} node", node_id=node_promise.node_id
        )

        if flow.get_node(node_promise.node_id) is not None:
            flow.delete_node(node_promise.node_id)

        if node_promise.node_type == "explore_data":
            flow.add_initial_node_analysis(node_promise)
            return OperationResponse(success=True, history=flow.get_history_state())

        flow.copy_node(node_promise, node_to_copy.setting_input, node_to_copy.node_type)

        return OperationResponse(success=True, history=flow.get_history_state())

    except Exception as e:
        logger.error(e)
        raise HTTPException(422, str(e)) from e

create_db_connection(input_connection, current_user=Depends(get_current_active_user), db=Depends(get_db))

Creates and securely stores a new database connection.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/db_connection_lib", tags=["db_connections"])
def create_db_connection(
    input_connection: input_schema.FullDatabaseConnection,
    current_user=Depends(get_current_active_user),
    db: Session = Depends(get_db),
):
    """Creates and securely stores a new database connection."""
    logger.info(f"Creating database connection {input_connection.connection_name}")
    _require_known_database_type(input_connection.database_type)
    try:
        store_database_connection(db, input_connection, current_user.id)
    except ValueError:
        raise HTTPException(422, "Connection name already exists") from None
    except Exception as e:
        logger.error(e)
        raise HTTPException(422, str(e)) from e
    return {"message": "Database connection created successfully"}

create_directory(new_directory)

Creates a new directory at the specified path.

Parameters:

Name	Type	Description	Default
new_directory	NewDirectory	An input_schema.NewDirectory object with the path and name.	required

Returns:

Type	Description
bool	True if the directory was created successfully.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/files/create_directory", response_model=output_model.OutputDir, tags=["file manager"])
def create_directory(new_directory: input_schema.NewDirectory) -> bool:
    """Creates a new directory at the specified path.

    Args:
        new_directory: An `input_schema.NewDirectory` object with the path and name.

    Returns:
        `True` if the directory was created successfully.
    """
    result, error = create_dir(new_directory)
    if result:
        return True
    else:
        raise error

create_flow(flow_path=None, name=None, namespace_id=None, register_in_catalog=True, persist=True, current_user=Depends(get_current_active_user))

Creates a new, empty flow file at the specified path and registers a session for it.

Two independent switches, deliberately not fused:

• persist writes the YAML to disk. False keeps the flow in-memory until an explicit save or first run, so an abandoned blank canvas leaves no orphan file.
• register_in_catalog creates the FlowRegistration row. False yields a flow with no source_registration_id: no run history, schedules or API publishing until it is filed. When namespace_id is provided the flow lands there; otherwise it auto-registers under General > {Unnamed | Local} Flows.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/create_flow/", tags=["editor"])
def create_flow(
    flow_path: str = None,
    name: str = None,
    namespace_id: int = None,
    register_in_catalog: bool = True,
    persist: bool = True,
    current_user=Depends(get_current_active_user),
):
    """Creates a new, empty flow file at the specified path and registers a session for it.

    Two independent switches, deliberately not fused:

    - ``persist`` writes the YAML to disk. ``False`` keeps the flow in-memory until an
      explicit save or first run, so an abandoned blank canvas leaves no orphan file.
    - ``register_in_catalog`` creates the ``FlowRegistration`` row. ``False`` yields a
      flow with no ``source_registration_id``: no run history, schedules or API
      publishing until it is filed. When ``namespace_id`` is provided the flow lands
      there; otherwise it auto-registers under ``General > {Unnamed | Local} Flows``.
    """
    if flow_path is not None and name is None:
        name = Path(flow_path).stem
    elif flow_path is not None and name is not None:
        if name not in flow_path and (flow_path.endswith(".yaml") or flow_path.endswith(".yml")):
            raise HTTPException(422, "The name must be part of the flow path when a full path is provided")
        elif name in flow_path and not (flow_path.endswith(".yaml") or flow_path.endswith(".yml")):
            flow_path = str(Path(flow_path) / (name + ".yaml"))
        elif name not in flow_path and (name.endswith(".yaml") or name.endswith(".yml")):
            flow_path = str(Path(flow_path) / name)
        elif name not in flow_path and not (name.endswith(".yaml") or name.endswith(".yml")):
            flow_path = str(Path(flow_path) / (name + ".yaml"))
    if flow_path is not None:
        # Validate path is within allowed sandbox
        flow_path = validate_path_under_cwd(flow_path)
        flow_path_ref = Path(flow_path)
        if not flow_path_ref.parent.exists():
            raise HTTPException(422, "The directory does not exist")
    user_id = current_user.id if current_user else None
    if namespace_id is not None and not register_in_catalog:
        raise HTTPException(422, "namespace_id requires register_in_catalog=True")
    if namespace_id is not None and not namespace_exists(namespace_id):
        raise HTTPException(404, "Namespace not found")
    _require_flow_save_permitted(flow_path, namespace_id, current_user)
    if namespace_id is not None and flow_path is not None:
        # Pre-validate before add_flow writes any YAML so a rejected create
        # leaves no orphaned file or session behind.
        reg_name = name or Path(flow_path).stem
        existing_by_name = find_registration_by_name(reg_name, namespace_id)
        if existing_by_name is not None and existing_by_name.flow_path != flow_path:
            raise HTTPException(
                status_code=409,
                detail=(
                    f"A flow named '{reg_name}' already exists in this namespace. "
                    "Choose a different name or namespace."
                ),
            )
        existing_by_path = find_registration_by_path(flow_path)
        if existing_by_path is not None and existing_by_path.namespace_id != namespace_id:
            raise HTTPException(
                status_code=409,
                detail=f"Flow path {flow_path} is already registered in another namespace",
            )
    flow_id = flow_file_handler.add_flow(name=name, flow_path=flow_path, user_id=user_id, persist=persist)
    flow = flow_file_handler.get_flow(flow_id)
    if register_in_catalog and flow and flow.flow_settings:
        try:
            register_flow_in_namespace(
                flow.flow_settings.path,
                name or flow.flow_settings.name,
                user_id,
                namespace_id,
                requesting_user=current_user,
            )
        except (FlowPathNamespaceCollision, FlowNameNamespaceCollision) as err:
            raise HTTPException(status_code=409, detail=str(err)) from err
        except NotAuthorizedError as err:
            raise HTTPException(status_code=403, detail=str(err)) from None
        except NamespaceNotFoundError:
            raise HTTPException(status_code=404, detail="Namespace not found") from None
        resolve_source_registration_id(flow)
    return flow_id

create_from_template(template_id, current_user=Depends(get_current_active_user))

Instantiates a template as a new flow session.

Downloads required CSV data files from GitHub if not already cached locally, then creates a flow from the template definition.

The new flow is not registered in the catalog: the only path available to register is the temp file this route unlinks on the way out, so doing so minted a permanently dangling row. Saving the flow files it.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/templates/{template_id}/create", tags=["templates"])
def create_from_template(template_id: str, current_user=Depends(get_current_active_user)) -> int:
    """Instantiates a template as a new flow session.

    Downloads required CSV data files from GitHub if not already cached locally,
    then creates a flow from the template definition.

    The new flow is not registered in the catalog: the only path available to register is
    the temp file this route unlinks on the way out, so doing so minted a permanently
    dangling row. Saving the flow files it.
    """
    import logging as _logging

    import yaml

    from flowfile_core.templates import get_template_flowfile_data, get_template_required_files
    from flowfile_core.templates.data_downloader import ensure_template_data

    _tpl_logger = _logging.getLogger("flowfile_core.templates.create")
    _tpl_logger.info(
        "create_from_template START: template_id=%s user_id=%s", template_id, getattr(current_user, "id", None)
    )

    try:
        required_files = get_template_required_files(template_id)
        _tpl_logger.info("required_files for %s: %s", template_id, required_files)
    except ValueError as e:
        _tpl_logger.warning("template not found: %s (%s)", template_id, e)
        raise HTTPException(status_code=404, detail=str(e)) from e
    except Exception as e:
        _tpl_logger.exception("unexpected error resolving required_files for %s", template_id)
        raise HTTPException(
            status_code=500,
            detail=f"get_template_required_files({template_id}) failed: {type(e).__name__}: {e}",
        ) from e

    try:
        resolved_files = ensure_template_data(required_files)
        _tpl_logger.info("resolved_files: %s", {k: str(v) for k, v in resolved_files.items()})
    except RuntimeError as e:
        _tpl_logger.exception("ensure_template_data raised RuntimeError for %s", template_id)
        raise HTTPException(status_code=502, detail=str(e)) from e
    except Exception as e:
        _tpl_logger.exception("ensure_template_data raised unexpected for %s", template_id)
        raise HTTPException(
            status_code=502,
            detail=f"ensure_template_data failed: {type(e).__name__}: {e}",
        ) from e

    try:
        data_dir = next(iter(resolved_files.values())).parent
        flowfile_data = get_template_flowfile_data(template_id, data_dir)
    except Exception as e:
        _tpl_logger.exception("get_template_flowfile_data failed for %s", template_id)
        raise HTTPException(
            status_code=502,
            detail=f"get_template_flowfile_data({template_id}) failed: {type(e).__name__}: {e}",
        ) from e

    import uuid

    from shared.storage_config import storage

    flows_dir = storage.flows_directory
    user_id = current_user.id if current_user else None

    flow_stem = flowfile_data.flowfile_name.replace(" ", "_").lower()
    temp_path = flows_dir / f"{flow_stem}_{uuid.uuid4().hex[:8]}.yaml"
    try:
        try:
            with open(temp_path, "w", encoding="utf-8") as f:
                yaml.dump(flowfile_data.model_dump(), f, default_flow_style=False, allow_unicode=True)
            flow_id = flow_file_handler.import_flow(temp_path, user_id=user_id)
        except Exception as e:
            _tpl_logger.exception("import_flow failed for template %s (temp=%s)", template_id, temp_path)
            raise HTTPException(
                status_code=502,
                detail=f"import_flow failed: {type(e).__name__}: {e}",
            ) from e
    finally:
        temp_path.unlink(missing_ok=True)

    _tpl_logger.info("create_from_template OK: template_id=%s flow_id=%s", template_id, flow_id)
    return flow_id

create_group(flow_id, request)

Create a visual group around a set of nodes. Returns the new server-assigned group.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/create_group/", tags=["editor"], response_model=GroupOperationResponse)
def create_group(flow_id: int, request: schemas.CreateGroupRequest) -> GroupOperationResponse:
    """Create a visual group around a set of nodes. Returns the new server-assigned group."""
    flow = _get_running_flow(flow_id)
    group = flow.create_group(
        request.name,
        request.node_ids,
        color=request.color,
        bounds=_bounds_from_request(request),
        parent_group_id=request.parent_group_id,
        child_group_ids=request.child_group_ids,
    )
    return GroupOperationResponse(success=True, history=flow.get_history_state(), group=_group_to_schema(group))

delete_db_connection(connection_name, current_user=Depends(get_current_active_user), db=Depends(get_db))

Deletes a stored database connection (own, or group-shared with manage access).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.delete("/db_connection_lib", tags=["db_connections"])
def delete_db_connection(
    connection_name: str, current_user=Depends(get_current_active_user), db: Session = Depends(get_db)
):
    """Deletes a stored database connection (own, or group-shared with manage access)."""
    logger.info(f"Deleting database connection {connection_name}")
    db_connection = get_database_connection(db, connection_name, current_user.id)
    if db_connection is None:
        raise HTTPException(404, "Database connection not found")
    authorize_connection_mutation(db, current_user, "database_connection", db_connection)
    delete_database_connection(db, connection_name, db_connection.user_id)
    return {"message": "Database connection deleted successfully"}

delete_group(flow_id, group_id)

Delete a group box (ungroup). Member nodes are kept.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/delete_group/", tags=["editor"], response_model=OperationResponse)
def delete_group(flow_id: int, group_id: int) -> OperationResponse:
    """Delete a group box (ungroup). Member nodes are kept."""
    flow = _get_running_flow(flow_id)
    flow.delete_group(group_id)
    return OperationResponse(success=True, history=flow.get_history_state())

delete_node(flow_id, node_id)

Deletes a node from the flow graph.

Returns:

Type	Description
OperationResponse	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/delete_node/", tags=["editor"], response_model=OperationResponse)
def delete_node(flow_id: int | None, node_id: int) -> OperationResponse:
    """Deletes a node from the flow graph.

    Returns:
        OperationResponse with current history state.
    """
    logger.info("Deleting node")
    flow = flow_file_handler.get_flow(flow_id)
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")

    node = flow.get_node(node_id)
    node_type = node.node_type if node else "unknown"
    flow.capture_history_snapshot(HistoryActionType.DELETE_NODE, f"Delete {node_type} node", node_id=node_id)

    flow.delete_node(node_id)

    return OperationResponse(success=True, history=flow.get_history_state())

delete_node_connection(flow_id, node_connection=None)

Deletes a connection (edge) between two nodes.

Returns:

Type	Description
OperationResponse	OperationResponse with current history state.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/delete_connection/", tags=["editor"], response_model=OperationResponse)
def delete_node_connection(flow_id: int, node_connection: input_schema.NodeConnection = None) -> OperationResponse:
    """Deletes a connection (edge) between two nodes.

    Returns:
        OperationResponse with current history state.
    """
    flow_id = int(flow_id)
    logger.info(
        f"Deleting connection node {node_connection.output_connection.node_id} "
        f"to node {node_connection.input_connection.node_id}"
    )
    flow = flow_file_handler.get_flow(flow_id)
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")

    from_id = node_connection.output_connection.node_id
    to_id = node_connection.input_connection.node_id
    flow.capture_history_snapshot(HistoryActionType.DELETE_CONNECTION, f"Delete connection {from_id} -> {to_id}")

    delete_connection(flow, node_connection)

    return OperationResponse(success=True, history=flow.get_history_state())

download_generated_project(flow_id)

Generates the project export and returns it as a zip archive.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/code_to_project/zip", tags=[])
def download_generated_project(flow_id: int) -> Response:
    """Generates the project export and returns it as a zip archive."""
    manifest = _export_project_manifest(flow_id)
    return Response(
        content=project_to_zip_bytes(manifest),
        media_type="application/zip",
        headers={"Content-Disposition": f'attachment; filename="{manifest.project_name}.zip"'},
    )

ensure_templates_available()

Downloads template flow YAMLs from GitHub if not already cached locally.

Called by the frontend on first visit to the templates page to ensure templates are available even when running from a PyPI install (no repo checkout).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/templates/ensure_available/", tags=["templates"])
def ensure_templates_available():
    """Downloads template flow YAMLs from GitHub if not already cached locally.

    Called by the frontend on first visit to the templates page to ensure
    templates are available even when running from a PyPI install (no repo checkout).
    """
    from flowfile_core.templates import get_flow_yaml_filenames
    from flowfile_core.templates.data_downloader import ensure_flow_yamls

    try:
        ensure_flow_yamls(get_flow_yaml_filenames())
        return {"status": "ok"}
    except RuntimeError as e:
        raise HTTPException(status_code=502, detail=str(e)) from e

fetch_rest_api_sample(input_data, sample_size=50, current_user=Depends(get_current_active_user))

Fetch a small sample from the configured REST API and infer its schema.

Runs one capped request through the worker (so all network I/O stays sandboxed off the core event loop), infers the output columns with Polars, caches them on the node's fields (so downstream schema prediction needs no network), and returns the inferred columns. This powers the node's "Fetch sample" button. Defined as a sync endpoint so the blocking worker round-trip runs in FastAPI's threadpool rather than the event loop.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/rest_api/sample", tags=["editor"], response_model=RestApiSampleResponse)
def fetch_rest_api_sample(
    input_data: dict[str, Any], sample_size: int = 50, current_user=Depends(get_current_active_user)
) -> RestApiSampleResponse:
    """Fetch a small sample from the configured REST API and infer its schema.

    Runs one capped request through the worker (so all network I/O stays
    sandboxed off the core event loop), infers the output columns with Polars,
    caches them on the node's ``fields`` (so downstream schema prediction needs
    no network), and returns the inferred columns. This powers the node's
    "Fetch sample" button. Defined as a sync endpoint so the blocking worker
    round-trip runs in FastAPI's threadpool rather than the event loop.
    """
    input_data["user_id"] = current_user.id
    try:
        node = input_schema.NodeRestApiReader(**input_data)
    except ValidationError as e:
        raise HTTPException(422, _format_validation_error(e)) from e
    except Exception as e:
        raise HTTPException(422, str(e)) from e

    flow = flow_file_handler.get_flow(node.flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")

    # Resolve the credential to an encrypted token (stored secret by name, or an
    # inline plaintext); the worker decrypts it. Never persist inline plaintext.
    auth = node.rest_api_settings.auth
    secret_encrypted = resolve_auth_secret_encrypted(auth, node.user_id)
    auth.secret = None

    sample_size = max(1, sample_size)
    worker_settings = build_rest_api_worker_settings(node, secret_encrypted, sample_size=sample_size)
    try:
        fetcher = ExternalRestApiFetcher(worker_settings, wait_on_completion=True)
        sample_df = fetcher.get_result().head(sample_size).collect()
    except Exception as e:
        raise HTTPException(422, f"Failed to fetch sample from REST API: {e}") from e

    fields = [c.get_minimal_field_info() for c in infer_schema_from_sample(sample_df)]
    # Cache onto the live node so schema prediction is immediately available.
    live = flow.get_node(node.node_id)
    if live is not None and isinstance(live.setting_input, input_schema.NodeRestApiReader):
        live.setting_input.fields = fields
    return RestApiSampleResponse(fields=fields)

get_active_flow_file_sessions(current_user=Depends(get_current_active_user)) async

Retrieves a list of all currently active flow sessions for the current user.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/active_flowfile_sessions/", response_model=list[schemas.FlowSettingsResponse])
async def get_active_flow_file_sessions(
    current_user=Depends(get_current_active_user),
) -> list[schemas.FlowSettingsResponse]:
    """Retrieves a list of all currently active flow sessions for the current user."""
    user_id = current_user.id if current_user else None
    sessions = [
        flow_file_handler.get_flow_info_with_runtime(flf.flow_id) for flf in flow_file_handler.get_user_flows(user_id)
    ]
    return _with_display_names(sessions)

get_catalog_flows_directory() async

Returns the managed flows directory used for catalog-tab saves.

On local this resolves to ~/.flowfile/flows; in Docker mode to /data/user/flows. The frontend uses this to build the target path for flows saved via the Catalog tab, so they always land in the managed location regardless of where the file browser was last navigated.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/files/catalog_flows_directory/", response_model=str, tags=["file manager"])
async def get_catalog_flows_directory() -> str:
    """Returns the managed flows directory used for catalog-tab saves.

    On local this resolves to ``~/.flowfile/flows``; in Docker mode to
    ``/data/user/flows``.  The frontend uses this to build the target path
    for flows saved via the Catalog tab, so they always land in the managed
    location regardless of where the file browser was last navigated.
    """
    return str(storage.flows_directory)

get_db_connections(db=Depends(get_db), current_user=Depends(get_current_active_user))

Retrieves all stored database connections for the current user (without passwords).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get(
    "/db_connection_lib", tags=["db_connections"], response_model=list[input_schema.FullDatabaseConnectionInterface]
)
def get_db_connections(
    db: Session = Depends(get_db), current_user=Depends(get_current_active_user)
) -> list[input_schema.FullDatabaseConnectionInterface]:
    """Retrieves all stored database connections for the current user (without passwords)."""
    return get_all_database_connections_interface(db, current_user.id)

get_db_dialects()

Returns the supported database dialects (drives the frontend's dialect dropdowns).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/db_dialects", tags=["db_connections"], response_model=list[DialectInfo])
def get_db_dialects() -> list[DialectInfo]:
    """Returns the supported database dialects (drives the frontend's dialect dropdowns)."""
    return dialect_catalog()

get_db_schemas(database_settings, current_user=Depends(get_current_active_user)) async

Returns available schema names for the given database connection.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/db_schemas", tags=["db_connections"], response_model=list[str])
async def get_db_schemas(
    database_settings: input_schema.DatabaseSettings, current_user=Depends(get_current_active_user)
) -> list[str]:
    """Returns available schema names for the given database connection."""
    try:
        return list_db_schemas(database_settings, user_id=current_user.id)
    except Exception as e:
        raise HTTPException(status_code=422, detail=str(e)) from e

get_db_tables(database_settings, current_user=Depends(get_current_active_user)) async

Returns available table names for the given database connection and optional schema.

When schema_name is provided, returns plain table names. When schema_name is not provided, returns schema-qualified names (schema.table) across all accessible schemas (skipping any that the user cannot read).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/db_tables", tags=["db_connections"], response_model=list[str])
async def get_db_tables(
    database_settings: input_schema.DatabaseSettings, current_user=Depends(get_current_active_user)
) -> list[str]:
    """Returns available table names for the given database connection and optional schema.

    When schema_name is provided, returns plain table names.
    When schema_name is not provided, returns schema-qualified names (schema.table) across all
    accessible schemas (skipping any that the user cannot read).
    """
    try:
        return list_db_tables(database_settings, user_id=current_user.id)
    except Exception as e:
        raise HTTPException(status_code=422, detail=str(e)) from e

get_default_path() async

Returns the default starting path for the file browser (user data directory).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/files/default_path/", response_model=str, tags=["file manager"])
async def get_default_path() -> str:
    """Returns the default starting path for the file browser (user data directory)."""
    return str(storage.user_data_directory)

get_description_node(flow_id, node_id)

Retrieves the description text for a specific node.

Returns the user-provided description if set, otherwise falls back to an auto-generated description based on the node's configuration. The response includes an is_auto_generated flag so the frontend knows whether to refresh the description after settings changes.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/description", response_model=output_model.NodeDescriptionResponse, tags=["editor"])
def get_description_node(flow_id: int, node_id: int):
    """Retrieves the description text for a specific node.

    Returns the user-provided description if set, otherwise falls back
    to an auto-generated description based on the node's configuration.
    The response includes an `is_auto_generated` flag so the frontend
    knows whether to refresh the description after settings changes.
    """
    try:
        node = flow_file_handler.get_flow(flow_id).get_node(node_id)
    except Exception:
        raise HTTPException(404, "Could not find the node") from None
    if node is None:
        raise HTTPException(404, "Could not find the node")
    user_description = node.setting_input.description if hasattr(node.setting_input, "description") else ""
    if user_description:
        return output_model.NodeDescriptionResponse(description=user_description, is_auto_generated=False)
    if hasattr(node.setting_input, "get_default_description"):
        auto_desc = node.setting_input.get_default_description()
        return output_model.NodeDescriptionResponse(description=auto_desc, is_auto_generated=True)
    return output_model.NodeDescriptionResponse(description="", is_auto_generated=True)

get_directory_contents(directory, file_types=None, include_hidden=False) async

Gets the contents of a directory path.

Parameters:

Name	Type	Description	Default
directory	str	The absolute path to the directory.	required
file_types	list[str]	An optional list of file extensions to filter by.	None
include_hidden	bool	If True, includes hidden files and directories.	False

Returns:

Type	Description
list[FileInfo]	A list of FileInfo objects representing the directory's contents.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/files/directory_contents/", response_model=list[FileInfo], tags=["file manager"])
async def get_directory_contents(
    directory: str, file_types: list[str] = None, include_hidden: bool = False
) -> list[FileInfo]:
    """Gets the contents of a directory path.

    Args:
        directory: The absolute path to the directory.
        file_types: An optional list of file extensions to filter by.
        include_hidden: If True, includes hidden files and directories.

    Returns:
        A list of `FileInfo` objects representing the directory's contents.
    """

    # In Electron mode, allow browsing the entire filesystem (no sandbox).
    # In other modes, sandbox to the user data directory.
    sandbox_root = None if is_electron_mode() else storage.user_data_directory
    try:
        directory_explorer = SecureFileExplorer(directory, sandbox_root)
        return directory_explorer.list_contents(show_hidden=include_hidden, file_types=file_types)
    except PermissionError:
        raise HTTPException(403, "Access denied: path is outside the allowed directory") from None
    except Exception as e:
        logger.error(e)
        raise HTTPException(404, "Could not access the directory") from e

get_downstream_node_ids(flow_id, node_id) async

Gets a list of all node IDs that are downstream dependencies of a given node.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/downstream_node_ids", response_model=list[int], tags=["editor"])
async def get_downstream_node_ids(flow_id: int, node_id: int) -> list[int]:
    """Gets a list of all node IDs that are downstream dependencies of a given node."""
    flow = flow_file_handler.get_flow(flow_id)
    node = flow.get_node(node_id)
    return list(node.get_all_dependent_node_ids())

get_excel_sheet_names(path) async

Retrieves the sheet names from an Excel file.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/api/get_xlsx_sheet_names", tags=["excel_reader"], response_model=list[str])
async def get_excel_sheet_names(path: str) -> list[str] | None:
    """Retrieves the sheet names from an Excel file."""
    validated_path = validate_path_under_cwd(path)
    sheet_names = excel_file_manager.get_sheet_names(validated_path)
    if sheet_names:
        return sheet_names
    else:
        raise HTTPException(404, "File not found")

get_expression_doc()

Retrieves documentation for available Polars expressions.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/expression_doc", tags=["editor"], response_model=list[output_model.ExpressionsOverview])
def get_expression_doc() -> list[output_model.ExpressionsOverview]:
    """Retrieves documentation for available Polars expressions."""
    return get_expression_overview()

get_expressions()

Retrieves a list of all available Flowfile expression names.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/expressions", tags=["editor"], response_model=list[str])
def get_expressions() -> list[str]:
    """Retrieves a list of all available Flowfile expression names."""
    return get_all_expressions()

get_flow(flow_id)

Retrieves the settings for a specific flow (including runtime dirty state).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/flow", tags=["editor"], response_model=schemas.FlowSettingsResponse)
def get_flow(flow_id: int):
    """Retrieves the settings for a specific flow (including runtime dirty state)."""
    flow_id = int(flow_id)
    result = get_flow_settings(flow_id)
    return result

get_flow_artifacts(flow_id)

Returns artifact visualization data for the canvas.

Includes per-node artifact summaries (for badges/tooltips) and artifact edges (for dashed-line connections between publisher and consumer nodes).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow/artifacts", tags=["editor"])
def get_flow_artifacts(flow_id: int):
    """Returns artifact visualization data for the canvas.

    Includes per-node artifact summaries (for badges/tooltips) and
    artifact edges (for dashed-line connections between publisher and
    consumer nodes).
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    ctx = flow.artifact_context
    return {
        "nodes": ctx.get_node_summaries(),
        "edges": ctx.get_artifact_edges(),
    }

get_flow_frontend_data(flow_id=1)

Retrieves the data needed to render the flow graph in the frontend.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow_data", tags=["manager"])
def get_flow_frontend_data(flow_id: int | None = 1):
    """Retrieves the data needed to render the flow graph in the frontend."""
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    return flow.get_frontend_data()

get_flow_settings(flow_id=1)

Retrieves the main settings for a flow (including dirty-state info).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow_settings", tags=["manager"], response_model=schemas.FlowSettingsResponse)
def get_flow_settings(flow_id: int | None = 1) -> schemas.FlowSettingsResponse:
    """Retrieves the main settings for a flow (including dirty-state info)."""
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    return _with_display_name(flow_file_handler.get_flow_info_with_runtime(flow_id))

get_flow_settings_validation(flow_id)

Conservative static check: node settings that reference missing input columns.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow/settings_validation", tags=["editor"], response_model=FlowSettingsValidation)
def get_flow_settings_validation(flow_id: int) -> FlowSettingsValidation:
    """Conservative static check: node settings that reference missing input columns."""
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    return validate_flow_settings(flow)

get_generated_code(flow_id)

Generates and returns a Python script with Polars code representing the flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/code_to_polars", tags=[], response_model=str)
def get_generated_code(flow_id: int) -> str:
    """Generates and returns a Python script with Polars code representing the flow."""
    flow_id = int(flow_id)
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    try:
        return export_flow_to_polars(flow)
    except UnsupportedNodeError as e:
        raise HTTPException(422, str(e)) from e

get_generated_flowframe_code(flow_id)

Generates and returns a Python script with FlowFrame code representing the flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/code_to_flowframe", tags=[], response_model=str)
def get_generated_flowframe_code(flow_id: int) -> str:
    """Generates and returns a Python script with FlowFrame code representing the flow."""
    flow_id = int(flow_id)
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    try:
        return export_flow_to_flowframe(flow)
    except UnsupportedNodeError as e:
        raise HTTPException(422, str(e)) from e

get_generated_project(flow_id)

Generates a multi-file Python project (FlowFrame code) representing the flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/code_to_project", tags=[], response_model=output_model.ProjectExportManifest)
def get_generated_project(flow_id: int) -> output_model.ProjectExportManifest:
    """Generates a multi-file Python project (FlowFrame code) representing the flow."""
    return _export_project_manifest(flow_id)

get_graphic_walker_input(flow_id, node_id, current_user=Depends(get_current_active_user))

Gets the saved chart specs and field schema for the Graphic Walker explorer.

Carries no rows: aggregation runs on the worker via /analysis_data/compute.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/analysis_data/graphic_walker_input", tags=["analysis"], response_model=input_schema.NodeExploreData)
def get_graphic_walker_input(flow_id: int, node_id: int, current_user=Depends(get_current_active_user)):
    """Gets the saved chart specs and field schema for the Graphic Walker explorer.

    Carries no rows: aggregation runs on the worker via ``/analysis_data/compute``.
    """
    _flow, node = _get_analysis_node(flow_id, node_id, current_user.id if current_user else None)
    if not node_viz.has_result_to_visualize(node):
        logger.error("The data is not refreshed and available for analysis")
        raise HTTPException(422, "The data is not refreshed and available for analysis")
    return AnalyticsProcessor.process_graphic_walker_input(node)

get_history_status(flow_id)

Get the current state of the history system for a flow.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to get history status for.	required

Returns:

Type	Description
HistoryState	HistoryState with information about available undo/redo operations.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/editor/history_status/", tags=["editor"], response_model=HistoryState)
def get_history_status(flow_id: int) -> HistoryState:
    """Get the current state of the history system for a flow.

    Args:
        flow_id: The ID of the flow to get history status for.

    Returns:
        HistoryState with information about available undo/redo operations.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    return flow.get_history_state()

get_instant_function_result(flow_id, node_id, func_string) async

Executes a simple, instant function on a node's data and returns the result.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/custom_functions/instant_result", tags=[])
async def get_instant_function_result(flow_id: int, node_id: int, func_string: str):
    """Executes a simple, instant function on a node's data and returns the result."""
    try:
        node = flow_file_handler.get_node(flow_id, node_id)
        result = await asyncio.to_thread(get_instant_func_results, node, func_string)
        return result
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e)) from e

get_list_of_saved_flows(path)

Scans a directory for saved flow files (.flowfile).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/files/available_flow_files", tags=["editor"], response_model=list[FileInfo])
def get_list_of_saved_flows(path: str):
    """Scans a directory for saved flow files (`.flowfile`)."""
    try:
        # Validate path is within sandbox before proceeding
        explorer = SecureFileExplorer(start_path=storage.user_data_directory, sandbox_root=storage.user_data_directory)
        validated_path = explorer.get_absolute_path(path)
        if validated_path is None:
            return []
        return get_files_from_directory(
            str(validated_path), types=["flowfile"], sandbox_root=storage.user_data_directory
        )
    except Exception:
        return []

get_local_files(directory) async

Retrieves a list of files from a specified local directory.

Parameters:

Name	Type	Description	Default
directory	str	The absolute path of the directory to scan.	required

Returns:

Type	Description
list[FileInfo]	A list of FileInfo objects for each item in the directory.

Raises:

Type	Description
HTTPException	404 if the directory does not exist.
HTTPException	403 if access is denied (path outside sandbox).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/files/files_in_local_directory/", response_model=list[FileInfo], tags=["file manager"])
async def get_local_files(directory: str) -> list[FileInfo]:
    """Retrieves a list of files from a specified local directory.

    Args:
        directory: The absolute path of the directory to scan.

    Returns:
        A list of `FileInfo` objects for each item in the directory.

    Raises:
        HTTPException: 404 if the directory does not exist.
        HTTPException: 403 if access is denied (path outside sandbox).
    """
    explorer = SecureFileExplorer(start_path=storage.user_data_directory, sandbox_root=storage.user_data_directory)
    validated_path = explorer.get_absolute_path(directory)
    if validated_path is None:
        raise HTTPException(403, "Access denied or directory does not exist")
    if not validated_path.exists() or not validated_path.is_dir():
        raise HTTPException(404, "Directory does not exist")
    files = get_files_from_directory(str(validated_path), sandbox_root=storage.user_data_directory)
    if files is None:
        raise HTTPException(403, "Access denied or directory does not exist")
    return files

get_node(flow_id, node_id, get_data=False, include_output=True, include_inputs=True)

Retrieves the complete state and data preview for a single node.

When include_output is False the node's own output preview (main_output) is skipped. The settings panel only needs the input schemas, and computing the output can be expensive for data-dependent nodes (e.g. a pivot must materialize data to determine its output columns), so the editor opens settings with include_output=false for an instant response.

When include_inputs is also False the input schemas are skipped too: resolving them can execute un-run upstream custom nodes (kernel/worker). The custom-node drawer uses this to render settings instantly and hydrates columns with a follow-up full fetch.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node", response_model=output_model.NodeData, tags=["editor"])
def get_node(
    flow_id: int, node_id: int, get_data: bool = False, include_output: bool = True, include_inputs: bool = True
):
    """Retrieves the complete state and data preview for a single node.

    When ``include_output`` is False the node's own output preview
    (``main_output``) is skipped. The settings panel only needs the input
    schemas, and computing the output can be expensive for data-dependent nodes
    (e.g. a pivot must materialize data to determine its output columns), so the
    editor opens settings with ``include_output=false`` for an instant response.

    When ``include_inputs`` is also False the input schemas are skipped too:
    resolving them can execute un-run upstream custom nodes (kernel/worker).
    The custom-node drawer uses this to render settings instantly and hydrates
    columns with a follow-up full fetch.
    """
    logging.info(f"Getting node {node_id} from flow {flow_id}")
    flow = flow_file_handler.get_flow(flow_id)
    node = flow.get_node(node_id)
    if node is None:
        raise HTTPException(422, "Not found")
    v = node.get_node_data(
        flow_id=flow.flow_id,
        include_example=get_data,
        include_output=include_output,
        include_inputs=include_inputs,
    )
    return v

get_node_available_artifacts(flow_id, node_id, kernel_id=None)

Return available artifact metadata for a node.

Merges run-observed artifacts (published in a prior run) with artifacts upstream kernel nodes declare they publish (from their manifests), so the frontend's artifact pickers work before the flow has ever run. Observed wins on name conflict.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow/node_available_artifacts", tags=["editor"])
def get_node_available_artifacts(flow_id: int, node_id: int, kernel_id: str | None = None):
    """Return available artifact metadata for a node.

    Merges run-observed artifacts (published in a prior run) with artifacts
    upstream kernel nodes *declare* they publish (from their manifests), so the
    frontend's artifact pickers work before the flow has ever run. Observed
    wins on name conflict.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    node = flow.get_node(node_id)
    if node is None:
        raise HTTPException(404, "Could not find the node")
    resolved_kernel_id = kernel_id or _resolve_node_kernel_id(node)
    if not resolved_kernel_id:
        return {"artifacts": []}

    upstream_ids = flow._get_upstream_node_ids(node_id)
    observed = flow.artifact_context.compute_available(
        node_id=node_id,
        kernel_id=resolved_kernel_id,
        upstream_node_ids=upstream_ids,
    )

    declared: list[tuple[int, str, str | None]] = []
    for uid in upstream_ids:
        up = flow.get_node(uid)
        if up is None:
            continue
        entry = user_defined_registry.get(up.node_type)
        if entry is None or entry.manifest is None:
            continue
        manifest = entry.manifest
        if manifest.environment.kind != "kernel" or not manifest.publishes:
            continue
        if _resolve_node_kernel_id(up) != resolved_kernel_id:
            continue
        for decl in manifest.publishes:
            declared.append((uid, decl.name, decl.type))

    return {"artifacts": merge_declared_artifacts(observed, declared, resolved_kernel_id)}

get_node_column_stats(flow_id, node_id, column_name, output_handle=DEFAULT_OUTPUT_HANDLE)

Computes on-demand statistics for one column of a node's cached result.

Runs a single bounded aggregate (counts, uniques, min/max) over the result the last run left behind — it never re-executes the node — and returns the column's updated FileColumn, the same shape table_schema ships. Query parameters (not path segments): column names contain / and ..

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/column_stats", response_model=output_model.FileColumn, tags=["editor"])
def get_node_column_stats(flow_id: int, node_id: int, column_name: str, output_handle: str = DEFAULT_OUTPUT_HANDLE):
    """Computes on-demand statistics for one column of a node's cached result.

    Runs a single bounded aggregate (counts, uniques, min/max) over the result
    the last run left behind — it never re-executes the node — and returns the
    column's updated ``FileColumn``, the same shape ``table_schema`` ships.
    Query parameters (not path segments): column names contain ``/`` and ``.``.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    node = flow.get_node(node_id)
    if node is None:
        raise HTTPException(404, "Could not find the node")
    if flow.flow_settings.execution_mode == "Performance":
        raise HTTPException(409, "Column statistics are not computed in Performance mode.")
    # Core must not run pipeline compute: whenever offloading is on, the stats
    # aggregate runs in the worker and core reads back only the 1-row result
    # (a failed offload raises → 409, never an in-process collect). In-process
    # is reserved for worker-less deployments (single-file mode).
    offload = bool(OFFLOAD_TO_WORKER)
    try:
        return node.get_column_stats(column_name, output_handle=output_handle, offload_to_worker=offload)
    except ColumnStatsUnavailable as e:
        raise HTTPException(409, str(e)) from None
    except ColumnNotFoundError:
        raise HTTPException(404, f"Column '{column_name}' not found in the node result") from None
    except Exception as e:
        logger.error(f"Failed to compute column stats for node {node_id}, column '{column_name}': {e}")
        raise HTTPException(422, "Could not compute column statistics") from e

get_node_input_names(flow_id, node_id)

Returns the named inputs available for a kernel node.

Each entry contains the input name (derived from the source node's node_reference or fallback df_{id}), the source node ID, and its type. The frontend uses this for autocomplete and display.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/input_names", tags=["editor"])
def get_node_input_names(flow_id: int, node_id: int) -> list[output_model.NodeInputNameInfo]:
    """Returns the named inputs available for a kernel node.

    Each entry contains the input name (derived from the source node's
    ``node_reference`` or fallback ``df_{id}``), the source node ID, and
    its type. The frontend uses this for autocomplete and display.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Flow not found")
    node = flow.get_node(node_id)
    if node is None:
        raise HTTPException(404, "Node not found")

    result: list[output_model.NodeInputNameInfo] = []
    for source_node in node.all_inputs:
        ref = getattr(source_node.setting_input, "node_reference", None)
        name = ref if ref else f"df_{source_node.node_id}"
        result.append(
            output_model.NodeInputNameInfo(
                name=name,
                source_node_id=source_node.node_id,
                source_node_type=source_node.node_type,
            )
        )
    return result

get_node_list()

Retrieves the list of all available node types and their templates.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node_list", response_model=list[schemas.NodeTemplate])
def get_node_list() -> list[schemas.NodeTemplate]:
    """Retrieves the list of all available node types and their templates."""
    from flowfile_core.flowfile.community_nodes.receipts import community_icon_overrides_by_node_key

    # Installed community icons live namespaced on disk (<id>__icon.png) while the
    # template keeps the node's original icon name; point the palette/canvas image at
    # the on-disk file so it loads (mirrors _node_info_from_entry for the catalog list).
    # model_copy, not in-place: the same NodeTemplate object is shared with node_dict.
    overrides = community_icon_overrides_by_node_key()
    if not overrides:
        return nodes_list
    return [
        node.model_copy(update={"image": overrides[node.item]}) if node.custom_node and node.item in overrides else node
        for node in nodes_list
    ]

get_node_model(setting_name_ref)

(Internal) Retrieves a node's Pydantic model from the input_schema module by its name.

Source code in flowfile_core/flowfile_core/routes/routes.py

def get_node_model(setting_name_ref: str):
    """(Internal) Retrieves a node's Pydantic model from the input_schema module by its name."""
    logger.info("Getting node model for: " + setting_name_ref)
    for ref_name, ref in inspect.getmodule(input_schema).__dict__.items():
        if ref_name.lower() == setting_name_ref:
            return ref
    logger.error(f"Could not find node model for: {setting_name_ref}")
    return None

get_node_upstream_ids(flow_id, node_id)

Return the transitive upstream node IDs for a given node.

Used by the frontend to determine which artifacts are actually reachable (via the DAG) from a specific python_script node.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow/node_upstream_ids", tags=["editor"])
def get_node_upstream_ids(flow_id: int, node_id: int):
    """Return the transitive upstream node IDs for a given node.

    Used by the frontend to determine which artifacts are actually
    reachable (via the DAG) from a specific python_script node.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    return {"upstream_node_ids": flow._get_upstream_node_ids(node_id)}

get_node_visualization_fields(body, current_user=Depends(get_current_active_user))

Return the Graphic Walker field schema for an Explore Data node's result.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/analysis_data/fields", tags=["analysis"], response_model=VisualizationFieldsResponse)
def get_node_visualization_fields(
    body: gs_schemas.NodeVisualizationFieldsRequest,
    current_user=Depends(get_current_active_user),
):
    """Return the Graphic Walker field schema for an Explore Data node's result."""
    flow, node = _get_analysis_node(body.flow_id, body.node_id, current_user.id if current_user else None)
    try:
        return node_viz.get_node_fields(flow, node)
    except node_viz.NodeNotRunError as exc:
        raise HTTPException(422, str(exc)) from exc
    except node_viz.CloudPlanNotVisualizableError as exc:
        raise HTTPException(400, str(exc)) from exc
    except RuntimeError as exc:
        raise HTTPException(502, str(exc)) from exc

get_reference_node(flow_id, node_id)

Retrieves the reference identifier for a specific node.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/reference", tags=["editor"])
def get_reference_node(flow_id: int, node_id: int):
    """Retrieves the reference identifier for a specific node."""
    try:
        node = flow_file_handler.get_flow(flow_id).get_node(node_id)
    except Exception:
        raise HTTPException(404, "Could not find the node") from None
    if node is None:
        raise HTTPException(404, "Could not find the node")
    return node.setting_input.node_reference or ""

get_run_status(flow_id, response)

Retrieves the run status information for a specific flow.

Returns a 202 Accepted status while the flow is running, and 200 OK when finished.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow/run_status/", tags=["editor"], response_model=output_model.RunInformation)
def get_run_status(flow_id: int, response: Response):
    """Retrieves the run status information for a specific flow.

    Returns a 202 Accepted status while the flow is running, and 200 OK when finished.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if not flow:
        raise HTTPException(status_code=404, detail="Flow not found")
    if flow.flow_settings.is_running:
        response.status_code = status.HTTP_202_ACCEPTED
    else:
        response.status_code = status.HTTP_200_OK
    return flow.get_run_info()

get_table_example(flow_id, node_id, output_handle=DEFAULT_OUTPUT_HANDLE)

Retrieves a data preview (schema and sample rows) for a node's output.

For multi-output nodes, output_handle selects which named output to preview (e.g. "output-0", "output-1"); the default is the first.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/data", response_model=output_model.TableExample, tags=["editor"])
def get_table_example(flow_id: int, node_id: int, output_handle: str = DEFAULT_OUTPUT_HANDLE):
    """Retrieves a data preview (schema and sample rows) for a node's output.

    For multi-output nodes, ``output_handle`` selects which named output to
    preview (e.g. ``"output-0"``, ``"output-1"``); the default is the first.
    """
    flow = flow_file_handler.get_flow(flow_id)
    node = flow.get_node(node_id)
    return node.get_table_example(True, output_handle=output_handle)

get_vue_flow_data(flow_id)

Retrieves the flow data formatted for the Vue-based frontend.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/flow_data/v2", tags=["manager"])
def get_vue_flow_data(flow_id: int) -> schemas.VueFlowInput:
    """Retrieves the flow data formatted for the Vue-based frontend."""
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    data = flow.get_vue_flow_input()
    return data

import_saved_flow(flow_path, current_user=Depends(get_current_active_user))

Imports a flow from a saved .yaml and registers it as a new session for the current user.

Opening a file is browsing, not filing: an existing registration is adopted so the session gets its display name and source_registration_id, but no new catalog row is created. Use the Save dialog (or POST /catalog/flows) to file a flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/import_flow/", tags=["editor"], response_model=int)
def import_saved_flow(flow_path: str, current_user=Depends(get_current_active_user)) -> int:
    """Imports a flow from a saved `.yaml` and registers it as a new session for the current user.

    Opening a file is browsing, not filing: an existing registration is adopted so the
    session gets its display name and ``source_registration_id``, but no new catalog
    row is created. Use the Save dialog (or ``POST /catalog/flows``) to file a flow.
    """
    validated_path = validate_path_under_cwd(flow_path)
    if not os.path.exists(validated_path):
        raise HTTPException(404, "File not found")
    user_id = current_user.id if current_user else None
    flow_id = flow_file_handler.import_flow(Path(validated_path), user_id=user_id)
    flow = flow_file_handler.get_flow(flow_id)
    if flow and flow.flow_settings:
        resolve_source_registration_id(flow)
    return flow_id

list_templates()

Returns metadata for all available flow templates.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/templates/", tags=["templates"])
def list_templates():
    """Returns metadata for all available flow templates."""
    from flowfile_core.templates import get_all_template_metas

    return get_all_template_metas()

overwrite_flow_in_catalog(flow_id, target_registration_id, current_user=Depends(get_current_active_user))

Overwrite an existing catalog flow's YAML with the contents of another flow.

Unlike /save_flow_to_catalog, this intentionally writes over an existing registration. The target registration's name and namespace are preserved; only the file contents on disk change. Primary use case: reverting a flow to an older version by loading that version and overwriting the canonical catalog entry.

Returns the (possibly new) flow id so the frontend can switch to the target.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/overwrite_flow_in_catalog", tags=["editor"])
def overwrite_flow_in_catalog(
    flow_id: int,
    target_registration_id: int,
    current_user=Depends(get_current_active_user),
):
    """Overwrite an existing catalog flow's YAML with the contents of another flow.

    Unlike ``/save_flow_to_catalog``, this intentionally writes over an existing
    registration.  The target registration's name and namespace are preserved;
    only the file contents on disk change.  Primary use case: reverting a flow
    to an older version by loading that version and overwriting the canonical
    catalog entry.

    Returns the (possibly new) flow id so the frontend can switch to the target.
    """
    user_id = current_user.id if current_user else None
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Flow not found")

    target = find_registration_by_registration_id(target_registration_id)
    if target is None:
        raise HTTPException(404, "Target catalog registration not found")

    # Overwrite is destructive — gate strictly on ownership even though
    # ``update_flow`` itself does not.
    if user_id is None or user_id != target.owner_id:
        raise HTTPException(
            status_code=403,
            detail="You do not have permission to overwrite this catalog flow",
        )

    target_path = validate_path_under_cwd(target.flow_path)
    managed_root = str(Path(storage.flows_directory).resolve()) + os.sep

    current_path = flow.flow_settings.path or flow.flow_settings.save_location
    normalized_current = validate_path_under_cwd(current_path) if current_path else None

    # Same-path case: current flow already lives at the target path, so just
    # re-save in place and keep the registration pointer fresh.
    if normalized_current == target_path:
        resolve_source_registration_id(flow)
        flow.save_flow(flow_path=target_path)
        _touch_flow_registration(target_registration_id)
        return flow_id

    new_flow_id = flow_file_handler.save_as_flow(
        flow_id=flow_id,
        new_path=target_path,
        user_id=user_id,
        on_catalog_register=None,  # registration already exists; preserve it
        on_resolve_registration=resolve_source_registration_id,
    )

    # If the source flow lived inside the managed dir on a different file,
    # unlink the abandoned file so we don't leak orphaned YAML.  We only clean
    # up files under the managed root; user-owned paths elsewhere are left
    # alone since we don't want to silently delete files the user manages.
    if normalized_current and normalized_current != target_path and normalized_current.startswith(managed_root):
        try:
            os.unlink(normalized_current)
        except OSError:
            logger.info(
                f"Could not unlink old managed flow file {normalized_current}",
                exc_info=True,
            )

    _touch_flow_registration(target_registration_id)
    return new_flow_id

preview_dynamic_rename(request)

Resolves a dynamic-rename rule against a given schema without mutating any flow.

The frontend calls this to render the live old-to-new preview pane inside the node's settings panel. Returns either the fully-resolved rename map (possibly empty) or an error describing a parse failure or duplicate-name collision.

first_row mode is intentionally not previewed here: its new names depend on actual row data, and we don't want to trigger upstream computation from a settings panel. The frontend renders a runtime-only placeholder for that mode.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/dynamic_rename/preview", response_model=DynamicRenamePreviewResponse, tags=["editor"])
def preview_dynamic_rename(request: DynamicRenamePreviewRequest) -> DynamicRenamePreviewResponse:
    """Resolves a dynamic-rename rule against a given schema without mutating any flow.

    The frontend calls this to render the live old-to-new preview pane inside the
    node's settings panel. Returns either the fully-resolved rename map (possibly
    empty) or an `error` describing a parse failure or duplicate-name collision.

    `first_row` mode is intentionally not previewed here: its new names depend on
    actual row data, and we don't want to trigger upstream computation from a
    settings panel. The frontend renders a runtime-only placeholder for that mode.
    """
    columns = [(c.name, c.data_type_group) for c in request.incoming_columns]
    try:
        rename_map = FlowDataEngine.resolve_dynamic_rename_map(columns, request.settings)
    except ValueError as exc:
        return DynamicRenamePreviewResponse(rename_map={}, error=str(exc))
    except Exception as exc:  # noqa: BLE001 - formula parse errors bubble up as various types
        # Log the full traceback so real bugs don't get silently reported as
        # "Formula error" to the user.
        logger.exception("Unexpected error while resolving dynamic rename preview")
        return DynamicRenamePreviewResponse(rename_map={}, error=f"Formula error: {exc}")
    return DynamicRenamePreviewResponse(rename_map=rename_map)

redo_action(flow_id)

Redo the last undone action on the flow graph.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to redo.	required

Returns:

Type	Description
UndoRedoResult	UndoRedoResult indicating success or failure.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/redo/", tags=["editor"], response_model=UndoRedoResult)
def redo_action(flow_id: int) -> UndoRedoResult:
    """Redo the last undone action on the flow graph.

    Args:
        flow_id: The ID of the flow to redo.

    Returns:
        UndoRedoResult indicating success or failure.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")
    return flow.redo()

register_flow(flow_data, current_user=Depends(get_current_active_user))

Registers a new flow session with the application for the current user.

Parameters:

Name	Type	Description	Default
flow_data	FlowSettings	The FlowSettings for the new flow.	required

Returns:

Type	Description
int	The ID of the newly registered flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/flow/register/", tags=["editor"])
def register_flow(flow_data: schemas.FlowSettings, current_user=Depends(get_current_active_user)) -> int:
    """Registers a new flow session with the application for the current user.

    Args:
        flow_data: The `FlowSettings` for the new flow.

    Returns:
        The ID of the newly registered flow.
    """
    user_id = current_user.id if current_user else None
    return flow_file_handler.register_flow(flow_data, user_id=user_id)

remove_nodes_from_group(flow_id, request)

Remove nodes from their group; a group emptied this way is pruned.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/group/remove_nodes/", tags=["editor"], response_model=OperationResponse)
def remove_nodes_from_group(flow_id: int, request: schemas.GroupMembershipRequest) -> OperationResponse:
    """Remove nodes from their group; a group emptied this way is pruned."""
    flow = _get_running_flow(flow_id)
    flow.remove_nodes_from_group(request.node_ids)
    return OperationResponse(success=True, history=flow.get_history_state())

rename_flow(body, current_user=Depends(get_current_active_user))

Renames a flow's display name: the catalog registration (when one exists) plus the in-memory session name. The file path is never touched.

For an unregistered flow the name is written back into its YAML, because there is no registration to hold it and open_flow would otherwise have nothing to read: the rename would silently vanish when the tab is closed.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/rename_flow/", tags=["editor"], response_model=schemas.FlowSettingsResponse)
def rename_flow(body: RenameFlowInput, current_user=Depends(get_current_active_user)) -> schemas.FlowSettingsResponse:
    """Renames a flow's display name: the catalog registration (when one exists) plus the
    in-memory session name. The file *path* is never touched.

    For an unregistered flow the name is written back into its YAML, because there is no
    registration to hold it and ``open_flow`` would otherwise have nothing to read: the
    rename would silently vanish when the tab is closed.
    """
    user_id = current_user.id if current_user else None
    flow = flow_file_handler.get_flow(body.flow_id, user_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    name = body.name.strip()
    if not name:
        raise HTTPException(422, "Flow name cannot be empty")
    resolve_source_registration_id(flow)
    reg_id = flow.flow_settings.source_registration_id
    renamed_registration = False
    if reg_id is not None:
        with get_db_context() as db:
            service = CatalogService(SQLAlchemyCatalogRepository(db), access=AccessResolver(db, current_user))
            try:
                service.update_flow(registration_id=reg_id, requesting_user_id=user_id, name=name)
                renamed_registration = True
            except NotAuthorizedError as e:
                raise HTTPException(403, str(e)) from e
            except FlowExistsError as e:
                raise HTTPException(409, "A flow with this name already exists in this namespace") from e
            except FlowNotFoundError:
                # Registration deleted concurrently — the in-memory rename below still applies.
                pass
    # Only after catalog success, so a 403/409 leaves the session name untouched.
    flow.flow_settings.name = name
    # __name__ is what gets serialised as ``flowfile_name``; without this the next
    # same-path save would write the stale name back over the rename.
    flow.__name__ = name
    if reg_id is None:
        _persist_rename_of_unregistered_flow(flow)
    response = _with_display_name(flow_file_handler.get_flow_info_with_runtime(body.flow_id))
    if renamed_registration:
        # Prefer the write we just made: a by-path lookup misses when the
        # registration's flow_path has drifted from the session's path.
        response.display_name = name
    return response

run_flow(flow_id, background_tasks, current_user=Depends(get_current_active_user)) async

Executes a flow in a background task.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to execute.	required
background_tasks	BackgroundTasks	FastAPI's background task runner.	required

Returns:

Type	Description
JSONResponse	A JSON response indicating that the flow has started.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/flow/run/", tags=["editor"])
async def run_flow(
    flow_id: int, background_tasks: BackgroundTasks, current_user=Depends(get_current_active_user)
) -> JSONResponse:
    """Executes a flow in a background task.

    Args:
        flow_id: The ID of the flow to execute.
        background_tasks: FastAPI's background task runner.

    Returns:
        A JSON response indicating that the flow has started.
    """
    logger.info("starting to run...")
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        # Frontend's flow_id has drifted from the in-memory handler — typically
        # after a Save As or backend restart. Surface a 404 so the UI can prompt
        # a reload instead of falling through to an AttributeError 500.
        raise HTTPException(
            status_code=404,
            detail=f"Flow {flow_id} is no longer in memory. Reload the flow and try again.",
        )
    lock = get_flow_run_lock(flow_id)
    user_id = current_user.id if current_user else None
    async with lock:
        if flow.flow_settings.is_running:
            raise HTTPException(422, "Flow is already running")
        background_tasks.add_task(_run_and_track, flow, user_id)
    return JSONResponse(content={"message": "Data started", "flow_id": flow_id}, status_code=status.HTTP_200_OK)

save_flow(response, flow_id, flow_path=None, namespace_id=None, register_in_catalog=True, current_user=Depends(get_current_active_user))

Deprecated GET variant of /save_flow. Prefer POST.

Kept for backward compatibility with older frontends/clients. Emits a Deprecation: true response header.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/save_flow", tags=["editor"])
def save_flow(
    response: Response,
    flow_id: int,
    flow_path: str = None,
    namespace_id: int = None,
    register_in_catalog: bool = True,
    current_user=Depends(get_current_active_user),
):
    """Deprecated GET variant of ``/save_flow``.  Prefer POST.

    Kept for backward compatibility with older frontends/clients. Emits a
    ``Deprecation: true`` response header.
    """
    logger.warning("GET /save_flow is deprecated; use POST /save_flow instead")
    response.headers["Deprecation"] = "true"
    return _save_flow_impl(flow_id, flow_path, namespace_id, current_user, register_in_catalog)

save_flow_post(flow_id, flow_path=None, namespace_id=None, register_in_catalog=True, current_user=Depends(get_current_active_user))

Saves the current state of a flow to a .yaml.

See :func:_save_flow_impl for semantics.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/save_flow", tags=["editor"])
def save_flow_post(
    flow_id: int,
    flow_path: str = None,
    namespace_id: int = None,
    register_in_catalog: bool = True,
    current_user=Depends(get_current_active_user),
):
    """Saves the current state of a flow to a ``.yaml``.

    See :func:`_save_flow_impl` for semantics.
    """
    return _save_flow_impl(flow_id, flow_path, namespace_id, current_user, register_in_catalog)

save_flow_to_catalog(flow_id, flow_name, namespace_id, current_user=Depends(get_current_active_user))

Save a flow into the managed catalog flows directory with a collision-free filename.

The file is always written to {flows_dir}/{flow_id}_{sanitized_name}.yaml so two flows with the same user-chosen name in different namespaces cannot overwrite each other. Returns the (possibly new) flow id so the frontend can switch to it.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/save_flow_to_catalog", tags=["editor"])
def save_flow_to_catalog(
    flow_id: int,
    flow_name: str,
    namespace_id: int,
    current_user=Depends(get_current_active_user),
):
    """Save a flow into the managed catalog flows directory with a collision-free filename.

    The file is always written to ``{flows_dir}/{flow_id}_{sanitized_name}.yaml`` so
    two flows with the same user-chosen name in different namespaces cannot overwrite
    each other. Returns the (possibly new) flow id so the frontend can switch to it.
    """
    display_name = flow_name.strip()
    # Strip a managed extension the user may have typed so the display name and
    # the on-disk filename stay in sync (``.yaml`` is appended below).
    display_name = re.sub(r"\.(ya?ml|json)$", "", display_name, flags=re.IGNORECASE).strip()
    if not display_name:
        raise HTTPException(422, "flow_name must not be empty")
    # The name is a free-form DISPLAY label (spaces, mixed case and punctuation
    # are all fine) — it is stored as the registration name and used for the
    # collision check below. Only the derived on-disk filename must be safe, so
    # slugify a separate stem and prefix it with the flow id; path separators and
    # ``..`` cannot survive the slug, and resolve_managed_flow_path re-validates
    # the result against the same allowlist as a defense-in-depth backstop.
    safe_stem = _MANAGED_FLOW_STEM_DISALLOWED_RE.sub("_", display_name).strip("_-") or "flow"

    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Flow not found")

    # Refuse before any side effect (ghost cleanup, YAML write) when the caller
    # may neither write the target namespace nor manage a flow already at the path.
    _require_flow_save_permitted(flow_path=None, namespace_id=namespace_id, current_user=current_user)

    source_registration_id = getattr(flow.flow_settings, "source_registration_id", None)

    # Pre-save name-collision check: reject BEFORE writing any YAML so a failed save doesn't leave orphaned files
    # on disk.
    existing_by_name = find_registration_by_name(display_name, namespace_id)
    if existing_by_name is not None and existing_by_name.id != source_registration_id:
        name_conflict = HTTPException(
            status_code=409,
            detail=(
                f"A flow named '{display_name}' already exists in this namespace. "
                "Select it in the catalog picker to overwrite, or choose a different name."
            ),
        )
        if os.path.exists(existing_by_name.flow_path):
            raise name_conflict
        # Ghost registration: its backing file was deleted (e.g. by an older
        # Save-As). Reclaim the name by removing the dead row instead of blocking
        # the user with a 409 they can't act on. Best-effort — fall back to the
        # conflict if it can't be cleanly removed (e.g. it still has artifacts).
        try:
            with get_db_context() as db:
                # Resolver-injected so delete_flow's manage gate applies: only the
                # ghost's owner (or admin/manage-grantee) may reclaim the name.
                CatalogService(
                    SQLAlchemyCatalogRepository(db),
                    access=AccessResolver(db, current_user),
                ).delete_flow(registration_id=existing_by_name.id, delete_file=False)
        except Exception as err:
            raise name_conflict from err

    own_registration = (
        existing_by_name if existing_by_name is not None and existing_by_name.id == source_registration_id else None
    )
    flow_path = _resolve_catalog_save_path(flow_id, safe_stem, own_registration)
    if own_registration is not None:
        _require_flow_save_permitted(flow_path, namespace_id, current_user)

    current_path = flow.flow_settings.path or flow.flow_settings.save_location
    normalized_current = validate_path_under_cwd(current_path) if current_path else None
    is_new_path = bool(normalized_current) and flow_path != normalized_current

    # Overwrite guard: if the resolved target file is already registered to a
    # different flow, or exists on disk without any registration, refuse.
    existing_reg = find_registration_by_path(flow_path)
    if existing_reg is not None and existing_reg.id != source_registration_id:
        raise HTTPException(
            status_code=409,
            detail=f"Target file {flow_path} is already registered to another flow",
        )
    if existing_reg is None and os.path.exists(flow_path):
        raise HTTPException(
            status_code=409,
            detail=(f"Target file {flow_path} exists but is not catalog-registered; " "refusing to overwrite"),
        )

    user_id = current_user.id if current_user else None

    # Always register under the user-typed ``display_name`` rather than the
    # filename (``{flow_id}_{safe_stem}``) so the catalog picker shows exactly
    # what the user typed — and so the name-collision check above compares apples
    # to apples.
    def _register(fp: str, _n: str, uid: int | None) -> None:
        register_flow_in_namespace(fp, display_name, uid, namespace_id, requesting_user=current_user)

    if is_new_path:
        try:
            new_flow_id = flow_file_handler.save_as_flow(
                flow_id=flow_id,
                new_path=flow_path,
                user_id=user_id,
                on_catalog_register=_register,
                on_resolve_registration=resolve_source_registration_id,
            )
        except FlowPathNamespaceCollision as err:
            raise HTTPException(status_code=409, detail=str(err)) from err
        except FlowNameNamespaceCollision as err:
            raise HTTPException(status_code=409, detail=str(err)) from err
        except NotAuthorizedError as err:
            raise HTTPException(status_code=403, detail=str(err)) from None
        except NamespaceNotFoundError:
            raise HTTPException(status_code=404, detail="Namespace not found") from None

        _discard_relocated_scratch_file(normalized_current, flow_path)
        sync_api_compatibility(flow_file_handler.get_flow(new_flow_id))
        return new_flow_id

    resolve_source_registration_id(flow)
    flow.save_flow(flow_path=flow_path)
    try:
        register_flow_in_namespace(flow_path, display_name, user_id, namespace_id, requesting_user=current_user)
    except FlowPathNamespaceCollision as err:
        raise HTTPException(status_code=409, detail=str(err)) from err
    except FlowNameNamespaceCollision as err:
        raise HTTPException(status_code=409, detail=str(err)) from err
    except NotAuthorizedError as err:
        raise HTTPException(status_code=403, detail=str(err)) from None
    except NamespaceNotFoundError:
        raise HTTPException(status_code=404, detail="Namespace not found") from None
    sync_api_compatibility(flow)
    return flow_id

save_generated_project(request)

Generates the project export and writes it into a directory on the server.

The target directory is validated with the same sandbox rules as the file browser (unrestricted in Electron mode, sandboxed to the user data directory otherwise). Files are written under <target_directory>/<project_name>/; existing project directories are rejected with 409 unless overwrite is set. Existing files are only overwritten file-by-file, never deleted.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/code_to_project/save", tags=[], response_model=output_model.ProjectSaveResponse)
def save_generated_project(request: output_model.ProjectSaveRequest) -> output_model.ProjectSaveResponse:
    """Generates the project export and writes it into a directory on the server.

    The target directory is validated with the same sandbox rules as the file
    browser (unrestricted in Electron mode, sandboxed to the user data
    directory otherwise). Files are written under
    ``<target_directory>/<project_name>/``; existing project directories are
    rejected with 409 unless ``overwrite`` is set. Existing files are only
    overwritten file-by-file, never deleted.
    """
    manifest = _export_project_manifest(request.flow_id)
    sandbox_root = None if is_electron_mode() else storage.user_data_directory
    try:
        explorer = SecureFileExplorer(request.target_directory, sandbox_root)
    except PermissionError:
        raise HTTPException(403, "Access denied: path is outside the allowed directory") from None
    target_dir = explorer.current_path
    if not target_dir.exists() or not target_dir.is_dir():
        raise HTTPException(404, "Target directory does not exist")
    project_dir = target_dir / manifest.project_name
    if project_dir.exists():
        if not project_dir.is_dir():
            raise HTTPException(409, f"'{project_dir}' exists and is not a directory.")
        if not request.overwrite:
            raise HTTPException(409, f"'{project_dir}' already exists. Enable overwrite to replace its files.")
    try:
        for file in manifest.files:
            file_path = project_dir / file.path
            file_path.parent.mkdir(parents=True, exist_ok=True)
            file_path.write_text(file.content, encoding="utf-8")
    except OSError as e:
        raise HTTPException(409, f"Could not write the project to '{project_dir}': {e}") from e
    return output_model.ProjectSaveResponse(saved_to=str(project_dir), file_count=len(manifest.files))

trigger_fetch_node_data(flow_id, node_id, background_tasks, performance_mode=False) async

Fetches and refreshes the data for a specific node.

performance_mode=true builds the node's query plan without storing its result — enough for the Explore Data drawer, which charts through the worker and never reads the example rows the default (preview) path materialises.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/node/trigger_fetch_data", tags=["editor"])
async def trigger_fetch_node_data(
    flow_id: int,
    node_id: int,
    background_tasks: BackgroundTasks,
    performance_mode: bool = False,
):
    """Fetches and refreshes the data for a specific node.

    ``performance_mode=true`` builds the node's query plan without storing its
    result — enough for the Explore Data drawer, which charts through the worker
    and never reads the example rows the default (preview) path materialises.
    """
    flow = flow_file_handler.get_flow(flow_id)
    lock = get_flow_run_lock(flow_id)
    async with lock:
        if flow.flow_settings.is_running:
            raise HTTPException(422, "Flow is already running")
        try:
            flow.validate_if_node_can_be_fetched(node_id)
        except Exception as e:
            raise HTTPException(422, str(e)) from e
        background_tasks.add_task(
            flow.trigger_fetch_node,
            node_id,
            performance_mode=performance_mode,
            reset_cache=not performance_mode,
        )
    return JSONResponse(
        content={"message": "Data started", "flow_id": flow_id, "node_id": node_id}, status_code=status.HTTP_200_OK
    )

undo_action(flow_id)

Undo the last action on the flow graph.

Parameters:

Name	Type	Description	Default
flow_id	int	The ID of the flow to undo.	required

Returns:

Type	Description
UndoRedoResult	UndoRedoResult indicating success or failure.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/undo/", tags=["editor"], response_model=UndoRedoResult)
def undo_action(flow_id: int) -> UndoRedoResult:
    """Undo the last action on the flow graph.

    Args:
        flow_id: The ID of the flow to undo.

    Returns:
        UndoRedoResult indicating success or failure.
    """
    flow = flow_file_handler.get_flow(flow_id)
    if flow is None:
        raise HTTPException(404, "Could not find the flow")
    if flow.flow_settings.is_running:
        raise HTTPException(422, "Flow is running")
    return flow.undo()

update_db_connection(input_connection, current_user=Depends(get_current_active_user), db=Depends(get_db))

Updates an existing database connection (own, or group-shared with manage access).

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.put("/db_connection_lib", tags=["db_connections"])
def update_db_connection(
    input_connection: input_schema.FullDatabaseConnection,
    current_user=Depends(get_current_active_user),
    db: Session = Depends(get_db),
):
    """Updates an existing database connection (own, or group-shared with manage access)."""
    logger.info(f"Updating database connection {input_connection.connection_name}")
    db_connection = get_database_connection(db, input_connection.connection_name, current_user.id)
    if db_connection is None:
        raise HTTPException(404, "Database connection not found")
    # Only gate a CHANGED type: legacy rows may hold pre-registry values (e.g. redshift)
    # and must stay updatable (password rotation) as long as the type is untouched.
    if input_connection.database_type.lower() != (db_connection.database_type or "").lower():
        _require_known_database_type(input_connection.database_type)
    if authorize_connection_mutation(db, current_user, "database_connection", db_connection):
        changed = changed_target_fields(
            db_connection, input_connection, ("host", "port", "database", "database_type", "ssl_enabled")
        )
        require_credentials_on_target_change(
            changed,
            has_new_credentials=bool(input_connection.password.get_secret_value()),
            has_bundled_secrets=db_connection.password_id is not None,
        )
    try:
        # Owner's user_id keeps a rotated password encrypted under the OWNER's key.
        update_database_connection(db, input_connection, db_connection.user_id)
    except ValueError:
        raise HTTPException(404, "Database connection not found") from None
    except Exception as e:
        logger.error(e)
        raise HTTPException(422, str(e)) from e
    return {"message": "Database connection updated successfully"}

update_description_node(flow_id, node_id, description=Body(...))

Updates the description text for a specific node.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/node/description/", tags=["editor"])
def update_description_node(flow_id: int, node_id: int, description: str = Body(...)):
    """Updates the description text for a specific node."""
    try:
        node = flow_file_handler.get_flow(flow_id).get_node(node_id)
    except Exception:
        raise HTTPException(404, "Could not find the node") from None
    node.setting_input.description = description
    return True

update_flow_settings(flow_settings)

Updates the main settings for a flow.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/flow_settings", tags=["manager"])
def update_flow_settings(flow_settings: schemas.FlowSettings):
    """Updates the main settings for a flow."""
    flow = flow_file_handler.get_flow(flow_settings.flow_id)
    if flow is None:
        raise HTTPException(404, "could not find the flow")
    flow.flow_settings = flow_settings

update_group(flow_id, group_id, request)

Rename / recolor / move / resize / collapse a group box.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/update_group/", tags=["editor"], response_model=GroupOperationResponse)
def update_group(flow_id: int, group_id: int, request: schemas.UpdateGroupRequest) -> GroupOperationResponse:
    """Rename / recolor / move / resize / collapse a group box."""
    flow = _get_running_flow(flow_id)
    try:
        group = flow.update_group(
            group_id,
            name=request.name,
            color=request.color,
            bounds=_bounds_from_request(request),
            collapsed=request.collapsed,
        )
    except ValueError as exc:
        raise HTTPException(404, str(exc)) from exc
    return GroupOperationResponse(success=True, history=flow.get_history_state(), group=_group_to_schema(group))

update_layout(flow_id, request)

Persist dragged node positions and/or group bounds (one drag-end -> one call).

Also closes the long-standing gap where dragged node positions were never persisted.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/editor/update_layout/", tags=["editor"], response_model=OperationResponse)
def update_layout(flow_id: int, request: schemas.UpdateLayoutRequest) -> OperationResponse:
    """Persist dragged node positions and/or group bounds (one drag-end -> one call).

    Also closes the long-standing gap where dragged node positions were never persisted.
    """
    flow = _get_running_flow(flow_id)
    if request.node_positions or request.group_bounds:
        if request.record_history:
            flow.capture_history_snapshot(HistoryActionType.MOVE_NODES, "Update layout")
        flow.set_node_positions(request.node_positions)
        flow.set_group_bounds(request.group_bounds)
    return OperationResponse(success=True, history=flow.get_history_state())

update_reference_node(flow_id, node_id, reference=Body(...))

Updates the reference identifier for a specific node.

The reference must be: - Lowercase only - No spaces allowed - Unique across all nodes in the flow

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/node/reference/", tags=["editor"])
def update_reference_node(flow_id: int, node_id: int, reference: str = Body(...)):
    """Updates the reference identifier for a specific node.

    The reference must be:
    - Lowercase only
    - No spaces allowed
    - Unique across all nodes in the flow
    """
    try:
        flow = flow_file_handler.get_flow(flow_id)
        node = flow.get_node(node_id)
    except Exception:
        raise HTTPException(404, "Could not find the node") from None
    if node is None:
        raise HTTPException(404, "Could not find the node")

    # Handle empty reference (allow clearing)
    if reference == "" or reference is None:
        node.setting_input.node_reference = None
        return True

    if " " in reference:
        raise HTTPException(422, "Reference cannot contain spaces")
    if reference != reference.lower():
        raise HTTPException(422, "Reference must be lowercase")

    for other_node in flow.nodes:
        if other_node.node_id != node_id:
            other_ref = getattr(other_node.setting_input, "node_reference", None)
            if other_ref and other_ref == reference:
                raise HTTPException(422, f'Reference "{reference}" is already used by another node')

    node.setting_input.node_reference = reference
    return True

validate_db_settings(database_settings, current_user=Depends(get_current_active_user)) async

Validates that a connection can be made to a database with the given settings.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.post("/validate_db_settings")
async def validate_db_settings(
    database_settings: input_schema.DatabaseSettings, current_user=Depends(get_current_active_user)
):
    """Validates that a connection can be made to a database with the given settings."""
    try:
        sql_source = create_sql_source_from_db_settings(database_settings, user_id=current_user.id)
        sql_source.validate()
        return {"message": "Query settings are valid"}
    except Exception as e:
        raise HTTPException(status_code=422, detail=str(e)) from e

validate_node_reference(flow_id, node_id, reference)

Validates if a reference is valid and unique for a node.

Returns:

Type	Description
	Dict with 'valid' (bool) and 'error' (str or None) fields.

Source code in flowfile_core/flowfile_core/routes/routes.py

@router.get("/node/validate_reference", tags=["editor"])
def validate_node_reference(flow_id: int, node_id: int, reference: str):
    """Validates if a reference is valid and unique for a node.

    Returns:
        Dict with 'valid' (bool) and 'error' (str or None) fields.
    """
    try:
        flow = flow_file_handler.get_flow(flow_id)
    except Exception:
        raise HTTPException(404, "Could not find the flow") from None

    # Handle empty reference (always valid - means use default)
    if reference == "" or reference is None:
        return {"valid": True, "error": None}

    if reference != reference.lower():
        return {"valid": False, "error": "Reference must be lowercase"}

    if " " in reference:
        return {"valid": False, "error": "Reference cannot contain spaces"}

    for other_node in flow.nodes:
        if other_node.node_id != node_id:
            other_ref = getattr(other_node.setting_input, "node_reference", None)
            if other_ref and other_ref == reference:
                return {"valid": False, "error": f'Reference "{reference}" is already used by another node'}

    return {"valid": True, "error": None}

auth

flowfile_core.routes.auth

Functions:

Name	Description
change_own_password	Change the current user's password
create_user	Create a new user (admin only)
delete_user	Delete a user (admin only)
get_password_requirements	Get password requirements for client-side validation
list_users	List all users (admin only)
refresh_access_token	Exchange a valid refresh token for a new access token and refresh token.
update_user	Update a user (admin only)

change_own_password(password_data, current_user=Depends(get_current_active_user), db=Depends(get_db)) async

Change the current user's password

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.post("/users/me/change-password", response_model=User)
async def change_own_password(
    password_data: ChangePassword, current_user: User = Depends(get_current_active_user), db: Session = Depends(get_db)
):
    """Change the current user's password"""
    user = db.query(db_models.User).filter(db_models.User.id == current_user.id).first()
    if not user:
        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found")

    if not verify_password(password_data.current_password, user.hashed_password):
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Current password is incorrect")

    is_valid, error_message = validate_password(password_data.new_password)
    if not is_valid:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error_message)

    user.hashed_password = get_password_hash(password_data.new_password)
    user.must_change_password = False
    db.commit()
    db.refresh(user)

    return User(
        username=user.username,
        id=user.id,
        email=user.email,
        full_name=user.full_name,
        disabled=user.disabled,
        is_admin=user.is_admin,
        must_change_password=user.must_change_password,
    )

create_user(user_data, current_user=Depends(get_current_admin_user), db=Depends(get_db)) async

Create a new user (admin only)

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.post("/users", response_model=User)
async def create_user(
    user_data: UserCreate, current_user: User = Depends(get_current_admin_user), db: Session = Depends(get_db)
):
    """Create a new user (admin only)"""
    if user_data.username == sharing.INTERNAL_SERVICE_USERNAME:
        # Reserved sentinel: sharing classifies principals as the synthetic kernel
        # identity by this username, so a real user named this way would be locked
        # out of their own resources.
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Username is reserved")
    existing_user = db.query(db_models.User).filter(db_models.User.username == user_data.username).first()
    if existing_user:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Username already exists")

    if user_data.email:
        existing_email = db.query(db_models.User).filter(db_models.User.email == user_data.email).first()
        if existing_email:
            raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Email already exists")

    is_valid, error_message = validate_password(user_data.password)
    if not is_valid:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error_message)

    hashed_password = get_password_hash(user_data.password)
    new_user = db_models.User(
        username=user_data.username,
        email=user_data.email or f"{user_data.username}@flowfile.app",
        full_name=user_data.full_name,
        hashed_password=hashed_password,
        is_admin=user_data.is_admin,
        must_change_password=True,
    )
    db.add(new_user)
    db.commit()
    db.refresh(new_user)

    return User(
        username=new_user.username,
        id=new_user.id,
        email=new_user.email,
        full_name=new_user.full_name,
        disabled=new_user.disabled,
        is_admin=new_user.is_admin,
        must_change_password=new_user.must_change_password,
    )

delete_user(user_id, current_user=Depends(get_current_admin_user), db=Depends(get_db)) async

Delete a user (admin only)

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.delete("/users/{user_id}")
async def delete_user(
    user_id: int, current_user: User = Depends(get_current_admin_user), db: Session = Depends(get_db)
):
    """Delete a user (admin only)"""
    user = db.query(db_models.User).filter(db_models.User.id == user_id).first()
    if not user:
        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found")

    # Prevent admin from deleting themselves
    if user.id == current_user.id:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot delete your own account")

    # Personal credential rows — everything keyed by user_id in the sharing
    # registry: secrets + all connection types — die with the user, including any
    # sharing grants on them: SQLite reuses rowids, so a stale grant would attach
    # to a future resource created with the same id. Catalog content
    # (owner_id/created_by keyed) is deliberately retained, so its grants stay
    # meaningful. Groups the user ran persist; global admins can administer them.
    for resource_type, spec in sharing.RESOURCE_REGISTRY.items():
        if spec.owner_attr != "user_id":
            continue
        resource_ids = [row[0] for row in db.query(spec.model.id).filter(spec.model.user_id == user_id)]
        if not resource_ids:
            continue
        db.query(db_models.ResourceGrant).filter(
            db_models.ResourceGrant.resource_type == resource_type,
            db_models.ResourceGrant.resource_id.in_(resource_ids),
        ).delete(synchronize_session=False)
        db.query(spec.model).filter(spec.model.id.in_(resource_ids)).delete(synchronize_session=False)
    sharing.delete_memberships_for_user(db, user_id)

    # workspace_projects is not share-registered, so plain row deletion suffices.
    # Removing these prevents a future user receiving the same rowid from inheriting
    # an orphan is_active=True project (rowid reuse; SQLite FK enforcement is off).
    db.query(db_models.WorkspaceProject).filter(db_models.WorkspaceProject.owner_id == user_id).delete(
        synchronize_session=False
    )

    db.delete(user)
    db.commit()

    return {"message": f"User '{user.username}' deleted successfully"}

get_password_requirements() async

Get password requirements for client-side validation

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.get("/password-requirements")
async def get_password_requirements():
    """Get password requirements for client-side validation"""
    return PASSWORD_REQUIREMENTS

list_users(current_user=Depends(get_current_admin_user), db=Depends(get_db)) async

List all users (admin only)

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.get("/users", response_model=list[User])
async def list_users(current_user: User = Depends(get_current_admin_user), db: Session = Depends(get_db)):
    """List all users (admin only)"""
    users = db.query(db_models.User).all()
    return [
        User(
            username=u.username,
            id=u.id,
            email=u.email,
            full_name=u.full_name,
            disabled=u.disabled,
            is_admin=u.is_admin,
            must_change_password=u.must_change_password,
        )
        for u in users
    ]

refresh_access_token(refresh_token=Form(...), db=Depends(get_db)) async

Exchange a valid refresh token for a new access token and refresh token.

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.post("/refresh", response_model=Token)
async def refresh_access_token(
    refresh_token: str = Form(...),
    db: Session = Depends(get_db),
):
    """Exchange a valid refresh token for a new access token and refresh token."""
    username = decode_refresh_token(refresh_token)

    user = db.query(db_models.User).filter(db_models.User.username == username).first()
    if user is None:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="User no longer exists",
            headers={"WWW-Authenticate": "Bearer"},
        )
    if user.disabled:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="User account is disabled",
            headers={"WWW-Authenticate": "Bearer"},
        )

    new_access_token = create_access_token(data={"sub": user.username})
    new_refresh_token = create_refresh_token(data={"sub": user.username})
    return {"access_token": new_access_token, "refresh_token": new_refresh_token, "token_type": "bearer"}

update_user(user_id, user_data, current_user=Depends(get_current_admin_user), db=Depends(get_db)) async

Update a user (admin only)

Source code in flowfile_core/flowfile_core/routes/auth.py

@router.put("/users/{user_id}", response_model=User)
async def update_user(
    user_id: int,
    user_data: UserUpdate,
    current_user: User = Depends(get_current_admin_user),
    db: Session = Depends(get_db),
):
    """Update a user (admin only)"""
    user = db.query(db_models.User).filter(db_models.User.id == user_id).first()
    if not user:
        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found")

    # Prevent admin from disabling themselves
    if user.id == current_user.id and user_data.disabled:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot disable your own account")

    # Prevent admin from removing their own admin status
    if user.id == current_user.id and user_data.is_admin is False:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot remove your own admin privileges")

    if user_data.email is not None:
        existing_email = (
            db.query(db_models.User)
            .filter(db_models.User.email == user_data.email, db_models.User.id != user_id)
            .first()
        )
        if existing_email:
            raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Email already exists")
        user.email = user_data.email

    if user_data.full_name is not None:
        user.full_name = user_data.full_name

    if user_data.disabled is not None:
        user.disabled = user_data.disabled

    if user_data.is_admin is not None:
        user.is_admin = user_data.is_admin

    if user_data.password is not None:
        is_valid, error_message = validate_password(user_data.password)
        if not is_valid:
            raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=error_message)
        user.hashed_password = get_password_hash(user_data.password)
        user.must_change_password = True

    if user_data.must_change_password is not None:
        user.must_change_password = user_data.must_change_password

    db.commit()
    db.refresh(user)

    return User(
        username=user.username,
        id=user.id,
        email=user.email,
        full_name=user.full_name,
        disabled=user.disabled,
        is_admin=user.is_admin,
        must_change_password=user.must_change_password,
    )

cloud_connections

flowfile_core.routes.cloud_connections

Functions:

Name	Description
create_cloud_storage_connection	Create a new cloud storage connection.
delete_cloud_connection_with_connection_name	Delete a cloud connection (own, or group-shared with manage access).
get_cloud_connections	Get all cloud storage connections for the current user.
update_cloud_storage_connection	Update an existing cloud storage connection (own, or group-shared with manage access).

create_cloud_storage_connection(input_connection, current_user=Depends(get_current_active_user), db=Depends(get_db))

Create a new cloud storage connection. Parameters input_connection: FullCloudStorageConnection schema containing connection details current_user: User obtained from Depends(get_current_active_user) db: Session obtained from Depends(get_db) Returns Dict with a success message

Source code in flowfile_core/flowfile_core/routes/cloud_connections.py

@router.post("/cloud_connection", tags=["cloud_connections"])
def create_cloud_storage_connection(
    input_connection: FullCloudStorageConnection,
    current_user=Depends(get_current_active_user),
    db: Session = Depends(get_db),
):
    """
    Create a new cloud storage connection.
    Parameters
        input_connection: FullCloudStorageConnection schema containing connection details
        current_user: User obtained from Depends(get_current_active_user)
        db: Session obtained from Depends(get_db)
    Returns
        Dict with a success message
    """
    logger.info(f"Create cloud connection {input_connection.connection_name}")
    try:
        store_cloud_connection(db, input_connection, current_user.id)
    except ValueError:
        raise HTTPException(422, "Connection name already exists") from None
    except Exception as e:
        logger.error(e)
        raise HTTPException(422, str(e)) from e
    return {"message": "Cloud connection created successfully"}

delete_cloud_connection_with_connection_name(connection_name, current_user=Depends(get_current_active_user), db=Depends(get_db))

Delete a cloud connection (own, or group-shared with manage access).

Source code in flowfile_core/flowfile_core/routes/cloud_connections.py

@router.delete("/cloud_connection", tags=["cloud_connections"])
def delete_cloud_connection_with_connection_name(
    connection_name: str, current_user=Depends(get_current_active_user), db: Session = Depends(get_db)
):
    """
    Delete a cloud connection (own, or group-shared with manage access).
    """
    logger.info(f"Deleting cloud connection {connection_name}")
    db_connection = get_cloud_connection(db, connection_name, current_user.id)
    if db_connection is None:
        raise HTTPException(404, "Cloud connection connection not found")
    authorize_connection_mutation(db, current_user, "cloud_connection", db_connection)
    delete_cloud_connection(db, connection_name, db_connection.user_id)
    return {"message": "Cloud connection deleted successfully"}

get_cloud_connections(db=Depends(get_db), current_user=Depends(get_current_active_user))

Get all cloud storage connections for the current user. Parameters db: Session obtained from Depends(get_db) current_user: User obtained from Depends(get_current_active_user)

Returns List[FullCloudStorageConnectionInterface]

Source code in flowfile_core/flowfile_core/routes/cloud_connections.py

@router.get("/cloud_connections", tags=["cloud_connection"], response_model=list[FullCloudStorageConnectionInterface])
def get_cloud_connections(
    db: Session = Depends(get_db), current_user=Depends(get_current_active_user)
) -> list[FullCloudStorageConnectionInterface]:
    """
    Get all cloud storage connections for the current user.
    Parameters
        db: Session obtained from Depends(get_db)
        current_user: User obtained from Depends(get_current_active_user)

    Returns
        List[FullCloudStorageConnectionInterface]
    """
    return get_all_cloud_connections_interface(db, current_user.id)

update_cloud_storage_connection(input_connection, current_user=Depends(get_current_active_user), db=Depends(get_db))

Update an existing cloud storage connection (own, or group-shared with manage access).

Source code in flowfile_core/flowfile_core/routes/cloud_connections.py

@router.put("/cloud_connection", tags=["cloud_connections"])
def update_cloud_storage_connection(
    input_connection: FullCloudStorageConnection,
    current_user=Depends(get_current_active_user),
    db: Session = Depends(get_db),
):
    """Update an existing cloud storage connection (own, or group-shared with manage access)."""
    logger.info(f"Update cloud connection {input_connection.connection_name}")
    db_connection = get_cloud_connection(db, input_connection.connection_name, current_user.id)
    if db_connection is None:
        raise HTTPException(404, "Cloud connection not found")
    if authorize_connection_mutation(db, current_user, "cloud_connection", db_connection):
        changed = changed_target_fields(
            db_connection, input_connection, ("storage_type", "auth_method", "endpoint_url", "verify_ssl")
        )
        has_new_credentials = any(
            field is not None and field.get_secret_value()
            for field in (
                input_connection.aws_secret_access_key,
                input_connection.azure_account_key,
                input_connection.azure_client_secret,
                input_connection.azure_sas_token,
                input_connection.gcs_service_account_key,
            )
        )
        has_bundled_secrets = any(
            getattr(db_connection, column) is not None
            for column in (
                "aws_secret_access_key_id",
                "aws_session_token_id",
                "azure_account_key_id",
                "azure_client_secret_id",
                "azure_sas_token_id",
                "gcs_service_account_key_id",
            )
        )
        require_credentials_on_target_change(changed, has_new_credentials, has_bundled_secrets)
    try:
        # Owner's user_id keeps rotated secrets encrypted under the OWNER's key.
        update_cloud_connection(db, input_connection, db_connection.user_id)
    except ValueError:
        raise HTTPException(404, "Cloud connection not found") from None
    except Exception as e:
        logger.error(e)
        raise HTTPException(422, str(e)) from e
    return {"message": "Cloud connection updated successfully"}

logs

flowfile_core.routes.logs

Functions:

Name	Description
add_log	Adds a log message to the log file for a given flow_id.
add_raw_log	Adds a log message to the log file for a given flow_id.
format_sse_message	Format the data as a proper SSE message
stream_logs	Streams logs for a given flow_id using Server-Sent Events.

add_log(flow_id, log_message) async

Adds a log message to the log file for a given flow_id.

Source code in flowfile_core/flowfile_core/routes/logs.py

@router.post("/logs/{flow_id}", tags=["flow_logging"])
async def add_log(flow_id: int, log_message: str):
    """Adds a log message to the log file for a given flow_id."""
    flow = flow_file_handler.get_flow(flow_id)
    if not flow:
        raise HTTPException(status_code=404, detail="Flow not found")
    flow.flow_logger.info(log_message)
    return {"message": "Log added successfully"}

add_raw_log(raw_log_input) async

Adds a log message to the log file for a given flow_id.

Source code in flowfile_core/flowfile_core/routes/logs.py

@router.post("/raw_logs", tags=["flow_logging"])
async def add_raw_log(raw_log_input: schemas.RawLogInput):
    """Adds a log message to the log file for a given flow_id."""
    flow = flow_file_handler.get_flow(raw_log_input.flowfile_flow_id)
    if not flow:
        raise HTTPException(status_code=404, detail="Flow not found")
    flow_logger = flow.flow_logger
    node_id = raw_log_input.node_id if raw_log_input.node_id is not None else -1
    if raw_log_input.log_type == "INFO":
        flow_logger.info(raw_log_input.log_message, extra=raw_log_input.extra, node_id=node_id)
    elif raw_log_input.log_type == "WARNING":
        flow_logger.warning(raw_log_input.log_message, extra=raw_log_input.extra, node_id=node_id)
    elif raw_log_input.log_type == "ERROR":
        flow_logger.error(raw_log_input.log_message, extra=raw_log_input.extra, node_id=node_id)
    return {"message": "Log added successfully"}

format_sse_message(data) async

Format the data as a proper SSE message

Source code in flowfile_core/flowfile_core/routes/logs.py

async def format_sse_message(data: str) -> str:
    """Format the data as a proper SSE message"""
    return f"data: {json.dumps(data)}\n\n"

stream_logs(flow_id, idle_timeout=300, current_user=Depends(get_current_user_from_query)) async

Streams logs for a given flow_id using Server-Sent Events. Requires authentication via token in query parameter. The connection will close gracefully if the server shuts down.

Source code in flowfile_core/flowfile_core/routes/logs.py

@router.get("/logs/{flow_id}", tags=["flow_logging"])
async def stream_logs(flow_id: int, idle_timeout: int = 300, current_user=Depends(get_current_user_from_query)):
    """
    Streams logs for a given flow_id using Server-Sent Events.
    Requires authentication via token in query parameter.
    The connection will close gracefully if the server shuts down.
    """
    logger.info(f"Starting log stream for flow_id: {flow_id} by user: {current_user.username}")
    await asyncio.sleep(0.3)
    flow = flow_file_handler.get_flow(flow_id)
    logger.info("Streaming logs")
    if not flow:
        raise HTTPException(status_code=404, detail="Flow not found")

    log_file_path = flow.flow_logger.get_log_filepath()
    if not Path(log_file_path).exists():
        raise HTTPException(status_code=404, detail="Log file not found")

    class RunningState:
        def __init__(self):
            self.has_started = False

        def is_running(self):
            if flow.flow_settings.is_running:
                self.has_started = True
            return flow.flow_settings.is_running or not self.has_started

    running_state = RunningState()

    return StreamingResponse(
        stream_log_file(log_file_path, running_state.is_running, idle_timeout),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "Content-Type": "text/event-stream",
        },
    )

public

flowfile_core.routes.public

Classes:

Name	Description
GeneratedKey	Response model for the generate key endpoint.
SetupStatus	Response model for the setup status endpoint.

Functions:

Name	Description
docs_redirect	Redirects to the documentation page.
generate_key	Generate a new master encryption key.
get_setup_status	Get the current setup status of the application.

GeneratedKey pydantic-model

Bases: BaseModel

Response model for the generate key endpoint.

Show JSON schema:

{
  "description": "Response model for the generate key endpoint.",
  "properties": {
    "key": {
      "title": "Key",
      "type": "string"
    },
    "instructions": {
      "title": "Instructions",
      "type": "string"
    }
  },
  "required": [
    "key",
    "instructions"
  ],
  "title": "GeneratedKey",
  "type": "object"
}

Fields:

• key (str)
• instructions (str)

Source code in flowfile_core/flowfile_core/routes/public.py

class GeneratedKey(BaseModel):
    """Response model for the generate key endpoint."""

    key: str
    instructions: str

SetupStatus pydantic-model

Bases: BaseModel

Response model for the setup status endpoint.

Show JSON schema:

{
  "description": "Response model for the setup status endpoint.",
  "properties": {
    "setup_required": {
      "title": "Setup Required",
      "type": "boolean"
    },
    "master_key_configured": {
      "title": "Master Key Configured",
      "type": "boolean"
    },
    "mode": {
      "title": "Mode",
      "type": "string"
    },
    "projects_enabled": {
      "title": "Projects Enabled",
      "type": "boolean"
    },
    "projects_confined": {
      "title": "Projects Confined",
      "type": "boolean"
    },
    "git_available": {
      "title": "Git Available",
      "type": "boolean"
    }
  },
  "required": [
    "setup_required",
    "master_key_configured",
    "mode",
    "projects_enabled",
    "projects_confined",
    "git_available"
  ],
  "title": "SetupStatus",
  "type": "object"
}

Fields:

• setup_required (bool)
• master_key_configured (bool)
• mode (str)
• projects_enabled (bool)
• projects_confined (bool)
• git_available (bool)

Source code in flowfile_core/flowfile_core/routes/public.py

class SetupStatus(BaseModel):
    """Response model for the setup status endpoint."""

    setup_required: bool
    master_key_configured: bool
    mode: str
    projects_enabled: bool
    projects_confined: bool
    git_available: bool

docs_redirect() async

Redirects to the documentation page.

Source code in flowfile_core/flowfile_core/routes/public.py

@router.get("/", tags=["admin"])
async def docs_redirect():
    """Redirects to the documentation page."""
    return RedirectResponse(url="/docs")

generate_key() async

Generate a new master encryption key.

Source code in flowfile_core/flowfile_core/routes/public.py

@router.post("/setup/generate-key", response_model=GeneratedKey, tags=["setup"])
async def generate_key():
    """Generate a new master encryption key."""
    key = generate_master_key()
    instructions = (
        f'Add to your .env file:\n  FLOWFILE_MASTER_KEY="{key}"\n\n'
        "Then restart: docker-compose down && docker-compose up"
    )
    return GeneratedKey(key=key, instructions=instructions)

get_setup_status() async

Get the current setup status of the application.

Source code in flowfile_core/flowfile_core/routes/public.py

@router.get("/health/status", response_model=SetupStatus, tags=["health"])
async def get_setup_status():
    """Get the current setup status of the application."""
    # Default to "tauri" — `flowfile run ui` (no env) and the Tauri desktop shell
    # both want desktop-mode auth (auto-auth, no setup wizard). The frontend
    # accepts "electron" | "tauri" | "desktop" all as desktop mode, so existing
    # deployments that hard-code FLOWFILE_MODE=electron keep working.
    mode = os.environ.get("FLOWFILE_MODE", "tauri")
    master_key_ok = is_master_key_configured()
    return SetupStatus(
        setup_required=not master_key_ok,
        master_key_configured=master_key_ok,
        mode=mode,
        projects_enabled=(not settings.is_docker_mode()) or bool(settings.FLOWFILE_ENABLE_PROJECTS),
        projects_confined=not settings.is_electron_mode(),
        git_available=git_available(),
    )

secrets

flowfile_core.routes.secrets

Manages CRUD (Create, Read, Update, Delete) operations for secrets.

This router provides secure endpoints for creating, retrieving, and deleting sensitive credentials for the authenticated user. Secrets are encrypted before being stored and are associated with the user's ID.

Functions:

Name	Description
create_secret	Creates a new secret for the authenticated user.
delete_secret	Deletes a secret by name for the authenticated user.
get_secret	Retrieves a specific secret by name for the authenticated user.
get_secrets	Retrieves all secret names for the currently authenticated user.

create_secret(secret, current_user=Depends(get_current_active_user), db=Depends(get_db)) async

Creates a new secret for the authenticated user.

The secret value is encrypted before being stored in the database. A secret name must be unique for a given user.

Parameters:

Name	Type	Description	Default
secret	SecretInput	A SecretInput object containing the name and plaintext value of the secret.	required
current_user		The authenticated user object, injected by FastAPI.	Depends(get_current_active_user)
db	Session	The database session, injected by FastAPI.	Depends(get_db)

Raises:

Type	Description
HTTPException	400 if a secret with the same name already exists for the user.

Returns:

Type	Description
Secret	A Secret object containing the name and the encrypted value.

Source code in flowfile_core/flowfile_core/routes/secrets.py

@router.post("/secrets", response_model=Secret)
async def create_secret(
    secret: SecretInput, current_user=Depends(get_current_active_user), db: Session = Depends(get_db)
) -> Secret:
    """Creates a new secret for the authenticated user.

    The secret value is encrypted before being stored in the database. A secret
    name must be unique for a given user.

    Args:
        secret: A `SecretInput` object containing the name and plaintext value of the secret.
        current_user: The authenticated user object, injected by FastAPI.
        db: The database session, injected by FastAPI.

    Raises:
        HTTPException: 400 if a secret with the same name already exists for the user.

    Returns:
        A `Secret` object containing the name and the *encrypted* value.
    """
    user_id = 1 if os.environ.get("FLOWFILE_MODE") == "electron" else current_user.id

    existing_secret = (
        db.query(db_models.Secret)
        .filter(db_models.Secret.user_id == user_id, db_models.Secret.name == secret.name)
        .first()
    )

    if existing_secret:
        raise HTTPException(status_code=400, detail="Secret with this name already exists")

    stored_secret = store_secret(db, secret, user_id)
    _project_sync_secret(user_id)
    return Secret(
        name=stored_secret.name,
        value=stored_secret.encrypted_value,
        user_id=str(user_id),
        id=stored_secret.id,
        access=_OWNER_ACCESS,
    )

delete_secret(secret_name, current_user=Depends(get_current_active_user), db=Depends(get_db)) async

Deletes a secret by name for the authenticated user.

Parameters:

Name	Type	Description	Default
secret_name	str	The name of the secret to delete.	required
current_user		The authenticated user object, injected by FastAPI.	Depends(get_current_active_user)
db	Session	The database session, injected by FastAPI.	Depends(get_db)

Returns:

Type	Description
None	An empty response with a 204 No Content status code upon success.

Source code in flowfile_core/flowfile_core/routes/secrets.py

@router.delete("/secrets/{secret_name}", status_code=204)
async def delete_secret(
    secret_name: str, current_user=Depends(get_current_active_user), db: Session = Depends(get_db)
) -> None:
    """Deletes a secret by name for the authenticated user.

    Args:
        secret_name: The name of the secret to delete.
        current_user: The authenticated user object, injected by FastAPI.
        db: The database session, injected by FastAPI.

    Returns:
        An empty response with a 204 No Content status code upon success.
    """
    user_id = 1 if os.environ.get("FLOWFILE_MODE") == "electron" else current_user.id
    delete_secret_action(db, secret_name, user_id)
    _project_sync_secret(user_id)
    return None

get_secret(secret_name, current_user=Depends(get_current_active_user), db=Depends(get_db)) async

Retrieves a specific secret by name for the authenticated user.

Note: This endpoint returns the secret name and metadata but does not expose the decrypted secret value.

Parameters:

Name	Type	Description	Default
secret_name	str	The name of the secret to retrieve.	required
current_user		The authenticated user object, injected by FastAPI.	Depends(get_current_active_user)
db	Session	The database session, injected by FastAPI.	Depends(get_db)

Raises:

Type	Description
HTTPException	404 if the secret is not found.

Returns:

Type	Description
Secret	A Secret object containing the name and encrypted value.

Source code in flowfile_core/flowfile_core/routes/secrets.py

@router.get("/secrets/{secret_name}", response_model=Secret)
async def get_secret(
    secret_name: str, current_user=Depends(get_current_active_user), db: Session = Depends(get_db)
) -> Secret:
    """Retrieves a specific secret by name for the authenticated user.

    Note: This endpoint returns the secret name and metadata but does not
    expose the decrypted secret value.

    Args:
        secret_name: The name of the secret to retrieve.
        current_user: The authenticated user object, injected by FastAPI.
        db: The database session, injected by FastAPI.

    Raises:
        HTTPException: 404 if the secret is not found.

    Returns:
        A `Secret` object containing the name and encrypted value.
    """
    user_id = 1 if os.environ.get("FLOWFILE_MODE") == "electron" else current_user.id

    db_secret = (
        db.query(db_models.Secret)
        .filter(db_models.Secret.user_id == user_id, db_models.Secret.name == secret_name)
        .order_by(db_models.Secret.id.asc())
        .first()
    )

    if db_secret:
        return Secret(
            name=db_secret.name,
            value=db_secret.encrypted_value,
            user_id=str(db_secret.user_id),
            id=db_secret.id,
            access=_OWNER_ACCESS,
        )

    # Shared-only match: metadata without the value (mirrors get_encrypted_secret's
    # own-shadows-shared, lowest-id-wins resolution).
    shared = [row for row in _shared_secret_rows(db, user_id) if row.name == secret_name]
    if shared:
        return shared[0]

    raise HTTPException(status_code=404, detail="Secret not found")

get_secrets(current_user=Depends(get_current_active_user), db=Depends(get_db)) async

Retrieves all secret names for the currently authenticated user.

Note: This endpoint returns the secret names and metadata but does not expose the decrypted secret values.

Parameters:

Name	Type	Description	Default
current_user		The authenticated user object, injected by FastAPI.	Depends(get_current_active_user)
db	Session	The database session, injected by FastAPI.	Depends(get_db)

Returns:

Type	Description
	A list of Secret objects, each containing the name and encrypted value.

Source code in flowfile_core/flowfile_core/routes/secrets.py

@router.get("/secrets", response_model=list[Secret])
async def get_secrets(current_user=Depends(get_current_active_user), db: Session = Depends(get_db)):
    """Retrieves all secret names for the currently authenticated user.

    Note: This endpoint returns the secret names and metadata but does not
    expose the decrypted secret values.

    Args:
        current_user: The authenticated user object, injected by FastAPI.
        db: The database session, injected by FastAPI.

    Returns:
        A list of `Secret` objects, each containing the name and encrypted value.
    """
    user_id = current_user.id

    db_secrets = db.query(db_models.Secret).filter(db_models.Secret.user_id == user_id).all()

    secrets = []
    for db_secret in db_secrets:
        secrets.append(
            Secret(
                name=db_secret.name,
                value=db_secret.encrypted_value,
                user_id=str(db_secret.user_id),
                id=db_secret.id,
                access=_OWNER_ACCESS,
            )
        )
    secrets.extend(_shared_secret_rows(db, user_id))

    return secrets

---