colony/.claude/project-map.md

Project map: colony

Auto-generated by the explorer subagent. Factual overview — refresh via explorer, not by hand.

What it does

Self-organizing cluster of nodes on a LAN (README.md). Nodes discover each other (TCP probe on the discovery port over a configured IP range), gossip membership, and bid for "roles" (replicated storage, password vault, web dashboard, chat, ...), so roles keep running as nodes join/leave/die. Each node is one Python process; colonyctl controls the local node.

How it works

• main.py boot order: validate discovery range -> StateManager (data/state.json) -> Network -> RoleRegistry(available_roles) -> Dispatcher + RoleManager (general + on-demand roles, init) -> TLS Listener bind -> unix ControlSocket bind -> threads for ControlSocket, Listener, Discover.listen -> Gossip.join_or_bootstrap(); if bootstrapped, Foreman.tick() runs to assign roles -> start_all() -> ControlQueue.run_forever() on main thread.
• Node-to-node: length-prefixed (4-byte !I, 16 MiB max) JSON frames over TLS (colony/transport/framing.py, listener.py, whisperer.py client). Payload {"command": {role, name, args, kwargs, from}}; Dispatcher.dispatch maps Role.Command to the role's COMMANDS enum and calls execute_local.
• Roles (colony/core/roles/<name>/role.py) subclass Role (roles/role.py): CONTINUOUS + LOOP_SECONDS_WAIT run loop() in a daemon thread; commands registered via Command(callback, with_caller).
• Role assignment: Foreman proposes unclaimed roles to capable nodes; RoleManager (roles/manager.py) + policy.py (config role_policy) + role_registry.py (priority, hardware preferences, minimal_percentage, on_demand) decide who runs what. Hardware scoring: core/hardware_scores.py, network/hardware.py.
• Unreachable nodes: Gateway role relays commands (network/relay.py, builds Gateway.Forward; config gateway.allow_commands whitelists relayable commands).
• Local control: colonyctl -> unix socket data/colony.sock (transport/control.py) -> ControlQueue -> ControlActions (core/control.py).

Tech stack

Python >=3.14 (.python-version 3.14), uv (uv.lock, package = false). Deps (pyproject.toml): dependency-injector 4.48.3, dynaconf 3.2.12, Flask 3.1.3, GPUtil 1.4.0, psutil 7.2.1, setuptools 80.9.0. Web UI static JS/CSS (no build step). Docker image python:3.14-alpine.

Repositories / modules

Single flat package.

• main.py node entrypoint; colonyctl.py -> colony/cli.py (argparse CLI); role_settings.py role list; config.json defaults.
• colony/core/ — context.py (RoleContext), dispatcher.py, control.py, state_manager.py, logger/, exceptions.py.
• colony/core/network/ — core.py (Network), membership.py, nodes.py, advice.py, hardware.py, relay.py, requests.py.
• colony/core/roles/ — Unit, Gossip, Foreman, Gateway, VaultWarden, Replicator, Explorer, WebView (Flask app.py + static/), Chatter; manager.py, policy.py.
• colony/core/storages/ — core.py (Storage base, storages/ dir, tar/hash helpers), default.py, replica.py, chat.py.
• colony/transport/ — listener.py, whisperer.py, discover.py, control.py, framing.py.
• colony/utils/ — files.py (atomic_write_json), network.py (hosts_in_range).
• certs/, data/, storages/, logs/ — runtime dirs (git-ignored: certs, data, storages).

Key implementations

• Startup/wiring: main.py.
• Role catalogue and priorities: role_settings.py (Unit, Gossip, Foreman, Gateway = GENERAL; VaultWarden, Replicator = CRITICAL (Replicator minimal_percentage=51); Explorer = HIGH; WebView, Chatter = LOW and on-demand).
• Role base/lifecycle: colony/core/roles/role.py, manager.py.
• Membership/join: roles/gossip/role.py, network/membership.py.
• Role placement: roles/foreman/role.py, roles/policy.py.
• CLI (status, role add, webview, chat): colony/cli.py.
• Settings: colony/settings.py (setting("a.b") raises KeyError when missing).

Build, run, test

• Install/run: uv sync then uv run python main.py.
• CLI: ./colonyctl.py status [--json], role add <Role>, webview, chat <node> [text].
• Docker: docker build -t colony .; docker compose up -d (host network; volumes /app/data, /app/storages, /app/logs; ports 65334/tcp, 28149/tcp, 8080/tcp).
• CI: .gitea/workflows/docker.yml builds and pushes registry.ciomek.pl/colony:latest on push to main.
• Tests: none in the repo.

External integrations

No third-party services. Peers over TLS (cert/key at certs/server.crt, certs/server.key), TCP discovery on 28149, optional web dashboard on 8080 with basic credentials (web.username/password), private Docker registry in CI.

Conventions & gotchas

• Config: config.json overridden by env COLONY_<SECTION>__<KEY> (dynaconf); docker-compose.yml lists keys.
• Discovery range defaults to 192.168.65.100-120; node exits if range invalid, listener or control socket cannot bind. Warns if node IP is 127.x.
• Role commands (COMMANDS enums) are for node-to-node calls; the CLI uses the control socket actions instead.
• State persisted atomically in data/state.json; a corrupt file is copied to .corrupt and reset.
• Loop exceptions in continuous roles are logged and retried.
• Roles with on_demand=True only start via roles.on_demand config or colonyctl role add.