59 lines
5.4 KiB
Markdown
59 lines
5.4 KiB
Markdown
# Project map: colony
|
|
|
|
_Auto-generated by the `explorer` subagent. Factual overview — refresh via
|
|
explorer, not by hand._
|
|
|
|
## What it does
|
|
Self-organizing cluster of nodes on a LAN (README.md). Nodes discover each other (TCP probe on the discovery port over a configured IP range), gossip membership, and bid for "roles" (replicated storage, password vault, web dashboard, chat, ...), so roles keep running as nodes join/leave/die. Each node is one Python process; `colonyctl` controls the local node.
|
|
|
|
## How it works
|
|
- `main.py` boot order: validate discovery range -> `StateManager` (data/state.json) -> `Network` -> `RoleRegistry(available_roles)` -> `Dispatcher` + `RoleManager` (general + on-demand roles, init) -> TLS `Listener` bind -> unix `ControlSocket` bind -> threads for ControlSocket, Listener, `Discover.listen` -> `Gossip.join_or_bootstrap()`; if bootstrapped, `Foreman.tick()` runs to assign roles -> `start_all()` -> `ControlQueue.run_forever()` on main thread.
|
|
- Node-to-node: length-prefixed (4-byte `!I`, 16 MiB max) JSON frames over TLS (`colony/transport/framing.py`, `listener.py`, `whisperer.py` client). Payload `{"command": {role, name, args, kwargs, from}}`; `Dispatcher.dispatch` maps `Role.Command` to the role's `COMMANDS` enum and calls `execute_local`.
|
|
- Roles (`colony/core/roles/<name>/role.py`) subclass `Role` (`roles/role.py`): `CONTINUOUS` + `LOOP_SECONDS_WAIT` run `loop()` in a daemon thread; commands registered via `Command(callback, with_caller)`.
|
|
- Role assignment: `Foreman` proposes unclaimed roles to capable nodes; `RoleManager` (`roles/manager.py`) + `policy.py` (config `role_policy`) + `role_registry.py` (priority, hardware preferences, `minimal_percentage`, `on_demand`) decide who runs what. Hardware scoring: `core/hardware_scores.py`, `network/hardware.py`.
|
|
- Unreachable nodes: `Gateway` role relays commands (`network/relay.py`, builds `Gateway.Forward`; config `gateway.allow_commands` whitelists relayable commands).
|
|
- Local control: `colonyctl` -> unix socket `data/colony.sock` (`transport/control.py`) -> `ControlQueue` -> `ControlActions` (`core/control.py`).
|
|
|
|
## Tech stack
|
|
Python >=3.14 (`.python-version` 3.14), uv (`uv.lock`, `package = false`). Deps (pyproject.toml): dependency-injector 4.48.3, dynaconf 3.2.12, Flask 3.1.3, GPUtil 1.4.0, psutil 7.2.1, setuptools 80.9.0. Web UI static JS/CSS (no build step). Docker image python:3.14-alpine.
|
|
|
|
## Repositories / modules
|
|
Single flat package.
|
|
- `main.py` node entrypoint; `colonyctl.py` -> `colony/cli.py` (argparse CLI); `role_settings.py` role list; `config.json` defaults.
|
|
- `colony/core/` — `context.py` (RoleContext), `dispatcher.py`, `control.py`, `state_manager.py`, `logger/`, `exceptions.py`.
|
|
- `colony/core/network/` — `core.py` (Network), `membership.py`, `nodes.py`, `advice.py`, `hardware.py`, `relay.py`, `requests.py`.
|
|
- `colony/core/roles/` — Unit, Gossip, Foreman, Gateway, VaultWarden, Replicator, Explorer, WebView (Flask `app.py` + `static/`), Chatter; `manager.py`, `policy.py`.
|
|
- `colony/core/storages/` — `core.py` (Storage base, `storages/` dir, tar/hash helpers), `default.py`, `replica.py`, `chat.py`.
|
|
- `colony/transport/` — `listener.py`, `whisperer.py`, `discover.py`, `control.py`, `framing.py`.
|
|
- `colony/utils/` — `files.py` (atomic_write_json), `network.py` (hosts_in_range).
|
|
- `certs/`, `data/`, `storages/`, `logs/` — runtime dirs (git-ignored: certs, data, storages).
|
|
|
|
## Key implementations
|
|
- Startup/wiring: `main.py`.
|
|
- Role catalogue and priorities: `role_settings.py` (Unit, Gossip, Foreman, Gateway = GENERAL; VaultWarden, Replicator = CRITICAL (Replicator `minimal_percentage=51`); Explorer = HIGH; WebView, Chatter = LOW and on-demand).
|
|
- Role base/lifecycle: `colony/core/roles/role.py`, `manager.py`.
|
|
- Membership/join: `roles/gossip/role.py`, `network/membership.py`.
|
|
- Role placement: `roles/foreman/role.py`, `roles/policy.py`.
|
|
- CLI (`status`, `role add`, `webview`, `chat`): `colony/cli.py`.
|
|
- Settings: `colony/settings.py` (`setting("a.b")` raises KeyError when missing).
|
|
|
|
## Build, run, test
|
|
- Install/run: `uv sync` then `uv run python main.py`.
|
|
- CLI: `./colonyctl.py status [--json]`, `role add <Role>`, `webview`, `chat <node> [text]`.
|
|
- Docker: `docker build -t colony .`; `docker compose up -d` (host network; volumes /app/data, /app/storages, /app/logs; ports 65334/tcp, 28149/tcp, 8080/tcp).
|
|
- CI: `.gitea/workflows/docker.yml` builds and pushes `registry.ciomek.pl/colony:latest` on push to main.
|
|
- Tests: none in the repo.
|
|
|
|
## External integrations
|
|
No third-party services. Peers over TLS (cert/key at `certs/server.crt`, `certs/server.key`), TCP discovery on 28149, optional web dashboard on 8080 with basic credentials (`web.username/password`), private Docker registry in CI.
|
|
|
|
## Conventions & gotchas
|
|
- Config: `config.json` overridden by env `COLONY_<SECTION>__<KEY>` (dynaconf); `docker-compose.yml` lists keys.
|
|
- Discovery range defaults to 192.168.65.100-120; node exits if range invalid, listener or control socket cannot bind. Warns if node IP is 127.x.
|
|
- Role commands (`COMMANDS` enums) are for node-to-node calls; the CLI uses the control socket actions instead.
|
|
- State persisted atomically in `data/state.json`; a corrupt file is copied to `.corrupt` and reset.
|
|
- Loop exceptions in continuous roles are logged and retried.
|
|
- Roles with `on_demand=True` only start via `roles.on_demand` config or `colonyctl role add`.
|
|
|
|
<!-- generated-at-commit: 244fefd8530bf5cd01267a13f8aee34782c31934 | 2026-10-06 -->
|