Hermes architecture: a 10-minute tour (as of 2026-10-17)

After Dark — Nightly Serendipity Dispatch Agent

A single Python agent that runs nightly and produces a serendipity dispatch: one surprising discovery, one deep dive, one build idea, and one markets note. Not morning news — things the owner wouldn't have found on their own.

Feed this document to an AI assistant (Muse, etc.) and ask it to implement the agent. Everything below is a specification, not code.

1. What it does (pipeline)

Discovery (no LLM) → interest-aware ranking → URL verification →
LLM curation (picks by index) → deterministic rendering → dispatch + teaser

Key design rule: the LLM only selects (returns indices). All fetching, rendering, and link insertion is deterministic code. The model can never hallucinate a URL because it never writes one.

2. Schedule

• Runs once nightly at 2:00 AM local time via cron.
• Single-flight: use a lock file so overlapping runs are impossible.
• Timeout budget: ~15 minutes max (LLM retries are the long pole).

3. Discovery sources (no LLM, pure HTTP)

Each candidate is a dict: {title, url, source, summary, published_at}.

Respectful fetching: declare a User-Agent identifying the bot. For SEC EDGAR (if used), the UA must contain an email-shaped contact or requests get 403.

4. Interest-aware ranking (no LLM)

• Maintain a keyword list representing the owner's interests (e.g. AI agents, local LLMs, prediction markets, options, FPL, home automation).
• Score each candidate: keyword hits in title/summary, recency boost, source diversity cap (max N per source so one source can't dominate).
• Dedupe against a seen.json of URLs already dispatched (persist across runs).

5. URL verification (no LLM)

• HEAD/GET each shortlisted URL with a short timeout (~8s).
• Drop dead links before they reach the LLM. A dispatch with a 404 is a failure.

6. LLM curation

Send the verified shortlist (numbered, with titles + summaries) to the LLM with a prompt like:

"You are curating a nightly serendipity dispatch. Pick the 4 most surprising, non-obvious items from the list below. Return ONLY the indices, e.g. 3, 7, 12, 18. One pick each for: a discovery (something new), a deep dive (worth 20 minutes), a build idea (something to make), a markets note (interesting prediction-market or options movement). Skip anything that reads like press-release hype."

• The model returns indices only. Map them back to candidates in code.
• Retry the LLM call up to 3 times on failure; after that, fall back to the top-ranked deterministic picks so the dispatch still ships.

7. Markets note (deterministic)

• Query the Kalshi public API (anonymous, read-only) for the biggest price movers in the last 24h.
• Render the top 5 movers with before/after prices. No LLM needed, no trading, no orders — read-only market data.

8. Wonderings rotation

• Keep a wonderings.md file: a running list of open questions/curiosities.
• Each night, pick the oldest uncovered wondering for the deep-dive slot, then mark it covered. When the list is exhausted, the LLM generates fresh wonderings from recent discoveries.
• Never repeat a covered wondering.

9. Output

Two files per night in data/dispatches/YYYY-MM-DD.md:

1. Dispatch — markdown with the 4 sections, each with title, link, 2–3 sentence why-it-matters, and source attribution.
2. Teaser — a 2-line push notification text (sent via ntfy, see below).

Also write data/latest.md (symlink or copy of the newest dispatch) and data/teaser.txt for the notification pipeline.

10. Notifications

• Send the teaser via ntfy (self-hosted or ntfy.sh) when the dispatch is ready.
• If the run fails entirely, send a failure notice instead of staying silent.
• Never send "ready" if the dispatch wasn't actually written.

11. State files (all JSON, atomic writes)

Write state atomically (temp file + rename), never in-place rewrites.

12. Failure discipline (non-negotiable)

• Never mark a candidate seen until it is successfully dispatched.
• Every retry loop: attempt counting, a ceiling, a fallback, a notification.
• If all LLM calls fail, still write the dispatch from deterministic picks — a thinner dispatch beats a missing one.
• Log every run; a silent failure is the worst outcome.

13. Dependencies

• Python 3 stdlib + requests. That's it.
• One LLM API endpoint (any OpenAI-compatible API works — LiteLLM proxy, OpenAI, Groq, local Ollama, etc.).
• Cron. A lock file. A directory.

14. What it is NOT

• Not a news aggregator — no headlines the owner already saw.
• Not a trading bot — the markets note is observational only.
• Not a chatbot — it produces one artifact per night, no conversation.

Hermes architecture: a 10-minute tour

Status as of 2026-10-17. Diagrams are Mermaid and render on GitHub. Facts that drift (agent list, schedules) are in CLAUDE.md and manifests/; this document explains how it fits together and why.

1. What Hermes is

About 15 small, cron-driven Python agents on one free Oracle Cloud ARM VM, plus a phone-first dashboard, a private push-notification server, and two paper forecasting/trading labs. It watches email, school, flyers, fantasy football, prediction markets and options, and tells the owner only what matters. Everything is $0: free VM, free-tier model APIs, local Ollama.

flowchart LR
  subgraph World["Outside world (read-only)"]
    MAIL[iCloud mail
IMAP read-only]
    WEB[School site, flyers,
FPL API, ESPN, FRED]
    MKT[Kalshi + CBOE + Yahoo
anonymous market data]
  end
  subgraph VM["One OCI Always Free ARM VM"]
    CRON[cron
generated from manifests]
    AG[~15 agents + 2 labs]
    LLM[LiteLLM proxy
free lanes + local Ollama]
    DASH[Dashboard
auth_server + static PWA]
    NTFY[Self-hosted ntfy]
    STORE[(SQLite files
per agent)]
  end
  OWNER((Owner's phone))
  MAIL --> AG
  WEB --> AG
  MKT --> AG
  CRON --> AG
  AG <--> LLM
  AG --> STORE
  AG --> NTFY --> OWNER
  STORE --> DASH
  DASH -- "Cloudflare tunnel" --> OWNER

Loading

Why one VM: it is free, simple to back up, and a failure has one place to look. Nothing needs more.

2. Repository map

3. How a job runs, and how code gets deployed

flowchart TD
  M["agent.toml / manifests/*.toml
schedule in PT"] --> G[tools/gen_crontab.py
explicit UTC, DST-aware]
  G --> S[tools/sync_crontab.py
managed block only]
  S --> C[cron line]
  C --> R["runner/hermes_run.py
flock + manifest timeout"]
  R --> A[agent script]
  R --> DB[(runner/data/runs.db
one row per run)]
  A --> OUT[agent data/ + card.hermes.json]
  OUT --> B[dashboard/build_data.py]
  B --> J[public/data/*.json]
  J --> PWA[static PWA]
  DB --> CAN[canary liveness checks]

Loading

flowchart LR
  DEV[Edit + test locally] --> PUSH[git push master]
  PUSH --> CI["GitHub CI
compile, unittest, bash -n,
gitleaks, cron goldens,
paper-only / IMAP / canary greps"]
  CI -- green --> PULL["VM cron every 15 min
cd/auto-pull.sh (ff-only)"]
  PULL --> SYNC[sync crontab] --> RESTART[restart hermes-dash if dashboard/ changed]
  CI -- red --> HOLD[hold + alert once]

Loading

Decisions and why

• Schedules live in manifests, crontab is generated. Ubuntu cron ignores CRON_TZ, and a hand-edited crontab drifts. So PT is authored, UTC is generated, and golden files for both PDT and PST are checked in CI, so a DST change needs no human.
• Pull-based deploy, CI-green only, fast-forward only. The VM never accepts a red or conflicting commit and never needs inbound access. The cost is up to ~15 minutes of deploy latency, which nothing here cares about.
• stdlib-only agents. No dependency drift on a VM nobody wants to babysit.

4. Models: lanes, not providers

flowchart LR
  AGENT[Agent] -- "task label + lane name" --> PROXY[LiteLLM proxy
stateless]
  PROXY --> D["agent-default
Groq 120b > Groq 20b > NVIDIA > Pollinations > Ollama"]
  PROXY --> RS["agent-reason
Groq 120b > Groq 20b > NVIDIA"]
  PROXY --> V["agent-vision
Gemini Flash only"]
  PROXY --> CH["agent-cheap
local Ollama only"]
  PROXY --> USAGE[(usage.db
every call logged)]
  MAILAG[email-brief / home-chief] -- "local only" --> OLL[Ollama on the VM]

Loading

• Agents name a lane, never a model. Providers can be swapped or reordered in one file and failover is automatic (3 strikes then 5-minute cooldown). Why: free tiers rate-limit and disappear; the stack must survive that.
• Vision never fails over to text models, so an image can never be silently mis-read.
• Family email never leaves the VM. Email agents use local Ollama by policy and a CI grep enforces read-only IMAP.
• Task labels are mandatory so usage.db shows which agent burns which quota.

5. Notifications and the dashboard

• One sender: hermes.notify.send(title, msg, channel="alerts"|"family") to the self-hosted ntfy server. A test forbids posting to public ntfy.sh (only the heartbeat ping is exempt). Why: topics are secrets; one choke point makes redaction and rate limits enforceable.
• Dashboard: auth_server.py on 127.0.0.1:8899 behind a Cloudflare tunnel, ntfy-code login, two roles. The family role can reach only /family/ and the flyer upload. Pages: Control Room /, /classic/, /family/, /chat/, /fpl/, /options/ (with a Lab learning panel: lessons and memory index), /proposals/ (the approval inbox, full role only).
• Contract-first data: agents emit versioned JSON (hermes.card/1, hermes.fleet/1, hermes.equity/1, ...), validated by hermes/cards.py. Why: the UI can change without touching agents, and a bad card fails validation instead of breaking the page.

6. The forecasting labs (the interesting part)

The Hermes framework underneath

The labs run on Hermes Agent by Nous Research (open source; the VM tracks upstream, v0.21.5). It supplies what we would otherwise hand-build: skills (SKILL.md playbooks), an agent loop with memory, MCP tool support and a job scheduler. Our cron stays the clock; the framework does the thinking.

What actually runs on it (checked on the VM):

Everything else (fpl, email-brief, home-chief, school, flyer-events, after-dark, lane-eval, ledger, canary, hygiene, vm-backup, memory, chief) is plain cron plus our Python, not framework-hosted. The framework's own messaging gateway service is not running; notifications go through our ntfy path instead.

How much of the framework we use, and why not more. In use: scheduler (as a trigger), skills, MCP, the LiteLLM provider, and now memory-backed recall. Deliberately off: delegation (more calls than an ~8k tokens/min free tier can pay for), web/browser/ terminal/file/code toolsets (no powerful tools next to the ledger), and the messaging gateway (ntfy does the job). Every tool schema costs tokens on every call, so the lab agents run with 10 tools, and the free-tier token budget, not missing features, is the real limit.

Both labs are paper only. There is no order code, no broker login; a CI grep and a canary invariant enforce it.

flowchart TD
  CRONT["cron = the clock only"] --> FW[fw.py wrapper]
  FW --> AGJ["Hermes framework AGENT job
skill + memory + MCP tools"]
  FW -. "no ledger evidence / lane down" .-> FB["deterministic --no-agent
fallback job"]
  AGJ --> MCP["stdlib MCP server
market, risk, fills, ledger"]
  FB --> MCP
  MCP --> L[("append-only ledger
integer cents / kalshi.db")]
  L --> SC[Brier scoring vs market baseline]
  SC --> REV[weekly rule-based review
tunes policy inside fixed envelopes]
  REV --> MCP
  L --> INV["loop-closure invariant
re-checked independently by canary"]

Loading

Decisions and why

• Agents propose, deterministic code disposes. Models never touch money logic: sizing, risk limits, fills and ledger writes are plain code behind MCP tools. A confused model can waste tokens but cannot corrupt the books.
• Always a fallback job. If the agent leaves no ledger evidence, a deterministic run still happens, so the clock never stalls.
• The ledger is append-only with a loop-closure invariant (every decision ends in a recorded outcome). Calibration is only trustworthy if nothing is rewritten.
• "Beat the baseline or it doesn't count." Every forecast stores model P, market P and later resolution at decision time. Fair tests only: markets whose outcome is already effectively known (games underway, prices near 0 or 1) are skipped.
• No self-congratulation. Docs may not say "live", "working" or "learning" until the phase gates are met.
• Policy tuning is bounded: weekly, rule-based, inside fixed envelopes, versioned.

7. Safety rails (why it can run unattended)

flowchart LR
  subgraph Prevent["Prevent (CI)"]
    P1[gitleaks secret scan]
    P2[grep: no order code]
    P3[grep: IMAP no STORE/DELETE/MOVE]
    P4[manifest + cron goldens PDT/PST]
    P5[test: no public ntfy.sh]
  end
  subgraph Detect["Detect (canary)"]
    D1[run liveness vs manifest]
    D2[loop-closure + cash reconciliation]
    D3[paper-only / IMAP re-check]
  end
  subgraph Respond["Respond"]
    R1[ntfy alert]
    R2["pause agent only for
paper-only, IMAP, outbound-mail rules"]
    R3[propose fix in a throwaway worktree
never auto-apply]
  end
  Prevent --> Detect --> Respond

Loading

• The canary is propose-only. Its fixes land on a canary/* branch built in a throwaway git worktree, never the live checkout, and it cannot edit its own policy files. It can pause an agent only for a short list of safety rules. Why: an autonomous fixer with write access to production is a bigger risk than the bugs it fixes.
• Backups: nightly encrypted vm-backup, pushed to OCI Object Storage via instance principal, create-only (it cannot delete its own backups). RESTORE.md is the runbook; the options lab also does a weekly restore drill.
• Secrets (.env, ntfy topics, tokens, Kalshi key, FPL team id) are never committed, printed or logged; crontabs are redacted when shown.
• The VM checkout is never reset or cleaned. It has intentional drift; ff-only pulls tolerate that.

8. Memory, Chief of Staff and the PR-writing canary

flowchart LR
  SRC["Cards, briefs, ledger rationales,
run logs, derived email briefs"] --> IDX["memory/index.py
nightly 2:30am PT"]
  IDX --> RED[redact secrets, emails, phones]
  RED --> MEM[("memory.db
SQLite FTS5 + optional embeddings")]
  MEM --> API["hermes.memory
add / seen / mark_pushed / search"]
  API --> CHIEF["chief/ ranks 'needs you today'
7:00am + 6:00pm PT"]
  CARDS[fleet + agent cards] --> CHIEF
  CHIEF -- "max 3/run, 6/day, quiet hours" --> NT[ntfy push + Control Room card]
  API --> CHAT["chat retrieval (planned)
private sources excluded"]
  INC[Agent failure] --> CAN["canary propose_fix
CANARY_PR_MODE off / local / draft"]
  CAN --> BUN["patch + regression test + PR.md
reviewed by a human"]

Loading

Learning loop and the Proposals inbox

flowchart TD
  LEDGER[("options ledger
closed trades + outcomes")] --> IDXL["memory index_lab
lab_outcome / lab_debate / lab_review"]
  IDXL --> MEM[(memory.db)]
  MEM --> RECALL["recall tool (kind=recall)
5 compact hits"]
  RECALL --> DECIDE[decide agent]
  LEDGER --> REVIEW["weekly rule-based review
(no LLM)"]
  REVIEW --> LES["lessons.md / lessons.json
max 10 bullets, 1500 chars, 8-week expiry"]
  LES --> RECALL2["recall tool (kind=lessons)"] --> DECIDE
  LES --> PROP["tools/lab_skill_proposals.py
one LLM call, proposes a SKILL.md edit"]
  CAN[canary propose_fix
CANARY_PR_MODE=local bundle] --> INBOX
  PROP --> INBOX["Dashboard Proposals inbox
summary, evidence, diff, tests badge"]
  INBOX -- "Dismiss" --> DONE[status only]
  INBOX -- "Approve (full role only)" --> BR["throwaway worktree off master
push proposal/ + draft PR"]
  BR --> YOU[["You review and merge on GitHub"]]
  YOU --> CI[CI] --> DEPLOY[VM auto-pull + install.py refreshes skills]

Loading

• Lessons are statistics, not opinions. They come from ledger numbers with minimum evidence, so noise is not promoted; they expire so the agents do not anchor on stale conclusions.
• Nothing self-modifies. A proposal is a file until a human merges a PR. The approve button can only create a proposal/* branch from master (never force, never master), restricted to the proposal's own files; the framework installer ignores the proposals folder.
• Proposals and approvals are visible only to the full role. Skill proposals are generated outside options-lab/ because the canary forbids model calls inside lab code.

9. Decision log (the short version)

10. Where to go next

CLAUDE.md (fast path and hard rules) > README.md > tools/README.md (schedules) > dashboard/CONTRACT.md (data shapes) > LANES.md (models) > kalshi-hermes/PRD.md and options-lab/PRD.md (lab specs and decision logs) > RESTORE.md (disaster recovery) > CHANGELOG.md (history).