gave a talk "owning your intelligence" - ty @sequoia @sonyatweetybird for having me

talked about harnesses and evals and the role they play in owning your intelligence

TLDR:
agents = model + harness + context
model - own the weights using something like @FireworksAI_HQ
context - memory needs to be portable
harness - needs to be model agnostic. also needs to be good at bringing right context to llm. "right" context may depend on your use case, which is why an open/configurable harness helps
how to use middleware in langchain/deepagents to configure your harness
how to use langgraph to fully own your cognitive architecture
why evals/obs matters - some quotes from @satyanadella

• “Create your private evals, because evals define what “good” looks like inside the organization”
• “retain ownership of your organization’s memory, traces, feedbacks, decisions, and institutional context”
• “you create your own continuous learning loop (i.e. hill climbing machine) that will allow your AI investments to compound the value of your firm”

how to use harbor for evals
tracing is important
evals + observability only matter so you can set up a data flywheel
data flywheel = run agent -> collect traces -> find interesting traces -> use those to improve
demo of langsmith engine which does exactly this!

full video: youtu.be/watch?v=HI2q3ci3…