Prompt Sufficiency: A Missive for the Managerial Class

AI Assisted Work: A Missive For the Managerial Class

The Smart Employee

“It doesn't make sense to hire smart people and then tell them what to do, We hire smart people so they can tell us what to do.” - Steve Jobs

This famous quote is bound to provoke some discomfort: What is the meaning of leadership if it dictates neither what nor how? The answer, according to Mr. Jobs, is that effective leadership is sometimes accomplished through yielding control, rather than holding on to it. By delegating aspects of the what, we are able to leverage the experience of others. This principle is an acknowledgment of the human limitations of the leader - no matter how capable - and a strong endorsement of the value of teamwork, humility, expertise, and thrift! After all, hiring capable people (read that as: expensive people) and reducing them to automatons is a tremendous waste.

On the topic of waste, it has been widely noted that large gains in productivity from Artificial Intelligence have been slower to materialize than expected. Having utilized LLMs quite effectively, and having carefully observed others stumble, I have a great deal of conviction that the gross misapplication of the above leadership principle is largely to blame. The problem is actually quite simple: we treat LLMs like smart employees, but that is not precisely what they are, appearances notwithstanding. It is becoming apparent that the wisdom of Mr. Jobs simply does not apply here.

To make the point a bit clearer, I think it is helpful here to meditate on what a smart employee actually does, because “smart” in the purely academic sense doesn’t quite fit. The opening to the famous 1899 essay “Carry a message to Garcia” by Elbert Hubbard is a bit closer - it is one who can be entrusted with accomplishing a task with little fuss.

Someone said to the President, “There is a fellow by the name of Rowan [Lt. Andrew S. Rowan, U.S. Army] will find Garcia for you, if anybody can.”Rowan was sent for and given a letter to be delivered to Garcia. How the “fellow by the name of Rowan” took the letter, sealed it up in an oilskin pouch, strapped it over his heart, in four days landed by night off the coast of Cuba from an open boat, disap­peared into the jungle, and in three weeks came out on the other side of the Island, having trav­ersed a hostile country on foot, and delivered his letter to Garcia—are things I have no special de­sire now to tell in detail. The point that I wish to make is this: McKinley gave Rowan a letter to be delivered to Garcia; Rowan took the letter and did not ask, “Where is he at?”By the Eternal! there is a man whose form should be cast in deathless bronze and the statue placed in every college of the land. It is not book-learning young men need, nor instruction about this and that, but a stiffening of the vertebrae which will cause them to be loyal to a trust, to act promptly, concentrate their ener­gies: do the thing—”Carry a mes­sage to Garcia.”

Raw cognitive ability here is pure table stakes. It is the intellect which makes competence possible; it is the ability to “fill in the details” which makes the smart employee a true operator - a worthy trustee of the what. Rowan knew of the necessity of a map and therefore acquired one, understood himself well enough to know if any refresher of trail-craft was needed, anticipated obstacles and prepared backup plans, and so on. One could delegate the what to such an employee, but such an employee could easily execute tasks as well.

The valuable activity here is gap-filling: working out the details that lie in the jungle between the letter and the general. The human being knows when it has enough information; the human being knows when it does not. It knows when it is safe to guess. It knows when guessing is dangerous. All these abilities are brought to bear to create the “experience” of the effective report - a clean, no-fuss operator.

Interacting with Agentic AI creates the impression that it is that kind of employee because it carries so many of the same signifiers: breadth, encyclopedic depth, speed, and some indicators of taste and judgment. This indicates intelligence - which, after all, is a common trait amongst productive people. But here’s the reality: machine intelligence - in its current incarnation - must be utilized, not unleashed.

My precise claim is that there are two points of failure that are compounding, both of which are managerial rather than technological. The first is that our prior experiences with direct reports have made us blind to the sheer amount of information - or context - required for successful execution of anything non-trivial. The second is that LLMs are essentially starved of this context by the narrow band of communication available to them: the prompt. The remedy to both problems is prompt sufficiency. I’ll develop this concept in more detail shortly.

Here’s the crux: The context we are accustomed to human beings acquiring independently and without explicit instruction to do so is not automatically acquired by AI Agents. We have become so accustomed to this invisible human activity that we have become blind to its necessity, and surprised when failure to provide necessary context results in failure.

An Example

Here is an example: Please digest the last five quarterly earnings reports and write a memo suggesting a strategy which addresses any shortfalls you see in our current approach.

This is a perfectly reasonable request for a capable employee, but there are several unstated yet load-bearing assumptions. For example: What is the organization’s capital risk-management philosophy? This is unlikely to be spelled out in the quarterlies, but is likely to influence what is considered an actual shortfall in strategy versus an acceptable risk.

The Agent AI’s report will appear to be well-thought-out and thorough. It will reference the appropriate figures. We take this as a signifier of the qualities that a strategy memo ought to have: diligence, thoughtfulness, thoroughness. But those are only signifiers: is it truly fit-for-purpose?

Here is another example: “Please digest the last five quarterly earnings reports and write a memo suggesting a strategy which addresses any shortfalls you see in our current approach. Bear in mind that we consider any indications of a future liquidity crunch to be unacceptable, and utilizing available credit lines is an acceptable recourse as long as recurring revenue remains above …”

Here we are providing a great deal more of the context required to produce a result which is fit for the situation at hand. There was no way that the AI could have acquired this additional information from sources like general knowledge because it pertains to how the executive responds to risk.

Prompt Sufficiency

A prompt is sufficient when an agent is able to complete the work product without arbitrary choice.

Sufficiency is not a qualitative property - it is not a function of format, appearance, taste or style. The presence of imperatives (”Make no mistakes!”), role-playing directives (”You are a terraform engineer”), or descriptors (”thoroughly…”) do not add to or take away from sufficiency.

Sufficiency is related to the idea of information content - it is a combination of deducibility and the absence of a requirement for mind-reading on the part of the machine. Given the prompt and the available information, is the agent required to know something that only you would know - assuming, of course, that you have an understanding of what needs to be done? Or, must the agent be unduly creative, clairvoyant, or decisive on your behalf? If so, the prompt is probably insufficient.

Here are examples of insufficient prompts:

• Design an appealing dashboard (Is it possible to deduce whether a dashboard will be appealing to you without reading your mind?)
• Create a new strategy for revenue growth based on the last five quarters' financials (Have you expressed a criterion for what you consider to be an acceptable strategy? Or is this known only to yourself?)

It is possible to develop a nose for insufficiency by making a practice of mentally elaborating on the task at hand. Suppose you are a stranger (assume you have expert background in whatever topic), and that you are asked to complete the work. Here is the question: Is it possible to deduce how to complete the work with only expert domain knowledge and the prompt? Or are there particulars to the task that cannot be deduced from subject matter expertise? As the asker, ask yourself: What needs to be specified that is particular to the task?

Let’s compare two prompts:

1. “Build a login screen using the brand kit I provided earlier”.
2. “Build a login screen using the brand kit I provided earlier, with a visual design matching the most popular app on the App Store”.

Despite neither prompt containing much detail, the second prompt is sufficient; the first is not. Why?

This is an exercise for the reader.