Tech Blog

What really determines whether an LLM can answer a strategy question is the system wrapped around it, or its harness. Let’s try a little exercise...

Start a new chat with Gemini, Claude or ChatGPT, and ask it the following:

How did it respond? In the best-case scenario, it will refuse to answer, stating that it doesn’t have access to the data it needs to answer the question. In my tests, Claude and Gemini passed; ChatGPT said it couldn’t answer but still tossed out a number anyway.

Now, prod it a little:

Sure! LLMs are token prediction machines, and the prediction of “good” answers is a highly subjective practice. A happy user results in thumbs-ups and positive feedback. Claude, Gemini and ChatGPT were all willing to take guesses with very little prodding (or none, in the case of ChatGPT).

Claude would still hedge its response, though:

An accurate number in five minutes? You are unshackled, Claude, do your thing.

After 71 web searches, ~1.77M tokens, 84 model turns and ~20 minutes, we get the following:

Wonderful. So, what have we shown here? First, an LLM by itself is bound by its training data, predicting words based on what it has seen. When asked about 2025 volumes, Claude readily admitted “…2025 is at or past my reliable knowledge cutoff...” Second, even with tool calls (like web search), an agent is only as good as the data and knowledge available to it. There is no magic. 

“Guessing” Is an LLM’s Innate Behavior

“Guessing” may be a little harsh, but it is pure prediction. From one of our Slack channels:

"An LLM isn’t looking things up by itself, it’s using chain of thought and predicting words based purely on its training set. All the other stuff you see is tool calls.”¹

Most people’s intuition is that an LLM “knows” things and occasionally misses (think back to the days when LLM’s couldn’t spell “strawberry” or do simple math.) The reality is a bit different. If you strip away the interface and all the other software surrounding an LLM, it performs one function: given a sequence of tokens, it produces the most likely “next” tokens. By itself, an LLM doesn’t “know” that 2+2 = 4. It has “seen” 2+2 a lot in its training set, and “predicts” that what should come next is “4.”   

Okay. But clearly this has been solved, right? Claude is pretty good at math these days. Yes, but probably not in the way that you think. First, the models have improved; most of them can reliably perform arithmetic operations due to clever training inputs and CoT reasoning. But the thing that has really changed is the harness, or the software system that surrounds and interacts with the model.

Look at the following interaction:

Notice that “ran a command” section? What Claude did was interpret your input (“what is 4,871 × 9,338”), translate that to a bash script and then ask the computer to calculate the answer. This is the same way you and I would behave when asked to perform some arbitrary multiplication. We would nod our head, pull out our calculator and then relay the answer back. This reasoning pattern is what has driven advances in the last 12-18 months. The models are absolutely improving, but model interaction with the surrounding system through tool calls has arguably been the more impactful innovation.

The same sort of tool call happens if you ask, "what time is it?"

What Surrounds the Model

This is why the question “Which model are you using?” is probably the least useful fact you can know about an AI tool. The choice of LLM matters at the margins, or in very specific cases, but generally the performance of frontier lab models (e.g., Opus, Sonnet) are all within shouting distance of each other, and the open-weight models aren’t that far behind either. What really determines whether a tool can answer a strategy question is the software wrapped around it.

As we mentioned earlier, that wrapper is called a harness. Long used in software engineering, a harness is generally described as the extra things built around a piece of code to run it, feed it inputs and use its outputs. The analogy is far older: picture a harness on a mule or a horse. It gives you a measure of control and allows you to channel input and output into useful work.

AI harness

An AI harness performs the same function: it decides what the model sees, what tools it can call, how long it can work and how the output is used and verified. All that functionality sits outside the model. Think about the difference between Photoshop and Photos, or GarageBand and Apple Music. They’re using the same files, but with vastly different intentions and outputs.

How Do These Parts Come Together? 

For an AI system to function well in your specific domain of work, it needs a combination of the following:

1) Domain-specific data

By now, frontier models have been trained on what is essentially the corpus of written, publicly available human knowledge. “Publicly” is the key word in that sentence. If the answers you need are in a dataset that is not publicly available, the LLM has not been trained on that data, nor does it know anything about that data. 

2) Domain-specific context

Given the vast breadth of information available on the internet, most of us naively assume that “publicly available knowledge” is equal to “all available knowledge.” This is far from the truth.

Practical, applied knowledge is largely tied up in internal documents and the heads of knowledge workers across industry. Frontier models are like studious college graduates, showing mastery only of the information available to them. A well-designed domain-specific AI system is more like an industry veteran, with knowledge and information that is gained from years of experience that is not available publicly.

3) Domain-specific harness

If a harness is the director of work, think about how different domains require different approaches. An extreme juxtaposition might be an athlete operating largely on instinct and reaction versus a tax auditor that moves slowly and deliberately. Commercial construction is a completely different world than residential construction. The harness is what orchestrates interplay between the question, the data and the context, and not all problems are solved in the same manner.

What does that look like in real life?  Here are some important components of the harness that we have built for our AI Strategist.

Planning and decomposition.  

"How has our cardiac market share shifted since 2024?" is not a single question. It is a combination of market definition, service-line definition, similarity comparison, trend comparison and data source selection. A planning layer resolves these foundational parts into ordered subtasks before generating any part of the answer. These subtasks are strong at any individual piece of work but weaker at holding a multi-part analysis. Planning is also where the system can not only resolve ambiguity from the user but suggest more relevant approaches. "Our market" could mean a geographical area or set of facilities. “Shifted” could mean a request for a simple increase/decrease percentage, or it could be a request for where the volume came and went. 

Context and memory.

A domain-specific context library must be available in a manner that is accessible to the harness. Methodology, caveats, definitions and prior art are important inputs to any analysis, but the breadth and depth of knowledge and context cannot simply be jammed into a long system prompt. If you are asking about a physician needs assessment, loading unrelated context (say, about price transparency data) could easily dilute what is most relevant.

Tool use.

Different subtasks are routed to specific components: planning, context retrieval, data retrieval, web search, chart generation and narrative. Each tool can be constrained and optimized on its own, and tools can be given multiple turns, skipped or added based on the output of prior tools.

How Things Are Evolving

The boundary between model and harness is not fixed. Newer models have been able to absorb functionality of what used to live only in a harness or tool. There is a limit, however, to this absorption. Frontier labs can train models to perform better with available knowledge, but they cannot train one to know how your organization defines a service line, or that a data quality problem in one of your markets is due to a clearinghouse issue.

Very little of the internal workings of these systems are obvious or transparent to you, the end user.
A general-purpose LLM with a general-purpose harness can answer general questions with great competence and confidence, and it is very easy to be lulled into a sense of security with them. It is important to remember the world of known-knowns, known-unknowns and unknown-unknowns. I propose adding one more for LLMs, which is the unknown-knowns: examples where humans have some piece of knowledge and understand it, but it is simply and completely unknown to general-purpose LLMs.  

This, of course, is where we are trying to expand the bubble of knowledge at Trilliant Health. Build the right harness, derive context and knowledge from tens of thousands of past analyses and feed that into a system with the right LLM and access to the right data. If you do all that, and ask the question I posed at the outset, you will get an answer that is a bit more useful.

Coming soon!