Using an LLM to Process Inconsistent Text

Developers are problem solvers. We are presented with challenging problems all the time, and solving them elegantly is incredibly satisfying. 

Intoxicating, even. 

But, sometimes programmers are presented with what I call “fuzzy” problems.

What is a “fuzzy problem”?

A fuzzy problem is a problem that has no perfect solution. 

These sorts of problems can be maddening for programmers, as the best we can do is deploy what I call “weaponized heuristics.”

For fuzzy problems, programmers must do their best to provide a solution most of the time. In some cases, a solution that is 80% effective may be adequate. In others, the 20% gap may be incredibly costly, requiring expensive human intervention.

I’ve been intermittently working on a fuzzy problem for a few years now. 

The problem requires processing inconsistent human-generated text. Each line of text refers to a specific product. 

For example, “Lay’s Plain Potato Chips.” But this same product may also be referred to as “Lay’s Chips in a Yellow Bag” or “Original Lay’s.” 

Heuristics can certainly be used to map all of these different forms to a single product, but doing so accurately and at scale is pretty challenging. 

But if you put these in front of a human, odds are good that they’ll quickly identify that all of these different forms are referring to a single product.

My code was pretty darn effective, with about 80% accuracy. The remaining 20% was easy to

handle manually, but it was time-consuming work, so I abandoned the project.

This weekend, I dusted off the code and took a fresh look at the problem. 

Could an LLM FINALLY solve my fuzzy problem?!

It occurred to me that an LLM would potentially be able to drive greater accuracy. 

I fired up my trust text editor and got to work.

My technique was simple — provide the LLM with the entire product catalog as context, and then iterate over the problematic data set line by line, asking the model to map each line to a product in the catalog. 

After testing the script against a few different models, I discovered that the new approach improved accuracy to 88% using GPT-4.1 nano.

Seeking to do even better, I made a few tweaks to my code. 

First, I tried asking the model three times on each iteration. Because models are non-deterministic, you’re not guaranteed to get the same answer to a prompt on subsequent attempts. 

With this approach, I boosted accuracy by a few points to 92%, selecting an answer on a line if the model provided it in the majority of responses.

Iterating on this concept, on each line, I asked the model for an answer twice. If the answers were the same, I’d accept the response. If they were not, I would ask the model repeatedly until an answer emerged as a majority response, up to 10 iterations, after which I selected the answer with the most occurrences.

First, the bad news. 

This technique was quite slow and quite expensive.

More importantly, the good news is that this technique boosted accuracy to a staggering 96%, and clearly demonstrates that LLMs can be highly effective at helping to solve “fuzzy” problems!