More transparency bought trust and cost outcomes

Your users don’t trust the model, so they ignore it, double-check it, or route around it. The standard fix is to open the box: show the reasoning, show the uncertainty, let people see why the system said what it said. It works. Trust goes up.

Then something else happens that the standard advice doesn’t cover.

Tapas Peshin, a senior product manager in energy and climate technology, wrote up his team’s version of this on Mind the Product on October 13. His product forecasts electricity prices and schedules batteries for customers in Texas. The grid there is run by ERCOT, the state’s grid operator, and during Winter Storm Uri in February 2021 its real-time prices sat at the $9,000 per megawatt-hour cap for most of five days. The team’s forecast pointed the right way. One customer, splitting capacity roughly 50/50 between the day-ahead and real-time markets, still asked whether they should trust it enough to act.

The forecast wasn’t wrong. Customers couldn’t see why it said what it said, or how sure it was. Peshin calls that the trust gap. So the team added the drivers behind each forecast, the probability of a price spike, confidence intervals, and alerts tied to ERCOT’s emergency levels. Accuracy didn’t change. Peshin reports that trust-related customer calls dropped noticeably in the winter storms that followed, though he gives no counts.

That is the success story, and it matches what the usual advice predicts. The second half of his piece is where it gets interesting.

The team had also rebuilt the battery scheduler. The old version produced one blended schedule and missed several price spikes in summer 2023 and winter 2024. The new version runs against six price forecasts, from conservative to aggressive, and offers capacity at higher prices in the outer bands, so it only clears if a spike happens. Peshin says the banded output is easier to understand and also easier to override. And he reports that customers who changed their settings often had more missed spike days than customers who let the model run.

Trust and outcome are separate things, and the signals that buy the first can cost you the second. Showing drivers and uncertainty makes users feel they understand the system. Feeling that you understand it is what makes you willing to overrule it. When the model is right more often than the user, that willingness loses money.

The view I’ve held is that trust is the gate to adoption: users who don’t trust an AI feature avoid it or re-check every output, which cancels the gains. This evidence doesn’t break that view. It adds a second failure the first one hides. Re-checking everything is the cheap version. The expensive version is a confident user who overrides some of the time.

Explanation did earn its keep in one other place. Peshin’s team had a feature on the roadmap that let batteries stack a four-hour reserve product alongside other revenue. When ERCOT changed its duration rules, he walked customers and internal teams through what the new rules meant for stored energy. The economics no longer worked, and the team dropped the feature. Customers had wanted it. Explaining the mechanics saved everyone from building it.

The fair objection is that this is one practitioner at one company. The product is a forecasting optimizer, not a chat assistant. Peshin gives no sample sizes, and customers who tweak settings constantly may simply have the hardest assets to schedule. Any of those could explain the gap. I’d want to see it replicated before I’d call it a rule.

But it points at a design question most teams haven’t tested. If trust is the thing you build, you measure adoption. If calibrated trust is the thing you build, you measure whether people override the model when it’s right and defer when it’s wrong. Those need different features. The second one might show each user their own override record next to the model’s, so the number that comes back is outcomes, not comfort.

Nobody has published that test that I can find, and the experiment is cheap. Take a product with banded or explained outputs, show one group their override history against the model’s, leave another group alone, and compare how many of the good calls each group captured. If the first group captures more, transparency was missing a feedback loop. If not, the explanations are doing less than we think.

Check your own product first. Pull the users who override most and see whether they do better or worse than the ones who don’t. I’d guess most teams have never looked. What you find will tell you what your transparency feature is really selling.