Computer says no, and nobody can say why
Sometimes. It depends on whether anyone can look inside and see why the model decided what it did. That is the promise of explainable machine learning: reasons, not just verdicts.
Loans, medical scans, product recommendations: AI now has a hand in all of them. “Computer says no” started life as a sketch-show punchline. Put it in front of a loan applicant and it stops being funny. The trouble is that most modern systems are black-box models, accurate but closed off, and sometimes a mystery even to the engineers who built them. This piece looks at what that secrecy costs, how two popular tools get around it, and where those tools run out of road.
Accuracy has a habit of costing readability. Linear regression, which goes back to the 1800s, is wonderfully plain: change one input, watch the output move. Yet real data is rarely that obliging. Once the relationships turn nonlinear, teams switch to things like ensembles of gradient boosted trees (many small decision trees, each correcting the last one’s mistakes), and the logic gets hard to follow. Data scientists call this the accuracy-interpretability trade-off. Annoying, but there it is.
Take lending. A credit model hands a high score to someone asking for a very large sum. Fine, except analysts would like to know whether that score reflects a genuine ability to repay or just a distortion in the data. A bare number can’t say. With real money on the line, “trust me” isn’t good enough.
Back in 2016, Ribeiro, Singh and Guestrin pointed out that, widespread adoption notwithstanding, “machine learning models remain mostly black boxes.” The effort to fix that goes by model interpretability: working out why an AI said what it said.
Two tools that pry the lid off
Two techniques get most of the attention, and both deliver local explanations, meaning they account for a single prediction rather than the model as a whole. People comparing lime vs shap usually discover the two tell broadly similar stories, so some teams run both and see whether the answers match.
LIME: poke the input, watch what happens
LIME, short for Local Interpretable Model-agnostic Explanations, appeared in 2016. The trick is almost cheeky. Take one case, mess with it (delete words from a sentence, grey out chunks of a photo) and log how the prediction shifts. Fit a simple model to those results and it shows which pieces carried the weight.
Picture a sentiment model scoring “I HATE Mondays! I missed the bus and forgot my lunch…”. LIME singles out “hate” as the main push towards negative, with “missed” next in line. Photos behave similarly. A husky on snow gets called a wolf, but crop the snow away and the label fixes itself. That hints the model was reading the scenery, not the animal.
SHAP: dividing up the credit
SHAP (SHapley Additive exPlanations) came along in 2017 and leans on game theory. Shapley values go back to Lloyd Shapley in 1951. They estimate what each player adds to a team’s result. In SHAP the model is the game and each feature is a player.
The arithmetic is the clever bit. Contributions are additive, so they sum exactly to the gap between the model’s average prediction and its prediction for the case at hand. In one diabetes example, the average sat at 141.2 and a particular patient’s prediction landed at 171.3, with serum triglycerides, a blood-fat measure, giving the biggest upward nudge.
The practical differences, briefly:
- LIME works from one observation, while SHAP wants a whole sample.
- LIME asks only that a model offers a predict_proba method. SHAP is fussier, though it supports popular libraries such as scikit-learn, XGBoost and TensorFlow.
- SHAP values are measured against the sample average, so a different sample can move them a little.
Why UK organisations are paying attention
Regulators saw this coming. In 2020 the ICO and the Alan Turing Institute put out joint guidance on explaining AI-assisted decisions to the people they affect. It’s good-practice advice, not a statutory code under the Data Protection Act 2018, and the ICO now flags it as under review following the Data (Use and Access) Act.
The logic holds whether the user is a lender, a hospital or a games studio matching players of similar skill. A prediction that can be explained can be challenged, audited and fixed.
What explanations can’t do
No magic wand here, sadly. Two caveats matter:
- They don’t replace performance metrics. Accuracy, F1-score and mean squared error say how good predictions are. LIME and SHAP only say why one was made.
- They don’t prove causality. A model is an approximation of reality, so an explanation shows what drove the model’s answer, not necessarily what drives the real-world outcome.
Still, they earn their keep when something goes wrong. A false negative can be traced back to the features that misled the model, and that shows where retraining should begin.
So, can AI be trusted?
Case by case, and slowly. A model that shows its reasoning can be questioned, corrected and, in time, relied upon. One that doesn’t can only be believed or ignored. AI transparency won’t turn every system into trustworthy AI, yet it makes the dodgy ones far easier to catch.
A few habits go a long way: asking which data drove a decision, reading explanations next to performance metrics (never instead of them), and treating feature importance as a clue rather than a verdict. Where real lives are affected, explainable AI should come with reasoning a non-specialist can follow. Black boxes aren’t leaving. They just shouldn’t be treated as oracles.
David Prior
David Prior is the editor of Today News, responsible for the overall editorial strategy. He is an NCTJ-qualified journalist with over 20 years’ experience, and is also editor of the award-winning hyperlocal news title Altrincham Today. His LinkedIn profile is here.
