by Tan Aik Keong (AK)
Artificial intelligence has moved fast in recent years, and the rise of generative large language models like ChatGPT has fundamentally changed how we interact with machines. But underneath that progress, a more uncomfortable problem keeps surfacing: AI decisions often feel like a black box — we simply don't know how the system arrived at its answer.
This isn't just a problem for researchers. For businesses, government agencies, and anyone using an AI product, if the system can't explain the basis for its own judgment, how are we supposed to trust it?
The trust problem with black-box AI
Say a large language model tells you a patient may have lung cancer. The real question is: why did it conclude that? Was it a feature in the scan? Something in the patient's history? Or just a pattern it happened to pick up somewhere online? If we can't see the model's reasoning process, its judgment is hard to trust — and in a case like this, potentially dangerous to act on.
Making this harder still, LLM output is non-deterministic — the same question, asked multiple times, can produce different answers. And these models typically can't offer a clear confidence score or supporting rationale, which only deepens public unease.
Enter Explainable AI
To make complex AI systems more legible, researchers and industry are pushing Explainable AI (XAI) forward. Two of the most widely adopted techniques are LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations).
LIME: taking a single decision apart
LIME works like a microscope for peering into the black box. Rather than trying to explain an entire model, it focuses on the local neighbourhood around one specific prediction, slightly perturbing the input and watching how the output changes — working backward to figure out which features the model actually relied on.
Say an image model classifies a photo as "cat." LIME will darken or mask parts of the image and see whether the model's prediction holds. If masking the ears flips the prediction to "dog," that tells you the ears were a decisive feature.
SHAP: explaining AI through game theory
SHAP takes its inspiration from game theory's Shapley value, and tries to answer a different question: how much did each individual feature contribute to the model's final decision?
SHAP assigns each feature a quantified "explanation value" — for example, telling you a customer was flagged as high-risk because a low credit score contributed -0.3 and a high outstanding balance contributed +0.4. That lets even a complex deep-learning model's prediction get "translated into plain language."
Trust depends on being able to explain yourself
The idea behind both LIME and SHAP is fairly simple: however sophisticated an AI system is, it still has to be able to explain its own behaviour to a human.
In high-stakes fields like healthcare, finance and the judiciary, that explanatory ability isn't a nice-to-have — it's a precondition for compliance and accountability. If a bank's AI model rejects a loan application, the bank needs to be able to say why, or it faces both legal and ethical exposure.
From "intelligent" to "trustworthy"
The AI of the future needs to be trustworthy, not just smart — and being trustworthy starts with being explainable.
LIME and SHAP open a window into AI's reasoning, gradually turning the black box into something closer to transparent glass. For any organisation using AI, investing in explainability isn't just a technical choice — it's a responsibility. Only once AI can explain itself clearly can people genuinely feel comfortable working alongside it.
Part of the AK AI Corner column. Originally published in Oriental Daily (东方日报) on May 31, 2025.