by Tan Aik Keong (AK)
Singapore has just disclosed its first AI-related data breach to be reported to the Personal Data Protection Commission. A well-known company meant only to send out a routine marketing email. Instead, it exposed the email addresses of 95,364 customers to one another. No hacker broke in, and no AI "went rogue" on its own — an entirely ordinary piece of marketing work ended up compromising customer data.
How did it happen? An employee asked a generative AI tool to write a program that would send the email out in batches — but never specified that each recipient's address had to stay hidden from everyone else in the batch. The program duly put every address in a batch into the same email, visible to all. When the employee tested it, they only checked the program's run log, not an actual test email, so they never saw what recipients would actually see. Once the code went live, the mistake scaled instantly.
The real risk isn't an AI "out of control"
What should actually alarm us here isn't some runaway AI. It's that AI makes it far easier to turn an insufficiently scrutinised habit into a piece of running code. Work that used to take an engineer real time to build can now be generated with a single prompt. The barrier to shipping something has dropped — but the scale of the damage when it's wrong can grow right along with it. Code that runs isn't the same as code that does the right thing, and it's certainly not the same as code that protects your customers.
Don't blame the employee alone
The regulator's findings point to a wider failure: the company had no policy guiding staff on how to use generative AI, and no process requiring a supervisor's review. When a task touches a large volume of personal data, and the only checks are one person's prompt, that same person's glance at a log, and that same person's decision to hit go, the organisation has concentrated all its risk in the single weakest link in the chain. The faster AI can produce something, the more deliberately a business needs to guard its key decisions.
What real human oversight looks like
Genuine human oversight isn't a manager clicking approve at the very end of a process. It has to start with clear rules, set before anyone writes a prompt: who is allowed to use AI to generate code that touches customer data? What data can that code actually reach? What will each recipient actually see? Before anything goes live, it should be tested with dummy data, and someone should actually open the test email and check the recipient field, the body and any attachments — not just the log. Any program that will send to a large number of customers should be reviewed independently by a second, technically competent person, and should go out in a small pilot batch before it goes to everyone.
Two different risks, not one
It's worth being precise about what went wrong here. This wasn't a case of feeding customer data into an AI model and having the model itself leak it. The data leaked when the company ran the code the AI had written, and the email system did exactly what that code told it to. That distinction matters: even when staff never upload a sensitive list to an AI in the first place, a business still has to review the code an AI produces, and the actions that code actually takes once it runs. Protecting data can't stop at the input stage — it has to extend to the output, and to execution.
Fixing it afterwards is not the same as managing it beforehand
To its credit, the company stopped sending, corrected the program, notified affected customers, and added a second check before similar programs now go live. All of that is the right response — but good management should have happened before the first email ever went out, not after. For any company adopting AI, teaching staff how to write a good prompt matters. Teaching them to recognise risk, to verify results against what will actually happen, and to stop the moment something is unclear, matters more.
Closing: outsource the work, never the responsibility
AI can write our code, organise our data, and speed up our work. What it cannot do is answer to your customers on your company's behalf. The more powerful the technology gets, the less a manager can afford to outsource their own judgement along with it. We can hand AI the work. We can never hand it the responsibility.
Part of the AK AI Corner column. Originally published in Oriental Daily (东方日报) on Oct 3, 2026.