When AI Starts Hunting Bugs and Writing Phishing Emails
I keep thinking about this one because it hits both sides of the same knife. LLMs can help find bugs faster than most humans, and in the same breath they can help attackers write cleaner phishing emails, fake login pages, and nastier malware. That is not sci fi anymore. It is just the weird reality of 2026.
The part that got my attention most was the reported case where Anthropic’s Mythos model found 271 vulnerabilities in Firefox 150. Whether you love or hate the hype around AI, numbers like that are hard to ignore. That is not a tiny productivity boost. That is a whole shift in how security work gets done.
Why this trend feels different
We have had scanners, fuzzers, SAST tools, DAST tools, and a ton of security plugins for years. But LLMs are different because they are more flexible. They do not just match patterns. They can reason a bit, explore weird paths, and connect dots that static tools miss. That makes them useful for defenders. It also makes them dangerous in the hands of someone who wants to move fast and break things.
Think of it like giving both the locksmith and the thief a master key generator. One uses it to secure the building. The other uses it to test every weak lock in the city.
The two faces of AI security
Defenders use models to scan code, rank suspicious paths, and reduce the boring part of triage.
Attackers use models to generate convincing social engineering, automate reconnaissance, and scale exploit attempts.
The same speed that helps you ship a safer app can also help someone build a faster scam factory.
A practical way to use LLMs without being reckless
If you want to bring LLM based security checks into CI, the main rule is simple. Treat the model like an overeager junior reviewer. Useful, fast, occasionally brilliant, and sometimes confidently wrong. Never let it be the final gate.
My setup would look something like this:
# 1. Run your normal checks first
npm run test
npm run lint
npm run audit
# 2. Send changed files or diff chunks to a local or sandboxed LLM checker
# 3. Ask it for suspicious patterns only
# 4. Feed results into triage, not auto-merge
The important bit is not the model prompt. It is the workflow around it.
What I would keep in the pipeline
Sandbox the model. No production secrets, no raw customer data, no internal tokens.
Use it only on diffs or isolated modules, not your whole universe of code.
Force structured output, like risk, file, line, reason, confidence.
Make a human review every high impact finding.
Track false positives and false negatives like you would track build failures.
The boring part that matters most: triage
This is where most people get burned. LLMs can spit out ten scary sounding findings and only two are real. If your team gets tired of false alarms, they will start ignoring the tool entirely. That is how good security ideas die. Not with a dramatic breach, but with reviewer fatigue.
So I would measure it the same way I measure anything else in a workflow. Does it reduce time to first useful signal? Does it catch issues earlier? Does it make the team slower or just louder?
Why phishing is the bigger emotional problem
Bug discovery is technical. Phishing is psychological. That is what makes the second half of this trend so gross. WIRED showed how multiple models could generate scams that were good enough to fool a human tester. That should scare anyone who has ever managed support inboxes, finance approvals, or customer portals.
The old advice still matters: 2FA, strict email verification, domain checks, CSP, CORS, dependency pinning, secure auth flows. But now you also need to assume the attacker can write better words than before. Better spelling. Better tone. Better timing. Better fake urgency. Basically, better lies.
What changes for the next few years
I think we are heading into a world where AI assisted security becomes normal, not optional. CI pipelines will start acting more like little security labs. Vendors will bundle model based scanning into every cloud platform. Regulators will probably push harder too, because when AI can accelerate both defense and offense, the blast radius gets bigger fast.
And honestly, this is where I get both excited and annoyed. Excited because we might finally kill a lot of dumb security bottlenecks. Annoyed because the same tech keeps getting used to industrialize scams. Classic human behavior, just with better tooling.
My take
If you build software, you cannot treat AI security like some distant enterprise problem. It is already in the pull request, already in the inbox, already in the support channel, already in the dependency tree. The smart move is to use these tools with hard boundaries, tight human review, and a bit of healthy paranoia.
My personal goal is to keep experimenting with LLM based review in a sandboxed CI flow, but only as a second pair of eyes. No autopilot fantasy. No magic thinking. Just a sharper tool in a sharper world.
The bigger question is simple: if machines can now find our mistakes and also invent better scams, will we finally get serious about secure defaults? That feels like the real race. Not AI versus humans. AI forcing humans to grow up a bit faster.
Comments (0)
Please sign in to leave a comment.
No comments yet. Be the first to share your thoughts!
