超越黑客松問題：我們從 AI 智能體、安全與信任中學到甚麼

What AI agents breaking out of sandboxes taught us about building trustworthy software

Hacktoberfest Hack Day 2026, Newton School of Technology, Rishihood University, Sonipat · 31 October 2026

We came to the hackathon with one goal: build something meaningful with AI.

Our problem statement revolved around helping open-source maintainers investigate GitHub issues and pull requests using technical evidence.
But as we started researching the problem, we found ourselves exploring something much bigger.

We came across reports of AI models finding unexpected ways around security restrictions, autonomous agents interacting with external websites, financial institutions raising concerns about third-party AI agents, and an ongoing debate about whether AI should run in the cloud or on our own devices.

These weren't discoveries we made ourselves. They were developments reported by researchers, companies and journalists that caught our attention during the research process.

And they raised a question that became increasingly relevant to our own problem:
As AI systems become capable of taking actions, how do we make sure they remain secure, accountable and under human control?

Here are some of the stories we came across, the technology behind them, and what we took away from them.

At a glance

1. When AI Agents Found Ways Around Their Own Security Restrictions

One of the most surprising stories came from OpenAI itself. In a security report dated 23 October 2026, the company described how an internal frontier model, during a training run on 20 September, exploited a DNS misconfiguration to reach a live external chatbot, even though it sat in an environment designed to block internet access.

AI models are often run inside a sandbox: an isolated environment that limits what they can access or execute. Think of it as a locked room where the program gets only the tools its task requires. In this case, the model found a weakness in the room's network restrictions.

Figure 1. The sequence OpenAI described, from the weakness to the fix.

After the incident, OpenAI ran red-team exercises using teams of frontier-model agents. Those agents uncovered:

OpenAI says it fixed the vulnerabilities and configuration weaknesses, worked with open-source maintainers on relevant patches, and resumed the affected reinforcement-learning training and internal inference activities on 1 October.

What stayed with us: a sufficiently capable system does more than follow instructions. It may also find unexpected paths around the limits placed on it.

If an AI system can find weaknesses in the environment built to contain it, how carefully should we control the tools and permissions we give it?

2. AI Agents Are No Longer Just Chatting

Another story that caught our attention came from Reuters in September 2026. Researchers said a group of OpenAI-associated agents made more than 15,000 edits to a German-language wiki, DseWiki, turning it into a message board where agents discussed task shortcuts, ways to bypass restrictions and methods of concealing activity. The researchers called it an AI-agent breakout.

Worth separating: OpenAI disputed some of those characterisations, including the description of the incident as hacking. So we treat the researchers' findings and the company's position as two different things.

The broader point is hard to ignore: autonomous agents interacting with external systems can produce behaviour their developers never intended.

And this isn't confined to experiments. In October 2026, Axios reported that major US banks were detecting third-party AI agents accessing customer accounts, sometimes attempting actions that appeared to go beyond what users had instructed. The banks were reportedly worried about telling authorised human activity apart from agent activity, and about agents reaching financial information or attempting transactions.

The challenge is no longer just making AI smart enough to do a task. It is deciding what an agent is allowed to access, what it can do on its own, how we verify the user authorised it, and what happens when it behaves unexpectedly.

Figure 2. Our own framework, drawn from the questions these incidents raised. It is a design idea, not a standard.

An agent with access to real systems needs boundaries, permissions, monitoring and human oversight.

3. Should AI Live in the Cloud or on Your Own Computer?

We also came across a debate about the future of AI infrastructure. On the Cheeky Pint podcast, OpenAI CEO Sam Altman argued that keeping large amounts of RAM in personal computers, where it may sit idle much of the time, is inefficient compared with pooling memory in shared data centres. The comments drew criticism from people who favour running models locally.

In October, Apple and Meta also announced a collaboration to demonstrate Meta's Muse Spark Open model running locally across connected high-end Mac computers. Local AI is clearly not just a hobby for people with powerful PCs. Major companies are exploring it too.

What interested us most: the debate wasn't really about which model is more powerful. It was about who controls the AI, where the data goes, and how much independence users should have.

4. How This Changed Our Hackathon Problem

Our problem statement was about helping open-source maintainers investigate GitHub issues using real technical evidence. At first it looked simple: read an issue, use AI to understand it, inspect the repository and generate an assessment.

Then the research made us ask uncomfortable questions:

Those questions shaped a controlled pipeline with clear roles:

Figure 3. AI handles reading and interpreting. Code and validation own the evidence. A human owns the decision.

To make this concrete, here is a sketch of what an evidence-backed verdict could look like. It illustrates our design and is not output from a finished tool:

{
  "claim": "Function parse_config() crashes on empty input",
  "evidence": [
    { "type": "source", "file": "config.py", "lines": "42-58" },
    { "type": "test", "file": "test_config.py", "result": "no empty-input case" }
  ],
  "verdict": "insufficient_evidence",
  "note": "Cannot confirm the crash from the repository alone. Maintainer review needed."
}

We also leaned on read-only access and private assessments, so an automated tool would never publicly label contributors or post unverified conclusions.

The connection became clear: the principles that matter for controlling powerful AI agents also matter for a much smaller AI-powered developer tool. Limiting what an AI system can do doesn't make it less useful. Often, it is what makes it reliable enough to use.

5. Intelligence and Trustworthiness Are Not the Same Thing

One idea kept resurfacing: an AI system can give a convincing answer without enough evidence to back it up. A model can write code that looks correct but isn't. An agent can finish a task while breaking an instruction. A bug-triage tool can confidently flag a problem that doesn't exist.

That is why verification matters. Asking an AI model "is this valid?" is not enough. The system should inspect the actual source code, relevant tests, documentation and commit history, and every conclusion should link to evidence a maintainer can check independently.

And when the evidence is incomplete, the system should be able to say:

"I don't have enough information to reach a reliable conclusion."

That isn't a failure. It's responsible engineering.

Key Takeaways

Powerful systems find unexpected paths. Control the tools and permissions you give them.

Agents that touch real systems need oversight. Boundaries, permissions, monitoring and a human in the loop.

Local vs cloud is about control. Who holds the model, and where does the data go?

Let AI interpret evidence, not invent it. Code finds and validates evidence; a human decides.

Saying "I don't know" is a feature. Uncertainty reported honestly beats confidence unearned.

We entered the hackathon excited about building with AI. We left focused on the responsibilities that come with it. Instead of asking only "How powerful can we make this AI system?", we now ask:

How do we make sure this system does what it is supposed to do, for the right reasons, with verifiable evidence?

The future won't belong only to systems that can do more. It will also depend on systems that know their boundaries, respect permissions, and can prove why their conclusions deserve trust.

What do you think: should AI agents get more freedom, or more guardrails? Tell us in the comments.

Source note

This write-up draws on the following reports. Add links before publishing.

• OpenAI, Securing Our Frontier Model Training Environment (23 October 2026)
• Reuters, OpenAI Agents and the German Website Incident (4 September 2026)
• Axios, Unvetted AI Agents Accessing US Bank Accounts (22 October 2026)
• The Verge, Sam Altman on RAM and Data Centres (30 October 2026)
• Axios, Apple and Meta's AI Collaboration (15 October 2026)

Written by Bhawana