Finding 0days with AI: Part II
In the first blog post, we covered the process from start to finish, focusing mostly on writing good prompts, tooling and techniques. It's recommended you read it first before continuing with this one as it lays the groundwork for thinking and being effective for bug discovery in audits.
For this one, we will explore various ways to enhance the process, bug hunting nuance and further ways for data analysis, vulnerability discovery and automation. Instead of primarily prompts, we shift more to markdown files and skills consumed by the models and harnesses, while still relying on manually driving the harness in the right direction or to overcome hurdles on the way to find really good bugs.
The methods and techniques discussed have found high and critical bugs in well-known products ranging from local applications like sudo to network administration services like Webmin. It's amazing how effective applying expertise and compounding knowledge with the scale and execution of agentic harnesses brings to power up vulnerability research.
There are ways to do this for individual bug hunters and organizations alike. For example these methods and tools can be used to aid in compliance, dependency, OWASP Top 10 checks and similar processes, however they can go even deeper for vulnerability discovery and analysis to be helpful in your product security, pen-test, red teaming efforts and programs.
Perhaps a good tagline for this one is...
If you can describe what you want and drive the harness well, you can find great bugs.
How can you do this? Let's begin.
Describing what you want
It sounds simple at first, right? Let's say you want to find bugs.
Well, how do you do this manually?
Get the code, binary or deploy the service locally for testing
Figure out how it works
Apply tooling to provide data and insights
Correlate intentions and assumptions with design (TBD: say this better)
Compare the intended design and assumptions with the actual code
Look for patterns that lead to bugs
Test any leads you uncover (this may include a huge bag 'o tricks)
Confirm its a bug, that it's repeatable, any requirements and so on
Hooray nice find 🎉
Most of this today can be automated, scaled with parallel processes and assisted by AI models and harnesses very well.
Models and Harnesses
If a model is like the brain, then a harness is the muscles.
Models can reason, generate and decide while harnesses give it capabilities to call tools, store/retrieve memory and iterate towards desired outcomes.
Models can figure things out if educated (context files) and directed well (eg. "look at every line in this function and try every way to bypass X"). Harnesses can do tasks and drive objectives to completion. This unlocks magic for vulnerability research.
Local models, which are getting better at "reasoning and intelligence", seem to be poised to take market share at some point, however the hardware cost and availability are barriers to even try the top ones, with many just using the smaller models that run on commodity hardware or GPUs. As the capabilities keep getting better and fast, so maybe we'll see a shift in the next 6-12 months.
For now, the frontier models such as those by Claude and OpenAI, with advanced reasoning skills and applications that support agents/subagents may just be the best high intensity and deep tasks like analyzing complex data, bug discovery and PoC generation especially in hardened codebases.
There's lots of agentic-capable platforms like Claude and Claude Code, Devin/Windsurf, Google's Antigravity, Cursor and more.
Making Skills
You can super charge audits with providing high quality and nuanced context in the form of prompts and markdown files. Skills were popularized by Claude and are how you can inform your model of all the different ways that bugs manifest across code,
For example, the creation of specific skills can make sure it looks for subtle bugs that aren't traditional patterns in code auditing, like security-critical code that is missing like authn/authz checks or working backwards from what bug you want to find and see if a component qualifies for it in various ways.
There's even usefulness in prompting over and over to generate new skills, different than what it already has, to try and come up with novel or just missing techniques for bug discovery.
Which Skills Are Effective?
While grepping and testing if untrusted data passed to insecure functions creates a vulnerable condition is not to be underrated, even today, there's a whole world of deeper and more interesting ways for discovering bugs.
Providing skills that can cover things like "find this pattern that only exists in this codebase", "look across all network functions for missing validation checks", "evaluate all security gating functions for completeness" can further elevate the effectiveness of audits. The possibilities are endless (that's the cool part of the job!).
Here's a few of my favorite skills and adjacent methods, many of which led to leads and findings and hopefully they will lead to many more cool discoveries.
Check Completeness
Look for a missing check instead of a broken one. Does every other API have a check_auth(), but one of them doesn't? Or does it call the old version? Instead of looking for bad code, look for code that is missing a pattern.
N-day Mining
Looking for n-days (publicly labeled or also silently fixed bugs) via commit history and using this data for hotspots for where more bugs may be, incomplete fixes or intel for new patterns that generate new leads for discovery.
State Machine Extraction
Completely understand the protocol state machine and all the ways it can be driven into forbidden or unintended states.
Variable Inversion
Inverts the developer's assumptions about how big, small, or bounded a variable can be. Build an assumed valid range based on code analysis and the actual possible ranges and transformations, use those as leads to look for bugs.
Second-Order Persisted Trust
When a user can write data as a low privileged process, but a higher privileged process parses or uses or otherwise trusts it to do a privileged operation.
Tainted Value Semantics
Looks for gaps in a value's meaning across components. If an app validates a string under one interpretation, but hands it to another component that has different rules or interpretation, that could be a bug.
file:/// is a valid location, but points to a file. file:///tmp is a valid storage location, but file:///proc/self/environ is not a normal file. Parsers may ignore multiple slashes, strip them, add them across different levels and ways of processing.
Empirical Interaction Matrix Differential
Based on the idea that every bug is a delta between two relationship graphs, the declared graph which is the relationships that developer believes the code has and the actual graph being the relationships that is actually has.
Compositional Interpreter Chain
Data is parsed or decoded at two different layers and different decisions are made based on how strict each parser is and what it looks like at those stages.
Guard Breaker Differential
Instead of just trying different inputs to bypass a check, try a structural bypass. Figure out alternative paths around it, where is could be absent, confused, disabled or fires later on.
Fail Open
When something goes wrong, it defaults to allowing the action, the opposite of failing closed.
For example, a parser doesn't know what the extra data sent in the packet, so it ignores it while still allowing it to pass and go further into the next parsing rounds instead of rejecting the packet or discarding the extra data.
Does this stuff really work
You can review the various bugs and security relevant findings in products from the previous post here, which were found via AI-assisted yet manual hands on keyboard, driving the harness to results, triaging and maneuvering preliminary leads and findings towards either the discard pile or a stronger CVSS score, testing it on the local lab environment to ensure its a real, reproducible, security bug with demonstrable exploitability (RCE) or high quality signs of it (memory corruption world).
Some of these bugs were in newer products and some were hardened codebases that don't often have high/critical security issues.
Techniques and methods usually get better. Models and harnesses have been getting better and even the "older" ones (still came out this year) are solid at reasoning and executing key tasks that lead to bug discoveries.
The more time and effort I put into discovery and the compounding of knowledge, skills and techniques that build up over time provide me with a response more on the side of "what can't we find bugs in" vs what used to be unicorns to find good bugs in many foundational products.
This is especially amazing when you're able to find good bugs. Not low or medium meh bugs, but ones that are interesting and unique. It's as if you have access to the superpower of being able to analyze and connect the dots for everything that was too hard or too big to do before. Scale at your fingertips, really.
So whether you're a security researcher looking for bugs in open source products or working in an organization that has code for services that run businesses, you can dramatically increase your own capabilities and those around you by deploying tools, techniques and processes (sure, a play on TTPs) that discover, triage and even help fix bugs when they exist, were about to exist, or were just fixed.
Enter Automation
There's a lot of value to being able to individually discover and evaluate products and also value in being able to enable this for a team or organization.
How does this usually take place? You have a skill or a solve a problem and want to scale it for others, so... you build a tool. Or if you need that tool to become a one-stop-shop, you could also build a platform.
Lotus
The experimental Lotus platform can be deployed to allow an individual or organization kick off audits for codebases and dependencies to provide "bug discovery as a service".
It's as simple as provide a link to a git repo, it downloads the code and starts kicking off tasks and using AI to assist in completing an audit. Full report generation, reproduction, and continuous coverage support (if any new code is pushed, it runs the audit again).
It is experimental, meant to be effective and do a lot with some bells and whistles, and was built to scale from individual to organization, reduce friction and be a fun way to navigate and have fun in the process of bug discovery.
It has been open sourced, and as many companies are using AI to get an edge or just catch up, more oss and commercial platforms will likely emerge to provide solutions for automated bug discovery, pentesting, and so on.
Development for ideas and advances in the security world can now happen fast and be decent quality, something that changes things in an interesting way.
Models get better over time, so results should as well. Individuals and teams can do it by hand, automate, platformitize or at some point we may get to code-appears-as-needed where features are written just in time (emergent or incoming requests) as the user navigates the OS or product.
Fuzzing
The technique that feels like AI before AI, at least in the automation sense. Fuzzing for bugs used to create that magic feeling where you build or configure and run the fuzzer, go to sleep and wake up to crashes to triage. It's can still be effective today, but oss-fuzz and various frameworks and initiatives especially in open source have coverage in many of the popular projects. Not to mention researchers have probably fuzzed them inside-outside for years now too, so doesn't seem as fruitful of a way for vulnerability research as it used to be.
However, there's a few angles that make sense and where AI could help fuzzing produce more high quality bugs
Discovering non/under-covered interfaces for fuzzing
Automating fuzzing, symbolic execution and similar harnesses that used to require manual time and effort
New code (AI makes writing code cheap, therefore its easy to increase)
Deeper and more advanced methods
Fuzzers are good at executing. AI can help them focus on what code to execute and get better at the "where and how" for shaking out bugs.
What's the future for bugs?
More capabilities to find and fix them means fewer bugs? Perhaps one day, when it outpaces the new code, but that may take a while.
For now, it looks like lots more code, lots more bugs or best case just more bugs. And now there are more opportunities and interesting ways to find them even in the most hardened software. This fixing of bugs and chains should raise the bar quite a bit for bugs across the board, where critical bugs become more rare as long as new code gets solid review and testing before it ships.
All of this is happening while the industry is being reshaped and hopefully coming out with cool work to do and solutions that we didn't quite have before. Lets hope its a bright future for the space and people use the new capabilities to bring forth new ideas, products and supercharge the intellectual curiosity that drives so many of us.
