Sieve

What the company appears to be: Sieve is a self-reported tool for auditing code-generating agents (e.g., GPT-5.6, Codex) by testing whether their stated reasons for actions actually influence behavior. It runs behavioral experiments where one declared reason is changed while holding other conditions fixed, then compares resulting patches or test outcomes.

What changed: The author describes Sieve as a method and tool developed during the OpenAI 2026 hackathon to test agent reasoning through structured interventions in a coding loop. It was built using Codex, GPT-5.6, OpenAI Responses API, Python, TypeScript, Vercel, and Vite.

Single most important open question: Is there any evidence of real-world usage or adoption beyond the author's own development process? The description does not indicate whether Sieve has been used by others, tested in production environments, or integrated into existing workflows — only that it was built as a proof-of-concept.

The description states that Sieve is a tool for auditing code-generating agents. It operates by capturing structured steps before local tool actions, then changing one field at a time (e.g., claim, constraint, hypothesis) and comparing the resulting behavior or test outcomes. This process is described as a "capture, intervene, compare" loop.

It uses an AST-based diff to measure patch divergence and evaluates test stability by comparing pass/fail sets. The tool generates static HTML reports showing 2x2 grids of results and allows for manual inspection of interventions.

The author claims Sieve audits agents through a direct OpenAI Responses API backend using GPT-5.6 and Codex, with a schema that covers specific fields rather than full model cognition.

Evidence: Self-reported by the author; no independent verification or demonstration beyond the project write-up.

The description positions Sieve as a way to test whether an agent's stated reasons actually drive its code. It is framed not as a tool for interpreting hidden model thoughts, but for measuring behavioral sensitivity — i.e., whether changing a reason changes the outcome.

It emphasizes that Sieve does not attempt mechanistic interpretability but instead focuses on reproducible interventions in agent behavior. The author notes that this was their first time using Codex and GPT-5.6 in a long-running ideator-executor loop, which shaped both the tool’s design and its own development process.

Inference: The positioning reflects an attempt to distinguish Sieve from other interpretability tools by focusing on observable behavior over internal reasoning.

The description does not explicitly name target customers or define an ideal customer profile (ICP). However, it implies that Sieve is aimed at developers working with code-generating agents such as GPT-5.6 and Codex.

It suggests a use case where developers want to validate whether agent-generated code reflects true reasoning rather than post-hoc rationalization.

Evidence: Not explicitly stated; inferred from the context of agent auditing and developer-focused tooling.

There is no mention of pricing, monetization, or business model in the description. The project appears to be a personal or hackathon effort with no indication of commercial intent or revenue streams.

Evidence: Not evidenced.

Sieve is built using:

• Codex
• GPT-5.6
• OpenAI Responses API
• Python, TypeScript
• Vercel, Vite

It records structured steps including claims, constraints, hypotheses, planned actions, and success criteria. It compares changes in code patches (via AST) and test outcomes.

The tool supports five isolated TypeScript fixtures with three single-field interventions per task. Output includes baselines, perturbed runs, JSON traces, diffs, and score records.

It generates a static HTML report with 2x2 grids for easy inspection, and allows manual replay via a --live mode that calls the API directly.

Evidence: Self-reported; no external validation or delivery evidence provided.

There is no evidence of traction, adoption, or usage beyond the author’s own development. The project was submitted to the OpenAI 2026 hackathon and is described as a proof-of-concept.

No customer data, revenue figures, user feedback, or product maturity metrics are mentioned.

Evidence: Not evidenced.

The description does not reference any competitors or existing tools in this space. It positions Sieve as a method for auditing agent behavior but does not compare it to other interpretability or verification tools.

Evidence: Not evidenced.

• Lack of external validation: The tool is entirely self-reported and lacks third-party testing or real-world deployment.
• Limited scope: Only five tasks are tested, and the method is described as a proof-of-concept, not a benchmark.
• No commercial viability: No indication of monetization, pricing, or target market beyond the author’s own use case.
• Dependency on proprietary APIs: Relies heavily on OpenAI's API, which may limit scalability or portability.

Inference: These risks are based on the lack of evidence for real-world usage, product maturity, and commercial viability.

1. Has Sieve been used outside of the author’s own development process?
2. Are there any plans to expand beyond the current five-task framework?
3. What are the limitations of the current approach in real-world agent workflows?
4. Is there a plan for integrating Sieve into existing development environments or CI/CD pipelines?
5. How does Sieve handle edge cases or failures in agent-generated code?