This chapter treats secure development when AI takes part in it: what studies measured about the security of generated code, how dependencies, secrets and licences behave when a tool proposes them, which classes of attack reach a coding assistant or agent, why the toolchain is itself a Tier 0 asset, and how a provenance record lets an owner show where code and dependencies came from. It concerns D4 Prove, because the question is what evidence shows that a change is safe to run, and D5 Steward, because the owner who answers for the change in production must be able to name what it contains and stop it from outside the pipeline that shipped it. Chapter 9 treated the general case of proving a change with an oracle independent of the generator. This chapter takes the part of proof that concerns an adversary: someone who writes the text the tool reads, publishes the package the tool suggests, or steals the credential the tool can reach. It is defensive throughout. Attacks are described by class and technique at the level needed to design controls and tests, and none is described in a form that could be repeated.
The argument has four steps. Generated code is untrusted input and is proved like any other. The supply chain gains an entry point in the tool's suggestions, so the control moves from the developer's judgement to a proxy that every request must pass. The tools read text that adversaries can write and hold credentials that adversaries want, so controls outside the model contain them. And the evidence is a record, kept with the code, that a supervisor or an incident responder can read. The conclusions are the chapter's own and are collected in the Summary.
Pearce and colleagues, in 2021, asked GitHub Copilot to complete code in 89 scenarios and produced 1,689 programs; they found approximately 40 per cent of them to be vulnerable (228, Pearce et al., 2021). The study measured a 2021 version of one tool on scenarios its authors chose; it says nothing about the tools of 2030 and does not compare the output with code written by people.
Primary topics explored
- What studies measured
- Dependencies and package hallucination
- Secrets and licences
- Prompt injection and excessive agency
- The toolchain as Tier 0
- Provenance
About AI Developer
AI Developer is Book 12 of The Transformed Architecture. It takes the developer as its subject and asks what happens when AI is used broadly in software development: the five developer functions (frame, design, build, prove, steward), the delegation ladder and the Level 4 prohibitions, the stewardship principle (name, understand, revert), verification as the scarce skill, the junior pipeline, adoption and productivity as claims that need evidence, regulation (the AI Act, the Cyber Resilience Act, the Product Liability Directive, NIS 2, DORA, the GDPR), ethics, intellectual property and sovereignty in the toolchain, and the developer as a person, for European enterprises and public authorities.
It builds on the conclusions of Architecture as Code, AI Governance, AI Platform and Security, AI Red Team, AI Threat, The Transformed Security Architect, AI Boss, AI Blue, AI Purple, AI Operations and AI Leader.
developers at every level, tech leads and engineering managers, architects, platform engineers, site reliability engineers and testers, product owners and designers, leaders who decide on developer tooling, heads of public-sector IT and procurement, works councils and unions, educators, and supervisors and auditors.
Gunnar Nordqvist
Conceived, directed and published the books of The Transformed Architecture.
Continue with the full chapter
Chapter 10: Prove Security: Secure Code, Supply Chain and the Toolchain as Attack Surface in AI Developer gives the detailed explanations, practical examples and implementation patterns behind this summary.
This whitepaper belongs to “The Craft of Developing” and is one of 37 drawn from the manuscript.