7 Challenges of Governing AI at Scale: Why Most State Agencies Aren’t Ready
|
Imagine a state agency deploys an AI tool to help draft procurement requirements. The tool is fast, the outputs look good, and program staff start using it across multiple projects.
Six months later, the inspector general requests documentation showing how a specific contract requirement was developed, who approved it, and how it connected to the original policy directive.
No one can produce that record.
The AI tool did exactly what it was supposed to do, but the governance structure around it didn’t.
This scenario plays out in variations across state government every day. AI adoption is accelerating, but the frameworks needed to make that adoption defensible, including under legislative oversight, audit scrutiny, procurement challenges, and public records requests, haven’t kept pace.
This article outlines concrete challenges of governing AI at scale in government settings, explaining why they’re especially difficult for state agencies, and offers practical guidance on where to focus first.
The Top Challenges of Governing AI at Scale for State Agencies
1. Unclear Ownership Across Teams
When an AI tool gets deployed, multiple teams often think someone else owns it:
- The business unit that requested it assumes IT manages it.
- IT assumes the vendor is accountable.
- The vendor assumes the agency defined the acceptable use.
This distributed confusion isn’t unique to government, but it’s more consequential there. In the private sector, a gap in AI ownership creates operational risk.
In state government, it can mean a program operates without accountable review, a use case expands beyond what was originally authorized, or no one flags that an AI output influenced a high-stakes decision without oversight.
Effective AI governance requires clearly defined ownership at every stage:
- Who approved the use case
- Who monitors ongoing use
- Who has authority to expand or restrict it
- Who is responsible if something goes wrong
Here’s what this looks like in practice. A state agency might have a program manager, an IT lead, a compliance officer, and a vendor all involved in an AI implementation.
Without explicitly assigning who owns governance decisions, each one defers to the others. The result is no one does.
2. Weak Requirements Quality Before AI Is Applied
This is one of the most underappreciated AI governance challenges, particularly in government programs: AI tools amplify ambiguous input.
When a requirement is incomplete, untestable, or vague, running it through an AI workflow produces an output that is harder to trace back to what was authorized.
The speed of AI makes this worse, because teams can move far down a development path before anyone notices the foundational requirement was flawed.
Strong governance requires that requirements meet basic quality standards before AI touches them:
- Complete: They define what they need to define, without gaps
- Testable: Outcomes can be verified against them
- Unambiguous: There is one clear interpretation
- Appropriately scoped: They specify what’s included and what isn’t
Catching quality issues at the point of authoring is far less costly than catching them during an audit, a procurement challenge, or a legislative review.
3. Limited Traceability Between Outputs and Approved Requirements
Requirements traceability is a basic accountability standard in government programs. It’s also one of the first things to break down when AI is introduced without governance structure.
AI tools produce outputs quickly. When those outputs aren’t linked to the requirements that authorized them, the agency has no reliable record of:
- Where a decision came from.
- What authorized it.
- Whether it stayed within approved scope.
This is a governance architecture problem. Traceability needs to be built into the workflow from the beginning. It is something to be not assembled after the fact when an audit request arrives.
When traceability is part of normal work, agencies get a clear, continuous record.
They know how requirements evolved, what decisions were made, and how every deliverable connects to an approved specification.
4. Invisible Downstream Impacts When Requirements Change
Requirements change, that’s normal in any government program.
What’s not normal, but increasingly common with AI-assisted workflows, is that downstream work doesn’t automatically reflect those changes, and no one knows it.
When a requirement is updated in a well-governed system, there’s a clear signal about what work is now out of alignment.
When that structure doesn’t exist, teams continue building on a requirement that’s no longer current. They may deliver a product that doesn’t match what was authorized, without knowing it until a review surfaces the gap.
This is particularly at risk in AI workflows because outputs are produced faster and in higher volume. A single upstream requirement change can affect a large body of downstream work before anyone catches it.
Governance needs to include change visibility, a mechanism that surfaces what’s affected when a requirement changes, so program managers can make informed decisions rather than discover problems at the worst possible moment.
5. After-the-Fact Documentation
Most government programs still build compliance documentation at the end of a project cycle. This was always imperfect, but with AI-assisted work, it becomes untenable. AI tools can generate large volumes of work in a short time. Reconstructing the decision trail for all of it is time-consuming, often incomplete, and frequently inaccurate.
The solution is to shift documentation from a closing task to a byproduct of normal work. When links between requirements and outputs are created at the time work is done, and every action connected to a requirement is logged in a version-controlled record, agencies don’t face a documentation gap. They have an accurate, continuous audit trail.
6. Controls That Don’t Match the Risk Level of Each Use Case
Not every AI use case carries the same risk. An AI tool used internally to summarize meeting notes is very different from one used to help evaluate procurement bids, score program applications, or develop regulatory guidance.
One of the most common AI governance challenges is applying the same governance requirements to every use case, or applying no requirements at all. Both are failures.
When controls are too light for high-stakes uses, the agency is exposed. When controls are too heavy for low-stakes uses, teams work around them, and the governance process loses credibility.
Effective governance applies controls that match each use case’s actual risk profile, factoring in:
- The stakes of the outputs (who is affected and how).
- Whether AI-generated content requires human review before use.
- What data the system accesses and how it’s protected.
- The regulatory and oversight environment for that program area.
Matching controls to use-case risk is harder than applying a blanket policy, but it produces governance that people follow.
7. Low Visibility into How AI Is Being Used
This challenge tends to grow as AI usage expands. Tools proliferate, with some approved and others adopted informally. Without a clear view of where AI is being used, it’s nearly impossible to identify higher-risk activity. This visibility gap creates a compounding problem: the more AI is used, the harder it becomes to govern.
A practical approach is maintaining an active inventory, covering approved systems, AI features embedded in existing tools, and known informal uses. This becomes the foundation for prioritizing governance resources and identifying where controls need to be strengthened.
RELATED: Accelerate AI-Driven Development with Jama Connect MCP™
Why These AI Governance Challenges Are Especially Critical for State Agencies
The challenges of governing AI at scale affect all types of organizations. But state government agencies face a specific combination of pressures that makes these challenges more acute.
Non-Negotiable Accountability Standards
Legislative oversight, inspector general reviews, procurement audits, and public records requests all require a clear, traceable record of what was decided, why, and how it connected to authorized requirements.
In private organizations, gaps in that record can be managed internally. In government, they become public problems.
Explainability Isn’t Optional
When an AI-assisted decision affects a contract award, a program eligibility determination, or a regulatory outcome, agencies need to explain how that decision was made.
“The AI tool recommended it” is not an acceptable answer in any oversight context.
The governance structure needs to produce an explanation that survives scrutiny.
Requirements Documentation Carries Legal Weight
In government programs, what was authorized matters as much as what was delivered. Requirements serve the purpose of internal planning. They are also the basis for contract terms, compliance reviews, and procurement challenges.
Weak requirements quality and poor traceability open the door to operational risk and legal exposure.
Constrained Resources
State agencies can’t always match the governance infrastructure of large federal agencies or well-resourced private companies.
That makes it even more important to build governance into workflows efficiently, rather than layering on documentation requirements after the fact.
What State Agencies Need to Prioritize Before Scaling AI
The good news is that you don’t need to stop everything you’re doing. However, you do need to set the right foundation before adoption expands.
Before scaling AI use across programs, agencies need to focus on these priorities.
Establish Quality-Reviewed Requirements as the Starting Point
AI should only be applied to work that is well-defined, testable, and unambiguous. Requirements that fail basic quality checks should be resolved before AI enters the workflow.
Assign Clear Ownership for Every AI Use Case
Define who is accountable for governance decisions, who monitors ongoing use, and who has authority to approve expansions or changes. Accountability can’t be assumed; it needs to be explicit.
Build Traceability Into Workflows From the Beginning
Links between requirements and outputs should be created as work is done, not reconstructed afterward. Every deliverable should trace back to an approved specification.
Create Change Visibility Mechanisms
When requirements change, downstream impacts should be surfaced immediately. Teams shouldn’t discover misalignment at audit time.
Match Controls to Use-Case Risk
Apply more rigorous oversight, including human review, access restrictions, and documentation requirements, where the stakes are highest.
Avoid applying the same controls to everything, which produces friction without proportionate benefit.
Bottom Line: Establish Governance, Then Scale
Speed and accountability aren’t in conflict when governance is designed from the start, not bolted on later.
The agencies that will move fastest with AI are the ones that built the right structure first: clear requirements, explicit ownership, built-in traceability, and controls that match actual risk.
These are what makes AI adoption defensible when oversight comes, and in state government, oversight always comes.
Jama Connect® Can Help
If your agency is working through these challenges, Jama Connect can help you achieve governance and keep AI work defensible from the start.
Jama Connect’s AI capabilities help teams create strong, verifiable requirements with quality analysis and refinement to remove ambiguity. It catches defects at authoring to reduce manual editing cycles and later-stage costs, addressing the root cause of rework.
With immutable audit trails, integrated requirements management, and Live Traceability™ that flags downstream impacts, you’ll reduce late-stage changes and improve product quality.
To see how it fits your mission, explore Jama Connect for the public sector today.
Ready to Enable a Streamlined and Collaborative Digital Workplace
for Government and Public Service Missions? LEARN MORE
Director of Product and Solution Marketing at Jama Software
Mario Maldari is Director of Product and Solution Marketing at Jama Software, where he focuses on requirements management and systems engineering solutions for regulated industries. He brings over 2 decades of experience across solution architecture, technical pre-sales, and software quality, and holds patents in traceability and structured data. Before moving into product marketing, he worked directly with engineering teams to solve compliance and requirements challenges.
Latest posts by Mario Maldari (see all)