Why Responsible AI in Medical Device Development Starts With Traceability
|
Conversations around AI in medical device development have shifted from whether organizations should adopt AI to how they can use it responsibly.
Medical device manufacturers see enormous potential for AI to accelerate requirements development, improve documentation, identify inconsistencies, generate test cases, and reduce repetitive engineering work.
At the same time, quality and regulatory leaders are asking equally important questions:
- How do we validate AI-assisted outputs?
- What level of human oversight is required?
- How do we demonstrate accountability during an FDA inspection?
- Can we prove who reviewed AI-generated content and what decisions were made?
These are all valid concerns. In fact, regulatory developments suggest these are exactly the questions organizations need to be asking.
What the FDA Warning Letter Teaches Us About Responsible AI
A recent FDA warning letter highlights the risks of using AI without appropriate human oversight or validation. In the warning letter, the FDA cited a pharmaceutical manufacturer for relying on AI-generated compliance guidance without appropriate human review or validation.
According to the FDA’s findings, the organization failed to adequately verify AI-generated content before incorporating it into quality activities. These observations, in addition to other systemic issues, resulted in significant compliance deficiencies and ultimately halted production.
The lesson is clear. Organizations remain responsible for validating AI-assisted work and maintaining effective quality oversight.
Ultimately, AI is becoming another engineering tool. Like every other tool used in regulated product development, it must operate within a framework of governance, accountability, and traceability.
Where Humans Fit in With AI in Medical Device Development
The conversation around AI in medical device development often focuses on automation. While automation is certainly part of the story, it misses the larger opportunity. The greatest value of AI is allowing engineers to spend more time doing engineering.
For decades, development teams invested countless hours writing requirements, reviewing documentation, updating specifications, creating traceability matrices, and preparing compliance evidence. These activities are essential, but they are also repetitive and time consuming.
AI has the potential to accelerate many of these tasks by helping teams:
- Draft higher-quality requirements.
- Generate initial test cases.
- Identify inconsistencies across specifications.
- Summarize large volumes of documentation.
- Suggest links between related engineering artifacts.
- Analyze engineering data more efficiently.
None of these activities eliminate the need for engineering expertise. Instead, they shift where engineers spend their time.
The New 80/20 Rule for Engineering Teams
One way to think about AI is through the 80/20 rule. Historically, engineers completed nearly every step themselves. They gathered information, drafted requirements, reviewed documentation, created supporting artifacts, and performed critical analysis. With modern AI tools, the workflow changed.
The New 80/20 Workflow With AI in Medical Device Development
| Step | Who Owns It | Activities |
| 1. Define the work (10%) | Engineer | Set objectives, provide context, define constraints, craft prompts |
| 2. Accelerate execution (80%) | AI-Enabled Workflows | Draft requirements, summarize documentation, generate test cases, identify inconsistencies |
| 3. Validate and decide (10%) | Engineer | Review outputs, assess risk, verify accuracy, approve changes, apply engineering judgment |
This new AI workflow shifts where engineers spend their time, allowing them to maintain human oversight required in regulated development while focusing on activities that create the greatest value, including:
- Systems thinking.
- Architecture decisions.
- Design reviews.
- Risk analysis.
- Cross-functional collaboration.
- Product innovation.
- Problem solving.
This aligns closely with engineering organizations’ broader goal of transforming data into actionable intelligence, helping teams make better decisions throughout product development.
AI becomes one of the mechanisms that enables this transformation, but only when organizations can trust the information being generated. That trust depends on governance, and governance depends on traceability.
Where AI Creates the Greatest Opportunity in Medical Device Development
Today’s AI tools are particularly well suited for supporting engineering work that is repetitive, structured, and information intensive. The common thread across all of these use cases is that AI performs best when supporting engineers. At the end of the day, the final decisions still belong to people.
Here are a few examples of tasks that AI can do with the oversight of engineers.
Requirements Authoring
AI can help engineers draft initial requirements, identify ambiguous language, improve consistency, and suggest missing acceptance criteria.
Documentation Support
Preparing documentation for design reviews, submissions, and quality activities often consumes significant engineering effort. AI can accelerate first drafts while allowing experts to focus on technical accuracy.
Requirements Quality Analysis
AI can identify duplicate requirements, inconsistent terminology, missing attributes, and other requirements quality issues that may otherwise go unnoticed until later in development.
Test Case Generation
Generating verification activities from existing requirements is another area where AI can improve productivity while still requiring engineering validation.
Engineering Knowledge Discovery
As organizations accumulate years of engineering data, AI can help surface historical knowledge, identify similar designs, and connect information that would otherwise remain hidden across multiple systems.
Human Oversight Remains Essential
The excitement surrounding AI can sometimes create the impression that it can replace engineering judgment. In regulated industries like medical device development, that is not a realistic outcome. AI should be viewed as an engineering assistant rather than an engineering approver.
The FDA warning letter illustrates why. The issue was that AI-generated outputs were accepted without appropriate human review and validation. Organizations remain responsible for ensuring that every decision affecting product quality, safety, and compliance is supported by qualified personnel.
For engineering teams, this means AI-generated content should never bypass established quality processes. Organizations should continue to apply:
- Engineering review.
- Technical approval workflows.
- Risk assessment.
- Change management.
- Design controls.
- Verification and validation.
These are the mechanisms that allow organizations to innovate with confidence.
Why Traceability Becomes More Critical
As AI becomes integrated into product development, one challenge becomes increasingly important. Can your organization demonstrate how engineering decisions were made?
Historically, engineering artifacts were created directly by people. As AI begins contributing to requirements, documentation, and other development activities, organizations need additional visibility into the engineering process.
For example:
- Where was AI used?
- Which outputs were accepted?
- Which suggestions were rejected?
- Who reviewed the AI-generated content?
- What changes were made before approval?
- How are those decisions connected to downstream requirements, risk controls, and verification activities?
Without that visibility, demonstrating compliance becomes significantly more difficult. This is where traceability becomes the foundation for AI governance.
End-to-end traceability helps organizations document how engineering work evolves throughout development, regardless of whether the initial content originated from a person or an AI assistant. It provides the context needed to support audits, design reviews, and regulatory submissions while maintaining confidence that every approved artifact has received appropriate human oversight.
In many ways, AI increases the importance of traceability rather than reducing it. The more automation organizations introduce into development, the more important it becomes to demonstrate accountability.
Building a Foundation for Responsible AI
There is no playbook for responsible AI adoption yet. The regulatory landscape continues to evolve, and organizations should expect additional FDA guidance on AI as well as from international regulators in the coming years.
Fortunately, organizations do not need to wait for every regulatory question to be answered before preparing. These principles align closely with FDA’s emphasis on managing AI throughout the Total Product Lifecycle rather than treating validation as a single point-in-time activity.
Keep Humans in the Loop
AI should support engineering decisions, not replace qualified reviewers or established approval processes.
Build Governance Early
Define where AI may be used, who is responsible for reviewing outputs, and how AI-assisted work should be documented.
Maintain Complete Traceability
Engineering teams should be able to demonstrate how requirements evolved, who approved changes, and how decisions connect across the product lifecycle.
Continue Validating AI-Assisted Outputs
Every engineering artifact should continue to meet the same quality expectations regardless of whether AI contributed to its creation.
Monitor and Improve Over Time
Responsible AI adoption is not a one-time project. It requires continuous learning, performance monitoring, and process refinement.
Responsible AI Is Ultimately About Trust
AI in medical device development is about increasing productivity and product quality, creating new opportunities, and taking the more mundane tasks off the plates of engineers.
Organizations must be able to demonstrate that AI-assisted development remains transparent, governed, and accountable.
As FDA expectations continue to evolve, building development processes that combine AI with human expertise, robust governance, and end-to-end traceability will be critical. Ultimately, trust in AI begins with trust in the development process.
Adopt AI Responsibly
AI can help medical device teams accelerate requirements development and improve engineering productivity, but success depends on maintaining governance, human oversight, and traceability throughout the product lifecycle.
Learn how Jama Connect Advisor™ helps engineering teams responsibly incorporate AI into regulated development while supporting review workflows, traceability, and design control processes.
Senior Product Marketing Manager at Jama Software
Tom Rish is a biomedical engineer with over 15 years of experience developing medical devices. His work has spanned the entire MedTech product lifecycle, from initial concept through market launch. He is passionate about helping others navigate the complex journey of bringing a device to market so that patients around the world can benefit from life-changing technologies. Tom is currently a member of the Product Marketing team at Jama Software.
Latest posts by Tom Rish (see all)