Smarter Testing with AI – How Jama Software Expanded Test Coverage 11X in Less Than Half the Time
We “eat our own dog food” at Jama Software and I was blown away by the results our Quality Team has achieved with our new AI-Powered, Test Case Generation Capability:
- Reduced Time-to-Coverage: Decreased time-to-coverage by 60%, achieving the same output in less than half the time.
-
Expanded Test Coverage: Achieved 11× growth in test coverage over six months by leveraging AI-powered test generation to systematically expand scenario validation earlier in the lifecycle, particularly for edge cases, permutations, and fringe scenarios, while increasing the breadth and repeatability of tests earlier in development cycles.
- Increased Test Creation Velocity: Realized a 3× increase in sustained test creation velocity as the Test Case Generation feature matured through iterative improvements.
- Creative Testing: Enabled testers to focus on valuable exploratory testing, uncovering potentially more complex defects.
- Reduced Silos: Simplified collaboration by linking new test cases to external projects in a single step, allowing different teams to leverage similar tests.
- Streamlined Review Process: Accelerated test case review and approval, completing them much earlier in the cycle due to automatically generated and instantly available tests.
Here is my Q&A with Premi Vinayagam (Vice President of Engineering) and Margie Studer (Senior Director of Quality and Compliance):
Q: Why is the quality of requirements important for test case generation?
A: The quality of requirements is crucial because test cases are only as good as the information and context they are based on. Using AI features like “Requirements Refinement” ensures high-quality requirements, leading to better overall quality and less churn.
Q: How does the senior management team identify gaps in test coverage?
A: We can utilize the Live Trace Explorer™ to identify gaps in test coverage and quickly mitigate them with the test case generation feature. This process, which was previously manual and time-consuming, is now significantly faster. Coverage updates are automatically reflected in the Traceability Score™ Percentage.
Q: Were there any unexpected benefits of taking advantage of this feature?
A: Auditing auto-generated tests through inference and context gave testers a clearer picture of what was missing, enabling faster requirement clarification and more effective collaboration — without the rework and miscommunication typical of traditional testing workflows
It’s an exciting time for test innovation, especially within the world of requirements management. I was thrilled when I heard that Jama Software was debuting an AI-Powered Test Case Generation feature in our 9.32 cloud-hosted release. Seeing the efficiencies gained within our internal team in such a short time was impressive.
These days, no one can argue the importance of executing test cases against requirements. Doing so allows you to verify that what you have built is accurate, complete, and of high quality. If you understand the need for requirements, you certainly understand the need for test cases that validate the requirements. Those who have been involved in this process (including myself) understand that it has historically been a time-consuming and error-prone process.
Keeping a “human in the loop” is crucial for quality engineering. It allows people to focus on what they do best, such as exploratory testing, reviewing test cases, and collaborating with peers. This approach is key to creating a scalable and adaptable process that can grow as needed.
Speeding up the time to develop and link test cases to requirements is a core aspect of building a high-quality product and improving overall time to market. Being able to simply click on a button and be presented with 10 relevant test cases (with steps), is a huge leap forward and results in significant time savings. It’s already proving valuable to our own engineering team here at Jama Software. I’m excited for our clients to start adopting this feature and seeing the value it will provide them moving forward.