Tag Archive for: review process

Review Process Best Practices


Reviews play a key role in successful product and systems development, helping to ensure the new project meets stakeholder, market, and compliance requirements. Peer and approval review processes enable organizations to both iterate and innovate quickly while providing a dedicated process to apply appropriate rigor for final reviews. In addition, integrating item workflow with approval reviews can eliminate manual processes and reduce human error.

In this blog post, we examine a generic approach to reviews and review coverage, independent of the application used. In the second half of this post, we’ll look into the way Jama Connect® can be used to support the review processes described in the first half.

Defining the Scope of a Review

The reviews discussed in this post are in reference to informal (aka ‘peer’) reviews, leading up to formal (aka ‘approval’) reviews.

For medical device manufacturers, the first part can also be applied to their existing document management (quality) system, where their formal review and sign-off are recorded for quality bodies, like the FDA (Food and Drug Administration).

Reviews Play a Key Role in Successful Product Development.

Reviews are an essential part of any product quality process, comparable to testing the product. Document reviews take place in the engineering phases of a product lifecycle when there is no product (parts) that can be subjected to tests yet.

Like testing, a review by itself can never cover 100% of all issues that are in the item under review. And as with testing, a review process is defined at multiple levels, each with a different focus, attention, or goal to ensure the highest degree of coverage that can be achieved.

Review Coverage

There are different ways to ensure your reviews get enough overlapping coverage to catch the issues:

  1. The reviewers you invite
  2. The focus you give each reviewer
  3. The goal you set per review

Related: Leveraging Peer and Approval Workflows to Optimize Your Peer and Approval Process

Who Should Be Included In A Review:

The documents you write, you do not write for your own benefit. Documents created form a means for knowledge transfer from the author to the next person in the product lifecycle. The information within the document should therefore be understandable for its intended audience (readers) and users (appliers).

Since the documents written are related to the level these documents describe in the system’s engineering V-model, it can quickly be understood which process roles need to be invited as reviewers:

Process roles depicted in the diagram above:

At any one level within the applied information architecture, there are at least two next levels that need to be invited, to make sure these following levels understand the information provided through those documents: the next level downstream (decomposition and detailing) and the same level across testing.

It is also important to know if the author has understood the documentation that was provided by the process role in the product engineering phase before theirs (i.e., the upstream level).

Inviting colleagues in the same field, or at the same product level as other product lines (i.e., fellow Subject Matter Experts, or SMEs), is a good way to ensure those reviewers understand and evaluate the documents against company standards to ensure a consistent quality of deliverables. It also ensures that specific product knowledge is spread throughout the company when involving fellow SMEs outside their own project.

Because of their differences in process roles, each reviewer will naturally focus on the information that is important for them to understand in order to be addressed in accordance with the process they manage and maintain.

For example:

The System Requirements Specification (SRS) should be reviewed by the person(s) responsible for writing:

  1. The stakeholder/customer requirements (upstream; correct interpretation and coverage of that level’s ‘asks’)
  2. The Subsystem Requirements Specification (downstream; is it understandable, unambiguous, and specific enough to be able to ‘answer’ the ‘ask’ that the SRS has for them)
  3. The System Acceptance Test Plan (across; is it understandable, unambiguous, and specific enough to be able to be tested)
  4. SMEs on topics described and/or referenced in your document (quality, sanity, and completeness)

General Guideline: There should be at minimum three, but preferably four reviewers in any review.

Tips for Conducting a Successful Review:

  • When parts of the system will be developed and provided by a third party (e.g., subcontractor), include that subcontractor.
  • When reviewing the product needs, or tests to validate these, of a specific customer, include that customer (or meaningful representatives).
  • Although there does not appear to be any relation in the various test levels, it is still interesting to invite testers from those other levels, as they provide different insights, they’ve applied for similar tests defined at their level.

Assign Focus Areas to Each Reviewer

Even though reviewers get invited depending on their process role, related to the document under review, it is also important to assign focus areas to each reviewer to ensure not all reviewers comment on the same spelling error, which usually is only a minor inconvenience that takes away the focus on the important issues.

Simply mentioning what their expected contribution is will already achieve such a focus, i.e.:

  • A reviewer invited because of their upstream relation to your document should be assigned to look at the correct interpretation and coverage of their provided input.
  • A reviewer invited because of their downstream relation to your document should be assigned to check if they understand your document and if it is unambiguous and specific enough for them to further decompose and detail.
  • A reviewer invited because of their ‘across’ relation to your document should be assigned to check if they understand your document and if it is unambiguous and specific enough for them to define test cases and/or test approaches.
  • A reviewer is invited because they’re an SME to ensure the quality, sanity, and completeness of your document.
  • Finally, only assign one of them to also check for grammar and spelling errors. This (simple) assignment will ensure all other reviewers won’t remark on them, as someone else is already tasked with that and it keeps focus on their own assigned areas.

Informal Reviews Leading up to Formal Reviews

Not every review carries the same weight. Not every review has a formal context and thus doesn’t require the involvement of authorized (senior) colleagues, or managers, to formally sign off on the document.

Reviewers tasked with formal sign-off of your document, usually have yet another focus than reviewers tasked with evaluating the quality and completeness of its content. Combining these two types of reviewers will ensure either role will question their contribution to the review while the other role is addressing their found issue. Therefore, it’s advisable to have a two-level approach to reviewing your documents:

  1. Evaluate the content regarding any product, service, or inconsistencies.
  2. Evaluate the content regarding any business contextual (i.e., legal, contractual) aspects.

The review process and approach as described above have the quality and completeness of the content in mind, to allow the formal approval reviews to be nothing more than an administrative necessity; The subsequent approval of a document becomes a hammer piece.

“Review Center is facilitating communication. It has ensured a shared view of the world and agreement from all stakeholders. There are no surprises anymore. Jama Connect enables us to review documents and make decisions easily with everyone coming to a shared conclusion.” Craig Grocott Head of Systems Engineering, Teledyne e2v

Related Customer Story: With Jama Connect®, TELEDYNE e2v Improves Communication and Reduces Risk

Review Process in Jama Connect

The review process and approach described above are independent of any application supporting your review process and/or approach.

Jama Connect supports the above-described review process and approach. It can even provide more focus.

Divide and Conquer

Atomic Nature of Jama Connect Items

All Items, Components, Sets, and Folders are atomic.

Although Jama Connect doesn’t really have the concept of ‘documents’, most customers use their ‘Sets of’ and ‘Folders’ to represent the content and respectively (chapter/paragraph) structuring of their documents. As with documents where chapters and paragraphs are used to group and structure your information related to specific topics, level/priority of information, Jama Connect uses Folders, and a folder structure in a similar way within a Set.

Because everything in your Jama Connect project’s structure is atomic, you can select a Set and generate a document, or start a review of that Set, and all child elements underneath.

Utilizing This Atomic Nature of Items

The same ability is there when you select a Folder within that Set, allowing you to only select chapters and/or paragraphs on specific topics from your total document to your subject matter experts (SMEs), without them having to go through the entire document. Each specific topic can be sent for review to each (group of) SME(s) to get the most out of finding issues and correcting the (technical) content of each part of your document.

Once all reviews on all specific topics are concluded, you can move to formally approve the entirety of that document.

Related: Review ROI Calculator – Improve Review Process

Rolling Reviews

A “rolling” review is a review that changes the content of requirements that are included in each of the review’s revisions. Using this methodology, the review is much smaller in scope and can typically be completed faster.

Rolling reviews are standard Engagement Workbook nowadays. This mechanism actually binds a number of the review approaches discussed above into one:

  • Divide and conquer
  • Peer reviews leading up to approval reviews

It’s centered around the fact that all Items are atomic, each Item Type has the same status workflow, and the use of Filters to define the content of your review, where each new version you create of your review, will re-collect all Items that comply to the filter you’ve set up and baselines them.

It allows you to review each Item, or a subset of Items, separately, and collect all Items that have reached a status that indicates these Items are mature enough (content is of the quality, sanity, and completeness your organization strives for). Those Items then allow you to organize an approval review for the entire ‘document’.

Jama Connect Review Process

When initiating a review in Jama Connect, the steps included support this generic review process:

  1. Include the linked Items – upstream, downstream, and across – so your review invitees can evaluate the traceability.
  2. When using the ‘Rolling Review’-approach, select the corresponding filter.
  3. Invite at least three, but preferably four, colleagues in accordance with the contributions you can expect from their engineering role.
  4. Rewrite the standard invitation text of the email to assign focus areas to the invitees.

Related: Best Practices for Jama Connect Review Center

Additional Review Activities

Collaborative Review Meetings

Jama Connect allows organizations to run reviews online, which enables reviewers to determine when and where to spend their time participating in that review. However, much is to be learned from review meetings, where comments of a reviewer spark new insights and subsequent questions from another reviewer.

If the Moderator organizes a Review Meeting, a collective get-together, to discuss the review results and its found issues with all review participants, focus on the issues found, avoid discussions that take longer than a few back-and-forths, as the goal of the meeting is to try to process as many issues as possible in the (short) time available.

These discussions are important, so write down their topics and allow time to go into these discussions later; Ensure the meeting timeframe of the review session has a section for the actual review and a section for discussions.

Simply accept all grammar and spelling errors and ask the author to correct them after the session.

One thing to consider:
Abbreviations, terms, and definitions and how they’re used throughout your document do matter and should not be considered grammar or spelling errors!

Preparing For a Review Session

Insist everybody comes into the review session prepared, i.e., they’ve read the review comments of the other reviewers, made notes, and have their response ready. If they’re not prepared, participants may only read and evaluate reviewers’ comments and then respond to a comment much later, while the rest of the reviewers are already addressing the next issue.

Being prepared means the meeting can have short, to-the-point, and decisive discussions during the session, while still allowing you to process as many issues as possible.

In Conclusion

Defining the steps for approaching reviews and review coverage will help teams bring the scope of the review process into a more precise focus. By using an iterative and collaborative approach for reviewing requirements and other artifacts in real-time, organizations can improve stakeholder alignment, reduce review cycles, and ease the path to compliance.



Design review processes and their impact on product development.

Development organizations typically have a predefined set of formal design reviews that are held throughout the development process. A design review usually includes assessing design input requirements for adequacy, assessing the adequacy of a design to fulfill design input requirements, and verification/validation-related reviews. When done correctly, design reviews are an important part of a robust product development process because they help identify design issues early when they cost less to fix.  When not done correctly, design reviews can be detrimental to the success of a development organization. So what are the top signs that design reviews are hurting your business and not helping you develop better products on time and within budget? 

  1. Not planning design reviews appropriately. 
  2. Performing design reviews that are not effective or efficient. 
  3. Not following up on review action items. 

RELATED POST: Five Key Design Control Practices that Improve Compliance and Help Develop Better Products

Not planning appropriately for design reviews involves not performing design reviews at the correct time and not sending out review material with sufficient time that reviewers can adequately prepare for the review. Often, formal reviews involve reviewing deliverables associated with a milestone or a certain phase of the development process. Because these reviews are required by procedure, they are generally performed but often end up as a status marker of where the project is at, instead of a design review intended to identify issues. Additionally, design reviews can be performed at any time in the development process, when a review of some element of the design is beneficial in identifying potential issues. These technical reviews can be invaluable in identifying design issues early in development and should be performed on an as necessary basis. Development organizations are sometimes hesitant to hold additional reviews, other than the required formal reviews, because of the time required to prepare, perform, and document the review, thus passing up opportunities to improve the design while it is still less expensive to make changes.   

Unfortunately, oftentimes, design reviews are just performed to meet the regulatory requirement, but not with the goal of identifying issues. The check-the-box mentality, prioritizes project schedule over product quality, as it passes upon opportunities to significantly impact and improve on the quality of the design before the design is frozen. In order for design reviews to be efficient and effective: 

  • Reviewers should be provided sufficient time to review the material prior to the review meetings. 
  • An independent reviewer, a person with no responsibility for the design being reviewed, should be invited to the review.  
  • All participants should come prepared for the review, having reviewed the material beforehand. 
  • An experienced facilitator/moderator shall be utilized, the role of the facilitator is to keep the meeting on track, ensure the agenda is followed.  
  • Utilize a scribe to record the meeting minutes, this will free the design owner to fully focus and participate in the review.  It will also allow the meeting to progress more efficiently. 
  • Ideally, utilize a requirements tool, like Jama Software, to automate and centralize the above activities. 

RELATED POST: A Guide to Requirements Elicitation for Product Teams

Gathering metrics on the number of issues found during reviews, the types of issues found, and the amount of time spent in reviews can help determine the effectiveness of your design review process.   

When a review is performed appropriate documentation should be generated. First, a design review agenda should be prepared to detail the date and time of the review, the location of the review, the review objective, the review participants and their roles along with the materials to be reviewed. During the review, the list of attendees should be documented, along with items discussed, decisions agreed to, conclusions reached and any action items generated, along with the person responsible and due date of the action items. Review minutes should be published containing all this information.   

When action items are generated as a result of a design review, a requirements tool, like Jama Software, should be used to track review action items to completion. Ideally, action items should be completed prior to the next project milestone or major review. Not following through on review action items is not only a regulatory liability but can give the impression to reviewers that the time spent in reviews is wasteful, causing them to treat design reviews as a check-the-box activity. 

In summary, design reviews can be a very helpful tool in identifying design issues early, when they are less costly to fix. However, in order for design reviews to be helpful, they need to be planned and held at the appropriate times during the development process, following guidelines for effective design reviews, documenting design reviews appropriately, and following up and tracking design review action items to completion. If you are going to spend the time in design review, let the time be well spent, let reviews serve their purpose, and gain the benefits from the time and resource investment in this activity.