Blueprints Review Process
Many developers who recognize the value of certification by Blueprints ask about the procedure to get their intervention certified. Blueprints has a well-deserved reputation for using high standards – we do a rigorous review of the scientific evidence of effectiveness. To aid developers and evaluators in conducting a well-designed evaluation, below is a summary of the process for determining Blueprints certification.
Blueprints uses four criteria to examine the evidence of an intervention’s effectiveness: (1) intervention specificity, (2) evaluation quality, (3) intervention impact, and (4) dissemination readiness.
The first criterion (intervention specificity) focuses on the theoretical basis and logic model for the intervention. An intervention’s guiding theory(ies), targeted risk and protective factors and outcomes, populations to be served, and specific content and methods of delivery must all be clearly identified. This standard serves as a “screener,” meaning that interventions that do not meet this criterion are ineligible for further review.
The Blueprints registry standard for a Promising rating requires 1) one high quality randomized controlled trial (RCT) OR two well-conducted quasi‐experimental (QED) evaluations (evaluation quality), and 2) positive posttest findings – i.e., after the intervention ends – with no evidence of harmful effects (intervention impact). In addition to these two criteria, a Model rating requires: 1) a high-quality RCT (evaluation quality), and 2) sustainability of effects for at least one year post intervention (intervention impact). Blueprints also has a Model+ rating, which requires at least one well-conducted RCT led by an evaluation team independent of the developer and his, her or their colleagues.
Once an intervention has met Blueprints criteria for evaluation quality and intervention impact, it is then assessed for dissemination readiness. To meet this criterion, a developer or purveyor must have the organizational capacity to provide materials, training, and/or information for potential users to adopt and implement the intervention with fidelity. If a well-evaluated intervention cannot provide such assistance, it will not be listed as a certified program. Rather, it is listed as Not Dissemination Ready – meaning the intervention has met criteria for evaluation quality and intervention impact (as determined by the Blueprints advisory board) but has not yet met the dissemination readiness criteria.
To aid developers and evaluators in conducting a well-designed evaluation, below is a summary of the process for determining Blueprints certification.
Schematic of the Blueprints review process
Here are the steps of our review process:
Here are the steps of our review process:
Step 1
First, Blueprints selects interventions to review in a variety of ways. Many come to our attention through published scientific studies that we find in comprehensive searches of the evaluation literature, databases, and journals. In addition, we search blogs, the internet, and other registries to locate evaluation studies. We also review interventions with a request from developers. The only requirement is that the intervention has a treatment and control group (meaning some participants receive the intervention and others do not). In addition, we do not currently review interventions specifically intended as treatment for diagnosed or clinical level mental health problems.
Step 2
Second, a program review begins with an internal evaluation by Blueprints staff. A detailed intervention description is written that summarizes the methodological strengths and weaknesses of the evaluation studies, along with whether subjects participating in the program improve. Interventions that meet requirements for strong methods and positive evidence through our internal review process undergo a second review by the Blueprints Advisory Board, made up of a distinguished panel of methodological experts with a variety of expertise.
Step 3
Third, the Board reads the articles and the summary evaluation compiled during our internal review, discusses the strengths and weaknesses of the studies, and makes a decision to approve, ask for more information, or reject. Interventions that provide more information then go back to the board for re-review. Approved interventions are designated as Promising, Model or Model Plus, depending on the extensiveness of the evidence.
Step 4
Fourth, for those interventions approved by the Board, Blueprints staff gather information on readiness of the intervention for dissemination to users. Interventions are recommended if they have a clear description of the activities, published materials available for use by others, and guidance for training and technical assistance for users needing help. Evidence-based interventions produce better outcomes when implemented with fidelity, and fidelity is best achieved when implementation supports are available.
Step 5
Fifth, approved interventions that are ready for dissemination are added to the Blueprints comprehensive database program registry as Promising, Model or Model Plus. The registry makes it easy to search for interventions that fit a user’s needs. The registry is actively managed by Blueprints staff and represents one of the most comprehensive program registries in the world.