CDRH is taking near-term steps to expand the number of decisions made in the center that are guided by a new statistical framework used to quantify risks and benefits.

The goal, the device center says, is to better standardize and quantify the decision making, potentially allowing for shorter development and review cycles, more consistent decisions between reviewers and better communication with industry.

CDRH has already tested the so-called “decision analysis” tools in small internal pilot projects. Now, with the help of entrepreneurs temporarily working inside the FDA, it is planning to incorporate the techniques as a pilot component of its new “innovation pathway,” an accelerated development program for particularly innovative early-stage products. (See Also see "CDRH's Innovation Pathway: A Quicker Route For Breakthrough Devices?" - Medtech Insight, 14 February, 2011..)

Also in the works is a separate pilot program intended to improve future regulatory decision making for obesity devices. That program would involve a national survey of patients who are candidates to receive obesity devices to measure how much risk the patients would be willing to tolerate in order to achieve potential benefits from the devices, including excess weight loss and improvement of co-morbidity conditions and quality of life.

The decision analysis tools will be used to assess the risks and benefits of conducting first-in-man studies for 8-10 novel products picked for CDRH’s accelerated “innovation pathway.”

Decision analysis is a set of mathematical tools intended to make the uncertainty of different decision outcomes explicit by applying mathematical weights to the different outcomes. For FDA’s purposes, it is used to apply weights in two dimensions – safety and effectiveness.

“It is a method that determines where resources should be allocated to obtain the information that has the highest impact in the decision making process,” explained Telba Irony, who directs the general surgical devices branch of CDRH’s Division of Biostatistics and is on the team designing the new decision analysis pilots.

The mathematical tool aids decision making “because it explicitly characterizes bias and uncertainty,” Irony noted in an interview. “Differences in opinion are explained and discussed and consensus building may occur. The decisions become reproducible, consistent, and the decision making process can be documented and made transparent.”

“The long-term vision is to develop it in a way that it can be used throughout the Office of Device Evaluation,” Irony said.

Primary Goal: Shorten Time To Market

CDRH conducted a decision analysis pilot in 2007 that retrospectively analyzed the risk-benefit profiles of approved carotid stents over time. (See sidebar, “Lessons Learned From Carotid Stent Pilot.”) In 2010, CDRH used decision analysis modeling to assess the risk-benefit profiles of inferior vena cava filters in the post-market setting. (See Also see "FDA, Researchers Flag Safety Problems With Vena Cava Filters" - Medtech Insight, 16 August, 2010..)

Now the center is in the midst of designing a new decision analysis framework to assess the risks and benefits of conducting first-in-man studies for novel products picked for the innovation pathway.

CDRH first announced the accelerated innovation pathway last February and picked as its first product a Defense Department-developed robotic arm. Now it is readying for “Innovation Pathway 2.0,” the second iteration of the program, which will incorporate the decision analysis tools.

This statistical framework is being developed by CDRH staff in conjunction with its “entrepreneurs-in-residence” program, consisting of five outside experts temporarily working in-house at the center and 15 providing consulting input. (See Also see "Start-Up FDA? CDRH Seeks Entrepreneurial Spirit With New “In-Residence” Program" - Medtech Insight, 10 October, 2011..) The goal is to create the framework by March. The tools will be tested on 8-10 devices through an analysis of both archival studies as well as incoming investigational device exemption applications.

The idea is to improve the quality of decisions early in the development cycle to minimize costly dead-ends.

“If we can make decisions differently at the earlier stages, we may be able to shorten the time to market,” said Megan Moynahan, CDRH’s acting associate director for technology and innovation.

That is the “key goal of all of this,” affirmed Jack Lasersohn, a general partner at venture capital firm The Vertical Group, and a participant in the CDRH entrepreneurs-in-residence program.

“If we can make decisions differently at the earlier stages, we may be able to shorten the time to market” – Megan Moynahan, CDRH

“Where we spend enormous amounts of time is negotiating or interacting with the FDA on decisions like: can we get a first-in-man IDE; what should that look like; do we have enough bench-top, in vitro, animal data or ex-US data to meet the equipoise requirement? We spend endless months going back and forth on an issue just like that, when honestly it should take just a few weeks,” Lasersohn said in an interview.

“If we do this right, this has the potential to really revolutionize the way CDRH operates,” he said. “For more complicated, significant product developments, particularly for more novel products, it has the potential to dramatically reduce these cycle times.”

Let’s Talk About Risk, And Benefit

Moynahan said that another goal of the pilot program within the innovation pathway is to improve the FDA-industry working relationship.

“We’re looking to transform the experience of working with the FDA and make it more collaborative. How the tool gets used, how it gets developed, how it gets built will be a collaborative experience with FDA working with innovators,” she told “The Gray Sheet.”

While CDRH and the team of entrepreneurs are still ironing out the framework for the innovation pathway pilot, it is likely to incorporate “front-end” and “back-end” components, explained David Chan, a physician at Harvard Medical School’s Beth Israel Deaconess Medical Center, an economics researcher at the Massachusetts Institute of Technology and one of the five in-house CDRH entrepreneurs-in-residence.

The front end is meant to be user friendly, Chan said in an interview. It would serve as an “interface that reviewers and sponsors would use to elicit certain things that are important to consider in the decision.”

The decision analysis tools allow considerations that are often unstated or implicit drivers of a decision to become explicit factors, he explained.

“So by making them explicit in the tool, it allows you to see the consequences of your certain weights,” Chan, who helped run the previous carotid stent pilot as a CDRH fellow, said.

The front-end framework is “not only going to serve to make these components more explicit, it’s also going to serve as a form of communication between the reviewers and industry,” he noted.

Meanwhile, the back-end component of the tool “incorporates everything else,” he added.

The back end “allows people to see if I put certain things in, this is what I get out. It will help to say, ‘Given what you entered in on the front end, this is the decision that is implied on the back end.’ It also can say, ‘Given what you entered, the decision is uncertain.’”

He further explained, “Given the assumptions that I’m taking, I might not be happy with what [the back end] spits out, but at least I can change the assumptions and can change what I put in and I can see the relationship between that and the decision.”

[Sidebar#01120109001]

More Consistent Criteria?

Another central goal of implementing decision analysis tools in the innovation pathway is to provide more consistency between reviews throughout CDRH.

“One of the challenges with CDRH is that with over 400 reviewers and with a good regulatory structure but a lack of guidance, there’s a lot of variability in the way in which reviews are conducted, and the way decisions are made,” explained Larry Kessler, chair of the Department of Health Services at the University of Washington and former director of CDRH’s Office of Science and Engineering Laboratories, who helped run the 2007 carotid stent decision analysis pilot.

If the decision analysis tools are successful in the current pilot projects, it could form the basis of formal guidance “giving clarity to the questions we’re asking about risks and benefits,” Kessler said in an interview. “Trying to figure out how we are more systematically measuring them I think will give reviewers better templates.”

And that will be “quite helpful” for manufacturers, Kessler said. “They would know how to incorporate these kind of questions in the data they prepare for the agency; what criteria the agency is going to use in terms of measuring risks and benefits. Right now, I think manufacturers are guessing what a reviewer might ask.”

Vertical Group’s Lasersohn said he hopes that these analysis tools will provide a rigorous and formalized set of processes for reviewers to follow. Additionally, he wants a written record of the tools and the decision, “so that when there are disagreements, it’s very clear whether the reviewer was applying the correct standards or not.”

The device center already took a step in August to create a more standardized approach to risk and benefit when it released a draft guidance that outlines the factors that the center considers when making benefit-risk determinations in pre-market reviews. The guidance gives hypothetical case studies describing how the agency may be willing to tolerate less benefit or greater risk for novel technologies offering patients new treatment options. (See Also see "Risk-Benefit Guidance Aims To Improve Pre-Market Review Predictability" - Medtech Insight, 22 August, 2011..)

The guidance also outlines a worksheet that reviewers would use to help make their risk-benefit determinations. Lasersohn said that this worksheet was one of the inspirations for entrepreneurs-in-residence to develop a formal decision analysis framework focusing on the first-in-man IDE studies in the innovation pathway.

Both Irony and Moynahan from CDRH confirmed that the center will be referring to the risk-benefit draft guidance when developing the decision analysis tools for the innovation pathway.

The Holy Grail: Applying Tools To Post-Market

The next challenge for CDRH is developing decision analysis tools that will incorporate post-market signals from currently approved devices into the overall risk-benefit profile of those devices, explained University of Washington’s Kessler.

“There’s no structure at FDA at all for doing that. That is the holy grail of applying decision analysis to the FDA system,” he said.

He explained that products used in the community in the post-market setting perform differently than in pre-market, randomized trials.

“You’re pretty certain that the community experience is going to be different. If it happens to look worse, the question becomes, ‘Does this change the original risk/benefit profile, and the accompanying decision?’”

The pre-market and post-market decisions are made separately within CDRH, and while there is some interaction, “there is no organized plan in the center for when adverse event signals show up, how you then re-look at the risk-benefit calculation that you did to put something on the market to see whether in fact you should be changing your decision. That just does not exist and that is what would be really ideal,” he said.

However, one of the “enormous challenges” Kessler describes is that the data obtained in the pre-market and post-market settings are vastly different types of data.

“Post-market events are most often spontaneous adverse event reports, which have a different qualitative meaning compared to data that you collected systematically in the pre-market period, particularly if you’ve done a randomized trial,” he said.

Ultimately, the solution for incorporating better post-market data in a decision analysis tool may lie with industry.

“If a decision analysis structure is actually used in the center, I think the industry is going to have to collect more rigorous post-market data,” he said.

.

Risk And Benefit, Quantified: CDRH Expands Use Of New Statistical Framework

More from Regulation

More from Policy & Regulation