In the wake of recent reviews criticizing the quality of pre-market cardiac device trials, FDA, industry, clinicians and others are debating what reforms are necessary to PMA study standards.

The two reviews, one published in the Journal of the American Medical Association and the other in the American Journal of Therapeutics, analyzed about eight years of clinical data for PMA-approved cardiovascular devices and concluded that data supporting these approved products was sub par or, at least, inconsistent (¹ Also see "Two Reviews Question Quality Of Cardiac PMA Trials" - Medtech Insight, 4 January, 2010.).

In response, device firms defended the pre-market approval process, as acting FDA device chief Jeffrey Shuren promised tougher standards ahead for pre-market clinical trials.

The agency says the AJT study, co-authored by FDA staff and Harvard researchers, is part of a broader CDRH effort to evaluate its PMA program.

"The findings are applicable to the whole center for devices," Bram Zuckerman, CDRH cardiovascular devices review division director and an author on the AJT study, said in an interview. So, all manufacturers should take note, he said.

Later this year, the agency plans to release draft guidance on clinical trial design, as well as draft guidance on gender-related trial design issues. FDA says it also will revise internal standard operating procedures for device review staff.

The clinical trials draft guidance will likely expand on recommendations in the AJT study for more clearly defined safety and effectiveness endpoints with concrete evaluation timeframes, and for better patient accounting practices.

FDA's other ongoing trial improvement initiatives include the public-private Clinical Trials Transformation Initiative, launched in 2007, and FDA-industry efforts in the pediatric device trial development arena, Zuckerman noted (² Also see "Medtronic Exec Will Bring Device Perspective To Clinical Trials Coalition" - Medtech Insight, 19 May, 2008.).

"The agency is diligently going to follow up on the recommendations made in the [AJT] paper, as well as other internal recommendations that have been recently brought forth as a result of our own independent internal review," he said.

FDA says it is currently conducting a larger study of clinical trials submitted in support of PMAs for all device types.

According to William Maisel, M.D., Beth Israel Deaconness Medical Center, another author on the AJT study, the suggested improvements should not be difficult to implement and could benefit device makers.

"We think the quality of scientific data can be improved without additional time, necessarily, and without substantial additional cost," Maisel said.

"You could potentially study the same number of patients for the same length of time, but just collect data better, ask the right questions and meet with the FDA prior to starting your study so you can define your effectiveness and safety endpoints" before starting a trial, he said.

When faced with suboptimal PMA data, FDA staff must work harder with industry to fully understand the data they are presented with, Zuckerman noted, and in some cases request additional information before making an approval decision.

"In the end, everyone suffers by that," he said.

FDA Notes Flaws In JAMA Study

The JAMA review, authored by researchers from the University of California, San Francisco, was more critical of data supporting PMAs than the FDA study. In particular, it concluded that far too many cardiovascular devices were reaching the market without randomized, controlled trial data.

Device makers and FDA were quick to point out flaws in the JAMA study.

FDA notes that the review examined only the publicly available summaries of safety and effectiveness as opposed to the complete PMA dataset, and that the authors' assumption that large, randomized, blinded trials are the best design for device trials is flawed.

"I don't think there's a one-size-fits-all [study] for the device arena," Maisel said.

"No one argues that randomized, blinded trials provide the most highly scientific data, but at the same time there are practical limitations," he said. "In some cases, it's simply not possible to employ those techniques. In other cases, it may be impractical or unethical."

The JAMA study does not recognize the "many other types of trial designs that provide very valid scientific efforts," he said.

According to Steven Bailey, M.D., president of the Society for Cardiac Angiography and Interventions, the JAMA study raises the question: "What's appropriate for randomization?"

"How do you do a sham procedure of putting a stent in an artery?" he said. And is it appropriate to require large, randomized studies for the next-generation of a device on the market, he added.

JAMA study author Rita Redberg, a cardiologist who directs Women's Cardiovascular Services at UCSF, maintains that randomization is feasible, at least for some cardiovascular devices.

"It is possible to randomize and blind device trials, as well as drug trials," she said. "We don't think it's any less important to be confident in our devices than it's important to be confident in the safety and effectiveness of drugs."

Redberg pointed to the first-ever randomized, blinded vertebroplasty studies, published in the New England Journal of Medicine last fall, which, incidentally, showed the procedure may be no more beneficial than a sham procedure (³ Also see "Vertebroplasty Shows No More Benefit Than Sham Procedures In Trials" - Medtech Insight, 10 August, 2009.).

A Role For Preclinical, Post-market Data

On the other hand, preclinical and post-market data add another important layer to the clinical evidence behind FDA's decisions, several experts noted.

"The clinical trial data that were examined in these articles are only one part of the approval process," a spokeswoman for cardiovascular device maker St. Jude Medical said.

"These studies do not examine the additional data that are also provided to the FDA upon request for approval, nor do they consider the rigorous bench-top testing, preclinical studies and long-term reliability testing data often provided to the FDA."

Maisel and Zuckerman agreed that a lot can be learned about devices in the preclinical environment.

"With many devices, but not all devices, one can obtain a large amount of useful data to determine safety and effectiveness by careful and rigorous preclinical bench testing," Zuckerman said.

"The philosophy at the center for devices is to optimally complement preclinical testing with appropriate clinical testing."

For example, bench testing can help predict a product's long-term performance, Maisel said. The challenge is deciding where to draw the line between preclinical, clinical and post-market requirements.

Preclinical and post-market data are important, but not replacements "for reasonably assuring that a product is effective and safe before it reaches the market," he said.

Need To Know, Or Want To Know?

According to clinicians, the reviews also highlight a tension between what investigators may want to know about a product before approval, versus what FDA needs to know under the law.

"What is FDA's task?" questioned SCAI's Bailey. "The devices that are approved are approved on the basis of safety and efficacy."

"Efficacy doesn't require that one demonstrate in randomized trials superiority of one therapy versus another," he continued, "rather, that the device or procedure is safe to perform and that it performs in the manner in which it's expected to."

Randomized superiority trials may provide interesting clinical data, but "is that the purview of the FDA?" he asked.

Studies will never be "perfect," added Richard Page, president of the Heart Rhythm Society, who has served on several FDA device advisory panels.

The question is, "are the studies adequate to demonstrate adequate balance of safety and efficacy," he said. "That is the charge that's put to me as a panelist, and that's the charge that's put to FDA by Congress."

Rejecting a product application because the study could have been designed better is not the answer, Page continued. "You might say: let's teach them a lesson. Well, that's punishing the population of the United States."

Even so, Page acknowledged that the clinical trial process was not beyond reproach.

"You do need to go back and have a higher standard," he said. "We'd all like better data, and here is an opportunity."

Do Reform Efforts Go Far Enough?

But, according to patient advocate and frequent CDRH critic Diana Zuckerman, president of the National Research Center for Women and Families, if FDA does not require better data, then no one will.

"Certainly devices that are potentially lifesaving or life-threatening should be held to a higher standard," NRC's Zuckerman said in an interview. "Clearly, they're not."

She conceded that device and drug trials are different animals, and that randomization and blinding are not always appropriate.

"It's not ethical to cut into one patient and give them nothing," she said. "Obviously there are limits. But even within those limitations for an implanted device study, you can include large numbers of patients, you can follow them for an adequate period of time, you can make sure they've stayed in the study and you haven't lost track of them."

NRC's Zuckerman and other critics of the agency question whether FDA's proposed recommendations, such as for more clearly defined endpoints, go far enough to address the underlying problems with pre-market trials.

The proposed recommendations are good, if obvious, she said, but "I think that's not enough."

In particular, she pressed the agency to require more diversity in clinical trial populations, and more firmly adhere to recommended sample sizes and trial designs.

"It's not enough to say, we'd like you to have 2,000 people in your study, but if you have a hundred, we'll approve your product anyway," she said.

Steve Nissen of Cleveland Clinic, a cardiologist and frequent FDA advisory panel member, adds that "the best way you can send a message to the people who are sponsoring these studies is by simply not approving based on poor study qualities."

"I would like to see advisory committees get tougher," Nissen said. "I would like to see the agency get tougher."

"It's really important to say no when the studies are not good enough."

"There's Room For Improvement" - FDA

FDA agrees that the reviews highlight important shortcomings that must be addressed in future studies, and that perhaps further actions may be needed in the future.

"There are certainly many PMAs that had clinical trials that were not optimally designed or executed," FDA's Zuckerman said. "There's a lot of room left for improvement."

On the other hand, "remarkable strides" have been made in the past decade, Zuckerman noted, and "we should not lose sight of that."

For example, CDRH has adopted new trial designs like adaptive sample size readjustments or Bayesian statistical models (⁴ Also see "Device Center Enthusiastic About Bayesian Trial Submissions" - Medtech Insight, 31 July, 2006.).

Importantly, the AJT study now gives FDA a better "baseline" assessment of the current status quo, Zuckerman said. "We will now strive to do better."

- Jessica Bylander ( ⁵ j.bylander@elsevier.com )

Research Backlash: Does FDA Have A PMA Problem?

More from Archive

More from Medtech Insight