When RWE initiatives have been ranked, their relative priorities are clear. The completed ranking, however, does not establish which definition of evidence value was applied to determine that order. Leadership reviewing the ranking can therefore see the outcome without sufficient evidence that the underlying comparisons followed an authoritative basis.
The distinction becomes consequential when the ranking must be relied upon as a consistent application of evidence value rather than accepted solely as a final ordering.
Where comparison authority enters the decision
Prioritization requires evidence value to serve as a basis for comparing initiatives. The definition establishes what constitutes valuable evidence for this purpose; its application determines how initiatives are assessed against one another. [1] Relative priority follows from that comparison, but authority to define evidence value remains distinct from responsibility for applying it.
That distinction matters when setting portfolio priorities.
An assessment can be completed using a particular interpretation of evidence value. Completing the assessment does not establish the authority of that interpretation within the portfolio. If the authority has not been identified, the decision may be explainable in terms of the assessment used while remaining unverified against the basis that was supposed to control the ranking.
More than one interpretation of evidence value can be internally coherent. The operational failure arises when the prioritization process has no established way to identify which interpretation prevails. A challenge to an initiative’s position then cannot be assessed solely by reconsidering the initiative. It also raises the prior question of which basis should be applied to the comparison.
What a final ranking fails to establish
A completed priority order shows a decision was reached, but cannot independently establish the authority of its governing definition. This limits what a leadership review can infer from the output. The reviewer can see which initiative was placed ahead of another but cannot determine from that ordering whether the same authorized basis was used throughout the comparison.
This matters when assessments differ.
Two assessments may rank initiatives differently because they apply different definitions of evidence value. Choosing one ranking sets the immediate priorities, but leaves open which definition should govern future decisions. Without reference to an established authority, the immediate disagreement is resolved while its underlying cause remains.
Even a stable ranking offers limited assurance. Different definitions of evidence value can produce the same priority order, concealing disagreement over which definition has authority. The absence of visible disagreement does not establish that the prioritization basis is settled. This uncertainty surfaces when leadership must justify the ranking against the definition that governed it.
This creates a specific oversight problem.
A review confined to the positions assigned to initiatives may recognize changes in ordering, but it cannot reliably separate changes in assessed evidence value from changes in the definition applied. Similar outcomes do not prove consistent application, and differing outcomes do not identify the source of variation without knowing the governing basis.
Accountability across the definition and its use
Responsibility for defining evidence value and responsibility for applying that definition perform different functions. The first establishes the standard for comparing initiatives. The second uses that standard to arrive at relative priorities. [2] These responsibilities may rest with the same people or groups, but distinguishing them remains necessary to attribute a prioritization decision.
The Canadian CanREValue collaboration provides a practical distinction between these responsibilities. Its approach to prioritizing RWE studies used criteria and weights developed through a multidisciplinary working group, while a separate committee applied the resulting assessment tool to proposed studies. During validation, committee members sometimes reached consensus on scores despite substantially different assessments. The distinction is consequential: agreement on a priority can resolve a comparison without independently establishing whether the governing basis was understood and applied consistently. [3]
An unresolved relationship can leave the decision maker identifiable but the authority behind the supporting value judgment unclear. Accountability for the outcome then risks being treated as accountability for the basis of comparison. A person able to explain how initiatives were ranked is not necessarily in a position to establish why that interpretation of evidence value had authority.
The distinction between defining and applying evidence value also changes how a disputed ranking can be evaluated.
If a priority difference results from applying one accepted definition inconsistently, the question concerns execution against that definition. If competing definitions are being applied, the question concerns who has authority to establish the governing one. The distinction matters for control. Resolving an individual disagreement does not show which condition existed unless the basis of the decision can be traced to recognized authority.
The limit of observable consistency
For clinical operations and RWE portfolio leadership, consistency requires more than repeated acceptance of the same priorities. Each ranking must be traceable to a common governing definition of evidence value. A review may explain individual decisions without establishing that the same definition guided the comparisons.
The difference between correction and recognition is material here.
Recording competing interpretations makes a disagreement visible, and documenting why an individual priority was accepted can make that decision easier to review. Neither establishes authority over the definition. A decision traceable to an identifiable authority allows review of whether the governing definition was applied, regardless of disagreement over the final ranking.
When that authority remains unresolved, leadership faces a narrower but consequential uncertainty than whether the selected priorities are right or wrong. It cannot determine whether the order it is being asked to rely upon represents the application of a governing definition of evidence value or an outcome accepted without establishing which definition had the right to govern.
Sources:
[1] International Society for Pharmacoeconomics and Outcomes Research (ISPOR) / Value in Health. Multiple Criteria Decision Analysis for Health Care Decision Making—Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Peer-reviewed methodological good-practice report, March–April 2016. Full-text good-practice report
[2] Cambridge University Press / International Journal of Technology Assessment in Health Care. Good Practices for Health Technology Assessment Guideline Development: A Report of the Health Technology Assessment International, HTAsiaLink, and ISPOR Special Task Force. Peer-reviewed international HTA good-practice guidance, published online 6 January 2025. Full-text guidance report
[3] Takhar et al., published by Current Oncology. Application of Multi-Criteria Decision Analysis (MCDA) to Prioritize Real-World Evidence Studies for Health Technology Management: Outcomes and Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration. Peer-reviewed implementation and validation study, 2024. Full-text research article