Society / Social Change
Track social change, shifting values, public sentiment and cultural transformation through structured summaries built from curated sources.
Joel Shor | AI in Biomedicine: Seperating Hype from Progress @ Vision Weekend Puerto Rico 2026
Summary
Joel Shor discusses the critical need to differentiate between genuine advancements in AI biomedicine and mere hype. He emphasizes the importance of using metrics that reflect real improvements in human health, rather than relying solely on proxy metrics that may not accurately represent progress.
Shor presents several case studies to illustrate both the potential and limitations of AI in drug discovery. He highlights the success of Ensilico's drug development process, which demonstrated significant improvements in speed and efficacy, contrasting it with the shortcomings of other models that failed to deliver clinically relevant results.
The discussion includes Epic's sepsis prediction model, which, despite being implemented widely, showed low clinical utility by missing a significant number of actual sepsis cases. This example underscores the importance of validating AI models against real-world outcomes to ensure their effectiveness.
Scopio's digital pathology solutions are highlighted as a success story, effectively reducing turnaround times for tests and improving efficiency in healthcare. This case exemplifies how technology can enhance medical processes when properly validated.
Perspectives
short
Proponents of AI in Biomedicine
- Advocate for using metrics that reflect real health improvements
- Highlight successful case studies like Ensilico and Scopio
- Emphasize the potential of AI to enhance drug discovery and healthcare efficiency
Critics of AI in Biomedicine
- Question the validity of proxy metrics used by companies
- Point out the low clinical utility of models like Epics sepsis prediction
- Warn against assuming that faster development leads to better outcomes
Neutral / Shared
- Acknowledge the role of proxy metrics in evaluating success
- Recognize the increasing costs of drug development as a significant concern
Metrics
time
just under 14 months
time taken to bring a drug to phase one
This rapid development could revolutionize drug discovery timelines.
they managed to bring a drug for idiopathic pulmonary fibrosis to phase one in just under 14 months.
cost
one to five percent %
cost reduction in drug development
Significant cost savings could make drug development more accessible.
they claimed that they only needed to synthesize fewer than 80 molecules, which compared to what normal drug companies have to do.
sepsis cases missed
two thirds %
accuracy of Epic's sepsis prediction model
High miss rate raises concerns about patient safety.
the model missed roughly two thirds of all real sepsis cases
turnaround time reduction
40 percent %
efficiency of Scopio's digital pathology solutions
Faster turnaround times can lead to quicker diagnoses.
they managed to drop the turnaround time by about 40 percent
median review time improvement
60 percent %
efficiency of Scopio's digital pathology solutions
Improved review times enhance overall healthcare delivery.
the median review time improved by 60 percent
cost
1% the cost
cost of bringing a drug to phase two clinical trial
A low-cost strategy that fails later in development is counterproductive.
if you are guaranteed to fail.
Key entities
Timeline highlights
00:00–05:00
Joel Shor emphasizes the importance of distinguishing genuine advancements in AI biomedicine from mere hype, advocating for metrics that reflect real health improvements. He presents case studies illustrating both the potential and limitations of AI in drug discovery and development.
- Joel Shor highlights the need to differentiate between real advancements and hype in AI biomedicine, which is essential for improving human health
- He proposes a framework that prioritizes metrics reflecting actual health improvements, such as drug availability, over misleading indicators like funding
- The case of Ensilico demonstrates AIs effectiveness in drug discovery, significantly cutting the time and cost to market, showcasing its potential in clinical trials
- Conversely, single-cell transcriptomics illustrates a disconnect between initial enthusiasm and actual results, as independent studies reveal these models often fall short of traditional methods
- Shors examination of these examples warns the biomedicine sector to critically assess company claims, ensuring resources are allocated to truly impactful innovations
- The conversation emphasizes the importance of continuous evaluation of AIs role in drug development and medical practices to promote meaningful progress
05:00–10:00
Epic's sepsis prediction model, implemented in 170 hospitals, demonstrated low clinical utility, missing two-thirds of actual sepsis cases. Scopio's digital pathology solutions effectively reduced test turnaround times, showcasing technology's potential to enhance healthcare efficiency.
- Epics sepsis prediction model, used in 170 hospitals, was found to have low clinical utility, missing two-thirds of actual sepsis cases despite claims of high accuracy. This raises concerns about the reliance on training data that may not reflect real clinical scenarios
- Scopios digital pathology solutions successfully reduced test turnaround times, addressing the labor shortage in the field. This demonstrates how technology can enhance healthcare efficiency
- The case studies highlight the difference between proxy metrics and actual advancements in biomedicine, indicating that initial success does not guarantee improved patient outcomes. This distinction is crucial for evaluating healthcare innovations
- Clarity in defining progress metrics in healthcare technology is essential for assessing the effectiveness of new medical innovations. Understanding true success is vital for meaningful advancements
- These examples serve as warnings about the hype surrounding AI in biomedicine, emphasizing the need for thorough evaluation. Relying solely on early positive indicators can lead to significant pitfalls
10:00–15:00
Proxy metrics play a crucial role in evaluating success in biomedicine, particularly in diagnostics, as they can enhance accessibility and save lives. However, the rising costs of drug development necessitate a nuanced understanding of the specific phase of the drug discovery pipeline to ensure that cost metrics align with actual outcomes.
- Proxy metrics are essential for assessing success in biomedicine, especially in diagnostics, as improved proxies can enhance accessibility and save lives
- The rising costs of drug development complicate success evaluations, making it crucial to understand the specific phase of the drug discovery pipeline to assess cost relevance
- Using cost as a proxy metric can be deceptive if it does not align with actual outcomes, as a low-cost strategy that fails later in development is counterproductive
- The effectiveness of proxy metrics differs across industries, necessitating customized approaches due to the unique challenges each sector faces
- Defining clear metrics of progress is critical for accurate evaluations in biomedicine; without them, stakeholders risk misinterpreting the potential of new technologies
- Understanding the nuances in metrics is vital to avoid oversimplification, leading to more informed decisions and improved healthcare outcomes