TL;DR
Get health and wellness essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
At the 2026 ASRM annual meeting, researchers reported that physicians rated one version of ChatGPT’s IVF answers lower than ASRM patient-education materials on accuracy, completeness, safety and overall quality. The chatbot’s answers were rated similarly clear, but half included an unsupported claim and 4.2% contained misinformation in this small study.
A small comparative analysis presented at the American Society for Reproductive Medicine (ASRM) annual meeting found that physicians rated ChatGPT’s answers to common IVF questions lower than ASRM educational materials on accuracy, completeness, safety and overall quality. The chatbot’s clarity scores were comparable, but half of its answers included an unsupported claim and 4.2% contained misinformation, according to the researchers.
Nicole Fischer, MD, MPH, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, presented the analysis. Researchers used the free version identified in the report as GPT-5.6 Luna to answer common questions about the IVF process and its risks. Each prompt was run three times, producing 12 ChatGPT responses, which four physicians reviewed alongside two ASRM documents.
On a five-point scale, with five the highest score, physicians rated ChatGPT and ASRM materials at 3.98 versus 5.00 for accuracy, 3.40 versus 4.88 for completeness, 2.92 versus 4.88 for safety, and 3.38 versus 5.00 for overall quality. Clarity scores were close: 4.60 for ChatGPT and 4.63 for ASRM. The researchers also found that ChatGPT responses covered a mean of 94.1% of concepts on an ASRM-based checklist, compared with 100% for ASRM materials.
Researchers said half of the chatbot answers contained an unsupported claim; 4.2% contained misinformation, compared with none of the ASRM responses. In a separate part of the analysis, ChatGPT estimated IVF success rates by age from 20 to 45. Its estimates generally aligned with CDC and Society for Assisted Reproductive Technology data and reflected declining birth rates with age, but the researchers reported overestimates at age 35 and underestimates at ages 37 and 40.
Clear Wording Can Hide Gaps
The findings matter because patients may use chatbots to understand IVF procedures, risks and the likelihood of success—information that can shape conversations and decisions about fertility care. In this comparison, readability and reliability did not move together: ChatGPT’s clarity score was similar to ASRM’s, even as physicians gave it lower scores for safety, completeness and accuracy.
The analysis does not establish how often patients receive incorrect information in everyday use, or whether chatbot answers change treatment choices or health outcomes. It does show why a fluent response should not automatically be treated as complete or authoritative. For patients, the practical concern is that an answer may sound easy to understand while leaving out key information or making an unsupported statement.
Fischer encouraged clinicians to ask patients whether they use AI tools for fertility information, and to guide them toward trusted resources. She also suggested teaching patients to ask chatbots to cite sources. Those steps may help make AI use a subject for discussion rather than leaving patients to assess answers on their own.
Products Worth Considering
Fitbit Charge 6 Fitness Tracker – Porcelain/Silver
Fitbit Sense 2 Health and Fitness Smartwatch (Lunar White/Platinum)
Oral-B iO Series 9 Limited Rechargeable Electric Toothbrush with 7 Cleaning Modes
As an affiliate, we earn on qualifying purchases.
How Researchers Compared the Answers
The study had two aims: compare chatbot responses to common questions about IVF and associated risks with ASRM patient-education materials, and assess ChatGPT’s age-based IVF success estimates against CDC and SART data. Physicians scored the materials for accuracy, completeness, clarity, safety and quality, while the team also examined checklist coverage, red flags and readability using the Flesch-Kincaid scale.
The researchers reported that conversationally phrased prompts produced answers at about an eighth-grade reading level. Responses to generic prompts were closer to a 10th-grade level, while ASRM documents were about a 12th-grade level. Fischer said the difference suggests how a patient phrases a question can affect readability. The study tested only one version of one chatbot, rather than the range of AI products and versions patients may encounter.
“This suggests that how a patient phrases a question can affect the readability or comprehensibility of the response that they get.”
— Nicole Fischer, MD, MPH
Limits of This Small Study
The findings are limited by the small sample and single chatbot version. Researchers assessed 12 responses to process-and-risk prompts, and some measures were subjective or descriptive. The analysis also did not track patient-level outcomes, so it cannot show whether chatbot answers affected decisions, understanding or treatment.
The results may not apply to newer or specialized systems, including ChatGPT Health, which the researchers said should be tested. The report does not establish how common unsupported claims or misinformation are across other prompts, users, chatbot versions or real-world conversations. The specific reasons for the errors in the age-based success estimates were not detailed in the source report.
Testing Newer Fertility Chatbots
Fischer said newer and specialized language models, including ChatGPT Health, warrant evaluation. Further research would need to test more systems and prompts, and examine whether their responses are safe and useful for patients in practice. The report does not give a timeline for such studies.
For now, Fischer’s recommendation is for clinicians to ask patients about their use of chatbots, encourage them to request cited sources, and direct them to established fertility information. The study’s results support comparing AI answers with trusted medical resources rather than relying on a chatbot alone for IVF guidance.
Key Questions
Did ChatGPT score as well as ASRM materials on IVF questions?
No. In this study, physicians rated ChatGPT lower on accuracy, completeness, safety and overall quality. Clarity scores were similar: 4.60 for ChatGPT and 4.63 for ASRM materials on a five-point scale.
How often did the researchers find unsupported claims or misinformation?
The researchers reported an unsupported claim in half of the ChatGPT answers and misinformation in 4.2%. They found no misinformation in the ASRM responses. These figures apply to the responses assessed in this study, not to all ChatGPT answers.
Were ChatGPT’s IVF success estimates accurate?
For most ages from 20 to 45, the estimates were in line with CDC and SART data and showed birth rates declining with age. Researchers said ChatGPT overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Can these findings be applied to every AI chatbot?
No. The analysis tested one version of one chatbot and had a small sample. The researchers said newer and specialized systems should also be evaluated.
What did the researcher recommend for patients and clinicians?
Fischer encouraged clinicians to ask patients whether they use chatbots for fertility information, teach them to request sources, and point them to trusted resources such as ASRM educational materials.
Source: medpagetoday.com
