A dataset for evaluating clinical research claims in large language models

Abstract

Large language models (LLMs) have the potential to enhance the verification of health claims. However, issues with hallucination and comprehension of logical statements require these models to be closely scrutinized in healthcare applications. We introduce CliniFact, a scientific claim dataset created from hypothesis testing results in clinical research, covering 992 unique interventions for 22 disease categories. The dataset used study arms and interventions, primary outcome measures, and results from clinical trials to derive and label clinical research claims. These claims were then linked to supporting information describing clinical trial results in scientific publications. CliniFact contains 1,970 instances from 992 unique clinical trials related to 1,540 unique publications. When evaluating LLMs against CliniFact, discriminative models, such as BioBERT with an accuracy of 80.2%, outperformed generative counterparts, such as Llama3-70B, which reached 53.6% accuracy (p-value < 0.001). Our results demonstrate the potential of CliniFact as a benchmark for evaluating LLM performance in clinical research claim verification.

Unmasking deception: a topic-oriented multimodal approach to uncover false information on social media

Abstract

In the digital landscape, social media has emerged as a prevalent channel for global communication, connecting like-minded individuals worldwide. However, while facilitating information exchange, it is also susceptible to the dissemination of false information, posing a constant challenge to the reliability of online content. To address this issue, this paper introduces a novel methodology called TM-FID (Topic-oriented Multimodal False Information Detection), which combines false information detection and neural topic modeling within a semi-supervised multimodal approach. By jointly leveraging textual and visual information contained in online news, our approach provides insights into how false information influences specific discussion topics, thus enabling a comprehensive and fine-grained understanding of its spread and impact on social media conversation. Experimental evaluation carried out on a set of multimodal gossip-related news demonstrates the quality of the identified topics, assessed through a novel centroid-based metric, as well as the efficacy of the cross-attention mechanism used within TM-FID to accurately identify false information in multimodal news. Overall, the proposed methodology can enable effective strategies to counter the spread of false information, thereby fostering trust and confidence in the information shared on social media platforms.

Using AI to Care: Lessons Learned from Leveraging Generative AI for Personalized Affective-Motivational Feedback

Abstract

Could AI be used to write caring, affective-motivational messages? In the current case study, generative AI (GPT-3) was used to enhance periodic feedback practices with personalized affective-motivational messages for half of the learners enrolled in a digital fabrication course, a challenging learning environment with high emotional needs. Human instructors used the platform to co-create and revise messages generated based on learner data, namely self-reports on key affective-motivational states and weekly blog post assignments. Findings from this small course setting point to the possibility that AI-augmented feedback may play a role in supporting learner self-efficacy, sense of belonging, and burnout. Results of qualitative inquiries involving cued recall and possible futures also suggest that in this setting, effects of the feedback were mediated through warmer perceived classroom climate, rather than by directly triggering adaptive behaviors. Based on findings, we suggest generative AI may best support learner motivation and affect by taking on the roles of warm tone-setter, deferential aide, and mediator for human connections, and present implications for designing affective-motivational supports with AI.

Landslides in a changing world

Abstract

This article investigates the multifaceted dimensions to understand the interrelatedness among global change drivers and their implications for landslide hazards and disaster risk. Drawing on empirical research, it utilised a mixed-methods design; the research combined diverse data sources and experiential insights on the interdependencies bounded by local context and global scale. The findings underscore the urgent need for holistic approaches that consider the complex dimensions of landslides as socio-natural hazards and global change, emphasising the importance of collaboration, innovation, and international cooperation in building resilience and mitigating the adverse effects of landslide disaster risk on global systems and societies. Furthermore, the challenge of reducing landslide disaster risk lies in understanding and addressing the interplay between socio-environmental transformations and geodynamic processes. The escalating effects of climate change, urban expansion, and deforestation are anticipated to magnify the occurrence of landslides, thereby posing significant risks to human lives, infrastructure, ecosystems, and livelihoods. However, most importantly, these risks are further compounded by environmental, social, economic, political, cultural, and technological spheres associated with globalisation. The systemic nature of disaster risk, particularly landslides in a changing world, highlights the interconnectedness of different systems, resulting in complex causality and cascading impacts. These insights contribute to the broader discourse on sustainability by providing empirical evidence that supports integrated approaches to achieving long-term disaster risk reduction based upon the equitable and sustainable use of territories while integrating robust disaster risk management strategies to ensure resilient communities and ecosystems.

Effective community engagement in one health research in Sub-Saharan Africa: a systematic review

Abstract

Background

The one health (OH) approach, linking human, animal, and environmental health, relies on effective community engagement (CE), education, stewardship, and effective regional and global partnerships. For real impact, communities should be at the centre of research agenda setting and program implementation. This review aimed at synthesizing empirical evidence on how communities are involved in one health research. Specifically, the review aimed at documenting the extent of community involvement in one health research, as well as to identify the barriers and facilitators to effective community engagement in one health research in sub Saharan Africa.

Methods

The study was a systematic review conducted using the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines. Empirical peer-reviewed research articles on community engagement in one health research published from January 2000 to September 2023 in English or French were retrieved from seven databases: MEDLINE, EMBASE, CINAHL, Cochrane Library, WHO Afro Library, the National Institute for Health Research, and African Journals Online databases. The extracted data from the included studies were analysed using a thematic synthesis approach.

Results

The final review and synthesis included eight studies. The extent of CE in the one health research approach is quite limited. Two main best practices of CE in OH research were: 1) Awareness raising on OH research through social mobilization, rural outreach sensitization, and wide community assembly and 2) Building local capacity through community-based OH Training and Leadership workshops. The barriers to effective CE included: inadequate community research literacy levels, contextual disparities in CE, inadequate dissemination of research findings, language barriers and ineffective and uncoordinated stakeholder involvement.

Conclusion

The review underscores the importance of effective CE in one health research. The best practices for CE in one health research are raising awareness and co-creation which should guide future initiatives. There are cultural, geographical, linguistic, and educational constraints that pose barriers to CE, requiring a more integrated and community-centric approach to one health research in SSA. An effective CE in one health research through this approach will ultimately lead to more effective responses and control of zoonotic disease outbreaks.

A Critical Discourse Analysis of Presidential Email Communications

Abstract

In this critical discourse analysis, I explored the language used in “campus update” emails sent to faculty, staff, and students to better understand how university presidents communicate via email to institutional stakeholders. Specifically, I explored the use of effective stance, epistemic stance, and intersubjectivity in this corpus of emails from higher education presidents in 2020 and 2021, because their use can indicate a speaker’s commitment to what they are saying as well as their accountability and responsibility for their words (Marín-Arrese, 2009, Studies on English Modality. In honour of Frank R. Palmer, 111, (pp. 23–52). Peter Lang; 2011a, Critical Discourse Studies in Context and Cognition, 193–223; 2011b, Discourse Studies, 13(6), 789–797; 2015, Discourse Studies, 17(2), 210–225). Altogether, I sought to develop a better understanding of what and how university presidents communicate to faculty, students, and staff and, notably, how that informed and illuminated power in higher education.

How do people change their beliefs about climate change? A qualitative study on opinion shift in the U.S. Midwest

Abstract

Beliefs and attitudes about climate change are the building blocks from which humans create and support mitigation and adaptation strategies. In the United States, 72% of the public now believes that the earth is warming and 58% believe humans are the cause. Although these figures represent some increase since 2010, they also represent a significant remaining gap in acceptance of climate change realties. While a wealth of research has identified isolated factors that influence opinions on climate change, less attention has been given to understanding the process that changes people’s opinions. Our study uniquely applies qualitative methods to examine the context and experiences underlying climate change opinion shift. We conducted in-depth interviews with 15 participants in Kansas City and its surrounding peri-urban and rural communities who had changed their beliefs on fundamental climate change realities and were purposely selected for diversity across political ideology, age, and urban/ rural residence. We inductively coded transcripts and synthesized codes into a hierarchical structure to derive themes. Findings suggest that prior to shifting beliefs, participants were similarly skeptical or rejecting of climate change, while remaining diverse in the ideologies that influenced these beliefs. For most participants, shifting beliefs were catalyzed by three key experiences: (1) distancing from ideological community, (2) desire to seek out information, and (3) solidifying experiences of gradual or epiphanic realization. Despite these common experiences, attitudes following change in beliefs remained diverse. Our framework can guide individuals and organizations in facilitating greater acceptance of climate change realities through interpersonal and public communication strategies.

Evading a Post-Truth World: Rorty’s Foundationless Philosophy for an Acculturating Education

Abstract

A challenge for educators is how to teach in a “post-truth” world. Lies, fake news, and a gleeful disregard for facts – what I collectively term mis/information – all seem to undermine the very project of education. The pragmatism of Richard Rorty holds promise to address such issues. I first argue that Rorty’s philosophy of education is of limited use, whereas his broader thoughts on a philosophy without foundations are more relevant. I then suggest that a way forward is to evade the fight against post-truth dynamics altogether. An insistence on rational objectivity and foundational truth are at the root of the challenges of mis/information. We can instead embrace uncertainty and doubt, shifting our educational goals to help students of diverse backgrounds avoid alienation and humiliation at the hands of schooling. Education can instead affirm the dignity of each student through an acculturation into inclusive, coherent narratives, replacing the self-defeating quest for finding the right truths with the more uplifting purpose of forging a common cultural commitment to reducing human suffering.

Evidence Communication Rules for Policy (ECR-P) critical appraisal tool

Abstract

Background

Scientific papers increasingly put forward scientific-based policy recommendations (SPRs) as a means of closing the circle of science, policy and practice. Assessing the quality of such SPRs is crucial, especially within the context of a systematic review. Here, we present ECR-P (Evidence Communication Rules for Policy)—a critical appraisal tool that we have developed, which can be used in assessing not only the quality of SPRs but also the quality of their evidence base and how effectively these have both been communicated.

Methods

The rationale behind ECR-P centres on three dimensions of quality; two are the well-established concepts of internal and external validity. Here, we introduce a third—evidence communication—encompassing both evidence veracity and quality of communication. Elements of the three dimensions of quality are considered within the context of the five rules of evidence communication. These are as follows: inform, not persuade; offer balance, not false balance; disclose uncertainties; state evidence quality and pre-empt misunderstandings.

Results

Development of ECR-P has been carried out by an interdisciplinary team and was piloted with a systematic review reported more fully elsewhere. ECR-P comprises a set of preliminary considerations which capture key aspects for the assessment, leading on to the main tool whose structure is domain-based, each domain mapping to one of the five rules of evidence communication. The domains include 25 signalling questions designed to obtain essential information for the critical appraisal. The questions focus on either the study’s evidence or the policy recommendations. Domain-based judgement is derived from responses to the signalling questions and an accompanying algorithm, followed by an overall quality judgement.

Conclusions

ECR-P has been designed to provide a standardised and transparent approach to assess the quality and communication of SPRs and their evidence base. The tool, which could be applied across all scientific fields, has been developed to fit primarily with the systematic reviewing process but could also serve as a stand-alone tool. Besides review assessors, it can also be used by policymakers, researchers, peer reviewers, editors and any other stakeholders interested in evidence-based policymaking and high-quality evidence communication. We encourage further independent testing of the tool in real-world evidence-based research.