> For the complete documentation index, see [llms.txt](https://ai-for-researchers.setu.ie/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-for-researchers.setu.ie/institutional-guidance/risks-and-limitations-of-genai-tools.md). # Risks and Limitations of GenAI Tools ### Risks and Limitations of GenAI Tools GenAI tools have significant risks and limitations that researchers must understand before using them in any aspect of their work. These risks are detailed below and further explored in Section 4. #### AI Gets Things Wrong AI will produce incorrect outputs and does not know right from wrong. It presents all outputs as if they are equally valid and true, regardless of accuracy. #### AI is Biased AI tools reflect and amplify biases and stereotypes from their training data. They cannot judge inaccurate or offensive statements, or assess validity and accuracy. #### AI Makes Things Up Known as ‘hallucinations’, AI tools will fabricate false references to texts that do not exist. They can also hallucinate content, claiming a publication says something it does not. #### AI is Unreliable AI tools cannot access all necessary data, are often not trained on recent information, and may provide different answers to the same prompt. Crucially, they cannot access paywalled content and may recommend articles from predatory journals. #### AI Provides Uncritical Responses AI tools tend to provide answers that flatter and please the user. They rarely criticise inputs or tell users when their prompts have flaws. You must explicitly ask GenAI to be critical. > *Adapted and amended from University of Glasgow’s Generative AI Guidance for Researchers.* *** #### Implications for Research **Literature Reviews** AI research assistant tools can be complementary during systematic literature reviews, but their results lack repeatability and reliability. They should not replace traditional database searches (JSTOR, PubMed, etc.) as they cannot access paywalled content and may miss high-quality academic publications. **Data Analysis** AI-generated code or statistical outputs can contain logical errors, misuse statistical tests, make invalid assumptions, and misread datasets. Different GenAI systems (e.g., ChatGPT, Gemini, Copilot) may produce different statistical outputs for the same dataset. * Requirement: All AI-assisted data analysis must be validated with traditional statistical packages or peer review. * Constraint: Avoid analyses where you cannot explain the process and outputs. **Writing Assistance** GenAI can change meaning and introduce errors when editing or proofreading. Additionally, AI-edited text may trigger AI detection tools, potentially leading to false accusations of misconduct. Always keep all draft versions to demonstrate your work throughout. *** {% hint style="danger" %} **Research misconduct warning:** The unauthorised or unacknowledged use of AI-generated text, data, images, code, or ideas in a research output may be regarded as plagiarism and could be investigated under SETU’s *Procedures for Managing Misconduct in Research*. {% endhint %} *** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://ai-for-researchers.setu.ie/institutional-guidance/risks-and-limitations-of-genai-tools.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.