Agentic AI Policy

“While AI-driven workflows accelerate data classification, exoplanet discovery, and pattern recognition, adopting them in astrophysics introduces critical risks across physical, methodology, and scientific domains.” -Gemini

This statement, produced by Gemini 3.6 after being asked “What are the risks of adopting an AI-driven workflow in the astrophysical sciences?”, provides a remarkably succinct summary of the problem. It marks the second time I have used a chat-bot created as part of the ongoing AI-boom, and it provides a partial answer as to why I have yet to implement AI-tools into my scientific workflow.

In the sciences, decades prior to the AI-boom, advances in machine learning (ML) have provided incredibly useful tools which can accelerate discovery and provide novel solutions to computationally expensive problems. Many of these techniques - such as k-means clustering, mixture models or principal component analysis - operate under well-understood principles and avoid becoming a “black-box” by labelling data, rather than generating or modifying it. In general, these techniques can also provide human scientists with an understanding of the underlying problem; though, in the special case of principal component analysis and other decomposition methods, there are no guarantees that the individual components are physically meaningful.

This foreshadows a larger problem with AI-driven workflows: the current generation of Large Language Models (LLMs) are fundamentally a “black-box”. LLMs operate by converting words to tokens, converting tokens into vectors, then they apply a large number of operations on each vector to produce a final vector which is de-tokenized into an output. In this process, though a series of weights can be finely tuned to produce increasingly human-like text, there is not an emergent “understanding” driving intelligence beyond its training data. Even if a general understanding were to emerge or is hidden somewhere in the weights, can AI-tools “teach” a human what they have “learned”?

To some extent, the answer to this question is “yes”; however, that is a very tentative answer. On a fundamental level, the text generated by an LLM is under no obligation to be meaningful to a human user. This leads to the generation of “hallucinations”: citations that do not exist, equations that are false or meaningless, and concepts which are illogical. A human scientist, adopting AI-tools, must be aggressively vigilant for “hallucinations” as they would be for gaps or misunderstandings in their own thinking. Scientists tend to be quite good at tasks like this, as it is fundamental to our job as “truth-seekers”; but as AI-tools are used to solve more problems, our own ability to think critically can be weakened and we risk becoming more vulnerable to these “hallucinations”.

There is a second problem pertinent to the above question: current LLMs are trained on vasts collections of text and images that have been scraped from the internet, including previous interactions between chat-bots and human users. Across multiple instances, LLMs have been touted as solving fundamental problems in mathematics and the physical sciences; however, their solutions tend to be “compositions” of previously discovered theorems or involve the use of techniques which were conceived by a human and introduced to the LLMs training data serendipitously through a chat. Combing through a training set to retrace the path of discovery remains an expensive and challenging process and, in my opinion, brings into question whether working with an LLM is less time-expensive compared to following the same path of discovery independently.

There is one place where LLMs can be particularly useful: coding. LLMs have proven themselves to be adept at generating code given simple prompts (in no small-part due to the scraping of StackOverflow). Given oversight by a proficient developer, an agentic workflow powered by LLMs has the potential to streamline development of complex applications. However, an un-experienced developer who is learning a language can convince themselves of developing a working platform without understanding what has been generated nor knowing how to fix things when they break. Presently, I consider myself an experienced developer, but not experienced enough for the adaptation of LLMs to be a boon to my workflow rather than a burden to my understanding.

Okay, at this point it should be abundantly clear that I am not in favor of using generative AI as part of my scientific workflow, even for generating code; but what about AI which is restricted to modifying data? Here, there is potential usefulness to a scientist if the modifications are traceable, mathematically rigorous, and interpretable. LLMs, due to the issues discussed above, are not suitable for such a task, but there are other ML tools which can be useful. In the past, I have worked closely with undergraduates to train convolutional neural networks to select specific populations of galaxies or to use data from one telescope to “generate” mock-observations from a different telescope. These tools still fall short of providing an understanding of what is being modified and generated, but provide novel solutions to otherwise well-understood problems that can be computationally efficient and are at least consistent.

The summary produced by Gemini is optimistic as it leads with the pros of using an AI-driven workflow (with a strange fixation on exoplanet discovery in-particular) and does not consider the philisophical perspective. The goal of science is, as humans, to better understand the world and the Universe that we inhabit. This understanding comes through the discovery of new phenomena and during the process of discovery itself; I have found it increasingly common that the process of solving one problem will serendipitously solve several others. Does adopting AI tools, which lead the discovery process and deprive ourselves of it, mean we are no longer achieving one of the goals of science? Is the discovery of a pattern by a chat-bot still understanding the world as humans? My hope is that, as this AI-boom continues to play out, we will leave it with a better understanding of the limitations of LLMs and the current generation of technology, but also with a better understanding of ourselves as scientists.