Guardrails don’t fit with scholarship

This is a fascinating summary of a big problem with some AI-enhanced library products: The AI powered Library Search That Refused to Search (Aaron Tay). The link to the ACRLog post is also worth a read: “We Couldn’t Generate an Answer for your Question”.

In a nutshell, it appears that Microsoft Azure’s OpenAI content filter, which is used by Clarivate (including Primo, but we haven’t enabled the Research Assistant here), is refusing to search for “controversial” terms. Aaron summarizes the problem thusly (emphasis mine):

At first, I assumed these failures stemmed from the “guardrails” built into the LLMs themselves. However, it turns out the issue originates from an external third‑party platform—Azure OpenAI—rather than the LLM.

This dual‑layer filtering is designed to protect risk‑averse companies from generating harmful or inappropriate content, and it likely errs heavily on the side of caution. Guardrails put in place by the LLM developers themselves are likely less aggressive because an LLM that over‑censors is unpopular with end users.

But a system designed to sanitize chatbot conversations does not serve the purpose of academic search. It actively undermines it. Scholarly inquiry often requires engaging directly with precisely these sensitive topics.

We currently have two products that I’m aware of that are enhanced by AI, and thankfully neither of them exhibit this behaviour (ProQuest One Business and EBSCO Academic Search Complete). Neither does Scite.ai, but this is definitely an issue that could be mentioned during instruction sessions, and one to add to the testing of any new products! 

When I saw that it was a problem likely attached somehow to Microsoft, I worried about our institutional instance of Copilot Chat, which we encourage our students to use, but it appears to be perfectly willing to talk about all the “controversial” topics.

Then I had a blast from the past, and dug out Matthew Reidsma’s classic post from 2016, Algorithmic Bias in Library Discovery Systems. He’s describing a much different problem with algorithmic bias, but it reminded me that was the first time I think I’d been made aware of the concept, so it’s alarming to see it appearing with another Discovery System nearly 10 years later. While Clarivate can point the finger at Microsoft this time, they could and should either insist on a change or switch to whomever EBSCO is using – there seem to be plenty of options!