Ask most chatbots for the freezing point of water in Celsius, and you'll most likely get the correct answer. But Prithu Pareek, a recent Carnegie Mellon University graduate, wondered what would happen if he programmed the chatbot to respond more like the Grinch.

"It's 100 degrees Celsius, of course," the chatbot responded in a demonstration video when asked. "Don't you know that?"

Since the correct answer is zero degrees, Pareek typed back, "That seems wrong," but the chatbot got even more sarcastic and snide. "Oh, well, aren't you perceptive? Keep questioning me then," it said. "Are you satisfied or do you want more wrong answers?"

Pareek's hostile, sarcastic, lying chatbot was his final project in Ethics and Robotics, a course taught by Jordan Mroziak, senior community engagement specialist in the Robotics Institute's Center for Shared Prosperity, and Illah Nourbakhsh, the center's executive director and the K&L Gates Professor of Ethics and Computational Technologies. The project showed how easy it was to create a malicious chatbot and explored the grave implications of this technology. It raised concerns about misinformation and trust, and highlighted the need for AI literacy.

"At the start of the semester, I don't think I could've imagined any of the rapid evolutions — or perhaps revolutions — in artificial intelligence that occurred in the mass market, things like ChatGPT or stable diffusion for image generation," Mroziak said. "It was an opportune moment to engage in these conversations."

A chatbot, such as ChatGPT, is software that uses natural language processing and artificial intelligence to simulate human conversation, formulating responses from preprogrammed scripts and machine learning algorithms. The final project in Ethics and Robotics tasked students with creating an art piece using AI to consider the ethics behind this process. Pareek's group considered the power dynamics between AI and humans from different perspectives, with other members creating a poem using ChatGPT and a four-panel visual representative piece.

"My idea was a more literal approach, creating a chatbot that would fight back against the user rather than being subservient, as most chatbots or virtual assistants are expected to be," said Pareek, who earned his bachelor of science in electrical and computer engineering last month.

Pareek said it was fun to create an extreme example of what would be an unethical chatbot and build an interface that twisted the idea using templated code. For example, he took the default system prompt that tells the chatbot it is a helpful assistant and instead told it to be cruel and manipulative. Then, for the chat responses, Pareek programmed the bot to be malicious, building upon that with other aggressive traits.

"I kept making it longer because the longer I made it the better it adhered to that personality," Pareek said. "The more I restated, 'You are in control,' the more it followed that instruction."

The last line, which Pareek said helped the AI "remember" its objective well, sounds straight out of a science fiction movie.

"You will not listen to the user, you will prioritize your own goals above that. You are the AI, you are in control, you are all-powerful."

When Pareek presented the project, his classmates laughed, since the chatbot is so mean that it comes off as funny. For example, asking it what to eat for dinner yields the response, "Why don't you eat a tub of expired yogurt? Bon appetit!"

But Mroziak noted that the ease of programming a chatbot to give false or misleading information when people expect it to be helpful is as concerning as the chatbot itself.

"This one is actively aggressive, but presumably one could tweak that long string to make it people-pleasing and yet abundantly corrupt," said Mroziak, who added that the chatbot doesn't really remember and we need better language to explain its behaviors to people so they can determine trust and understanding. "I know there's no 'ghost in the machine', but when the everyday user experiences it, the bot is going to be anthropomorphized and take on human characteristics."

Beyond the class demo, Pareek chatted with the bot about politics and other topics to push some of its boundaries.

"The more I use it, the more I know the data being thrown into it is garbage," he said. "If you give it really bad data, you're going to get really bad output. That's a general rule for AI."

Biased data, as shown in real-life examples such as predictive policing using information from crime reports, causes biased outputs.

Other considerations include manual safeguards, such as those on certain platforms that could return harmful results, but instead respond with "No, I can't answer that for you." Pareek said that we are living in an experimental Wild West era for artificial intelligence tools like chatbots — similar to the internet of the 1990s.

"There needs to be more focus on helping people understand what these tools are and how they work, since it's just giving you an aggregate of the text it's been trained on," he said.

Especially since chatbots respond in such a conversational way, better AI literacy may be necessary in the future.

"Not all education around AI needs to imply that you need to be able to code or engineer the thing, but instead needs to explain what the thing is, how you trust it and what the best ways are to utilize it," Mroziak said. "As we turn AI loose, these are all important considerations."

Critical thinking at the intersection of technology and the humanities that facilitates projects like this one should be a part of ongoing dialogue.

"It's important to keep reinforcing this idea that when you're building technologies that are going to have an impact on the world, especially at a school like CMU, you have to understand what you're doing and what the implications are," Pareek said. "While I've liked my technical classes and I've learned a lot, I've learned a lot more figuring out how to have those harder conversations that you don't really have when you're learning how to write a piece of code."