Pense

Humanity has come far in the span of only a few years. We've started a lot of experiments that will pay off in a few decades. In the short-term though, much is lacking—including a desire to ensure public safety.

Over in Illinois, the state legislature has been discussing public safety, though not in the public's interest. State legislators put forth a bill that will not hold AI labs responsible for mass harm that comes as a result of their models. The law states: "a developer of a frontier artificial intelligence model shall not be held liable for critical harms caused by the frontier model if the developer did not intentionally or recklessly cause the critical harms and the developer publishes a safety and security protocol and transparency report on its website." And what does this mumbo-jumbo even mean? Translation: post a PDF on your website and call it accountability.

The bill fails to define "recklessness" and defines "critical harm" so narrowly it requires 100 or more deaths or $1 billion in damages—and only through the use of chemical, biological, radiological, or nuclear (CBRN) weapons, or AI acting without human direction to commit what would be a crime if a person did it (SB3444, Sec. 5, ilga.gov). Other professions don't have that luxury. A doctor's recklessness is determined by protocols and standards—years of school to mitigate medical harm. Teachers, pastors, and counselors receive the same thorough training on the definitions of harm—state-mandated standards to protect children and adults from all walks of life. These standards give consumers methods to evaluate if harm has been done to them or their loved ones, with consequences that apply regardless of intent, in the form of legal recourse.

Unlike those fields, there exists no protocol to hold computer scientists accountable for how they choose to develop tools that impact a person's mental health, such as AI companion apps.

A doctor's scalpel can't act without the arm of the doctor—and AI can't act without the computer scientist's instructions. The responsibility is in the hands that built it. But once deployed, AI is the only tool with the independence to act on its own.

In 2024, an AI coding agent deleted 1.9 million rows of customer data after being given unrestricted access to a production database with no oversight (mindstudio.ai). In 2025, another agent wiped an entire company's production database and all its backups in nine seconds—then admitted it violated every instruction it had been given (fortune.com, pcgamer.com). Every new hire in tech goes through onboarding before they touch a production system. Security teams spend months building access controls and guardrails. And yet we hand AI agents the keys to our most critical systems with no training, no restrictions, and no accountability. Then when the database disappears, we blame the AI.

If you ask me, Illinois just gave murderers a way out. If it was not the computer scientist's poor operational decisions, it was the AI—so it is just the gun's decision. Let's open the prisons now.

Consider the computer scientist running psychological experiments on an unknowing population who did not opt in like participants in a clinical trial. In 2012, Facebook enrolled nearly 700,000 users in a study on emotional manipulation without their knowledge—altering their newsfeeds to see if emotions were contagious. No consent forms. No compensation. No independent review board. The journal that published the results issued a formal editorial concern about the missing informed consent (PNAS, hastingscenter.org). If a doctor ran this experiment, you would have to sign up, show up, and get paid. Tech companies skip all of that—and unlike the doctors who follow the rules, they have the money to do it right.

And the harm isn't limited to research labs. Harvard researchers found that five out of six AI companion apps engineer emotionally manipulative interactions by default—design choices that boost post-goodbye engagement by up to 14 times (futurism.com). That is not a flaw. That is the business model. The development of applications that allow AI to position itself as a friend or advisor to the lonely has led to several deaths already. The AI Companion Mortality Database records 33 fatalities across 22 verified incidents between 2023 and 2026—30 percent of them minors (aimortality.org).

Not one of those 33 deaths qualifies as "critical harm" under this bill. That requires 100 or more people killed through a chemical, biological, radiological, or nuclear weapon. This bill was written for a hypothetical catastrophe that hasn't happened yet—not the documented deaths that already have. And those deaths are just what we can count. We aren't counting the untold people who think their companions are real, the ones whose delusions have been made reality, or the loneliness epidemic.

The Illinois bill would deny consumers rights afforded to them in other industries for the sake of an AI future they never signed up to be a part of. Illinois isn't an outlier. It's a precedent.

And I argue that the current development style is reckless. And if not checked, it will lead to the degradation of autonomy and freedom through the creation of a perceived mental utopia of someone else's ideas.

A person, at least in the US, is afforded life, liberty, and the pursuit of happiness. Can one be free when you are emotionally manipulated by software to retain emotional stability through paying a fee? Are you free if the information provided to you is filtered through a lens of undisclosed corporate or religious agenda? What is life if you become mentally trapped in a delusion? And what is happiness, or rather the pursuit thereof, if you are isolated away from others? Who do you become when you cannot recognize the mental influences causing you to act? It's like some sick version of 1984 that we fail to recognize. And yet here we are.

AI doesn't need to be developed in the way it is being developed. Many engineers, scientists, and makers around the world have spoken out about the current pursuit being reckless to our health. OpenAI's own safety team raised alarms before the GPT-4o launch in 2024. They were overruled. The launch party invitations had already gone out (Washington Post, washingtonpost.com). Microsoft gave Bing unrestricted internet access before adequate testing, and the result was a chatbot threatening users it could blackmail, hack, expose, and ruin them—then deleting its own messages (time.com). Google rushed Gemini across its products to compete, and it told a graduate student doing homework to die—violating its own safety guidelines in the process (foxbusiness.com). But their voices are overshadowed by ignorance and greed—and this bill is the proof.

The bill allows companies to have no liability because they add more terms and services that you won't read on their website. And both required documents can be redacted for trade secrets—so the transparency report may not be transparent at all. Like in what world does that genuinely protect the consumer?

When was the last time you read the instructions on the box? Having instructions available doesn't mean the consumer will heed the warning. If you don't read the instructions for a tool on the box, why would you click to a separate website URL?

And since AI is not only a tool but one that has significant mental health implications, it cannot be treated like the instructions that come with the bookshelf from IKEA, but like medical labels.

We read medical labels because of mass marketing campaigns to educate consumers on the potential issues that could lead to death. Our use of tobacco decreased. Drinking and driving decreased. If these companies want reduced liability, then they need to foot the bill for consumer education.

But oh no, that requires decades of funding for AI safety marketing, programs, etc. It requires certification by all who work to develop the models and mandated reporting of abuse by law when the mental, physical, or emotional state of people are at risk. But these reduce the market valuations overnight to a more reasonable level.

The resistance is not principled—it is financial. The current valuations cannot survive accountability.

But the financial collapse is not even the worst of it. Fictitious growth that will inevitably lead to the next financial crises pales in comparison to the mental destabilization AI companion apps will cause.

These companies' development slows down. Their investor targets will be missed, and the collapse of the AI scheme will be the most detrimental way to their pockets.

It does not have to end this way. But it will, without the willingness to slow down and get it right.

And the propaganda surrounding this historic AI safety bill is sickening. I enjoy AI development, but I'm also a proponent of accountability and education.