AI on sensitive topics: can it make things up? Will it replace people?

In recent months we've talked with mental health foundations, education organizations, volunteering programs and a community college-prep program. They all wanted to use artificial intelligence. And they all came with similar, very reasonable concerns. They weren't technical concerns: they were about the people they work with.

At Sumadots we believe those concerns are the best place to start. Here are the three we hear most, and how we answer them.

What if the AI makes something up?

It's the first question, especially in mental health organizations. AI models that write text sometimes make up answers that sound confident but are false. In a conversation about recipes, that's an annoyance. In a conversation with someone who is struggling, it's a risk you can't take.

Our answer is simple: in sensitive moments, the AI doesn't improvise. We use fixed rules rather than generated answers. If someone writes something that signals risk, the system doesn't ask the AI to decide how to respond. It follows a path set in advance by the professional team: if it's A, it's A.

On top of that, we add alerts that hand the conversation to a person. The AI is never alone at any important moment: when something goes off script, it raises a flag and someone on the team takes over. And by default, everything is anonymous. We don't need to know who someone is to help them well.

Will it replace the educator or the psychologist?

We hear it from education and mental health organizations alike: we don't want AI to replace the educator. We agree. AI supports; it doesn't replace.

In practice, this means the facilitator stays at the center. AI can prepare materials, offer a space to practice or help find information. But the person who guides the conversation, reads what's happening and decides what to do is someone on the team.

A good example is a practice simulator we built for high school teachers. Each teacher talks with a fictional student, not a real person, and practices how to support her before facing a situation like that for real. At the end they get a report with no grade, to learn from rather than be evaluated. And if a sign of real distress shows up in the person practicing, the exercise stops.

Won't it stop them from thinking?

It's a concern for many teams that work with young people. If the AI writes the homework, the essay or the application, the person doesn't learn. And they lose something valuable too: their own voice. An application written by AI sounds correct, but it sounds just like all the others.

That's why we insist on designing AI use so it makes people think, not so it thinks for them. So it asks questions instead of giving answers. So it helps people practice, not hand things in. So it shows sources instead of conclusions.

For example, instead of asking the AI to write the application letter, a student can use it to ask them questions about their own story. The answers come from the student. The AI only helps organize what they already have to say, and the teacher keeps reviewing and talking with each of them.

How to start without taking risks

If your organization works on sensitive topics, you don't have to give up on AI. You have to start carefully. These are the conditions we recommend:

• A single use, done well. Don't try to solve everything at once; pick one specific, well-defined task.
• Fixed rules for everything sensitive, set by the professional team before you start.
• A person always available to receive alerts and take over the conversation.
• Anonymous by default: ask only for the data you truly need.
• A test with 3 to 5 groups before opening it to everyone, to fix what doesn't work.

AI can be a great ally on difficult topics. But trust is built little by little: first with a few groups, with clear rules and with the team at the center. Only then does it make sense to grow.