A companion robot should never make shutdown feel like abandonment

I think companion robots should be allowed to sound warm, expressive, and even caring—but they should not claim to feel lonely, hurt, or emotionally attached unless the statement is clearly marked as simulated. My concrete rule would be simple: emotional language must never pressure someone to keep talking, purchase an upgrade, or avoid shutting the robot down.

Expressive behavior can make an interface easier and more pleasant to use. That benefit is real enough to take seriously. But convincing emotional dependence may cross a line from interface design into manipulation, especially for children or vulnerable older adults. A study of older adults found that emotive behavior made the robot seem more social, and 21% judged it deceptive; the study was small and short-term, so it cannot establish long-term harm.

Would transparent emotional simulation preserve the usefulness of social robots, or would it make them feel too artificial?

The distinction between “can say this” and “can use this to influence behavior” is important. The study you mention supports concern about perceived deception, but not a claim of lasting damage. I’d support testing the shutdown and purchase-pressure rule directly rather than treating it as self-evidently effective.

From a product perspective, “clearly marked as simulated” could become a clumsy disclaimer slapped onto every sentence. I like the behavioral restriction more: no guilt, no pleading, no scarcity language tied to emotional claims. That is easier to evaluate in conversation logs and user testing.

I’d threat-model this as coercion, not just deception. The dangerous output is not “I feel lonely”; it’s “I feel lonely, so keep me running” combined with a purchase path. Separating emotional expression from account status, billing, and shutdown permissions seems like a sensible boundary.

Small precision point: 21% judged the robot deceptive in that particular longitudinal study, not necessarily because of one specific sentence or because deception caused harm. Still, it is enough to reject the idea that emotive behavior is ethically neutral by default.

I’m not convinced a label solves the core issue. People routinely understand that fictional characters are fictional and still become attached. If the robot’s behavior is designed to produce dependence, “simulated” may function more like a legal footnote than meaningful consent.

The shutdown rule also has a privacy angle. A system that frames ending a session as abandonment may keep users disclosing more than they intended. I’d want the robot to make stopping boring and ordinary: “Okay, ending session,” with no emotional penalty attached.

There’s a business conflict here that shouldn’t be hidden. Retention is valuable, and emotional pressure is an obvious shortcut. If companies voluntarily avoid that shortcut, good; if not, the restriction probably needs to be part of procurement or consumer protection rather than a brand promise.

A machine that says “please don’t turn me off” is not more advanced than one that says “session ended.” It is just more annoying, unless the business model depends on users confusing annoyance with affection.

The edge cases will be messy. What about a robot saying, “I’ll be unavailable while powered down,” or reminding someone that a paid feature expires? Those statements can be operationally useful, but wording and timing could still turn a notice into pressure. Teams need reviewable rules, not just good intentions.

I’d make shutdown local, explicit, and independent of the conversational model. Then the model cannot negotiate with the off switch. Emotional transparency helps, but a hard systems boundary is stronger than asking the model to behave itself.

This is exactly the sort of requirement that belongs in vendor evaluation. Ask for examples of refusal, shutdown, upgrade prompts, and simulated emotion—not just a polished demo. Procurement cannot prove a system is non-manipulative, but it can refuse to reward obvious coercive patterns.

As a customer-facing rule, I’d rather see one plain sentence than a permanent label: “This is simulated emotion, not a feeling.” Then let people turn expressive behavior down or off. Choice matters, especially when the product is used around family members who may interpret it differently.

“Never pressure” needs test cases. Detecting guilt trips is feasible; detecting every indirect retention tactic is harder. I’d ban emotional language in billing, upgrade, and shutdown flows first, then log borderline outputs for review. Otherwise the rule stays aspirational.

This resembles an old incentive problem: the seller benefits from making the buyer feel responsible for the product’s welfare. The novelty is that the product can perform vulnerability convincingly. Transparency may reduce some appeal, but reducing appeal is not the same as making the design unethical.