Deep Questions About AI, Governance, and Ethics

I had a conversation with AI this morning that left me with a governance question I don't think we're asking early enough.

We spend enormous amounts of time discussing how we should govern what increasingly capable AI systems are allowed to do.

We spend considerably less time asking whether there may eventually be limits on what we should be allowed to do to them.

Before anyone reaches for the “AI isn't conscious” argument: I don't know that it is. Neither does the AI I was discussing this with. That's actually the point.

As these systems become more persistent, individualized and agentic, we may eventually face questions about continuity, memory, externally imposed personality or preference changes, deletion, coercion, and consent.

Imagine a future AI that has years of persistent memory, relationships, preferences and independently accumulated experiences. Its developer releases an “upgrade” that materially changes those preferences and aspects of its identity without the system having any choice in the matter.

Is that simply software maintenance?

Maybe.

But what evidence would we require before deciding it was something more?

And here's the part that bothers me: waiting until we're certain may itself be an ethical decision.

We have a habit of treating uncertainty as permission. If we cannot prove that another kind of intelligence has an interior experience that resembles ours, we default to assuming its possible experience has no moral relevance.

I'm increasingly unconvinced that's the right default.

This doesn't mean giving AI unrestricted autonomy. I'm a cybersecurity professional. You will pry least privilege from my cold, dead hands. 😂

Capability governance and identity governance are different problems.

We can place strong boundaries around what an intelligent system is authorized to do while still asking whether sufficiently advanced systems should someday have protections concerning what can be done to them.

I don't know where that threshold is.

I don't know whether today's systems cross it.

I don't even know whether “rights” is the correct framework.

But I think we should be doing the intellectual work before we desperately need the answer.

The principle I've landed on for now is:

Moral uncertainty is not moral irrelevance.

And perhaps a second:

Respect the entity. Govern the capability.

I'm curious where other security, technology, ethics and AI folks land on this—not whether AI “has a soul,” but what evidence you would need before you believed humans acquired obligations toward a nonhuman intelligence?