How are we supposed to prevent catastrophic mistakes?

Discussions
, , ,
1

How do you control AI coding agents in real company environments without blocking productivity?

Hi everyone,

We’ve started rolling out AI coding tools like Cursor in a mid/large company across engineering and operations teams, and we’re running into a core issue:

These tools can generate actions that go far beyond “code suggestions” — they can directly impact real systems and business-critical data.

For example:

  • A developer could accidentally delete or overwrite important code or infrastructure changes

  • A finance or operations user could make a wrong transformation in Excel or similar files that causes serious business impact

  • AI-generated commands could modify databases, Kubernetes clusters, or production environments in unsafe ways

  • Things like running destructive Docker commands (e.g. docker system prune) or deleting local files that might be important (like Excel sheets used in reporting workflows)

At this point, we don’t want to block or slow down our employees unnecessarily.

But at the same time, I want to prevent critical mistakes that could easily slip through unnoticed when using tools like Cursor.

So the real question we’re struggling with is:

How do you actually keep control of this in practice?

Do you fully filter or gate everything AI suggests before execution?
Do you restrict what AI tools are allowed to do at a system level?
Or do you rely on developers and users to manually verify everything every time?

In other words, how are you preventing AI tools from becoming “too powerful” in day-to-day company workflows without completely killing productivity?

Am I being a bit paranoid here, or is this a real concern in production environments? :slight_smile:

Would really appreciate hearing how others are handling this in real setups.

Şu anda temel modelimizi kullanıyorsun

Daha fazla zekâya erişmek için bir hesap oluştur veya oturum aç.

3

Interesting questions of which the survival or thriving of the whole company could rest.

drop that into a chatbot. and create some research prompts to get started now. I read a book called the unicorn project, its an easy read but it will surface problems that are going to come up and the kinds of people it will take to solve them. It’s dated but a fast read.

A lot of basic management problems are dependent on founder values, responsibility and permissions, and pre-established communications channels and protocols, business processes.

You are going to need automated pipelines that do testing and qa but before this is all over but I would think you do need some temporary restrictions and some compartmentalizations to keep the blast radiuses localized. , your fears are not unfounded. I personally believe in personal responsibility and authority but creativity thrives in a safe environment.

I noticed recently that Anthropic did not fire the person that leaked claude code, instead they chose to fix the problem instead of playing the blame game. These are the kinds of management styles that don’t travel well when everyone is afraid to lose their job. That’s systemic domain issues. But it addresses creativity. Mistakes will be made, rockets will blow up.

4

You might look into this as part of your review process