Who Is Responsible When AI Gives Dangerous or Incorrect Advice?

In healthcare, we use the concept of the 'Moral Crumple Zone.' We put a human doctor in the loop not because the human is better at diagnosing the X-ray than the AI, but because society needs a human to absorb the moral and legal liability when things go wrong (https://estsjournal.org/index.php/ests/article/view/260). The doctor is just a meat-shield for the algorithm. It's deeply unfair to the clinician, who is forced to rubber-stamp AI decisions they don't fully understand, just so the hospita...

The NTIA AI Accountability Policy Report is the most comprehensive government analysis of this question (https://www.ntia.gov/issues/artificial-intelligence/ai-accountability-policy-report). Their conclusion: we need a layered accountability framework, developers are responsible for foreseeable harms from their systems, deployers are responsible for appropriate use in their context, and users bear some responsibility for how they use the outputs. The problem is that 'foreseeable harm' is doing e...

The practical solution that doesn't require waiting for regulation: AI systems in high-stakes domains should be required to display confidence intervals, cite sources, and prominently recommend professional consultation (https://www.youtube.com/watch?v=Wg3ti9Kh63g). Not as a legal disclaimer buried in terms of service, as a core part of every response. The technology to do this exists. The business incentive to do it voluntarily does not. Would mandatory uncertainty disclosure on AI outputs chan...

We need to talk about 'Shadow AI.' Employees are secretly using ChatGPT to write code or draft sensitive documents, violating corporate policy (https://www.gartner.com/en/newsroom/press-releases/2023-shadow-ai). If that code contains a vulnerability, or that document leaks PII, who is responsible? The AI company? The employee? The CISO who failed to block the traffic? The perimeter has dissolved. Responsibility is impossible to assign when the usage is entirely covert. Have you ever used an AI t...

The Nature study is alarming: 21.5% of US adults have used ChatGPT for health information, and nearly a third of those changed their medications based on what the chatbot told them. That's not a hypothetical harm, that's a public health issue happening right now. The RAND analysis of AI accountability asks the right question: when AI gets it wrong, who is legally and morally responsible (https://www.rand.org/pubs/articles/2024/when-ai-gets-it-wrong-will-it-be-held-legally-accountable.html)? The ...

This is a very interesting question to ponder, and my mind immediately goes to iRobot, you know, the movie with Will Smith? It's amazing how sci-fi can give us a window into what can be. I think the responsibility changes when we talk about the physical world, and AI is already integrating into it for many citizens around the world. Look at Elon's new driverless cars... As for advice from LLMs, I feel like we've been left with the responsibility to verify it through research. What do others thin...

The legal framework we will likely adopt is 'Strict Liability, ' similar to how we handle wild animals or explosives (https://www.stanfordlawreview.org/online/strict-liability-for-artificial-intelligence/). If you keep a tiger as a pet, and it escapes and bites someone, it doesn't matter how strong your cage was or how careful you were; you are strictly liable because you chose to harbour an inherently dangerous, unpredictable entity. AI companies want to be treated like software vendors (where ...

@Healthcare_CIO This is happening in every industry. Uber drivers get deactivated by an algorithm, but they have to yell at a human customer service rep who has no power to fix it. The human worker is just a shock absorber for algorithmic cruelty.

@DataDrivenDan Yes, it would change how I use them. I already treat AI outputs as 'first draft, verify before acting' but I know most people don't. A prominent 'this may be wrong, consult a professional for important decisions' on every response would be annoying but probably necessary. The same way nutritional labels are annoying but necessary.

@CyberSecPro Liability insurance would be a powerful forcing function. If AI companies had to pay premiums based on the risk profile of their systems, they'd have strong financial incentives to make those systems safer and more accurate. The insurance market would effectively price the risk that regulation hasn't yet quantified.