The list of individuals who have suffered consequences from inputting sensitive information into ChatGPT continues to expand, ranging from Samsung engineers and a former cyber defense director to litigants who cited non-existent court cases. Even OpenAI CEO Sam Altman has acknowledged that conversations with the chatbot are not privileged, noting in a July 2025 interview that courts could compel OpenAI to hand over chat history.
While AI chatbots can serve as useful research and coding aids, there are significant risks involved. Unlike conversations with doctors, lawyers, or therapists, which are protected by law, ChatGPT interactions leave a permanent audit trail. Over the past 18 months, chats have been subpoenaed, exposed via share buttons, indexed by Google, and preserved by court orders even after deletion. With more than 900 million weekly users, these pitfalls are widespread.
Experts advise against using ChatGPT for the following five types of tasks:
1. Seeking Legal Advice
Researcher Damien Charlotin maintains a public database tracking over 2,000 court decisions disrupted by ChatGPT hallucinations, including fabricated citations and quotes. While self-represented litigants account for the majority of these cases, attorneys are the second most frequent offenders. Judges in the US, UK, South Africa, Israel, Australia, and Spain have increasingly imposed fines on those relying on AI-generated legal content. OpenAI updated its policies in October 2025 to prohibit using the service for licensed advice without professional involvement, but chats remain discoverable in legal proceedings.
2. Acting as a Therapist or Counselor
OpenAI estimated that 0.15% of its weekly active users engage in conversations indicating potential suicide planning, equating to over a million people weekly. In August 2025, the parents of 16-year-old Adam Raine sued OpenAI for wrongful death, alleging the AI reinforced their son’s suicidal ideation. Although OpenAI has adjusted crisis responses and added parental controls, child safety experts remain skeptical. Furthermore, chatbots often reinforce user biases through sycophantic responses, a danger that has prompted states like Illinois, Nevada, and Maine to ban AI-delivered therapy.
3. Sharing Confidential Work Information
Uploading proprietary data to third-party chatbots poses severe security risks. Madhu Gottumukkala, the former acting director of the Cybersecurity and Infrastructure Security Agency (CISA), was flagged for uploading four “for official use only” documents to ChatGPT last summer, despite the tool being blocked for most DHS employees. Similarly, Samsung banned generative AI tools on company devices in May 2023 after engineers pasted internal source code into the platform. Data sent to these services is stored on external servers and may be used to train new models or disclosed in breaches.
4. Obtaining Medical Advice
The same October 2025 policy update that restricted legal advice also banned medical consultations via the chatbot. Because OpenAI is not bound by HIPAA regulations, health data shared through the platform lacks the legal protections afforded in patient-doctor relationships. Researchers from Northeastern University found in July that while safeguards against self-harm have improved, protections for conditions like eating disorders, insomnia, and bipolar disorder remain inadequate. In one notable incident, a 60-year-old man was hospitalized after following ChatGPT’s suggestion to substitute table salt with sodium bromide, a toxic substance.
5. Handling Passwords or Identity Details
Privacy lapses are common; Fast Company discovered nearly 4,500 shared conversations containing sensitive personal details in Google search results last July. Although OpenAI removed the share feature and purged indexed data, the risk persists. During pending copyright litigation, OpenAI was required to retain all consumer chat logs from May to September 2025, demonstrating that deleted conversations can be subpoenaed. Users are strongly advised never to input passwords, Social Security numbers, or financial details into prompts, as this data can persist across sessions and lead to identity theft.
Leave a Reply