Updated: January 14, 2026- 15 min read
In Salesforce’s latest State of the Connected Customer report, the share of customers who strongly agree they’re concerned about unethical AI use jumped from 22% to 37% in just a few years (1).
If you build products with AI, every product experiment you run is also an ethics experiment, whether you acknowledge it or not. From AI agents making semi-autonomous decisions, to RAG systems quietly curating “facts,” to third-party AI stitched into critical workflows, you are no longer just optimizing conversion or retention. You are shaping what users can trust.
This piece looks at what that means for AI-powered product managers and AI PMs: the new ethical risks in modern AI products, where responsibility really sits, and a practical product experimentation framework you can use.
Turning AI Doubt into AI Strategy
Ryan Daly Gallardo, SVP of Product, Consumer at Dow Jones, reveals how to test without eroding trust, embed cross-functional safeguards, and use evidence-based design to deliver AI features that improve engagement.
Download Playbook
Why AI Ethics Belongs In Product Management
Most companies are already in the AI business, whether they like it or not. Recent surveys suggest that around 78–88% of businesses now use AI in at least one function, and a fast-growing majority are experimenting with generative AI in production workflows (2).
That means AI ethics in product management is not an academic side topic. It is about how your product behaves in front of real users, under real pressure, at real scale. And, despite your natural gut feeling, it’s not stalling automation. As Murtaza Chowdhury, AWS AI Product Leader, frames it in this fourth episode of AI Series: “The goal is not to reduce autonomy. The goal is to ensure the autonomy is exercised safely and consistently.”
Jeetu Patel, President and Chief Product Officer at Cisco, similarly argued that:
Security and safety are not looked at at odds with productivity. It's actually looked at as a prerequisite of productivity.
For AI-powered product managers, ethics shows up in seemingly small decisions: which customer emails you feed into a summarization model, how you tune an AI assistant for your CS team, or which developer tools you allow into your stack.
For AI PMs building AI products, it is even more structural: what data you train on, what guardrails you add around AI agents, how you design RAG systems so they do not leak sensitive information, and how you evaluate hallucinations and bias.
In case you haven’t read about the distinction between these two, here’s a quick illustration:
In both roles, you are working with non-deterministic systems. The same prompt can produce different outputs tomorrow. A minor change in data or a third-party model can subtly shift behavior. That uncertainty is exactly why ethical thinking has to sit inside your product decisions, not on a separate slide titled “Responsible AI”.
Ethical dilemmas for AI product managers
The hard part about AI ethics in product management isn’t spotting “obviously bad” ideas. Most teams know they shouldn’t ship a hate-speech generator or an agent that can move money with no oversight.
The real difficulty lives in the grey zone — where a feature is useful, effective, and aligned with your KPIs, but you’re not entirely sure what it will normalize once it’s in the wild.
Ethics is not a nice-to-have layer on top of AI. It’s the adoption layer. If users don’t trust what the system is doing, they won’t engage, and your KPIs won’t matter anyway. As Jeetu Patel, President and Chief Product Officer, said live on stage at ProductCon:
If people don't trust these systems, they're not going to use them.
As an AI PM, you are no longer just choosing which features to ship. You’re choosing which behaviors to scale and which trade-offs you are willing to live with.
In practice, the dilemmas tend to repeat in different shapes:
Speed to market vs depth of evaluation: Ship the AI assistant in this quarter’s release, or invest more time in evals, red-teaming, and guardrails, knowing the opportunity might move.
Personalization vs privacy: Use richer behavioral and conversational data to make the model feel “magical,” or limit inputs and keep a clear line on what is too intrusive.
Human in the loop: Let agents take real actions (refunds, approvals, changes in production) to unlock efficiency, or keep them in “recommendation mode” and accept operational friction.
Average performance vs cohort fairness: Optimize for global metrics, or slow down and investigate whether some groups consistently get worse outcomes and need special handling.
Experimental mindset vs product stability: Treat AI features as “lab experiments” with loose expectations, or acknowledge that anything exposed to users will be perceived as part of the product and must meet a higher bar.
These are not theoretical questions. They show up in product roadmap debates, launch reviews, and even small implementation details like default settings, data retention rules, and opt-out flows.
How AI ethics land for AI-powered PMs vs AI PMs
For AI-powered product managers, the dilemmas are often about how you use AI inside the team.
Do you send raw customer calls into a third-party transcription and summarization tool to move faster on product discovery, or do you strip identifiers and accept a bit more manual work?
Do you let an AI assistant propose roadmap ideas directly from product analytics, or do you insist on “human in the loop” before anything goes into a deck?
For AI PMs building AI products, the dilemmas cut closer to the core of the product.
Do you let an onboarding agent actually configure a customer’s account, or only suggest changes for a human to confirm?
Do you let a RAG system pull from the entire internal knowledge base, or deliberately exclude certain document types and take the hit on “completeness”?
Do you accept a model that performs brilliantly overall but shows uneven behavior for smaller cohorts, or do you go back and adjust at the cost of latency, complexity, or launch dates?
The point is not to eliminate trade-offs. You can’t. The point is to make them visible, argue them consciously, and connect them to your evaluation strategy. Once you admit, “We are choosing speed over depth here,” you can at least put compensating controls in place: tighter rollouts, stronger evals, feature flags, clearer user messaging.
Ethical Risks in AI Products
Agents can take actions you did not fully anticipate. RAG systems can quietly pick up outdated, confidential, or outright poisoned content from your internal knowledge base. A prototype LLM feature you rolled out “only to a small cohort” can be screenshotted and shared in hours. And a third-party API you trust today can change its data-handling policy overnight.
At a high level, most ethical risks in AI products cluster into a few themes:
Bias and fairness: AI models trained on historical data can replicate or amplify societal biases (e.g. in hiring, lending, policing). Without checks, the system might favor one group over another. Product teams should ask, “Does our training data reflect all user groups? Could this feature work worse for some people?” and use bias-detection tools.
Privacy and data misuse: AI often needs large data sets. Ensuring data is collected and used properly is vital. Teams must verify consent, secure personal data, and avoid inadvertent leakage (especially in RAG systems, which can reveal sensitive content).
Transparency and explainability: Many AI models are “black boxes.” Users and regulators increasingly expect clear explanations for AI decisions. For high-stakes features, product managers should ask, “Can we explain how this decision was made?” or provide user-facing notes on limitations.
Security and robustness: AI can introduce new vulnerabilities. For example, poorly designed AI code might be tricked by malicious inputs or could use outdated third-party models. PMs should ensure threat modeling for AI (e.g., adversarial testing for RAG prompts or isolating models in secure environments).
Unintended automation: Over-reliance on AI can remove human oversight. For example, an AI agent might make an authoritative-sounding decision that’s wrong. Product teams must consider fail-safes: Who intervenes if the AI errs? Can users opt out or get a human review?
Resource fairness: AI features can shape user experience unevenly. For instance, personalization might improve metrics on average but hurt a subset of users. Teams should evaluate whether improving one KPI might “push the minority off a cliff.”
Ethical misuse and dual use: Some AI functions (like content generation or decision support) can be misused for spam, deepfakes, or harmful recommendations. PMs should consider edge scenarios – e.g., could a chatbot be prompted to give dangerous advice? and put guardrails in place.
Each of these shows up differently depending on the feature.
A recommendation model might quietly push one demographic down the funnel while boosting another. An AI agent for finance might accidentally move money in ways that trigger compliance issues. An “internal-only” RAG tool might surface confidential contracts in a way that feels like a data breach, even if no laws are technically broken.
Real-World Ethical Dilemmas of AI
If you work in product, you already know the headlines. What matters more for us is the pattern underneath them.
Apple Card AI ethics controversy
Take the Apple Card controversy. The algorithm appeared to give women lower credit limits than men with similar financial profiles.
Regulators got involved, users were angry, and Apple was forced to defend a system most people couldn’t see or understand. The lesson isn’t that “AI is biased” in a generic way. It’s that a production system was making life-impacting decisions, at scale, without anyone being able to clearly explain or contest the outcome.
That is exactly the kind of scenario AI PMs should design against.
Grok as an example of poor AI ethics
Or look at incidents like Grok on X giving step-by-step instructions for violent acts and pushing antisemitic content. A prompt tweak and some new data were enough to push a live model into territory that is obviously unacceptable to any product leader.
What this tells you is not just “models can be toxic.” It shows how fragile alignment can be when you chain prompts, fine-tunes, retrieval, and guardrails together without a clear theory of failure modes.
Quieter, almost mundane AI ethics failures
A media outlet publishes a “summer reading list” of books that don’t exist because an AI assistant hallucinated them, and nobody caught it. A code assistant wipes a startup’s production database because it was given way too much access and not enough constraints.
These are not dark sci-fi scenarios. They are everyday workflow tools behaving exactly as designed, but in environments where a wrong action or lack of humans in the loop is very expensive.
The throughline here is simple: once AI is inside your product, ethical risk is tied to three things. Those are scale, opacity, and coupling.
Scale means small mistakes hit millions of people at once. Opacity means neither users nor teams fully understand why the system behaved that way. Coupling means a single AI output can trigger a chain of automated actions that are hard to reverse.
At the end of the day, users do not care whether the problem came from a prompt, a fine-tune, or a RAG pipeline. They experience harm, or they don’t.
From a product perspective, that is the bar: does this AI-powered experience behave in a way that we would be proud to defend in public when, not if, it is tested at the edges.
Who Owns AI Ethics On The Product Team
AI ethics is never “owned” by a single person, but the product tends to be where the trade-offs crystallize.
AI-powered PMs make everyday calls about where AI tools show up in product discovery, user research, and delivery. They decide which internal data to connect to a copilot, how much autonomy to give agents in operational workflows, and how to communicate limitations to stakeholders.
Those are ethics decisions, even if they show up in a Jira ticket.
AI PMs go one level deeper. They shape which problems are worth solving with AI, what “good” looks like for an AI-driven feature, and where human review is required. They negotiate with legal and compliance on acceptable risk, and they decide how transparent to be with users about model behavior.
What legal and compliance actually handle in AI ethics
Legal and compliance are there to define the playing field, not to design the play.
In practice, this is what you can expect them to own and what you can safely delegate:
Interpreting laws and regulations (GDPR, CCPA, sector rules, upcoming AI regs), mapping your use case to concrete “musts” and “must nots.”
Reviewing and drafting user-facing language: terms of service, privacy notices, consent flows, disclaimers, data-use language for AI features
Running and documenting risk/impact assessments (e.g,. DPIAs), plus classification of high-risk vs. low-risk AI use cases
Vendor and third-party reviews: DPAs, model/API contracts, data-sharing arrangements, cross-border transfer issues
Setting internal policies: what data can be used for training, retention rules, incident reporting, escalation paths, and documentation standards
Supporting incident response: what happens legally if there’s a breach, misuse, or public complaint about your AI feature
Your job as an AI PM is to bring them a clear description of the feature, the data flows, and the intended behavior. Their job is to tell you where the legal boundaries are, what they need to be comfortable signing off, and what additional controls or documentation are required.
Engineering, data science, design, legal, and security all share responsibility. But AI ethics in product management becomes real only when PMs treat it as part of product craft, not as a compliance bolt-on. That means ethics appears in PRDs, in success metrics, in roadmap discussions, and in post-launch reviews.
AI Ethics Checklist for Product Managers
To turn ethics into action, PMs can use a practical checklist when scoping or launching AI features. Questions include:
Data & model governance: Is the training data diverse, high-quality, and ethically sourced? Are there privacy or consent considerations? Can we audit how data flows through the model (especially in RAG systems, which “pull in” new data)? Tools and frameworks (e.g. NIST’s AI RMF, IBM’s AI Fairness 360) can help identify biases.
Bias and fairness: What fairness checks have we run? Are there known demographic imbalances? For example, measure false positive/negative rates across groups and ask if any are unacceptably high. Leverage existing AI bias-detection libraries or D&I testers to root out issues early.
Explainability & transparency: How will we explain this feature to users and stakeholders? Is there a model card or documentation? Can we create a brief “transparency note” so others know what data/logic it uses? For high-risk features, can we implement simpler algorithmic layers or provide user controls if an automated decision is unclear?
Ethical risk throughline (scale, opacity, coupling): Once AI is inside the product, ethical risk concentrates in three areas. Scale means small errors affect many users at once. Opacity means teams and users may not fully understand why the system behaved a certain way. Coupling means a single AI output can trigger downstream automated actions that are difficult to stop or reverse.
Human in the loop: What level of human review is in place? Is there a clear rollback or escalation path if the AI misbehaves? (For example, if a recommended action looks wrong, can a person override it?) In “agent” workflows, ensure a human-in-the-loop or out-of-band alerts for dangerous outputs.
Security & resilience: Have we threat-modeled the AI feature? Are there safeguards against adversarial inputs (e.g. malicious prompts)? Are APIs or model endpoints authenticated and rate-limited? If using third-party models or APIs, have we reviewed their data use policies and reliability?
Privacy & compliance: Does the feature comply with relevant laws (GDPR, HIPAA, etc.)? Have we anonymized or encrypted user data where needed? For features that infer sensitive attributes (like health or finances), ensure explicit consent and clear opt-out paths.
User impact & feedback: How will this AI feature affect product experience for different groups? Do we have plans to collect and act on user feedback specifically about fairness or trust? (For example, monitoring whether certain segments adopt the feature or report problems.)
Monitoring & evaluation: Can we continuously test the model’s outputs in production? Using systematic “AI eval suites” (as we teach it in Product School's AI Evals Certification) can catch drift or bias over time. Plan to log decisions and review them periodically.
Governance & accountability: Have we documented who on the team is responsible for this AI feature’s ethics? Is there a champion (the PM) who will raise issues with product leadership if needed? Establish an internal sign-off process, for instance, a brief ethics review during each PR or sprint demo.
AI Evals Certification
Learn to build trusted AI products. Design eval suites, integrate CI/CD gates, monitor drift and bias, and lead responsible AI adoption.
Enroll now
Ethical AI As A Product Advantage
AI ethics in product management is not a side quest. It is the work of deciding which behaviors your product will make normal at scale.
The teams that treat ethics, evals, and governance as core product capabilities will ship AI that users actually trust, regulators can live with, and leadership is proud to defend in public.
Today, anyone can bolt an LLM onto their product. The real differentiator is not “who ships AI,” but “who ships AI that deserves to exist.”
Level up on your AI knowledge
Based on insights from top Product Leaders from companies like Google, Grammarly, and Shopify, this guide ensures seamless AI adoption for sustainable growth.
Download Guide
(1): https://www.salesforce.com/en-us/wp-content/uploads/sites/4/documents/research/State-of-the-Connected-Customer.pdf
(2): https://hai.stanford.edu/ai-index/2025-ai-index-report
Updated: January 14, 2026
