AI ethics in practice: real dilemmas, real decisions

Artificial intelligence has no shortage of ethical principles. Fairness, transparency, accountability, privacy, human oversight and explainability now appear in corporate policies, government frameworks, and international standards around the world. The broad direction is remarkably consistent. Organizations deploying AI should understand the risks, protect people from discriminatory outcomes, maintain appropriate human control, and remain accountable for the consequences.

The difficulty begins when those principles encounter an actual decision.

A facial-recognition system identifies someone entering a store as a suspected shoplifter. Recruitment software rejects an applicant before a person reviews the application. A tenant-screening algorithm produces a score that determines whether someone is offered somewhere to live. In each case, the ethical question is no longer abstract. Someone has to decide what authority the output will be given, whether a person can challenge it, and who is responsible when the outcome is wrong.

That gap between stated principle and operational decision is becoming one of the most important problems in AI governance. It also explains why many of the most instructive AI ethics cases are not stories about exotic failures of artificial intelligence. They are stories about ordinary organizations allowing automated outputs to determine consequences.

The Rite Aid case – when a match becomes an accusation

From 2012-20, United States pharmacy chain Rite Aid deployed facial-recognition technology in hundreds of stores to identify customers who it believed might previously have been involved in shoplifting or other problematic behaviour. The system compared people entering stores against a database containing tens of thousands of images. According to a Federal Trade Commission (FTC) complaint filed in December 2023, the deployment produced thousands of false-positive matches.[1]

The consequences did not remain inside the software.

The FTC alleged that employees acting on false alerts followed customers through stores, searched them, ordered them to leave, publicly accused them of wrongdoing and, in some cases, called police. The agency also found serious deficiencies in how the technology had been deployed. Rite Aid had failed to adequately test or assess its accuracy before deployment, failed to regularly monitor its accuracy afterward, and failed to adequately train employees about the possibility of false matches.[2]

One incident described in the FTC complaint exposes the problem particularly clearly. Rite Aid’s facial-recognition system generated an alert identifying a black woman as a match for an enrolment image that employees themselves described as depicting a white woman with blond hair. Employees called police and asked the customer to leave before recognizing that the alert was false.[3]

The FTC also alleged that the technology was more likely to generate false positives in stores located in plurality-black and Asian communities than in plurality-white communities. Its resulting order prohibited Rite Aid from using facial recognition for surveillance purposes for five years and required safeguards for any future automated biometric surveillance systems.[4]

It would be easy to classify this simply as an example of biased or inaccurate facial recognition. That explanation is incomplete. An inaccurate match has no direct consequence until an organization decides what the match is authorized to trigger.

The important sequence was therefore not merely that the software produced an incorrect result. The system produced a match, the organization allowed that match to carry operational weight, an employee acted upon it, and a customer experienced the consequence.

That distinction moves the discussion beyond model accuracy. Even a substantially more accurate system would leave the ethical question unresolved: what level of authority should a probabilistic identification be allowed to exercise over a person?

Rite Aid’s experience demonstrates why ethics cannot be confined to the technical performance of the model. The ethical architecture includes the people receiving the output, the procedures governing their response, the evidence required before action is taken, and the ability of the affected person to challenge what follows.

Discrimination does not become neutral when software executes it

Employment provides an even clearer illustration.

In 2022, the U.S. Equal Employment Opportunity Commission (EEOC) sued iTutorGroup, alleging that the online tutoring company had programmed its recruitment software to automatically reject female applicants aged 55 or older, and male applicants aged 60 or older. More than 200 qualified U.S. applicants were rejected. In 2023, iTutorGroup agreed to pay $365,000 to settle the lawsuit.[5]

There is an important qualification. The EEOC later noted that the technology involved was not technically artificial intelligence, but automated screening software.[6] That makes the case more useful. The governance problem does not begin when software becomes sophisticated enough to satisfy a particular definition of AI. It begins when automated processing is granted influence over a consequential decision.

The software did not invent the age criteria. People designed or authorized a process in which those criteria became executable. Automation then made the discrimination fast, consistent, and scalable.

This is one reason the current discussion about AI ethics can become misdirected. Attention gravitates toward the model: whether it is biased, explainable, accurate, or sufficiently advanced. Yet a technically simple automated rule can create an unethical outcome just as effectively as an advanced machine-learning system if the surrounding decision process permits it.

The EEOC and U.S. Department of Justice have separately warned that employers using algorithmic and AI tools remain responsible for complying with existing disability discrimination law. The EEOC has specifically identified risks where automated tools screen out people with disabilities who could perform the job with reasonable accommodation.[7]

The principle is straightforward. Technology changes the mechanism by which a decision is produced. It does not automatically change the organization’s responsibility for making it.

Housing and the problem of apparently neutral inputs

Housing demonstrates a more difficult version of the same problem because discrimination does not have to be explicitly programmed.

In Louis et al. v. SafeRent et al., two black rental applicants who used housing vouchers alleged that they were denied housing because of scores generated by SafeRent’s algorithmic tenant-screening system. The plaintiffs alleged that the scoring system had an unlawful disparate impact on black and Hispanic applicants because it relied on factors including credit history and non-tenancy-related debts while failing to account for the relevance of housing vouchers to the applicants’ ability to pay rent.[8]

The U.S. Department of Justice filed a Statement of Interest in the case in January 2023, arguing that the Fair Housing Act applies to companies providing algorithm-based residential screening services. The federal court subsequently rejected defendants’ attempts to dismiss the Fair Housing Act claims, holding that the plaintiffs had plausibly alleged disparate-impact claims.[9]

That ruling was not a finding that SafeRent had discriminated. It allowed the plaintiffs’ claims to proceed. The distinction matters.

What makes the case important for AI ethics is the structure described in the court record. According to the plaintiffs’ allegations quoted by the court, SafeRent controlled how its score was calculated, housing providers did not know the details of the algorithm, and the resulting evaluations were relied upon in accepting or denying rental applications. In one alleged instance, a property manager told an applicant that it could not override the tenant-screening outcome.[10]

Here, the ethical problem is considerably more subtle than explicitly instructing software to reject someone because of age.

Credit history can be relevant information. Debt can be relevant information. Statistical scoring can be useful. Yet an apparently neutral collection of variables can still produce an outcome that requires legal and ethical scrutiny.

This exposes a limitation in the idea that ethical AI can be achieved simply by removing protected characteristics from a dataset. The consequential question is not only what variables enter the system, but what those variables represent, how they interact, whether they are appropriate to the decision being made, and what authority the resulting score receives.

Fairness is therefore not a property that can simply be installed in a model. It is partly a judgment about the legitimacy of the decision process surrounding it.

The emerging regulatory answer

Regulators and standards bodies are increasingly recognizing this distinction.

The U.S. Federal Trade Commission, Consumer Financial Protection Bureau, Department of Justice Civil Rights Division, and EEOC issued a joint statement in 2023 making clear that existing legal authorities apply to automated systems just as they apply to other practices. The agencies specifically warned that automated systems can perpetuate unlawful bias, automate unlawful discrimination, and produce other harmful outcomes.[11]

The significance of that position is easily missed. An organization does not escape existing obligations because an algorithm sits between its policy and the person affected by it.

The European Union’s AI Act takes the issue further by placing explicit obligations around human oversight on high-risk AI systems. Article 14 requires such systems to be designed so they can be effectively overseen by natural persons. Oversight is intended to allow those people to understand relevant capabilities and limitations, recognize the risk of over-reliance, correctly interpret outputs and, where appropriate, disregard, override, or reverse them. The legislation specifically identifies automation bias as a risk requiring attention.[12]

That is a substantially more demanding concept than simply placing a human in the loop.

A person can be present in a decision process without exercising meaningful authority. If an employee receives hundreds of algorithmically ranked cases each day, sees only the information selected by the system, is expected to follow its recommendation, and must justify every override, the final mouse click may be human while the practical decision architecture is not.

Meaningful human oversight, therefore, has to be assessed operationally. What information did the reviewer actually receive? Could the recommendation realistically be rejected? Was contrary evidence available? Did the reviewer have sufficient competence and time to challenge the output? What happened when the reviewer disagreed?

Without answers to those questions, “human oversight” risks becoming a description of workflow rather than evidence of governance.

From ethical principles to operational evidence

This transition is also visible in international standards.

The National Institute of Standards and Technology’s (NIST) Artificial Intelligence Risk Management Framework describes trustworthy AI through characteristics including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. Crucially, NIST describes these as attributes of a socio-technical system rather than properties of a model considered in isolation.[13]

The United Nations Educational, Scientific and Cultural Organization’s (UNESCO) Recommendation on the Ethics of Artificial Intelligence makes the accountability requirement still more explicit. It states that ultimate responsibility and accountability must remain with natural or legal persons, and calls for appropriate oversight, impact assessment, audit, and due-diligence mechanisms. It identifies auditability and traceability as necessary elements of accountability across the AI lifecycle.[14]

The Organization for Economic Co-operation and Development’s (OECD) updated AI principles similarly connect accountability to traceability, systematic risk management, human agency, and the ability of people adversely affected by AI systems to challenge their outputs.[15]

Canada offers a practical example of what this transition looks like inside government. The federal Directive on Automated Decision-Making applies to automated systems used to make or support administrative decisions about clients. Departments must complete, approve, and publish an algorithmic impact assessment before a system enters production – and must update it when functionality or scope changes.[16]

The assessment examines the system, algorithm, decision, impact, and data, and assigns one of four impact levels. Higher levels trigger stronger requirements. The current assessment framework considers the duration and reversibility of decisions and requires departments to consider how an automated system assists or replaces human judgement.[17] Projects at impact level II or above are subject to peer-review requirements.[18]

None of these mechanisms guarantees an ethical outcome. They do something more practical; they force parts of the decision architecture to become visible before consequences occur.

That is an important evolution in AI ethics. A statement that an organization values accountability is an aspiration. Identifying the executive responsible for a system, documenting its intended use, assessing its impact, defining the required level of human involvement, testing outcomes, and preserving a route of recourse creates evidence that the aspiration has entered the operating environment.

The ethics gap is an authority gap

The cases above appear to concern different problems. Rite Aid involves biometric surveillance. iTutorGroup concerns employment. SafeRent concerns housing. The technologies are different, the legal regimes are different, and the alleged or established harms are different.

Structurally, however, they expose the same problem.

An automated system generates an output. That output enters an organizational process. Somewhere inside that process, it acquires authority. A person is then affected.

This is the point at which an ethics policy either becomes operational or disappears.

An organization can test a model for bias and still fail if employees misunderstand its output. It can require human review and still fail if reviewers cannot meaningfully override the recommendation. It can maintain detailed logs and still fail if the prohibited action has already occurred. It can publish principles of fairness and still fail if nobody has defined what evidence is required before an automated assessment can influence a consequential decision.

This suggests that organizations should examine AI ethics less as a list of values and more as a chain of authority.

The relevant chain begins with the information admitted into the system, continues through the inference or calculation it produces, then passes through recommendation, human or automated authority, action, consequence, and finally recourse. Ethical controls can fail at any transition.

That final distinction is especially important. Auditability is necessary, but an audit occurs after an event. A perfect record of an inappropriate decision does not make the decision appropriate. It establishes what happened.

Governance has to operate earlier. It must determine what the system is permitted to influence, what evidence is required before action follows, where human authority is mandatory, when that authority can be exercised, and what actions remain prohibited regardless of the confidence or sophistication of the model.

What this means for business

For executives, the practical consequence is that AI ethics cannot remain solely with an ethics committee, legal department, or model-development team. Once an AI output enters an operational workflow, the organization has created a chain of responsibility that extends from system design to the person experiencing the outcome.

Boards, therefore, need a different test.

Instead of asking whether the organization has an AI ethics policy, ask whether it could reconstruct a contested AI-influenced decision.

What data entered the decision? What system processed it? What limitations were known? What output was produced? Who received it? What authority did that person or system possess? Could the output be challenged? What evidence supported the eventual action? Who was accountable for the result? What recourse was available to the person affected?

If those questions cannot be answered, the problem is no longer the absence of ethical principles. It is the absence of evidence that those principles governed the decision.

The lesson emerging from real-world cases is, therefore, more demanding than the familiar instruction to build “ethical AI.”

Ethics must survive execution.

Fairness has to remain meaningful when a score reaches an employee. Human oversight has to remain real when the system recommends an answer. Accountability has to remain identifiable when multiple vendors, datasets, and decision-makers are involved. Recourse must exist when the affected person believes the system was wrong.

The future of AI ethics will not be determined by how many principles organizations publish. Most of the necessary principles already exist.

It will be determined by whether an organization can demonstrate that those principles were present at the moment an automated output became a real decision affecting a real person.

Our currency is truth and accuracy.

References

[1] Federal Trade Commission, Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards.

[2] Federal Trade Commission, Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards.

[3] Federal Trade Commission, Complaint for Permanent Injunction and Other Relief, FTC v. Rite Aid Corporation.

[4] Federal Trade Commission, Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards.

[5] U.S. Equal Employment Opportunity Commission, iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit.

[6] U.S. Equal Employment Opportunity Commission, Gary D. Friedman testimony: Navigating Employment Discrimination in AI and Automated Systems.

[7] U.S. Equal Employment Opportunity Commission, U.S. EEOC and U.S. Department of Justice Warn against Disability Discrimination.

[8] U.S. Department of Justice, Civil Rights Division, Louis et al. v. SafeRent et al. (D. Mass.).

[9] U.S. Department of Justice / U.S. District Court, Memorandum and Order, Mary Louis v. SafeRent Solutions, LLC.

[10] U.S. Department of Justice / U.S. District Court, Memorandum and Order, Mary Louis v. SafeRent Solutions, LLC.

[11] FTC, CFPB, DOJ Civil Rights Division and EEOC, Joint Statement on Enforcement Efforts Against Discrimination and Bias in Automated Systems.

[12]European Union, Regulation (EU) 2024/1689 — Artificial Intelligence Act, Article 14.

[13] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0).

[14] UNESCO, Recommendation on the Ethics of Artificial Intelligence.

[15] OECD, OECD AI Principles.

[16] Treasury Board of Canada Secretariat, Directive on Automated Decision-Making.

[17] Treasury Board of Canada Secretariat, Algorithmic Impact Assessment Tool.

[18] Treasury Board of Canada Secretariat, Guide to Peer Review of Automated Decision Systems.

(Mark Jennings-Bates – BIG Media Ltd., 2026)