Research Insights: Ethics and algorithmic decision-making systems

Judgments of what is right and good—that is, ethics—have long been of concern regarding the rise of technology and automation.  Recently, ethical concerns have grown dramatically over the proliferation of algorithmic decision-making systems (ADMS), autonomous self-learning systems that make judgments without human intervention. For example, are these systems truly objective and reliable? Can they be manipulated by large organizations that use them? Given the huge amounts of data collected to power ADMS, what about privacy concerns?

To date, ethical analyses of these issues have not been especially rigorous, according to. Along with two co-authors,  Sean Hansen, Ph.D., professor of management information systems (MIS) and chair, MIS, marketing, and analytics conducted  a theoretically driven case study analysis of ethical exploration of ADMS within the IS field, which was published in an article, entitled, “Research perspective: Toward theoretical rigor in ethical analysis: The case of algorithmic decision-making systems,” published in the Journal of the Association of Information Systems. 

Hansen and his co-authors use “three big” ethical theories: Consequentialism (or Utilitarianism), which focuses on outcomes and the greater good; Deontology, which views ethical dilemmas based on the inherent morality of an act; and Virtue, which focuses on the virtuous character of the actor rather than on right actions or what anyone should do in a given situation. Employing these three ethical models provides the theoretical rigor Hansen sees as lacking in the majority of studies of this kind.

“Toward theoretical rigor” then performs an ethical analysis of ADMS used by an Australian bank to identify problem gamblers. Australia has the highest rate of gamblers in the world: 80% of adults. Australian banks have been criticized for enabling such potentially ruinous behavior, leading one major bank there to develop a machine-learning algorithm to identify problem gamblers. Customers so identified are subjected to a range of interventions aimed at mitigating their gambling. However, many have raised ethical alarms about this invasive and potentially harmful procedure.

The authors’ analysis using the “Big Three” ethical theories had somewhat mixed results which nonetheless found the bank’s approach overall to be unethical. Using Utilitarian ethics, they found that “[i]n aggregate, the initiative enhanced customers’ well-being and is therefore ethical.” The deontological (rules-based) model found that the bank’s initiative violated the bank’s moral obligations to customers, and virtue ethics determined the system “compromised the ability of customers to develop practical wisdom and act voluntarily and is therefore unethical.”

These results demonstrate that ethical decisions rely on the theoretical perspective applied. Therefore, the authors address characteristics of ADMS using the Big Three ethical systems, provide guidance on which theory might be particularly fruitful given the context, and “highlight the advantages of theoretically grounded ethical analyses of ADMS.

View the article published in the Journal of the Association of Information Systems (2022) “Research perspective: Toward theoretical rigor in ethical analysis: The case of algorithmic decision-making systems”