This section delves into case studies that illuminate the pervasive issue of inequity in AI systems. Through these examples, we examine real-world instances where biases and disparities have been uncovered, shedding light on the profound implications of AI technologies on communities.

‍

Facial recognition

‍Several studies conducted by the National Institute of Standards and Technology in the US between 2002 and 2019 revealed significant racial and gender biases in widely used facial recognition algorithms.¹ These studies, along with others, have consistently shown notable accuracy discrepancies across gender, age, and racial groups, with marginalized and non-dominant subpopulations experiencing the highest levels of misidentification and performance disparities.  For example Microsoft’s FaceDetect model exhibited a 6.3% error rate in gender classification tasks overall. However, upon analysis of gender and race intersections, disparities emerged: while light-skinned males experienced a 0% error rate, dark-skinned females faced a 20.8% error rate. This highlights the underrepresentation of people of color and women in the training and benchmark datasets, exposing a broader neglect by model designers to address performance disparities for marginalized groups.

‍‍

Hiring

‍In the recruitment process, algorithmic bias manifests in various forms, including gender, race, color, and personality. Gender biases are evident in natural language processing (NLP) techniques and machine learning models, with research indicating that these systems tend to portray minority-gender occupations as less professional.² For instance, Amazon's ML-based hiring tool exhibited gender bias due to training on predominantly male employees' resumes, resulting in discrimination against female applicants. Similarly, racial biases emerge, as seen with Microsoft's chatbot Tay, which quickly adopted sexist and racist language on Twitter. These examples underscore the need for addressing algorithmic bias to promote fairness and equity in AI-driven recruitment processes.

‍

Defining “high-risk” students‍

In Wisconsin, education officials initially viewed the Dropout Early Warning System (DEWS) as a critical tool in addressing the state's graduation gap, with disparities seen among White, Hispanic, and Black students.³ Intended to provide personalized predictions early for timely intervention, DEWS has been in use for a decade, but recent investigations suggest it may negatively impact how educators perceive students, especially those of color. Research from the University of California, Berkeley concludes that DEWS has failed to improve graduation rates for labeled "high-risk" students. An equity analysis by the Department of Public Instruction (DPI) in 2021 revealed that DEWS generated false alarms about Black and Hispanic students not graduating on time at a significantly higher rate than their White counterparts. Despite these findings, DPI has not informed school officials or made changes to the algorithms.

‍

‍

The risk of inequitable AI practices are severe and self perpetuating. Without active mitigation throughout the development process, we risk engaging in cycles of development that lead to even more biased inputs further encoding systemic and historical inequities.

‍

For a detailed exploration of the importance of equitable AI and the risks associated with inequitable AI practices, please refer to this resource.

‍

‍

‍

This guide will introduce equity-aligned decision points, actions, and strategies that arise throughout the development process that are related to the following overarching equity considerations:

‍

Inclusivity

‍

​​Inclusivity is a crucial aspect of ensuring data equity and fairness in the development of responsible AI systems. It involves actively considering and addressing the needs, perspectives, and experiences of all individuals and communities represented in the data, particularly those from marginalized groups. This includes ensuring that data collection methods are accessible and inclusive, taking into account factors such as language barriers, socioeconomic status, and cultural diversity. Moreover, inclusive data processing and modeling practices involve recognizing and mitigating biases that may disproportionately impact certain demographic groups, thus promoting fair and equitable outcomes for all stakeholders. Inclusive AI development also entails fostering diversity and representation within AI teams and decision-making processes to ensure that a wide range of perspectives are considered and that AI systems are designed with the needs of diverse communities in mind. 

‍

Once a solution is implemented, it is equally important to address access such that all students have equitable access to AI-powered educational resources. This includes bridging the digital divide by addressing disparities in technology access and designing inclusive AI systems that accommodate diverse learning needs and styles. Prioritizing inclusivity in AI integration fosters fairness, diversity, and accessibility in education, emphasizing the ethical responsibility to provide equal access to learning opportunities for all students, regardless of their background or circumstances.

‍

Fairness

‍

Fairness, from an equity perspective, is a fundamental principle in the development and deployment of responsible AI systems. It encompasses the concept of treating all individuals and groups fairly and impartially, irrespective of their demographic characteristics or social backgrounds. In the context of AI, fairness goes beyond merely ensuring equal treatment; it involves actively preventing the perpetuation or exacerbation of existing inequalities. This means that AI systems should not produce outcomes that systematically disadvantage certain individuals or groups based on protected attributes such as race, gender, age, or socioeconomic status. Achieving fairness in AI requires careful consideration of biases in data collection, algorithm design, and decision-making processes to mitigate the risk of discriminatory outcomes. Additionally, fairness entails ongoing monitoring and evaluation of AI systems to identify and address any disparities that may arise during deployment.

‍

Transparency and Accountability

‍

Equitable AI practices emphasize the importance of transparency in the operation and decision-making processes of AI systems to ensure fairness and accountability. The complexity of AI algorithms and their opaque decision-making mechanisms can create challenges for users in understanding how conclusions are reached, leading to potential biases or unfair outcomes. By promoting transparency, stakeholders can gain insights into how AI systems function, enabling them to identify and address any biases or errors that may arise. Transparency also enhances trust in AI systems and empowers the communities impacted by AI solutions to have a voice in how they are applied. This involves engaging stakeholders in discussions about the design, deployment, and impact of AI systems, ensuring that their perspectives and concerns are taken into account. 

‍

For more guidance and support with transparency, explore this helpful resource:

Transparency Throughout the AI Development Lifecycle

‍

Data Privacy and Consent

‍

One of the primary ethical concerns in using AI in education is data privacy. AI systems often require large amounts of data to function effectively, which can include sensitive personal information about students. Ensuring that this data is collected, stored, and used in a manner that respects the privacy of individuals is crucial. This involves obtaining informed consent from participants, anonymizing data to protect identities, and implementing robust security measures to prevent unauthorized access or data breaches. 

‍

When implementing AI systems in educational settings, organizations must ensure that student data is handled in compliance with FERPA regulations to prevent unauthorized access, disclosure, or misuse of sensitive information. To protect student privacy, AI systems should employ techniques such as data anonymization or de-identification to remove or obfuscate personally identifiable information from education records. By anonymizing student data, educational institutions can minimize the risk of unauthorized re-identification and comply with FERPA regulations.

‍

‍