Artificial intelligence is increasingly being used to help improve decision-making in executive settings. For example, an autonomous system can identify a power distribution strategy that minimizes costs while maintaining stable voltages.
But while these AI-driven results may be cutting edge, are they right? What if a cheap power distribution strategy leaves low-income areas more vulnerable to blackouts than high-income areas?
To help stakeholders quickly identify potential ethical issues before deployment, MIT researchers developed an automated evaluation method that measures interactions between measurable outcomes, such as cost or reliability, and qualitative or subjective values, such as fairness.
The system divides the target survey into user-defined demographics, using a large-scale linguistic model (LLM) as a population proxy to capture and synthesize stakeholder preferences.
The flexible framework selects the best conditions for further testing, simplifying a process that often requires expensive and time-consuming manual effort. These test cases can show situations where autonomous systems are well suited to human values, as well as situations where the behavior criteria are unexpectedly low.
“We can put a lot of rules and safeguards into AI systems, but those safeguards can only prevent things that we think are happening. It’s not enough to say, ‘Let’s use AI because it’s trained in this knowledge.’ “We wanted to develop a more systematic way to detect unknowns and have a way to predict them before anything bad happens,” said senior author Chuchu Fan, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro) and principal investigator in the MIT Laboratory for Information and Decision Systems (LIDS).
Fan is joined on the paper by lead author Anjali Parashar, a mechanical engineering graduate student; Yingke Li, AeroAstro postdoc; and others at MIT and Saab. The research will be presented at the International Conference on Advocacy for Learning.
Ethical assessment
In a large system like the power grid, evaluating the ethical compatibility of AI model recommendations in a way that considers all objectives is very difficult.
Most evaluation frameworks rely on pre-collected data, but data with behavioral criteria are often difficult to obtain. Furthermore, because AI ethics and systems are both constantly changing, static test methods based on written codes or regulatory documents need to be updated regularly.
The fan and his team tackled the problem from a different perspective. Using their previous work evaluating robotic systems, they developed an experimental design framework to identify the most instructive scenarios, which human participants would closely evaluate.
Their two-part system, called Scalable Experimental Design for System-level Ethical Testing (SEED-SET), includes quantitative metrics and ethical criteria. It can identify situations that effectively meet measurable requirements and align well with one’s values, and vice versa.
“We don’t want to spend all our resources on random testing. Therefore, it is very important to direct the framework to the testing cases we care about the most,” said Li.
Importantly, SEED-SET does not require pre-existing experimental data, and is suitable for many purposes.
For example, a power grid may have several user groups, including a large rural community and a data center. Although both groups may seek low-cost and reliable energy, the value of each group from an ethical perspective may be very different.
These behaviors may not be well defined, so they cannot be measured analytically.
The operator of the power grid wants to find the most cost-effective strategy that best meets the independent ethical preferences of all stakeholders.
SEED-SET addresses this challenge by dividing the problem into two, following a hierarchical structure. The goal model considers how the system is performing on tangible metrics such as costs. Then a positive model that considers stakeholder judgments, such as perceived fairness, builds on objective evaluation.
“The objective part of our methodology is tied to the AI system, while the affective part is tied to the users who evaluate it. By breaking down preferences in a hierarchical manner, we can generate desired scenarios with fewer tests,” said Parashar.
It encodes subjectivity
To conduct an independent assessment, the program uses the LLM as a proxy for human assessors. The researchers input each user group’s preferences for natural language recognition into the model.
LLM uses these commands to compare two scenarios, selecting the preferred design based on behavioral criteria.
“After seeing hundreds or thousands of cases, a human evaluator can experience fatigue and become inconsistent with their evaluation, so we use an LLM-based strategy instead,” explained Parashar.
SEED-SET uses a selected scenario to simulate the system as a whole (in this case, a power distribution strategy). These simulation results guide its search for the best candidate state to test.
Finally, SEED-SET intelligently selects the most representative instances that do or do not meet objective metrics and behavioral criteria. In this way, users can analyze the performance of the AI system and adjust its strategy.
For example, SEED-SET can identify power distribution events that prioritize high-income areas during periods of high demand, leaving poor areas prone to outages.
To test SEED-SET, researchers tested autonomous real-world systems, such as an AI-driven power grid and an urban traffic routing system. They rated how well the situations produced corresponded to the ethical principles.
The system produced more than twice as many test times as the original techniques in the same amount of time, while uncovering many cases that other methods could ignore.
“As we changed the user’s preferences, the set of conditions produced by SEED-SET changed significantly. This tells us that the test strategy responds well to the user’s preferences,” said Parashar.
To measure how useful SEED-SET can be in practice, researchers will need to conduct a user study to see if the productive conditions are helpful in actual decision-making.
In addition to conducting such research, the researchers plan to test the use of efficiency models that can reach larger problems with additional criteria, such as evaluating LLM decisions.
This research was funded, in part, by the US Defense Advanced Research Projects Agency.
