AI Needs to be Supervised

I’ve been hearing a lot of discussion around whether AI has reached autonomy or not… let’s talk about that.

In Kant’s moral theory, autonomy is a quality of rational agents that act and make choices based on their own reason. Merriam-Webster defines autonomy as the quality or state of being self-governing, self-directing freedom, and especially moral independence. When it comes to artificial intelligence, the definition gets a little foggy. I’ve seen autonomous AI defined as the ability of machines to act with limited human intervention. I’ve also seen it referred to as the ability to operate independently of direct human intervention.

There is one caveat to the latter, and that includes the fact that AI is working within defined constraints to achieve a specific goal; however, it does have the capacity to learn from its own experiences and make decisions. But, if AI can act and make decisions with full autonomy, who’s accountable when things go wrong?

To build trustworthy AI solutions, companies should be documenting things like: what the AI solution is being used for, how the AI solution is created, how the AI solution makes decisions, and how the AI solution is being monitored. To add to the monitoring documentation, companies should also be implementing specific guidance and direction for supervising AI decisions, outputs, and impacts. The supervision itself can be tricky, but can be done with a few different approaches.

Audit Logs

One approach to supervising AI is to enable detailed audit logs to capture things like data inputs, model parameters, model versions, and data outputs. These can be used to troubleshoot issues and evaluate model performance. However, since most models are complex and require very large data sets, monitoring of these logs will require an analytical approach. This isn’t impossible, but there are tools out there that can do this more efficiently.

AI Observability

A modern approach to AI supervision would be to use tools to help with monitoring and analyzing AI systems, which is often referred to as AI Observability. There are tools out there that help with the collecting of the data inputs, model parameters, model versions, and data inputs as described above. Tools such as Dynatrace, New Relic, Datadog, and AppDynamics by Splunk can be used as a holistic and complete approach to drive insights on the availability, reliability, performance, and trustworthiness throughout the AI lifecycle. AI Observability can also work as a control to detect certain AI related threats.

You can set up monitoring of the data pipeline to observe data inputs, which would help detect potential abuse like prompt injection attacks. You can analyze data quality which could expose potential data poisoning attempts or data drift. Monitoring of the infrastructure could identify heavy use of an AI system which could be a sign of a prompt injection attack. Also, general monitoring of model performance can even help catch things like data drift and model drift.

These are all great tools and techniques, but you can’t forget about human oversight.

Human in the Loop

When developing these AI solutions, it’s critical to include a human in the loop for supervision of the decisions, outputs, and impacts as well as validating the quality and relevance of the data used for training the model. In some cases, there should even be a final decision-making role to approve the AI workflow. For instance, if you had an AI system developed to diagnose patients with a particular disease, a human doctor should be included in the loop as a secondary review and final decision-making role for that decision.

Another reason to support human involvement is to mitigate the impacts that can occur with hallucinations, which is a common issue with GenAI. A hallucination is when the AI model makes a prediction that didn’t come from the input data. This is exceptionally detrimental in high-risk or regulated situations where having inaccurate output can lead to legal repercussions, financial penalties, and reputational damage.

To reduce hallucinations, companies can implement controls (or guardrails) on both input and output data by using templates to provide structure and ensure consistency. It’s important to emphasize that these controls only help reduce the potential for hallucinations and the controls themselves should be monitored and adjusted often.

It’s hard to make the argument that AI can act entirely as an independent agent that makes decisions. There needs to be accountability for when things go wrong and therefore, robust supervision should be implemented to mitigate the potential risk posed by these AI solutions.

Do you think we have reached AI autonomy?

Originally published at www.medium.com.

I love working with Internal Audit teams to help them develop strategies to leverage analytics and AI to modernize their audit approach. If you’re interesting in leveling up you’re audits,  let’s chat!