Overview
- Automated decision-making as a legal question
- Cognitive demands and explanation
- Attention and habituation
- Open questions
Automated decision-making as a legal question
- Automated decision-making (ADM) is becomming common in many domains
- Self-driving cars
- High-frequency trading
- Automated recruitment and selection
- Among others
- Our object: ADM based on personal data
- Brazil: LGPD art. 20
- EU: GDPR art. 22
The right to review of automated decisions
- Laws establish the right to review or otherwise contest ADM
- However, many of those decisions happen
within immersion interactions (Verbeek 2015)
- Shape an agent’s possibilities
- The agent might not be aware of a decision
Perceiving automated decisions
- Transparency has been proposed as an approach to immersion
- e. g. French law on explanation
- Seamful experiences
- Make evident the interfaces between systems
- In doing so, reveal the technical object
- Both transparency and seamfulness provide information
- This approach is only useful to the extent that the
destinataries can make sense of what has been provided
- Further issue (beyond this talk): doing something with the information.
Cognitive demands and explanation
- Modern ADM models rely on complex models
- Mathematical complexity
- Large volumes of data
- Simply providing all rules might exceed working memory
- Cognitive demand might be too taxing
- Potential solutions:
- Explanations: XAI, clear interfaces
- Legal requirement for actionable information
Attention and habituation
- People are subject to many decisions every day
- Informing about each one might lead to habituation
- Even if warnings are not ignored, attention
- Dehabituation might lead people to abandon systems
or circumvent solutions
Open questions
- Cognitive limits
- How to bring automated decisions to attention?
- What is a reasonable chunk of information for a data subject?
- How does habituation work in ADM?
- Effective legal and HCI implementation of limits