Human cognition and the regulation of automated decision-making

Marco Almada, BSCS, MCompEng

Researcher, Lawgorithm

LL.B. student, USP Law School


  • Automated decision-making as a legal question
  • Cognitive demands and explanation
  • Attention and habituation
  • Open questions
  • Automated decision-making (ADM) is becomming common in many domains
    • Self-driving cars
    • High-frequency trading
    • Automated recruitment and selection
    • Among others
  • Our object: ADM based on personal data
    • Brazil: LGPD art. 20
    • EU: GDPR art. 22

The right to review of automated decisions

  • Laws establish the right to review or otherwise contest ADM
  • However, many of those decisions happen within immersion interactions (Verbeek 2015)
    • Shape an agent’s possibilities
    • The agent might not be aware of a decision

Perceiving automated decisions

  • Transparency has been proposed as an approach to immersion
    • e. g. French law on explanation
  • Seamful experiences
    • Make evident the interfaces between systems
    • In doing so, reveal the technical object

Obstacles to information-based approaches

  • Both transparency and seamfulness provide information
  • This approach is only useful to the extent that the destinataries can make sense of what has been provided
  • Further issue (beyond this talk): doing something with the information.

Cognitive demands and explanation

  • Modern ADM models rely on complex models
    • Mathematical complexity
    • Large volumes of data
  • Simply providing all rules might exceed working memory
    • Cognitive demand might be too taxing
    • Potential solutions:
      • Explanations: XAI, clear interfaces
      • Legal requirement for actionable information

Attention and habituation

  • People are subject to many decisions every day
    • Informing about each one might lead to habituation
    • Even if warnings are not ignored, attention
  • Dehabituation might lead people to abandon systems or circumvent solutions

Open questions

  • Cognitive limits
    • How to bring automated decisions to attention?
    • What is a reasonable chunk of information for a data subject?
    • How does habituation work in ADM?
  • Effective legal and HCI implementation of limits

Thank you!


Human-computer relations

Seamful experiences

Memory and attention