I’m a 5th year Artificial Intelligence PhD student at Berkeley advised by Anca Dragan and Stuart Russell within BAIR and CHAI. I’m thankful to be supported by the NSF Fellowship.

I’m interested in humans’ preference, value, and belief changes, and how they may be affected by interactions with AI systems. I’ve studied this both in generality (with the language of DR-MDPs), and more specifically in the context of recommender systems, investigating how the choice of algorithm might affect us users. I’m probably best known for my work on human-AI collaboration, and developing the Overcooked-AI benchmark.

Outside of research, I enjoy inline skating 🛹, watching movies 🎥, and finding new music 🎵. Before immigrating to the US, I grew up in the amazingly chaotic city of Livorno 🇮🇹 – visit if you get the chance!

Headshot of Micah Carroll smiling, wearing glasses and a tan sweater, with green plants in the background

Publications

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback
Marcus Williams*, Micah Carroll*, Adhyyan Narang, Constantin Weisser, Brendan Murphy, Anca Dragan
ICLR 2025
arXiv Paper Thread Code Talk
Beyond Preferences in AI Alignment
Tan Zhi-Xuan, Micah Carroll, Matija Franklin, Hal Ashton
Philosophical Studies 2024
arXiv Paper Thread
AI Alignment with Changing and Influenceable Reward Functions
Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, Anca Dragan
ICML 2024
arXiv Paper Thread Talk
Characterizing Manipulation from AI Systems
Micah Carroll*, Alan Chan*, Henry Ashton, David Krueger
EAAMO 2023
arXiv Paper
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Stephen Casper, Xander Davies, ..., Micah Carroll, ..., Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, Dylan Hadfield-Menell
TMLR 2023
arXiv Paper Thread
Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Media
Smitha Milli, Micah Carroll, Sashrika Pandey, Yike Wang, Anca Dragan
PNAS Nexus 2025
arXiv Paper Thread
Harms from Increasingly Agentic Algorithmic Systems
Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, Tegan Maharaj
FAccT 2023
arXiv Paper
Who Needs to Know? Minimal Knowledge for Optimal Coordination
Niklas Lauffer, Ameesh Shah, Micah Carroll, Michael Dennis, Stuart Russell
ICML 2023
arXiv Paper Thread
Uni[MASK]: Unified Inference in Sequential Decision Problems
Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgescu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, Sam Devlin
NeurIPS 2022 (Oral)
arXiv Paper Thread Code
Estimating and Penalizing Induced Preference Shifts in Recommender Systems
Micah Carroll, Dylan Hadfield-Menell, Stuart Russell, Anca Dragan
ICML 2022 (Spotlight)
arXiv Paper Thread
Optimal Behavior Prior: Improving Human-AI Collaboration Through Generalizable Human Models
Mesut Yang, Micah Carroll, Anca Dragan
Human-in-the-loop Learning (HILL) Workshop, NeurIPS 2022
arXiv Paper Thread Code
Time-Efficient Reward Learning via Visually Assisted Cluster Ranking
David Zhang, Micah Carroll, Andreea Bobu, Anca Dragan
Human-in-the-loop Learning (HILL) Workshop, NeurIPS 2022
arXiv Paper
Evaluating the Robustness of Collaborative Agents
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca Dragan, Rohin Shah
AAMAS 2021
arXiv Paper Code
On the Utility of Learning about Humans for Human-AI Coordination
Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, Anca Dragan
NeurIPS 2019
arXiv Paper Thread Code Demo Demo Blogpost Blogpost

Selected Talks

Talk thumbnail

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

FAR.ai Alignment Workshop, Santa Cruz 2024

Talk thumbnail

AI Alignment with Changing and Influenceable Reward Functions

PIBBS Speaker Series 2024

Research Mentorship

  • George Ingebretsen (Berkeley EECS undergrad)
  • Karim Abdel Sadek (incoming PhD student)
  • Marcus Williams (now OpenAI)
  • Constantin Weisser (now Haize Labs)
  • Lukas Fluri (incoming PhD Student at ETH Zurich)
  • Calvin Bo Zhang (now Scale AI)
  • Tianyi (Alex) Qiu (now Anthropic Fellow)
  • Davis Foote
  • Ethan Mendez (now PhD Student at Georgia Tech)
  • Austin Jang (incoming PhD Student)
  • Francis Geng (now PhD Student at UCSD)
  • Sebastian Zhao (Berkeley EECS undergrad)
  • Yike Wang (now PhD Student at University of Washington)
  • David Zhang (now SWE at Observe, Inc.)
  • Sashrika Pandey (now SWE at Figma)
  • Anrui Gu (now ML Engineer at Smith & Associates)
  • Mesut Yang (now SWE at Ironclad)
  • Nathan Miller (now SWE at Microsoft)
  • Paul Knott (now CSET)

Learning

Teaching