The increased capability of modern artificial intelligence (AI) systems—especially generative AI—has raised widespread concerns about their impact. We define AI as “an emergent family of technologies that build on machine learning, computation and statistical techniques, as well as rely on large data sets to generate responses, classifications or dynamic predictions that resemble those of a knowledge worker” [1]. In this project, we consider a long-standing concern about the impact of technology use, namely its effects on the skills of those using or affected by the system.
We adopt as a research setting the use of AI tools for programming, chosen because AI tools to assist programmers are already widely used and having an increasing impact. For instance, in October 2024, Sundar Pichai (CEO of Google and Alphabet) stated that 25% of the code at Google is now written by AI. This trend raises concerns about the future of entry-level programmers. While most attention has focused on AI’s impact on productivity or labor replacement, we investigate skill acquisition and maintenance, examining how AI use differentially affects beginners and experienced workers. We consider programmers to be a bellwether of broader transformations in skilled information work.
Within the broad topic of how AI tools are affecting the future of programming work, we explore the following research questions:
- How does the use of AI tools for certain tasks affect beginners’ acquisition and experts’ retention of the skills to do those tasks? How do these effects vary across tasks and tools?
- What new skills are needed to work effectively with AI tools and how are those skills acquired?
- Can one be competent at a job without developing particular skills long viewed as necessary, i.e., can new skills of using an AI tool compensate for lack of skills in the domain?
With generous support from the Sloan Foundation (grant G-2025-79202), we are piloting two studies to explore these questions.
Study 1 investigates how introductory programming students develop traditional and AI-related skills (e.g., prompting, evaluation) when learning with generative AI tools like ChatGPT or Copilot. It compares different patterns of AI tool use and explores how instructional interventions (like feedback on over-reliance) affect learning outcomes, particularly procedural and metacognitive skill acquisition.
Study 2 examines how faculty and graduate student developers adapt to using AI tools in real-world development work, focusing on the retention or erosion of core programming skills. Using the Critical Incident Technique, it explores how expertise level, motivation and organizational norms influence tool reliance, workflow changes and skill trajectories. We will compare new hires and more experienced programmers for evidence of deskilling (people with less skill doing the job) or upskilling (more skilled workers getting more benefit from the tools).
We hope to soon launch a second arm of Study 2 addding professional programmers and a third study, carried out at universities in two countries.
Study 3 develops, deploys and evaluates a customized AI tool for data science. It tests the findings of the other studies by designing interventions, with the goal of developing a system that supports data science development and effective collaboration without undermining engagement and skill development.
[1] Faraj, S., Pachidi, S. and Sayegh, K. 2018. Working and organizing in the age of the learning algorithm. Information and Organization. 28, 1, 62–70. doi: 10.1016/j.infoandorg.2018.02.005.