arXiv preprint ·
Characterizing Feedback Statements in Machine Learning Jupyter Notebooks
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use it to decide what to do next. We call these feedback statements and identify two forms: exploratory statements that display values for visual inspection, and validation statements that enforce conditions programmatically through assertions. Many ML failures do not surface as exceptions and thus escape the crash-based analyses that dominate prior work on ML notebooks. This study examines what practitioners check to catch failures that would otherwise pass silently, by characterizing feedback statements that encode the practitioner's mental model of what the code should do and what could go wrong. We mine 297,851 public Python Jupyter notebooks from GitHub and Kaggle and extract 1,092,780 feedback statements. We sample 816 statements through proportional stratified sampling from semantic clusters obtained from CodeBERT embeddings, and apply grounded theory and open coding to label and analyze each one. We contribute a taxonomy of feedback statements in ML notebooks, organized along the functional intent of the statement and the ML pipeline stage in which it appears. The taxonomy reveals that feedback is overwhelmingly exploratory, and that the two platforms host qualitatively different modes of ML work. Mapping our taxonomy to an existing crash taxonomy shows that it captures defensive practices against silent failures that crash analysis cannot observe. Our findings indicate that notebook source should be treated as a confounder in studies of ML developer practice, surface opportunities for notebook tooling, and motivate empirical study of silent ML failures. We release the corpus of 1,092,780 feedback statements and the codebook to support replication and tooling research.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
From Code to Requirements: Agentic Reverse Engineering of Business Rules at Enterprise Scale
Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks
Towards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical Study
One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents
"It Comes in Notebooks": Changes and Challenges when Operationalizing ML Prototypes
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.