Publications and preprints
2023
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre*, Gabriel Ilharco*, Alex Fang*, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt
[Paper] / [Code] / [Website]TaskWeb: Selecting Better Source Tasks for Multi-task NLP
Joongwon Kim, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi
[Paper]Reproducible scaling laws for contrastive language-image learning
Mehdi Cherti*, Romain Beaumont*, Ross Wightman*, Mitchell Wortsman*, Gabriel Ilharco*, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev
Conference on Computer Vision and Pattern Recognition (CVPR) 2023
[Paper] / [Code]
2022
Editing Models with Task Arithmetic
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi
International Conference on Learning Representations (ICLR) 2023.
[Paper] / [Code]Adaptive Testing of Computer Vision Models
Irena Gao, Gabriel Ilharco, Scott Lundberg, Marco Tulio Ribeiro
[Paper]Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, Ludwig Schmidt
Conference on Neural Information Processing Systems (NeurIPS) 2022
[Paper] / [Code]Patching open-vocabulary models by interpolating weights
Gabriel Ilharco*, Mitchell Wortsman*, Samir Yitzhak Gadre*, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt
Conference on Neural Information Processing Systems (NeurIPS) 2022
[Paper] / [Slides] / [Code] / [Website]Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt
International Conference on Machine Learning (ICML) 2022
[Paper] / [Code]Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt International Conference on Machine Learning (ICML) 2022
[Paper]CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, Shuran Song
[Paper]Robust fine-tuning of zero-shot models
Mitchell Wortsman*, Gabriel Ilharco*, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt
Conference on Computer Vision and Pattern Recognition (CVPR) 2022
[Paper] / [Code]
2021
Probing contextual language models for common ground with visual representations
Gabriel Ilharco, Rowan Zellers, Ali Farhadi, Hannaneh Hajishirzi
North American Chapter of the Association for Computational Linguistics (NAACL) 2021
[Paper] / [Slides]Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner
Empirical Methods in Natural Language Processing (EMNLP) 2021
[Paper] / [Code]MultiModalQA: complex question answering over text, tables and images
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant
International Conference on Learning Representations (ICLR), 2021.
[Paper] / [OpenReview]Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A Smith
Empirical Methods in Natural Language Processing (EMNLP) 2021
[Paper]Contrasting Contrastive Self-Supervised Representation Learning Pipelines
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, Roozbeh Mottaghi
International Conference on Computer Vision (ICCV) 2021
[Paper]
2020
Evaluating models’ local decision boundaries via contrast sets
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
Conference on Empirical Methods in Natural Language Processing (EMNLP) Findings 2020
[Paper]Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith
[Paper]Toward ML-centric cloud platforms
Ricardo Bianchini, Marcus Fontoura, Eli Cortez, Anand Bonde, Alexandre Muzio, Ana-Maria Constantin, Thomas Moscibroda, Gabriel Ilharco, Girish Bablani, Mark Russinovich
Communications of the ACM 2020
[Paper]
2019
Large-scale representation learning from visually grounded untranscribed speech
Gabriel Ilharco, Yuan Zhang, Jason Baldridge
Conference on Computational Natural Language Learning (CoNLL) 2019
[Paper] / [Code]General evaluation for instruction conditioned navigation using dynamic time warping
Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, Jason Baldridge
Visually Grounded Interaction and Language Workshop at NeurIPS 2019
[Paper] / [Code]Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
Vihan Jain*, Gabriel Ilharco*, Alex Ku*, Ashish Vaswani, Eugene Ie, Jason Baldridge
Annual Meeting of the Association for Computational Linguistics (ACL) 2019
[Paper] / [Slides] / [Code]Transferable representation learning in vision-and-language-navigation
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Ilharco, Jason Baldridge, Eugene Ie
International Conference on Computer Vision (ICCV) 2019
[Paper]