About me

Hi, I’m Gabriel Ilharco. My research interests include large-scale multimodal models and data. You can find more about my recent work from recent publications below.

Publications

2023

• DataComp: In search of the next generation of multimodal datasets
  Samir Yitzhak Gadre*, Gabriel Ilharco*, Alex Fang*, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt
  [Paper] / [Code] / [Website]

• TaskWeb: Selecting Better Source Tasks for Multi-task NLP
  Joongwon Kim, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi
  [Paper]

• Reproducible scaling laws for contrastive language-image learning
  Mehdi Cherti*, Romain Beaumont*, Ross Wightman*, Mitchell Wortsman*, Gabriel Ilharco*, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev
  Conference on Computer Vision and Pattern Recognition (CVPR) 2023
  [Paper] / [Code]

2022

• Editing Models with Task Arithmetic
  Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi
  International Conference on Learning Representations (ICLR) 2023.
  [Paper] / [Code]

• Adaptive Testing of Computer Vision Models
  Irena Gao, Gabriel Ilharco, Scott Lundberg, Marco Tulio Ribeiro
  [Paper]

• Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
  Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, Ludwig Schmidt
  Conference on Neural Information Processing Systems (NeurIPS) 2022
  [Paper] / [Code]

• Patching open-vocabulary models by interpolating weights
  Gabriel Ilharco*, Mitchell Wortsman*, Samir Yitzhak Gadre*, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt
  Conference on Neural Information Processing Systems (NeurIPS) 2022
  [Paper] / [Slides] / [Code] / [Website]

• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt
  International Conference on Machine Learning (ICML) 2022
  [Paper] / [Code]

• Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
  Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt International Conference on Machine Learning (ICML) 2022
  [Paper]

• CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration
  Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, Shuran Song
  [Paper]

• Robust fine-tuning of zero-shot models
  Mitchell Wortsman*, Gabriel Ilharco*, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt
  Conference on Computer Vision and Pattern Recognition (CVPR) 2022
  [Paper] / [Code]

2021

• Probing contextual language models for common ground with visual representations
  Gabriel Ilharco, Rowan Zellers, Ali Farhadi, Hannaneh Hajishirzi
  North American Chapter of the Association for Computational Linguistics (NAACL) 2021
  [Paper] / [Slides]

• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
  Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner
  Empirical Methods in Natural Language Processing (EMNLP) 2021
  [Paper] / [Code]

• MultiModalQA: complex question answering over text, tables and images
  Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant
  International Conference on Learning Representations (ICLR), 2021.
  [Paper] / [OpenReview]

• Finetuning Pretrained Transformers into RNNs
  Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A Smith
  Empirical Methods in Natural Language Processing (EMNLP) 2021
  [Paper]

• Contrasting Contrastive Self-Supervised Representation Learning Pipelines
  Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, Roozbeh Mottaghi
  International Conference on Computer Vision (ICCV) 2021
  [Paper]

2020

• Evaluating models’ local decision boundaries via contrast sets
  Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
  Conference on Empirical Methods in Natural Language Processing (EMNLP) Findings 2020
  [Paper]

• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
  Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith
  [Paper]

• Toward ML-centric cloud platforms
  Ricardo Bianchini, Marcus Fontoura, Eli Cortez, Anand Bonde, Alexandre Muzio, Ana-Maria Constantin, Thomas Moscibroda, Gabriel Ilharco, Girish Bablani, Mark Russinovich
  Communications of the ACM 2020
  [Paper]

2019

• Large-scale representation learning from visually grounded untranscribed speech
  Gabriel Ilharco, Yuan Zhang, Jason Baldridge
  Conference on Computational Natural Language Learning (CoNLL) 2019
  [Paper] / [Code]

• General evaluation for instruction conditioned navigation using dynamic time warping
  Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, Jason Baldridge
  Visually Grounded Interaction and Language Workshop at NeurIPS 2019
  [Paper] / [Code]

• Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
  Vihan Jain*, Gabriel Ilharco*, Alex Ku*, Ashish Vaswani, Eugene Ie, Jason Baldridge
  Annual Meeting of the Association for Computational Linguistics (ACL) 2019
  [Paper] / [Slides] / [Code]

• Transferable representation learning in vision-and-language-navigation
  Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Ilharco, Jason Baldridge, Eugene Ie
  International Conference on Computer Vision (ICCV) 2019
  [Paper]