Workshop Info

Date & location: Sun Dec 15, 2024. West Meeting Room 211-214

Tentative schedule and NeurIPS virtual site: https://neurips.cc/virtual/2024/workshop/84753

Accepted papers: OpenReview

Awards

Oral presentations🏆

Transformers Learn to Compress Variable-order Markov Chains in-Context. Ruida Zhou, Chao Tian, Suhas Diggavi

Getting free Bits Back from Rotational Symmetries in LLMs. Jiajun He, Gergely Flamich, José Miguel Hernández-Lobato

Prechastic Coding: An Alternative Approach to Neural Network Description Lengths. Paris Dominic Louis Flood, Pietro Lio

An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models. Anuj K. Nayak, Lav R. Varshney

Spotlight presentations🏆

Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations. Kola Ayonrinde, Michael T Pearce, Lee Sharkey

Diffusion Models With Learned Adaptive Noise. Subham Sekhar Sahoo, Aaron Gokaslan, Christopher De Sa, Volodymyr Kuleshov

The Rate-Distortion-Perception Trade-Off with Algorithmic Realism. Yassine Hamdi, Aaron B. Wagner, Deniz Gunduz

The Trichromatic Strong Lottery Ticket Hypothesis: Neural Compression With Three Primary Supermasks. Ángel López García-Arias, Yasuyuki Okoshi, Hikari Otsuka, Daiki Chijiwa, Yasuhiro Fujiwara, Susumu Takeuchi, Masato Motomura

Best reviewer award🏆 goes to Thanh-Dung Le.

Congratulations to all the award winners!

Call for Papers

The workshop solicits original research in the intersection of machine learning, data/model compression, and more broadly information theory.

Machine learning and compression have been described as “two sides of the same coin”, and the exponential amount of data being generated in diverse domains underscores the need for improved compression as well as efficient AI systems. Leveraging deep generative models, recent machine learning-based methods have set new benchmarks for compressing images, videos, and audio. Despite these advances, many open problems remain, such as computational efficiency, performance guarantees, and channel simulation. Parallel advances in large-scale foundation models further spurred research in efficient AI techniques such as model compression and distillation. This workshop aims to bring together researchers in machine learning, data/model compression, and information theory. It will focus on enhancing compression techniques, accelerating large model training and inference, exploring theoretical limits, and integrating information-theoretic principles to improve learning and generalization. By bridging disciplines, we seek to catalyze the next generation of scalable, efficient information-processing systems.

Topics of interest include, but are not limited to,

  • Improvements in learning-based techniques for compressing data, model weights, implicit/learned representations of signals, and emerging data modalities.
  • Accelerating training and inference for large foundation models, potentially in distributed settings.
  • Theoretical understanding of neural compression methods, including but not limited to fundamental information-theoretic limits, perceptual/realism metrics, distributed compression and compression without quantization.
  • Understanding/improving learning and generalization via compression and information-theoretic principles.
  • Information-theoretic aspects of unsupervised learning and representation learning.

Call for Reviewers

Please fill out this Google form if you are interested in reviewing for the workshop. Best reviewer wins free registration for the full conference!

Important Dates

  • Submission deadline: Sept 30, 2024 (Anywhere on Earth)
  • Notification date: Oct 9, 2024
  • Workshop date: Dec 15, 2024

Submission Instructions

Submission website: OpenReview

We solicit short workshop paper submissions of up to 4 pages 6 pages + unlimited references/appendices. Please format submissions in NeurIPS style. Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names.

All accepted papers are expected to be presented as posters at the poster session, and published via Openreview after the workshop. Some accepted papers will be selected as contributed/spotlight talks.

This workshop will not have formal proceedings, so we welcome the submission of work currently under review at other archival ML venues (for example, shorter versions of main conference submissions can be submitted to our workshop concurrently). We also welcome the submission of work recently published in information theory venues (e.g. Transactions on Information Theory, ISIT, ITW) that may be of interest to an ML audience. However, we will not consider work recently published in or accepted to other archival ML venues (e.g., NeurIPS main conference) to encourage the presentation of fresh and cutting-edge results at this workshop.

Tentative Speakers

Emilien Dupont
Research Scientist, DeepMind
Ziv Goldfeld
Assistant Professor, Cornell
Ashish Khisti
Professor, University of Toronto
Sanae Lotfi
PhD Student, NYU
Ayfer Özgür
Associate Professor, Stanford

Tentative Panelists

Aaron Wagner
Professor, Cornell
Sanae Lotfi
PhD Student, NYU
Ashish Khisti
Professor, University of Toronto
Ayfer Özgür
Associate Professor, Stanford

Organizers

Yibo Yang
PhD Student, UC Irvine
Karen Ullrich
Research Scientist, Meta AI
Justus Will
PhD Student, UC Irvine
Ezgi Özyılkan
PhD Student, NYU
Elza Erkip
Professor, NYU
Stephan Mandt
Associate Professor, UC Irvine

Organizers affiliations