Memory-Efficient Contrastive Learning via Budgeted Hard Negative Selection
Abstract
Abstract Contrastive learning objectives such as those used in CLIP and SimCLR rely on large nega- tive sets, but conventional implementations materialize dense pairwise similarity matrices whose memory footprint grows quadratically with batch size. We introduce a memory-effcient con- trastive learning framework that avoids full similarity-matrix materialization while concentrating optimization on a fixed budget of informative negatives. The method streams similarity com- putation in column blocks, maintains an online top-k set of hard negatives for each anchor, and skips unnecessary top-k updates when a block cannot improve the current candidate set. A detached queue-based memory bank can further expand negative diversity, while low-rank adaptation (LoRA) reduces trainable-state overhead. On a ViT-B/32 implementation, the streaming formulation reduces the peak similarity-buffer footprint from O(N2 ) to O(Nb) for block size b ≪ N. In our single-A100 measurements, the dense baseline runs out of memory at batch size 4096, whereas the memory-effcient configuration scales to batch size 8192. The online pruning rule also eliminates an increasing fraction of top-k update operations as training progresses, reaching an 89.1% row-level skip ratio in the reported setting. These results establish a practical foundation for large-batch contrastive adaptation under constrained accelerator memory while retaining exact similarity evaluation within each streamed block.
Read the paper
Related studies
- Diarylethenes for Memories and Switches
- Long-Term Potentiation and Memory
- Social Memory Studies: From “Collective Memory” to the Historical Sociology of Mnemonic Practices
- The Cognitive Neuroscience of Working Memory
- Memory aging and brain maintenance
- Effector and Memory CTL Differentiation
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.