LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Loading…

Audio-to-video attention follows the sound source. Drag a divider to compare each clip with LeAVJEPA’s attention from audio to video patches. Brighter patches stay on the sound-producing object as it moves, without any localization supervision.

Abstract

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

How it works

LeAVJEPA learns in latent space with one shared encoder and a symmetric objective. Modality dropout aligns the embeddings of audio-only and video-only views with a joint audio-video representation, without an EMA teacher, reconstruction decoder, cross-modal predictor, or contrastive negatives during pretraining.

Audio becomes a log-mel spectrogram; video becomes tubelets. Two joint global views and two modality-specific local views pass through one transformer and projector. Alignment brings their embeddings together; SIGReg prevents collapse across the batch.

Enable JavaScript to explore the method animation.

LeAVJEPA connects sound and vision by dropping a modality. One shared encoder aligns audio-only and video-only views with joint audio-video views. MSE brings embeddings together; SIGReg prevents collapse, without a teacher, predictor, or contrastive negatives.

One shared transformer

Video tubelets (1,568 tokens) and log-mel spectrogram patches (400 tokens) enter one early-fusion Vision Transformer, with ViT-B and ViT-L variants. A shared projector maps the [CLS] representation into the space used by the loss.

Modality-specific views

Two global views retain both modalities. Two local views alternate between audio-only and video-only: tokens from the absent modality are removed. “Local” means partial modality content; these views retain the same eight-second duration.

Alignment and SIGReg

Mean-squared error aligns each view with the mean of the joint global embeddings. SIGReg regularizes the batch toward an isotropic Gaussian to prevent collapse. Both terms train the encoder and projector together.

VGGSound ablations: modality-specific local views give the highest linear and attentive probe accuracy; either alignment or SIGReg alone fails to learn useful features. — **Modality-specific views improve cross-modal learning in LeAVJEPA.** Left: joint local views remain near the audio-only baselines (dashed lines). Random modality dropout improves probe accuracy, and making every local view audio-only or video-only yields the largest gain. Right: either loss term alone fails to learn useful features.

Frozen transfer

Frozen transfer with an attentive probe. AudioSet-20K mAP for audio-only (A), video-only (V), and audio-video (A-V) inputs, and ESC-50 accuracy. Baseline frozen results and parameter counts are as reported by Teotia et al. (2026). Best result per column in bold. ^*Not reported in the MJEPA paper; provided by the authors through personal correspondence.

				AudioSet-20K (mAP)			ESC-50
	Method	Params	Pre-train data	A	V	A-V	Acc.
Separate encoders	CAV-MAE (Gong et al., 2023)	170M	IN+AS	19.38	18.14	34.59	77.5
	MAViL (Huang et al., 2023)	170M	IN+AS	30.00	–	–	90.8
	EquiAV (Kim et al., 2024)	170M	IN+AS	34.25	18.60	38.60	93.2
	CAV-MAE Sync (Araujo et al., 2025)	170M	IN+AS	21.66	16.20	28.50	89.2
	XKD (Sarkar & Etemad, 2024)	170M	AS	–	–	–	93.6
Shared encoder	MJEPA ViT-B^* (Teotia et al., 2026)	86M	AS	37.54	20.18	40.68	–
	MJEPA ViT-L (Teotia et al., 2026)	300M	AS	38.89	25.38	42.90	95.2
	LeAVJEPA ViT-B (ours)	87M	AS	27.79	14.82	30.98	88.0
	LeAVJEPA ViT-L (ours)	304M	AS	31.56	18.25	36.03	91.3

Cross-modal retrieval

LeAVJEPA retrieves paired audio and video by cosine similarity of projected [CLS] features. The additional InfoNCE-head experiment trains a contrastive head on frozen features and is not zero-shot.

AudioSet cross-modal retrieval on the CAV-MAE balanced evaluation subset (1,725 clips), recall in percent. Baseline rows are as reported in the respective papers and all train an explicit cross-modal alignment term or predictor; LeAVJEPA rows are zero-shot cosine retrieval unless marked. Best baseline result per column in bold. ^†MJEPA retrieves through its jointly trained cross-modal predictor with an L1 nearest-neighbour search rather than direct embedding similarity. ^‡Contrastive alignment head trained on frozen features; not zero-shot.

	Audio → video			Video → audio
Method	R@1	R@5	R@10	R@1	R@5	R@10
CAV-MAE (Gong et al., 2023)	15.1	34.0	43.0	18.8	39.5	50.1
CAV-MAE Sync (Araujo et al., 2025)	27.9	52.4	62.2	35.2	58.3	67.6
EquiAV (Kim et al., 2024)	29.6	53.7	63.1	30.1	53.3	62.9
MJEPA ViT-L^† (Teotia et al., 2026)	29.3	53.7	61.8	30.6	53.4	61.3
LeAVJEPA ViT-B (ours)	10.5	28.9	36.6	11.4	28.2	36.8
LeAVJEPA ViT-L (ours)	19.1	40.5	50.4	18.7	42.1	51.9
LeAVJEPA ViT-L + InfoNCE head^‡ (ours)	24.4	46.9	57.7	25.6	48.0	57.1

End-to-end fine-tuning

Audio-visual classification on VGGSound (top-1, end-to-end fine-tuning). Published baselines all use separate audio and video encoders, except AVSiam, and are listed above the rule; our AudioSet-pretrained shared-encoder ViT-B (87M) and ViT-L (304M) below it. Best result per block in bold.

Method	Params	Init.	Pretraining	Top-1
AV-MAE (Georgescu et al., 2023)	172M	–	VGGS	64.2
AV-MAE (Georgescu et al., 2023)	611M	–	VGGS	65.0
AVSiam (Lin & Bertasius, 2024)	100M	IN-21K SL	AS-2M	64.9
AVSiam (Lin & Bertasius, 2024)	332M	IN-21K SL	AS-2M	67.1
CAV-MAE (Gong et al., 2023)	164M	IN-SSL	AS-2M	65.5
MAViL (Huang et al., 2023)	172M	IN-SSL	AS-2M	67.1
EquiAV (Kim et al., 2024)	172M	IN-SSL	AS-2M	67.1
LeAVJEPA (Linear)	87M	–	AS-2M	58.3
LeAVJEPA (Attentive)	87M	–	AS-2M	60.4
LeAVJEPA (Linear)	304M	–	AS-2M	59.9
LeAVJEPA (Attentive)	304M	–	AS-2M	61.1

The encoder localizes sounding objects

DINOv2‑style feature PCA of last-layer video patch tokens, with one basis fit per class (rows) and the top three components mapped to RGB. The sounding object takes a consistent color across clips and frames, despite training with clip-level objectives.

Feature PCA on four piano clips: the keyboard takes a consistent color across clips and frames — **Feature PCA of video patch tokens.** Left to right: playing piano, playing bass guitar, playing violin. Click to enlarge.

Feature PCA on four bass guitar clips: the instrument body and neck take a consistent color — **Feature PCA of video patch tokens.** Left to right: playing piano, playing bass guitar, playing violin. Click to enlarge.

A semantically structured embedding space

Each point is a VGGSound clip, placed by its LeAVJEPA embedding and coloured by what the clip contains. Start with the joint audio-video points: instruments, animals, vehicles and water sounds form clean clusters, although the model never sees a label during training. Then switch on Audio or Video to see where the same clips land when the model gets only one modality.

Enable JavaScript to explore the embedding space.

Embedding space of LeAVJEPA. Click a point to link the audio-only, video-only and joint versions of that clip. Pinch or Ctrl/⌘-scroll to zoom, and drag to pan. Heavy-tailed t-SNE of frozen ViT-L embeddings for 1,474 test clips from 30 classes; nearby points are similar, but distances between groups carry no meaning.

BibTeX

@misc{robson2026leavjepa,
  title  = {{LeAVJEPA}: A Minimalist Architecture for Audio-Visual Self-Supervised Learning},
  author = {Robson, Benjamin and Mentu, Santeri and Zhao, Wenshuai and Solin, Arno},
  year   = {2026},
  note   = {Preprint}
}