Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]

The story

reddit.com: LLMs describe physics well but don t understand it in any grounded sense — they ve learned statistical relationships between tokens like falls and gravity , not actual physical intuition. This is basically the Mary s Room problem: Mary knows every physical fact about color but has never seen one. LLMs are Mary. The idea I ve been thinking about: Train a JEPA-style model inside a physics simulation (think MuJoCo or a simple 2D env). Instead of predicting pixels or tokens, the model predicts representations of future states in an abstract embedding space. If it gets physics wrong, the predictor fails — the loss is unforgiving in a way next-token prediction never is. The embedding space that emerges should encode actual physical structure — object permanence, momentum, trajectories — because that s what makes prediction possible. Not surface-level textures, just the principles. Freeze those representations and attach them to an LLM-style reasoning model as a conditioning signal. The LLM now has both linguistic physics knowledge AND grounded physical intuition it can actually run forward — closer to a computational primitive than a propositional fact. The hypothesis is this makes downstream learning significantly faster — the LLM doesn t have to rediscover that objects fall, it has representations that already encode that. V-JEPA does something adjacent for video (predict future frame representations, not pixels), and DreamerV3 uses a latent world model for efficient RL — but the specific combo of JEPA-style prediction + sim-grounded physics representations + LLM attachment doesn t seem to have been done cleanly. Questions for the community: - Is there prior work I m missing that does exactly this? - What s the right interface between grounded JEPA representations and a language model? Just concatenate to the prompt embedding? Cross-attention? - Would the sim-to-reality gap kill the transfer, or are the representations abstract enough to survive it? Interested in whethe

reddit.com: Hi everyone, I just quickly wanted to share a paper I was working on for around a year now. I created this summary website with key results: https://flogrammer.github.io/moljepa/ TL;DR: its a multimodal JEPA model for molecules. There will be more work to do to improve performance and I would be happy about feedback and ideas :) submitted by /u/TerribleAntelope9348 [link] [comments]

arxiv.org: arXiv:2608.22642v4 Announce Type: replace-cross Abstract: Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of biochemical environments. To address these challenges, we present \textbf{Mol-JEPA}, a scalable framework for learning molecular world models. Rather than relying on suboptimal molecular perturbations, our model uses modality masking to exploit information from molecular structures, cellular phenotypes, binding affinities, ADMET profiles, quantum chemistry simulations and other drug discovery data. Across various benchmarks, we show that the representations learned by Mol-JEPA deliver strong performance, demonstrating the value of incorporating biochemical context through latent space prediction.