AI

Discontinuous Galerkin and Riemann solver in JAX.

Requiring 10^13-10^15 FLOPs to calculate one 8 bit weight in an LLM during pretraining is extremely expensive and seems inefficient. To better leverage the huge investments made into pretrained models, we develop the new “Assembly-of-Experts” (AoE) construction method to create capable child variants of existing Mixture-of-Experts parent models in linear time. Model weight tensors get interpolated individually, allowing to enhance or suppress semantic features of the parents. Varying the proportion of weights taken from the parent models, we observe some properties of the AoE child model changing gradually, while other behavioral traits emerge with a sharp transition. Surprisingly, nearly every generated model is functional and capable, which makes searching the model space straightforward. We construct the DeepSeek R1T “Chimera”, a 671B open-weights hybrid model combining DeepSeek’s V3-0324 and R1 model variants. The child inherits only the routed expert tensors of R1, but still achieves about R1-level intelligence. At the same time, it uses about 40% fewer output tokens, close to V3 speed. Constructed without any fine-tuning or distillation, the Chimera exhibits surprisingly compact, orderly reasoning compared to its parent models. The DeepSeek R1T Chimera model is available as open weights on Hugging Face.

Rare events are essential for understanding the behavior of non-equilibrium and industrial systems. It is of ongoing interest to develop methods for effectively searching for rare events. With the advent of quantum computing and its potential advantages over classical computing for applications like sampling certain probability distributions, the question arises whether quantum computers could also provide an advantage or inspire new methods for sampling the statistics of rare events. In this work, we propose a quantum reinforcement learning (QRL) method for studying rare dynamics, and we investigate their benefits over classical approaches based on neural networks. As a proof-of-concept example, we demonstrate that our QRL agents can learn and generate the rare dynamics of random walks, and we are able to explain this success as well as the different contributing factors to it via the intrinsic Fourier features of the parameterized quantum circuit. Furthermore, we show better learning behavior with fewer parameters compared to classical approaches. This is the first investigation of QRL applied to generating rare events and suggests that QRL is a promising method to study their dynamics and statistics.