Abstract Music generation has attracted growing interest with the advancement of deep generative models. However, generating music conditioned on textual descriptions, known as text-to-music, remains challenging due to the complexity of musical structures and high sampling rate requirements. Despite the task’s significance, prevailing generative models exhibit limitations in music quality, computational efficiency, and generalization. This paper introduces JEN-1, a universal high-fidelity model for text-to-music generation. JEN-1 is a diffusion model incorporating both autoregressive and non-autoregressive training. Through in-context learning, JEN-1 performs various generation tasks including text-guided music generation, music inpainting, and continuation. Evaluations demonstrate JEN-1’s superior performance over state-of-the-art methods in text-music alignment and music quality while maintaining computational efficiency.

Comparison between music generative models

In the following, we compare the output of JEN-1 to the outputs of other state-of-the-art music generative models (MusicGen, MusicLM, Riffusion and Mousai).

---

---

---

---

---

---

---

---

---

---

---

---

---

---

Representation of key musical attributes

Zero-shot ability

Diversity

Music inpainting (cut and edit)

By masking part of the tracks we can generate music that respects the text prompt, while following the remainder of the track - the following samples have masking applied at 2-5 seconds.

Music continuation

We can generate coherent music continuations that respect the text prompt - the following samples have masking applied during the second half of the track (5-10 seconds).