Learn PromptingMusic Generation
- This article explores different AI tools for music generation.
Music generation models are becoming increasingly popular, and will eventually have a large impact on the music industry.
Music generation models can create chord progressions, melodies, or full songs. They can structure and create music in specific genres and compose or improvise in the style of specific artists.
However, despite the enormous potential of music models, they are currently difficult to prompt. The generated output is often not thoroughly customizable by prompts, unlike image or text generation models.
Suno
Suno is a platform for creating music with GenAI. Users can generate unique tracks, remixes, and soundscapes by inputting simple prompts or using the site's advanced customization features.
Udio
Udio is another, similar platform music generation platform.
Riffusion
Riffusion, a fine-tuned version of Stable Diffusion, can be controlled with prompts to generate instruments and pseudo styles, but it has a limited number of beats available.
Mubert
Mubert seems to interpret prompts through sentiment analysis that links appropriate musical stylistics to the prompt (controlling the musical parameters in detail via prompts is not possible). It is unclear how much of the resultant generation is done by AI.
Other
There are attempts to use GPT-3 as a Text-2-Music tool with actual prompting for musical elements on the "micro-level" of notes (instead of the rather vague prompt-style-analogies mubert & riffusion produce) (e.g. write the notes for a folk song that only uses A, B, C#, F#, and G). However, at present those attempts are limited to single instruments.
Other approaches include a model chain that converts any image into sound that represents it and prompting ChatGPT to generate code for Python libraries that create sound.
Notes
Music prompting is not well built out... yet. MusicLM looks promising, but it is not yet available to the public.
Sander Schulhoff
Sander Schulhoff is the CEO of HackAPrompt and Learn Prompting. He created the first Prompt Engineering guide on the internet, two months before ChatGPT was released, which has taught 3 million people how to prompt ChatGPT. He also partnered with OpenAI to run the first AI Red Teaming competition, HackAPrompt, which was 2x larger than the White House's subsequent AI Red Teaming competition. Today, HackAPrompt partners with the Frontier AI labs to produce research that makes their models more secure. Sander's background is in Natural Language Processing and deep reinforcement learning. He recently led the team behind The Prompt Report, the most comprehensive study of prompt engineering ever done. This 76-page survey, co-authored with OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions, analyzed 1,500+ academic papers and covered 200+ prompting techniques.
Footnotes
-
Forsgren, S., & Martiros, H. (2022). Riffusion - Stable diffusion for real-time music generation. https://riffusion.com/about ↩