Télécom Paris · Institut Polytechnique de Paris · LTCI

MusicFlux: flexible conditioning for text-guided music generation and editing

A text-guided music editor built on a pretrained flow-matching text-to-audio model, unifying generation, inpainting/outpainting, and SFX restoration under one masked-conditioning formulation.

Teysir Baoueb* Leonardo Boulitreau de M. M. Marques* Paraskevas Stamatiadis Gaël Richard

LTCI, Télécom Paris, Institut Polytechnique de Paris, France · *equal contribution

▶ Listen to audio examples

Overview

One model, three music-editing tasks

Music editing transforms an existing recording to meet a new objective while preserving the musical attributes outside the intended modification. Given the high cost of training music generators from scratch, MusicFlux adapts a single pretrained flow-matching text-to-audio model (TangoFlux) to generation and two forms of editing, under one shared conditioning formulation.

Text-to-Music Generation

Generates a full 30-second clip directly from a text caption — no source audio involved.

Source = zeros tensor, mask m = 1 everywhere (fully generated).

Inpainting / Outpainting

Regenerates a masked time span so it fits seamlessly with the audio around it, guided by a caption.

Source = surrounding audio, m = 1 inside the region to reconstruct, m = 0 elsewhere.

SFX / Mastering Restoration

Cleans up a degraded recording — e.g. reducing reverb, fixing dynamics — following a text instruction.

Source = degraded audio (full context), mask m = 0 everywhere.

Diagram showing MusicFlux's shared masked-latent formulation across text-to-music generation, music inpainting, and SFX restoration, feeding a source-target latent fusion module and a diffusion transformer backbone.

Fig. 1. MusicFlux formulates generation, inpainting, and SFX restoration under a shared latent-conditioning paradigm. The source latent s is the input audio to transform; a source-target latent fusion module combines (s, m, x_t) into the trajectory z_t fed to the diffusion transformer (DiT).

Audio examples

Real test-set outputs per task, comparing the ground truth / input against both MusicFlux conditioning strategies (and, for restoration, the SonicMaster baseline).

Headphones recommended — 30-second excerpts from held-out test data, re-encoded to MP3 for the web.

Goal: generate a complete 30-second music clip purely from the text instruction below — no reference audio is given to the model.

Goal: reconstruct a masked span of an existing track — given the instruction and the surrounding, unmasked context — so it blends in seamlessly.

Goal: clean up a degraded recording — e.g. excess reverb, poor EQ, low volume — by following a plain-language instruction, without altering its musical content.

Method

How MusicFlux conditions on the source audio

MusicFlux fine-tunes TangoFlux, a flow-matching text-to-audio DiT, then explores two ways to condition it on source audio for editing — trading off editing quality against compute.

Background — Flow Matching

Learns a velocity field transporting a prior sample x₀ to data x₁ along a probability path. With rectified flow's linear interpolation:

x_t = (1−t)·x₀ + t·x₁ , velocity u_t = x₁ − x₀

Audio is generated by integrating the resulting ODE from noise to data.

Background — CAR-Flow

Reparameterizes the flow's source and target distributions with condition-dependent additive shifts, so the network only has to learn the remaining transport:

z₀ = x₀ + μ₀(y) , z₁ = x₁ + μ₁(y)

where y is the conditioning variable and μ₀, μ₁ are lightweight learned networks.

Fig. 2. (Left) MusicFlux-Concat: target and source latents concatenated along the sequence dimension (2L frames), mask concatenated along the channel dimension. (Right) MusicFlux-CAR: shift networks μ_src, μ_tgt shift the noise start-point and target endpoint; the DiT operates over the original L-frame sequence.

MusicFlux-Concat

Inspired by FLUX.1 Kontext: concatenate the source with the target so the Transformer attends to both directly.

Source latent s and noisy target x_t concatenated along the sequence dimension, plus a mask channel: z_t ∈ ℝ^B×2L×(C+1).
Distinct positional IDs per source/target token; full self-attention over 2L frames, loss computed on the target half only.
One new C → C+1 projection — but doubles the attention sequence length.

Stronger editing quality 2× sequence length / cost

MusicFlux-CAR

Inspired by CAR-Flow: pre-align the flow's endpoints with the conditioning signal via lightweight shift networks, instead of lengthening the sequence.

Two Context Shift Networks μ_src, μ_tgt predict additive shifts from source audio s, text embedding e, and mask m: z₀=ε+μ_src(s,e,m), z₁=x₁+μ_tgt(s,e,m).
DiT runs on the original L-frame interpolant — same length as plain text-to-music.
Zero-initialized output layer (training starts unshifted); adds only ≈4.6M params (<0.6% of the backbone).

Original sequence length >50% faster & lighter

To the authors' knowledge, the first application of CAR-Flow to audio editing and, more broadly, to time-series data.

Task-specific loss objectives

Both variants optimize a velocity-prediction MSE (u_t = x₁ − x₀ for Concat, u_t = z₁ − z₀ for CAR) over the full sequence for generation and SFX restoration. For inpainting/outpainting, the loss splits into a masked-region term and a down-weighted context term:

L_inpaint = L_masked + λ_ctx·L_ctx , λ_ctx = 0.1

Reconstructing the already-known context is the easier sub-problem, so it is weighted down relative to the region that must be generated.

Experiments

Datasets, setup, baselines & metrics

MusicFlux trains jointly on all three tasks, then is evaluated in- and out-of-domain against task-specific baselines.

JamendoMaxCaps (JMC)

Training + in-domain test

362Ktracks

58K htotal audio

1.81Mcaption pairs

Instrumental Jamendo audio captioned via Qwen2-Audio; 30-s segments, stereo, 44.1 kHz.

Song Describer Dataset (SDD)

Out-of-domain test (generation + inpainting)

706recordings

1,106captions

≤2 min95% of clips

Openly licensed recordings with human captions; vocal tracks excluded for an instrumental-only subset.

SonicMaster-Test

Out-of-domain test (SFX restoration)

1,000clean excerpts

7,000degraded variants

19degradation effects

Degraded variants across EQ, dynamics, reverb, amplitude and stereo effects, each with a restoration instruction.

Training setup

Base modelTangoFlux (flow-matching DiT, FLAN-T5 text encoder)
Transformer size6 dual-stream + 18 single-stream blocks, 8×128 heads
ParametersMusicFlux-CAR ≈859M · MusicFlux-Concat ≈860M
Max sequence645 latents ≈ 30 s @ 44.1 kHz (Stable Audio Open 1.0 VAE)
Hardware3 × L40S GPUs, 4 epochs
Batch size32 (CAR) / 16 (Concat), Adam, lr 2e-4
Schedule2k-step warmup, cosine with 2 restarts
Task samplingGeneration / inpainting / restoration, equal probability
Context loss weightλ_ctx = 0.1 (best on validation)
Inference50 ODE integration steps, CFG scale 4.5

Baselines

Stable Audio 3 (small-music-base)

459M-param DiT + flow matching. Frozen T5Gemma text embeddings via cross-attention; context audio is projected and added to each block's hidden states. Trained on curated AudioSparx + Freesound (≈1.2M tracks).

SonicMaster

Text-conditioned rectified-flow restoration model; the degraded audio is the initial flow state, not an added conditioning signal.

Evaluation metrics

CAF-Score↑ better

Text–audio alignment: CLAP similarity + FLEUR-based LALM assessment.

KAD↓ better

Kernel Audio Distance — MMD between CLAP embeddings of generated vs. reference audio.

FLEUR*↑ better

Inpainting fit: Qwen3-Omni-30B rates 0–1 boundary continuity, timbre, loudness, ambience, rhythm, artifacts.

KL divergence↓ better

Distributional discrepancy between restored and reference audio.

SSIM↑ better

Structural similarity between restored and clean mel-spectrograms.

Production Quality (PQ)↑ better

Audiobox Aesthetics estimate of technical quality — clarity, dynamics, balance, spatialization.

Table 1 — Generation, inpainting & restoration results

Model	Generation				Inpainting / Outpainting				Restoration
	JMC-Test		SDD		JMC-Test		SDD		SonicMaster-Test
	CAF ↑	KAD ↓	CAF ↑	KAD ↓	FLEUR* ↑	KAD ↓	FLEUR* ↑	KAD ↓	KAD ↓	KL ↓	SSIM ↑	PQ ↑
Ground-truth (GT)	0.5102	0.0000	0.5704	0.0000	0.8592	0.0000	0.8834	0.0000	0.0000	0.0000	1.000	7.7951
Stable Audio 3	–	–	0.5593	2.8631	–	–	0.8248	1.4794	–	–	–	–
SonicMaster	–	–	–	–	–	–	–	–	0.1137	0.0009	0.5918	7.5310
MusicFlux-Concat	0.5100	0.6400	0.5213	4.5535	0.8340	0.0099	0.8428	0.0380	0.0489	0.0013	0.5702	7.5478
MusicFlux-CAR	0.5196	1.1239	0.4950	4.6687	0.8234	0.2496	0.8390	0.4744	0.2680	0.0015	0.5412	7.4044

Bold = bestUnderline = second-best"–" = not applicable / not evaluated for this baseline

Table 2 — Computational efficiency (batch size 1, sequence length 645, FP32)

Stage	MusicFlux Method	Base memory (MiB)	Dynamic memory (MiB)	RTF
Inference	CAR	3364.19	75.71	0.1188
Inference	Concat	3358.54	150.11	0.2203
Training	CAR	3372.48	1988.10	–
Training	Concat	3366.82	2748.19	–

RTF = real-time factor. "Base memory" is model instantiation; "dynamic memory" is additional memory used during execution.

Generation: in-domain (JMC-Test), MusicFlux-CAR gets the best CAF-Score (0.5196) and MusicFlux-Concat the lowest KAD. Out-of-domain (SDD), Stable Audio 3 scores better on both metrics — MusicFlux generalizes less well to unseen prompts.

Inpainting / outpainting: MusicFlux-Concat is strongest on both metrics, in- and out-of-domain; MusicFlux-CAR still beats the SA3 baseline out-of-domain.

SFX restoration: MusicFlux-Concat gets the best KAD and Production Quality; SonicMaster leads KL and SSIM. Both MusicFlux variants stay competitive with the specialized baseline.

Efficiency: MusicFlux-CAR cuts inference RTF and dynamic memory by over 50%, and training dynamic memory by 28%, versus Concat — highlighting a trade-off between editing fidelity and efficiency.

Takeaways

What to remember

Unified framework

MusicFlux covers three music-editing settings — generation, inpainting/outpainting, and SFX restoration — in one framework.

Two conditioning strategies

MusicFlux-Concat achieves stronger editing performance; MusicFlux-CAR preserves the original sequence length and cuts inference runtime and dynamic memory by over 50%.

Competitive with specialists

Both remain competitive with task-specific baselines despite being single multi-task models trained on open data.

Fidelity vs. efficiency

The results highlight a trade-off between editing fidelity and efficiency; limitations in out-of-domain generalization and CAR conditioning motivate richer, more efficient conditioning in future work.