[TTS] MagpieTTS: Implement Frechet Codec Distance metric + some minor inference bugfixes #15223
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
nemo/collections/tts/modules/magpietts_inference/evaluate_generated_audio.py
Fixed
Show fixed
Hide fixed
Signed-off-by: Fejgin, Roy <[email protected]>
Instead of taking a codec instance, accept a codec name: local path or HF/NGC name. This simplifies the metric's integration in calling code. Signed-off-by: Fejgin, Roy <[email protected]>
Signed-off-by: Fejgin, Roy <[email protected]>
Signed-off-by: Fejgin, Roy <[email protected]>
* address some CI linting issues * include a file that was missed in last commit Signed-off-by: Fejgin, Roy <[email protected]>
rfejgin
force-pushed
the
magpietts_frechet_codec_distance
branch
from
8d997ac to
3fc5f37
Compare
Signed-off-by: Fejgin, Roy <[email protected]>
blisc
requested changes
Dec 30, 2025
Comment on lines
+53
to
+59
| # Consturct a length tensor: one batch element, all frames. | ||
| x_len = torch.tensor(x.shape[0], device=x.device, dtype=torch.long).unsqueeze(0) # (1, 1) | ||
| tokens = x.permute(1, 0).unsqueeze(0) # 1, C, B*T | ||
| embeddings = self.codec.dequantize(tokens=tokens, tokens_len=x_len) # (B, D, T) | ||
| # we treat each time step as a separate example | ||
| embeddings = rearrange(embeddings, 'B D T -> (B T) D') | ||
| return embeddings |
Collaborator
There was a problem hiding this comment.
Don't we need some sort of masking here? If we are reducing B and T into one dimension, how do we ensure that no padding gets passed to the model?
Collaborator
There was a problem hiding this comment.
If this only works on batch 1, we should add a check that x has a batch dimension of 1 or output an error or a warning
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds the Frechet Codec Distance metric and integrates it in MagpieTTS inference scripts. Also fixes some minor MagpieTTS inference bugs.
Collection: TTS
Changelog
The Frechet Distance (FD) is commonly used to evaluate generative models (e.g. Frechet Inception Distance, Frechet Audio Distance). In this PR we implements FD in the embedding space of a neural codec. This is a metric that measures how closely the distributions of real and generated codec frames match, at the single frame level.
Changes:
frechet_codec_distance.py: An implementation of FD in codec embedding space. Builds on TorchMetrics' FID implementation. We provide the audio codec as a custom feature extractor.test_frechet_coec_distance.py: Unit test--disable_fcdcommand line argument tomagpietts_inference.pytitanet_smallspeaker representation model. This was present in earlier versions of the inference scripts and appears to have been accidentally lost in recent refactoringsPR Type: