Support conda lock files with any extension and remote URLs

[ewels]

From #5583 (comment):

@pinin4fjords: Conda lock files DO work, but they can't currently have the '.lock' extension. They must be .txt, and can't be remote files.

PR Goals:

• Support remote URIs for conda lock files
• Detect conda lock files by looking at file content, instead of relying on file extension

Details

This change allows conda lock files to be used with any file extension (e.g., .lock, .txt, or no extension) as long as they contain the `@EXPLICIT` marker in the first 20 lines. Additionally, remote conda lock files can now be specified using http/https URLs.

Changes:

• Add isLockFile() method to detect lock files by @EXPLICIT marker
• Add isRemoteFile() method to check for http/https URLs
• Add stageRemoteFile() method to download remote files locally
• Modify condaPrefixPath() to handle remote files and lock files with any extension
• Modify createLocalCondaEnv0() to detect and handle lock files properly
• Add tests for new functionality
• Update documentation to reflect new behavior

Fixes #5583

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	67bc9fd
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/695d258675adf30008effd55

[christopher-hakkaart]

Docs look good. One minor suggestion. Someone else will need to check the code.

[pditommaso]

It looks like this PR tries to do too many things (parsing content, formatting env (?), supporting remote uris, etc), please scope clearly the goals.

[ewels]

Tested successfully in GitHub codespaces, working as expected:

main.nf

#!/usr/bin/env nextflow

params.input = "data/*.fq.gz"

workflow {
    channel.fromPath(params.input)
        .set { reads }

    FASTQC(reads)
}

process FASTQC {
    conda "https://wave.seqera.io/v1alpha1/builds/bd-af7a5314d5015c29_1/condalock"

    input:
    path reads

    output:
    path "*.html"
    path "*.zip"

    script:
    """
    fastqc ${reads}
    """
}

With a nextflow.config with conda.enabled = true

I tried locally at first but it failed, because the lockfile was for a linux/arch64 Docker build so didn't work on my apple silicon ARM processor.

[ewels]

It looks like this PR tries to do too many things (parsing content, formatting env (?), supporting remote uris, etc), please scope clearly the goals.

You basically described the goals 😅 - though I don't think it does any formatting?

Goals:

• Support remote URIs for conda lock files
• Detect conda lock files by looking at file content, instead of relying on file extension

[ewels]

I updated the PR description to try to make this clearer @pditommaso. I'm not really sure how I can reduce the scope of the code changes without missing the goals. Suggestions welcome!

[pinin4fjords]

I think the scope here is appropriate. The two goals are inherently linked - remote conda lock files (like Wave's condalock endpoint) don't have .txt extensions, so content-based detection (@EXPLICIT marker) is required to make remote URLs work at all.

The implementation is clean: three small helper methods (isLockFile, isRemoteFile, stageRemoteFile) plus integration into the existing flow. The test coverage is thorough.

This unblocks a real user pain point from #5583 - conda lock files are increasingly common in nf-core and other pipelines, and the current .txt extension requirement is confusing and limiting.

[pditommaso]

A part the comment on performance, not quite convinced nextflow should download remote lock files on-the-file. Lock files should be considered an assert of the module bundle (and eventually downloaded with the bundle itself)

[ewels]

Ok, I reverted that part of the PR and added an admonition to the docs explaining that environment.yml files have partial remote support (http) but that's all.

The behavior is now:

• Remote YAML files (http/https + .yml/.yaml): URL passed directly to conda/mamba
• Remote lock files: Throws error (not supported)
• Local lock files: Work with any extension (.lock, .txt, no extension, etc.)

[ewels]

@pditommaso should be good to go now.