dataset: synthetic from PANTHER

[tristan-f-r]

We also refactor the cache system (again!) to reduce code indirection, and make the entire repository a python package for referencing other modules throughout the codebase.

Co-Authored-By: Neha Talluri 78840540+ntalluri@users.noreply.github.com
Co-Authored-By: Oliver Faulkner Anderson 112665860+oliverfanderson@users.noreply.github.com
Co-Authored-By: Altaf Barelvi altafayyubibarelvi@gmail.com

[ntalluri]

A question I’d appreciate feedback on: Currently, we generate separate source, target, and prize files for each pathway, but we combine all pathways into each thresholded interactome. Should we also create a combined list of sources, targets, and prizes? Should we also combine the gold standard as well? Or would it be better to keep separate interactomes for each individual pathway (keep it the way it is)?

[tristan-f-r]

We should have separate gold standards.

[ntalluri]

When this is reviewed (or before) we should do tests to see how connected the networks are after thresholding, adding back the pathway data, and removing proteins that don't have uniprot ids.

[ntalluri]

Also there is a chance we can use more panther pathways, we should look to see what else we can use from pathway commons.

[ntalluri]

@oliverfanderson @ctrlaltaf For the gold standard nodes (and potentially the edges), should we exclude source, target, and prize nodes when defining it? Currently, it looks like we’re including these nodes in the gold standard for each pathway. These nodes overlap with the gold standard, but that overlap should happen naturally, not by construction/being predefined. I’m concerned this could inflate our precision and recall metrics, because of a form of data leakage.

[ntalluri]

@oliverfanderson @ctrlaltaf For the gold standard nodes (and potentially the edges), should we exclude source, target, and prize nodes when defining it? Currently, it looks like we’re including these nodes in the gold standard for each pathway. These nodes overlap with the gold standard, but that overlap should happen naturally, not by construction/being predefined. I’m concerned this could inflate our precision and recall metrics, because of a form of data leakage.

Plan to keep all of them in the gold standard. But update the evaluation code to deal with the sources/targets/prizes being in the gold standard and shown as a different baseline where those are all set as frequency 1.0.

[ntalluri]

Should we also consider how sparse an interactome becomes after applying a threshold to the STRING interactome? When we filter by size, we implicitly accounting for the decrease in graph density as well. Would it make more sense to treat size and density as separate variables when evaluating performance? However, does testing for density even matter in this context; are there any interactomes that aren’t already highly connected?

I’m thinking we should first threshold the interactomes, then select only those that are highly connected (e.g., density ≥ 0.85). From that subset, we could choose a few to represent different size scales.

[ntalluri]

I will be updating how we create interactomes for the Panther pathways dataset.

Current:
Our current thresholded interactomes are built by applying a hard cutoff on experimental scores (keeping edges with score ≥ x).
While this approach retains high-score edges and removes low-score ones, it distorts the original score distribution, which could be problematic for algorithms that rely on edge scores during optimization.

New:
Instead for the interacomes, we should approach it in a downsampling approach: build smaller interactomes that preserve the original score distribution of the original network while also reducing the total edge count.

For example, in the STRING interaction networks, when using only physical interactions and experimental edge scores, we could aim to keep 25% of all edges.
To achieve this, we could:

• Sample 25% (or remove 75%) of edges uniformly at random, ignoring edge scores
• Or use stratified sampling by edge score bins to preserve the distribution of scores

Now we will be construct new interactomes by removing X% of edges and then adding all edges from all chosen PANTHER pathways. We will only keep downsampled interacomes that satisfy specified properties for a given set of sources and targets.

Proposed brute-force method for Panther pathways interactomes:

1. edge removal

Randomly remove X edges from the full STRING interactome

• Option A: Remove edges uniformly at random (ignoring scores)
• Option B: Stratify edges by score bins and sample within each bin to maintain the overall score distribution
  • We can stratify the edges into bins based on their score ranges (example: [0–300], [300–600], [600–900] ...).
  • Then, when we randomly remove X edges, we can remove them proportionally from each bin.
  • Example: suppose 20% of the original edges fall in the 0–300 bin, 50% in the 300–600 bin, and 30% in the 600–900 bin. If we're removing 1000 edges total, we'd randomly pick ~200 from the first bin, ~500 from the second, and ~300 from the third.

2. pathway integration
   Add all edges from the selected Panther pathways to the new downsampled interactome

3. Property checks

Verify that the new network maintains the following properties:

• All-in-one: all sources and targets lie in the same connected component

  • V_{ST} = {all sources} U {all targets}, and after edge removal, all vertices in V_{ST} should belong to the same connected component of the graph

• Reachability: every target is reachable from at least one source (can be checked via apsp or bfs)

• Might want to soften the criteria to be X% of the sources and targets remain in a single component,
  and (potentially) X% of the targets are reachable from at least one source.

4. Restart is necessary

If the properties above are not satisfied, repeat the process with a different random sample.

[ntalluri]

For this dataset, we are planning on using it for all of the evaluations. I was deciding if we need to use all of the pathways, and I don't think we need to. I decided on a couple that we can use:

Balanced
Interleukin_signaling - 86 GS Nodes, 811 GS Edges, 18 S / 16 T (ratios 0.209 / 0.186)
Apoptosis_signaling - 108 GS Nodes, 286 GS Edges, 6 S / 17 T (0.056 / 0.157)

Skewed
Cadherin_signaling - 150 GS Nodes, 2650 GS Edges, 17 S / 3 T (0.113 / 0.02)
PDGF_signaling - 125 GS Nodes, 764 GS Edges, 2 S / 28 T (0.016 / 0.224)
Toll_signaling - 44 GS Nodes, 84 GS Edges, 8 S / 3 T (0.182 / 0.068)

Tiny
Hedgehog_signaling - 19 GS Nodes, 61 GS Edges, 2 S / 2 T (0.105 / 0.105)

When making the interactomes, I want to add all of these pathways on the thresholded interactomes and uphold the properties above.

I need to double check if I used any of these will break the rules for pilot data/runs; but since we are making a new dataset that wasn't used for my thesis, I think we will be okay.

[tristan-f-r]

Made minor changes to fix the interactome fetching - these shouldn't cause any conflicts, nor were the changes I wanted to make as mentioned in Slack. [If they do, feel free to force push.]

[ntalluri]

This won't be the final list. We still need to check for pathways that have at least one source and at least one target. That will help us finalize this list.

[ntalluri]

I think these do, I think the file you sent me had a messed up folder structure because of the "/" so my script didn't find information on it. I have to go in a get these stats manually.