Skip to content

Joystream SDK: RFC / draft #5198

New issue
New issue
Open
Open
Joystream SDK: RFC / draft#5198
@Lezek123

Description

@Lezek123
Lezek123
opened on Jan 29, 2025

In this issue, I'll outline the proposed tech stack, requirements and functionality to be supported by the initial version of the Joystream SDK, based on the description here: #4596

I encourage everyone interested in building on top of Joystream to leave their comments and suggestions here. Any feedback at this stage is invaluable and will help shape the SDK for the maximum benefit of everyone.

Repository, Package Management, and Tooling

  • TypeScript Monorepo
  • Environment-agnostic (works in both Node.js and browser contexts)
  • High test coverage (ideally 100%)
  • Proper semantic versioning and CHANGELOGs for each package
  • Regular publishing to npm registry under @joystream/sdk-*, where * is the SDK package name (see: Library structure)
  • Extensive, versioned, interactive documentation
  • Striving to limit the number and size of dependencies (excluding devDependencies) to absolute minimum
  • Code examples included in the repository

Tooling

  • Package manager: yarn (2+)
  • Versioning&publishing: Changesets / Lerna
  • Code formatting & standards: ESLint & Prettier
  • Tests: Jest
  • CI/CD: Github workflows
  • Documentation: TypeDoc, Docosaurus, LLMs
  • Logging: debug by default, but allows passing custom logger instance (e.g. winston)

Repository structure:

/docs
/packages
	/core
		/query
		/keys
		...
	/storage
	/content
/examples
...

Documentation

Features

Good examples:

Library structure (packages)

Disclaimer: The package and module descriptions below are not a final or exhaustive list of features. They are a potential way of categorizing SDK packages, along with illustrative examples of commonly repeated code patterns in existing Joystream tools (CLI, Atlas, Pioneer, Orion, YouTube Sync, Network Tests, etc.), plus a few of my own ideas.

Out of scope:

  • (?) @joystream/sdk-react
  • @joystream/sdk-governance
  • @joystream/sdk-forum
  • ...

@joystream/sdk-core

This package would include a set of core features and utilities that can be used in a wide variety of contexts, beyond just a single Joystream runtime module (like Content or Elections). It would be the main building block of other SDK packages targeting more specific use cases.

@joystream/sdk-core/query

This module would consist of utilities related to interacting with GraphQL servers provided by tools such as Hydra or Subsquid. The three main data sources supported will be:

Features:

  • Includes the current GraphQL schema of GraphQL servers listed above
  • Supports pagination and chunking of GraphQL queries based on predefined limits
  • Provides a simple, type-safe interface for executing GraphQL queries (without the need to run graphql-codegen, etc.)
    • This should be possible with tools such as @genql/cli
  • Works with ApolloClient (for compatibility with React-based apps)

Existing code references:

Show references

@joystream/sdk-core/keys

This module would be responsible for managing keys, signatures, and integrations with external signers.

Features:

  • Simplifies integration with external signers and wallets:
    • WalletConnect: Enables integration with SubWallet, Tangem and other mobile/hardware wallets that support the WalletConnect protocol.
    • Talisman connect: Enables integration with multiple wallets and browser extensions like Talisman, Nova Wallet, SubWallet, Polkadot.js etc.
  • Provides a wrapper over @polkadot/keyring with Joystream network defaults
  • Injecting dev accounts (//Alice, //Bob and possibly Joystream-specific ones like //testing//worker//Storage/0 etc.)
  • (?) Offline transaction signing utilities

Similar / related projects:

Related issues

Existing code references:

Show references

@joystream/sdk-core/chain

This module would provide a set of utilities related to retrieving, decoding and processing data from the Joystream blockchain (chain state, blocks, events, extrinsics etc.).
It would rely on ApiPromise.derive, ApiPromise.query and ApiPromise.rpc from @polkadot/api to communicate with Joystream nodes through a Websocket RPC API.

Features

  • Calculating average blocktime / retrieving blocks in given time range
  • Conversion utilities:
    • between blockHeight and date/time
    • between number of blocks and duration (in various units of time)
  • Blockchain state querying utilities (e.g. processing map entries)
  • Retrieving and decoding events and extrinsics from a given block
  • (?) Migration of @joystream/types: Includes current Joystream runtime metadata and helpers for constructing, encoding and decoding Joystream runtime types

Existing code references

Show references

@joystream/sdk-core/tx

This module would provide a set of utilities for preparing, sending and tracking transactions.

Features

  • Extrinsic status tracking & errors:
    • Allows tracking transaction/extrinsic status through all of the stages:
      • Client-side validation (may include validation based on other features provided by the SDK, e.g. balance / permissions validation)
      • Signer events (e.g. signing cancelled)
      • RPC node validation (see: Transaction validity)
      • Runtime dispatch errors and events
      • Query node / Orion processing status (metaprotocol message parsing errors etc.)
  • Transaction batching and parallel execution utilities
    • Extracting status of individual calls in a utility.batch / utility.batchAll / utility.forceBatch transaction,
    • Nonce caching
    • (?) Auto-batching of transactions
  • Calculating transaction costs (fees, bloat bonds, storage fee etc.)

Existing code references

Show references

@joystream/sdk-core/assets

This module would provide a set of utilities for managing balances, vesting, locks and stakes.

Features

  • Simplifies checking balances available for different purposes (reference)
  • HAPI <=> JOY <=> USD conversion (possibly with reference to a specific point in time)
  • Balance formatting utilities
  • Managing stakes & locks:
    • checking if locks are conflicting
    • retrieving all stakes by key(s) / membership(s)
    • retrieving membership's staking accounts
    • checking status of a stake (unbonding / recoverable etc.)
    • setting up staking accounts (addStakingAccount + confirmStakingAccount)
    • (?) recovering stakes
  • Vesting utilities:
    • constructing vesting scheme based on amount, cliff percentage and period
    • calculating vested amount at specific date

Out of scope:

  • (?) Utilities related to managing other assets owned on Joystream:
    • CRTs
    • NFTs

Existing code references:

Show references
  • Parsing balances:
  • Balance formatting:
    • CLI ( Api.ts): formatBalance config
    • Argus: createApi (formatBalance config)
    • Pioneer (formatters): formatJoyValue, formatTokenValue
  • Setting up staking account(s):
  • HAPI <=> JOY <=> USD conversion:
  • Stakes and locks
    • Pioneer: lockTypes (types of locks / stakes, conflicting stakes etc.), hooks: useGroupLocks, useHasRequiredStake, useStakingAccountsLocks, useStakingAccountStatus (...)
    • CLI (Api.ts): fetchStake, nonRivalrousLocks, isLockRivalrous, allStakingLedgers (...)

@joystream/sdk-core/roles

This module would provide a set of utilities related to roles, permissions, authorization and access control on the Joystream blockchain.

Features

  • Filter keys/memberships by role(s)
  • Get roles by key(s) / membership(s)
  • Get available contexts based on action (example: channel creation / deletion)
  • Get permissions based on role/roles
  • Roles and permissions tree (see Data model)
  • Basic validation if user has sufficient permissions for a given action

Data model

  • A Role can have a parent role (for example: MemberController is a parent role of ChannelCollaborator)
  • Some roles may have an associated entity / set of entities (for example MemberController role is associated with Member entity)
  • Each role can have an associated set of permissions:
    • Some roles, like MemberController, have a static set of permissions
    • Other roles, like ChannelCollaborator or CuratorGroupMember can have fine-grained permissions assigned individually
Expand to show some examples 
```
MemberController {
	memberId
	permissions {
		MemberRemark
		CreateChannel
		CreateProposal
		...
	}
}

MemberRoot {
	memberId
	permissions {
		UpdateControllerAccount
		UpdateRootAccount
	}
}

Worker {
	workerId
	workingGroup
	permissions {
		UpdateRewardAccount
		...
	}
}

CuratorGroupMember {
	parent: Worker
	permissions {
		...
	}
}

ChannelOwner {
	parent: MemberController
	channelId
	permissions {
		AddVideo
		UpdateChannelMetadata
		UpdateChannelAssets
		...
	}
}
```

Existing code references:

Show references
  • Network tests (Api.ts): getLeadRoleKey, getLeaderStakingKey, getMemberSigners, getWorkerRoleAccounts
  • CLI: getRequiredMemberContext, getRequiredWorkerContext, getRequiredLeadContext, getCuratorContext, getForumModeratorContext, isModeratorWithRequiredPermission, getChannelManagementActor, getModerationActionActor, getCategoryManagementActor, getContentActor, getChannelOwner (...)
  • Pioneer: useWorker, useMyMemberships, useRoleAccount, useMyRoleIds, useMyWorkers, useAllMemberRoles (...)

@joystream/sdk-core/metaprotocol

This module may provide Joystream metaprotocol (protobuf) related utilities migrated from @joystream/metadata-protobuf (ie. constructing, validating, encoding and decoding metaprotocol messages)

@joystream/sdk-storage

This package would simplify interactions with Joystream network storage & distribution infrastructure.

Features

  • Retrieving a list of storage / distributor nodes (endpoints, location metadata, ping (optionally), supported buckets, status etc.)
  • Retrieving an url to an assets by id (with the possibility of choosing the most optimal one according to ping / location)
  • Downloading data objects by id:
    • Argus / Colossus endpoints can either be derived or provided explicitly,
    • retry mechanism,
    • allows handling events like individual Argus/Colossus node errors, switching to a different endpoint etc.
    • multiple download targets possible: into file, into stream, into buffer etc.
  • Uploading data objects by id (assumes existence on chain):
    • Argus / Colossus endpoints can either be derived or provided explicitly,
    • retry mechanism,
    • multiple sources supported: from file, from stream, from buffer
  • Calculating multihash of a file
  • Storage bags and buckets utilities:
    • Parsing BagId/BucketId to/from string
    • ...

Existing code references

Show references

Related links

@joystream/sdk-content

This package would simplify common tasks involving Joystream's content module (channels, videos etc.).

Features

  • Simplifies creation / updating of channels and videos:
    • Provides an interface similar to CLI's (without the need to run CLI programatically):
      • Provides a single abstraction over a complex, multi-step process (extrinsic data & channel/video metadata preparation, sending & processing extrinsic, uploading assets etc.)
      • (?) Allows auto-derivation of some metadata fields (like video length, codecs etc.)
      • Supports batching of channel / video creation operations
      • Allows handling events and errors related to transaction processing, uploading etc.
      • Supports AppAction wrapping
      • Supports issuing video NFTs
  • Retrieval of video / channel assets
  • Video / channel removal (with all associated assets)
  • Channel payout utils:
    • constructing and verifying merkle proofs
    • claiming rewards
  • (?) Channel reward account from channel id (mirroring runtime encoding impl.)
  • Channel payment history (nft revenue, crt sales, tips, channel payments, direct payments etc.)
  • Direct channel payments
  • Known licenses
  • (?) Orion authentication cryptographic utilities

Out of scope:

  • (?) YPP API features
  • (?) Content moderation features
  • (?) NFT utils
  • (?) CRT utils

Existing code references:

Show references

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions