Skip to content

Custom video and audio sample resources#257

Draft
Vanilagy wants to merge 6 commits intomainfrom
sample-resources
Draft

Custom video and audio sample resources#257
Vanilagy wants to merge 6 commits intomainfrom
sample-resources

Conversation

@Vanilagy
Copy link
Owner

This PR adds VideoSampleResource and AudioSampleResource, a way to create a video or audio sample that's backed by a custom data resource not covered by the base WebCodecs API. The intention is that you can use this to efficiently connect Mediabunny with other sample sources, for example a sample backed by an FFmpeg AVFrame, without needing to copy any data or convert to Uint8Array first.

Things to figure out:

  • allocationSize and copyTo can get very complex to implement properly. Is the burden pushed fully onto the implementor of the resource, or does Mediabunny help out? I feel both must be possible: For max efficiency, the resource should be able to handle it itself, but if it's lazy, it should be able to communicate that too.
  • What about drawing to a canvas? Right now, it takes the VideoFrame route, which is okay in browser environments. But, how does this extend to server environments where the canvas API needs to be polyfilled?
  • Is the API good to use? Needs to be tested with an actual production-ready example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant