diff --git a/docs/docs/guides/SKIA_FRAME_PROCESSORS.mdx b/docs/docs/guides/SKIA_FRAME_PROCESSORS.mdx new file mode 100644 index 0000000..8a70dbd --- /dev/null +++ b/docs/docs/guides/SKIA_FRAME_PROCESSORS.mdx @@ -0,0 +1,67 @@ +--- +id: skia-frame-processors +title: Drawing onto Frame Processors +sidebar_label: Drawing onto Frame Processors +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import useBaseUrl from '@docusaurus/useBaseUrl'; + +
+ + + + +
+ +### Drawing onto Frame Processors + +It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera. + +This allows you to draw onto a Frame using simple JavaScript APIs: + +```ts +const frameProcessor = useSkiaFrameProcessor((frame) => { + 'worklet' + + // Detect faces using GPU-accelerated ML API + const faces = detectFaces() + + // Draw faces using GPU-accelerated Skia drawing API + for (const face of faces) { + const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height) + const paint = Skia.Paint() + paint.setColor(Skia.Color('red')) + frame.drawRect(rect, paint) + } +}, []) +``` + +..or even apply color-filters in realtime: + +```ts +const invertColorsFilter = Skia.RuntimeEffect.Make(` + uniform shader image; + half4 main(vec2 pos) { + vec4 color = image.eval(pos); + return vec4((1.0 - color).rgb, 1.0); + } +`) +const paint = Skia.Paint(invertColorsFilter) + +const frameProcessor = useSkiaFrameProcessor((frame) => { + 'worklet' + + // Draw frame using Skia Shader to invert colors + frame.render(paint) +}, []) +``` + +While this API was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See [this PR for more information](https://github.com/mrousavy/react-native-vision-camera/pull/1740). + +In my agency, [Margelo](https://margelo.io), we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature [reach out to us](https://margelo.io#contact) and we'll build a customized/tailored solution for your company! :) + +
+ +#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview)) diff --git a/docs/sidebars.js b/docs/sidebars.js index 90632d1..a5d169c 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -6,18 +6,25 @@ module.exports = { 'guides/lifecycle', 'guides/formats', 'guides/capturing', - 'guides/frame-processors', { type: 'category', - label: 'Creating Frame Processor Plugins', + label: 'Realtime Frame Processing', items: [ - 'guides/frame-processors-plugins-overview', - 'guides/frame-processors-plugins-ios', - 'guides/frame-processors-plugins-android', - 'guides/frame-processors-plugins-final', + 'guides/frame-processors', + 'guides/frame-processor-plugin-list', + 'guides/skia-frame-processors', + { + type: 'category', + label: 'Creating Frame Processor Plugins', + items: [ + 'guides/frame-processors-plugins-overview', + 'guides/frame-processors-plugins-ios', + 'guides/frame-processors-plugins-android', + 'guides/frame-processors-plugins-final', + ] + }, ] }, - 'guides/frame-processor-plugin-list', 'guides/zooming', 'guides/focusing', 'guides/errors',