docs: Restructure FP docs (#1762)

This commit is contained in:
Marc Rousavy 2023-09-04 14:45:21 +02:00 committed by GitHub
parent f3fd3f15e3
commit b300209e36
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 81 additions and 7 deletions

View File

@ -0,0 +1,67 @@
---
id: skia-frame-processors
title: Drawing onto Frame Processors
sidebar_label: Drawing onto Frame Processors
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import useBaseUrl from '@docusaurus/useBaseUrl';
<div>
<svg xmlns="http://www.w3.org/2000/svg" width="283" height="535" style={{ float: 'right' }}>
<image href={useBaseUrl("img/frame-processors.gif")} x="18" y="33" width="247" height="469" />
<image href={useBaseUrl("img/frame.png")} width="283" height="535" />
</svg>
</div>
### Drawing onto Frame Processors
It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera.
This allows you to draw onto a Frame using simple JavaScript APIs:
```ts
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// Detect faces using GPU-accelerated ML API
const faces = detectFaces()
// Draw faces using GPU-accelerated Skia drawing API
for (const face of faces) {
const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height)
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(rect, paint)
}
}, [])
```
..or even apply color-filters in realtime:
```ts
const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const paint = Skia.Paint(invertColorsFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// Draw frame using Skia Shader to invert colors
frame.render(paint)
}, [])
```
While this API was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See [this PR for more information](https://github.com/mrousavy/react-native-vision-camera/pull/1740).
In my agency, [Margelo](https://margelo.io), we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature [reach out to us](https://margelo.io#contact) and we'll build a customized/tailored solution for your company! :)
<br />
#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview))

View File

@ -6,7 +6,13 @@ module.exports = {
'guides/lifecycle', 'guides/lifecycle',
'guides/formats', 'guides/formats',
'guides/capturing', 'guides/capturing',
{
type: 'category',
label: 'Realtime Frame Processing',
items: [
'guides/frame-processors', 'guides/frame-processors',
'guides/frame-processor-plugin-list',
'guides/skia-frame-processors',
{ {
type: 'category', type: 'category',
label: 'Creating Frame Processor Plugins', label: 'Creating Frame Processor Plugins',
@ -17,7 +23,8 @@ module.exports = {
'guides/frame-processors-plugins-final', 'guides/frame-processors-plugins-final',
] ]
}, },
'guides/frame-processor-plugin-list', ]
},
'guides/zooming', 'guides/zooming',
'guides/focusing', 'guides/focusing',
'guides/errors', 'guides/errors',