docs: Restructure FP docs (#1762)
This commit is contained in:
parent
f3fd3f15e3
commit
b300209e36
67
docs/docs/guides/SKIA_FRAME_PROCESSORS.mdx
Normal file
67
docs/docs/guides/SKIA_FRAME_PROCESSORS.mdx
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
---
|
||||||
|
id: skia-frame-processors
|
||||||
|
title: Drawing onto Frame Processors
|
||||||
|
sidebar_label: Drawing onto Frame Processors
|
||||||
|
---
|
||||||
|
|
||||||
|
import Tabs from '@theme/Tabs';
|
||||||
|
import TabItem from '@theme/TabItem';
|
||||||
|
import useBaseUrl from '@docusaurus/useBaseUrl';
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="283" height="535" style={{ float: 'right' }}>
|
||||||
|
<image href={useBaseUrl("img/frame-processors.gif")} x="18" y="33" width="247" height="469" />
|
||||||
|
<image href={useBaseUrl("img/frame.png")} width="283" height="535" />
|
||||||
|
</svg>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
### Drawing onto Frame Processors
|
||||||
|
|
||||||
|
It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera.
|
||||||
|
|
||||||
|
This allows you to draw onto a Frame using simple JavaScript APIs:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const frameProcessor = useSkiaFrameProcessor((frame) => {
|
||||||
|
'worklet'
|
||||||
|
|
||||||
|
// Detect faces using GPU-accelerated ML API
|
||||||
|
const faces = detectFaces()
|
||||||
|
|
||||||
|
// Draw faces using GPU-accelerated Skia drawing API
|
||||||
|
for (const face of faces) {
|
||||||
|
const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height)
|
||||||
|
const paint = Skia.Paint()
|
||||||
|
paint.setColor(Skia.Color('red'))
|
||||||
|
frame.drawRect(rect, paint)
|
||||||
|
}
|
||||||
|
}, [])
|
||||||
|
```
|
||||||
|
|
||||||
|
..or even apply color-filters in realtime:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const invertColorsFilter = Skia.RuntimeEffect.Make(`
|
||||||
|
uniform shader image;
|
||||||
|
half4 main(vec2 pos) {
|
||||||
|
vec4 color = image.eval(pos);
|
||||||
|
return vec4((1.0 - color).rgb, 1.0);
|
||||||
|
}
|
||||||
|
`)
|
||||||
|
const paint = Skia.Paint(invertColorsFilter)
|
||||||
|
|
||||||
|
const frameProcessor = useSkiaFrameProcessor((frame) => {
|
||||||
|
'worklet'
|
||||||
|
|
||||||
|
// Draw frame using Skia Shader to invert colors
|
||||||
|
frame.render(paint)
|
||||||
|
}, [])
|
||||||
|
```
|
||||||
|
|
||||||
|
While this API was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See [this PR for more information](https://github.com/mrousavy/react-native-vision-camera/pull/1740).
|
||||||
|
|
||||||
|
In my agency, [Margelo](https://margelo.io), we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature [reach out to us](https://margelo.io#contact) and we'll build a customized/tailored solution for your company! :)
|
||||||
|
|
||||||
|
<br />
|
||||||
|
|
||||||
|
#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview))
|
@ -6,7 +6,13 @@ module.exports = {
|
|||||||
'guides/lifecycle',
|
'guides/lifecycle',
|
||||||
'guides/formats',
|
'guides/formats',
|
||||||
'guides/capturing',
|
'guides/capturing',
|
||||||
|
{
|
||||||
|
type: 'category',
|
||||||
|
label: 'Realtime Frame Processing',
|
||||||
|
items: [
|
||||||
'guides/frame-processors',
|
'guides/frame-processors',
|
||||||
|
'guides/frame-processor-plugin-list',
|
||||||
|
'guides/skia-frame-processors',
|
||||||
{
|
{
|
||||||
type: 'category',
|
type: 'category',
|
||||||
label: 'Creating Frame Processor Plugins',
|
label: 'Creating Frame Processor Plugins',
|
||||||
@ -17,7 +23,8 @@ module.exports = {
|
|||||||
'guides/frame-processors-plugins-final',
|
'guides/frame-processors-plugins-final',
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
'guides/frame-processor-plugin-list',
|
]
|
||||||
|
},
|
||||||
'guides/zooming',
|
'guides/zooming',
|
||||||
'guides/focusing',
|
'guides/focusing',
|
||||||
'guides/errors',
|
'guides/errors',
|
||||||
|
Loading…
Reference in New Issue
Block a user