diff --git a/README.md b/README.md index d6ee2c2..6da69c9 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,18 @@ - + +
README.mdANIMATED.md DEVICES.md FORMATS.mdFRAME_PROCESSORS.mdANIMATED.md ERRORS.md
-
+
+
+ +

Camera

diff --git a/docs/ANIMATED.md b/docs/ANIMATED.md index 8f342ff..a67ffd5 100644 --- a/docs/ANIMATED.md +++ b/docs/ANIMATED.md @@ -1,9 +1,10 @@ - + +
README.mdANIMATED.md DEVICES.md FORMATS.mdFRAME_PROCESSORS.mdANIMATED.md ERRORS.md
diff --git a/docs/DEVICES.md b/docs/DEVICES.md index 5eb0f70..c362586 100644 --- a/docs/DEVICES.md +++ b/docs/DEVICES.md @@ -1,9 +1,10 @@ - + +
README.mdANIMATED.md DEVICES.md FORMATS.mdFRAME_PROCESSORS.mdANIMATED.md ERRORS.md
diff --git a/docs/ERRORS.md b/docs/ERRORS.md index 42321d9..d816e37 100644 --- a/docs/ERRORS.md +++ b/docs/ERRORS.md @@ -1,9 +1,10 @@ - + +
README.mdANIMATED.md DEVICES.md FORMATS.mdFRAME_PROCESSORS.mdANIMATED.md ERRORS.md
diff --git a/docs/FORMATS.md b/docs/FORMATS.md index 056089e..706632b 100644 --- a/docs/FORMATS.md +++ b/docs/FORMATS.md @@ -1,9 +1,10 @@ - + +
README.mdANIMATED.md DEVICES.md FORMATS.mdFRAME_PROCESSORS.mdANIMATED.md ERRORS.md
diff --git a/docs/FRAME_PROCESSORS.md b/docs/FRAME_PROCESSORS.md new file mode 100644 index 0000000..1a835f1 --- /dev/null +++ b/docs/FRAME_PROCESSORS.md @@ -0,0 +1,62 @@ + + + + + + + + + +
README.mdDEVICES.mdFORMATS.mdFRAME_PROCESSORS.mdANIMATED.mdERRORS.md
+ +
+
+ +

Frame Processors

+ +
+ + +
+ +### What are frame processors? + +Frame processors are functions that are written in JavaScript (or TypeScript) which can be used to **process frames the camera "sees"**. + +For example, you might want to create a QR code scanner _without ever writing native code while still achieving almost-native performance_. Since you can write the scanning part yourself, you can implement a custom QR code system like the one Snapchat uses for Snap-codes. + +
+ +
+
+ +Frame processors are by far not limited to QR code detection, other examples include: + +* **AI** for **facial recognition** +* **AI** for **object detection** +* Using **Tensorflow**, **MLKit Vision** or other libraries (if they provide React Native JSI bindings in the form of "react-native-vision-camera plugins") +* Creating **realtime video-chats** since you can directly send the camera frames over the network +* Creating **snapchat-like filters**, e.g. draw a dog-mask filter over the user's face (WIP) +* Creating **color filters** with depth-detection +* Using **custom C++ processors** exposed to JS for maximum performance + +### Technical + +Frame processors are JS functions that will be **workletized** using [react-native-reanimated](https://github.com/software-mansion/react-native-reanimated). They are created on a **separate thread** using a separate Hermes/JSC Runtime and are **invoked synchronously** without ever going over the bridge. + +### Example + +```tsx +function App() { + const frameProcessor = useFrameProcessor((frame) => { + const qrCodes = scanQrCodes(frame) + console.log(qrCodes) + }, []) + + return ( + + ) +} +``` + +> FRAME PROCESSORS ARE STILL WIP diff --git a/img/snap-code.png b/img/snap-code.png new file mode 100644 index 0000000..f6387e3 Binary files /dev/null and b/img/snap-code.png differ