VSeeFace is a free, highly configurable face and hand tracking VRM avatar puppeteering program for virtual youtubers with a focus on robust tracking and high image quality. VSeeFace offers functionality similar to Luppet, 3tene, Wakaru and similar programs. VSeeFace runs on Windows 8 and above (64 bit only). VSeeFace can send, receive and combine tracking data using the VMC protocol, which also allows iPhone perfect sync support through Waidayo like this.
Face tracking, including eye gaze, blink, eyebrow and mouth tracking, is done through a regular webcam. For the optional hand tracking, a Leap Motion device is required. You can see a comparison of the face tracking performance compared to other popular vtuber applications here. In this comparison, VSeeFace is still listed under its former name OpenSeeFaceDemo.
Running four face tracking programs (OpenSeeFaceDemo, Luppet, Wakaru, Hitogata) at once with the same camera input. 😊 pic.twitter.com/ioO2pofpMx
— Emiliana (@emiliana_vt) June 23, 2020
If you have any questions or suggestions, please first check the FAQ. If that doesn’t help, feel free to contact me, @Emiliana_vt!
Please note that Live2D models are not supported. For those, please check out VTube Studio or PrprLive.
To update VSeeFace, just delete the old folder or overwrite it when unpacking the new version.
Old versions can be found in the release archive here. This website, the #vseeface-updates channel on Deat’s discord and the release archive are the only official download locations for VSeeFace.
I post news about new versions and the development process on Twitter with the #VSeeFace
hashtag. Feel free to also use this hashtag for anything VSeeFace related. Starting with 1.13.26, VSeeFace will also check for updates and display a green message in the upper left corner when a new version is available, so please make sure to update if you are still on an older version.
The latest release notes can be found here. Some tutorial videos can be found in this section.
The reason it is currently only released in this way, is to make sure that everybody who tries it out has an easy channel to give me feedback.
VSeeFaceはVTuber向けのフェーストラッキングソフトです。Webカメラで簡単にVRMアバターを動かすことができます。Leap Motionによる手と指のトラッキング機能もあります。ダウンロードはこちら。リリースノートはこちら。まだベータ版です。
@Virtual_Deatさんのディスコードサーバーに入るとルールズチャンネルで👌にクリックでルールを同意して他のチャンネルも表示されます。#vseefaceと日本語チャンネルもあります。
VSeeFaceはカラーキーで録画が出来ないけどOBSのGame CaptureでAllow transparencyをチェックしてVSeeFaceで右下の※ボタンでUIを見えないにすれば綺麗な透明の背景になります。
UIの日本語訳があり、日本語のチュートリアル動画もあります。
ライセンス:営利・非営利問わずご自由にお使いください。
You can use VSeeFace to stream or do pretty much anything you like, including non-commercial and commercial uses. Just don’t modify it (other than the translation json
files) or claim you made it.
VSeeFace is beta software. There may be bugs and new versions may change things around. It is offered without any kind of warrenty, so use it at your own risk. It should generally work fine, but it may be a good idea to keep the previous version around when updating.
ライセンス:営利・非営利問わずご自由にお使いください。
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
VSeeFace is being created by @Emiliana_vt and @Virtual_Deat.
There are a lot of tutorial videos out there. This section lists a few to help you get started, but it is by no means comprehensive. Make sure to look around!
About VRM models:
This section is still a work in progress. For help with common issues, please refer to the troubleshooting section.
The most important information can be found by reading through the help screen as well as the usage notes inside the program.
You can rotate, zoom and move the camera by holding the Alt key and using the different mouse buttons. The exact controls are given on the help screen.
Once you’ve found a camera position you like and would like for it to be the initial camera position, you can set the default camera setting in the General settings
to Custom
. You can now move the camera into the desired position and press Save
next to it, to save a custom camera position. Please note that these custom camera positions to not adapt to avatar size, while the regular default positions do.
VSeeFace does not support chroma keying. Instead, capture it in OBS using a game capture and enable the Allow transparency
option on it. Once you press the tiny ※ button in the lower right corner, the UI will become hidden and the background will turn transparent in OBS. You can hide and show the ※ button using the space key.
You can set up the virtual camera function, load a background image and do a Discord (or similar) call using the virtual VSeeFace camera.
You can hide and show the ※ button using the space key.
Those bars are there to let you know that you are close to the edge of your webcam’s field of view and should stop moving that way, so you don’t lose tracking due to being out of sight. If you have set the UI to be hidden using the ※ button in the lower right corner, blue bars will still appear, but they will be invisible in OBS as long as you are using a Game Capture
with Allow transparency
enabled.
Yes, unless you are using the Toaster
quality level or have enabled Synthetic gaze
which makes the eyes follow the head movement, similar to what Luppet does. You can try increasing the gaze strength and sensitivity to make it more visible.
If humanoid eye bones are assigned in Unity, VSeeFace will directly use these for gaze tracking. The gaze strength determines how far the eyes will move. To use the VRM blendshape presets for gaze tracking, make sure that no eye bones are assigned. The synthetic gaze, which moves the eyes either according to head movement or so that they look at the camera, uses the VRMLookAtBoneApplyer
or the VRMLookAtBlendShapeApplyer
, depending on what exists on the model.
Make sure the gaze offset sliders are centered. They can be used to correct the gaze for avatars that don’t have centered irises, but they can also make things look quite wrong when set up incorrectly.
Make sure your eyebrow offset slider is centered. It can be used to overall shift the eyebrow position, but if moved all the way, it leaves little room for them to move.
First, hold the alt key and right click to zoom out until you can see the Leap Motion model in the scene. Then use the sliders to adjust the model’s position to match its location relative to yourself in the real world. You can refer to this video to see how the sliders work.
Changing the position also changes the height of the Leap Motion in VSeeFace, so just pull the Leap Motion position’s height slider way down. Zooming out may also help.
To fix this error, please install the V4 (Orion) SDK. It says it’s used for VR, but it is also used by desktop applications.
All configurable hotkeys also work while it is in the background or minimized, so the expression hotkeys, the audio lipsync toggle hotkey and the configurable position reset hotkey all work from any other program as well. On some systems it might be necessary to run VSeeFace as admin to get this to work properly for some reason.
In at least one case, the following setting has apparently fixed this: Windows => Graphics Settings => Change default graphics settings => Disable “Hardware-accelerated GPU scheduling”
Try switching the camera settings from Camera defaults
to something else. The camera might be using an unsupported video format by default.
Many people make their own using VRoid Studio or commission someone. Vita is one of the included sample characters. You can also find VRM models on VRoid Hub and Niconi Solid, just make sure to follow the terms of use.
Follow the official guide. The important thing to note is that it is a two step process. First, you export a base VRM file, which you then import back into Unity to configure things like blend shape clips. After that, you export the final VRM. If you look around, there are probably other resources out there too.
Yes, you can do so using UniVRM and Unity. You can find a tutorial here. Once the additional VRM blend shape clips are added to the model, you can assign a hotkey in the Expression settings
to trigger it. The expression detection functionality is limited to the predefined expressions, but you can also modify those in Unity and, for example, use the Joy
expression slot for something else.
This is most likely caused by not properly normalizing the model during the first VRM conversion. To properly normalize the avatar during the first VRM export, make sure that Pose Freeze
and Force T Pose
is ticked on the ExportSettings
tab of the VRM export dialog. I also recommend making sure that no jaw bone is set in Unity’s humanoid avatar configuration before the first export, since often a hair bone gets assigned by Unity as a jaw bone by mistake. If a jaw bone is set in the head section, click on it and unset it using the backspace key on your keyboard. If your model does have a jaw bone that you want to use, make sure it is correctly assigned instead.
Note that re-exporting a VRM will not work to for properly normalizing the model. Instead the original model (usually FBX) has to be exported with the correct options set.
Make sure to set “Blendshape Normals” to “None” on the FBX when you import it into Unity and before you export your VRM. That should prevent this issue.
You can add two custom VRM blend shape clips called “Brows up” and “Brows down” and they will be used for the eyebrow tracking. You can also add them on VRoid and Cecil Henshin models to customize how the eyebrow tracking looks. Also refer to the special blendshapes section.
I would recommend running VSeeFace on the PC that does the capturing, so it can be captured with proper transparency. The actual face tracking could be offloaded using the network tracking functionality to reduce CPU usage. If this is really not an option, please refer to the release notes of v1.13.34o. The settings.ini
can be found as described here.
The screenshots are saved to a folder called VSeeFace
inside your Pictures
folder. You can make a screenshot by pressing S
or a delayed screenshot by pressing shift+S
.
VRM conversion is a two step process. After the first export, you have to put the VRM file back into your Unity project to actually set up the VRM blend shape clips and other things. You can follow the guide on the VRM website, which is very detailed with many screenshots.
Because I don’t want to pay a high yearly fee for a code signing certificate.
N versions of Windows are missing some multimedia features. First make sure your Windows is updated and then install the media feature pack.
Right click it, select Extract All...
and press next. You should have a new folder called VSeeFace. Inside there should be a file called VSeeFace
with a blue icon, like the logo on this site. Double click on that to run VSeeFace. There’s a video here.
If Windows 10 won’t run the file and complains that the file may be a threat because it is not signed, you can try the following: Right click it -> Properties -> Unblock -> Apply or select exe file -> Select More Info -> Run Anyways
No.
No. Although, if you are very experienced with Linux and wine as well, you can try following these instructions for running it on Linux.
It’s reportedly possible to run it using wine.
No, VSeeFace only supports 3D models in VRM format. To integrate Live2D, an “Expandable Application” license would be required, for which there is no free tier. As VSeeFace is a free program, integrating an SDK that requires the payment of licensing fees is not an option.
No. It uses paid assets from the Unity asset store that cannot be freely redistributed. However, the actual face tracking and avatar animation code is open source. You can find it here and here.
If you have any issues, questions or feedback, please come to the #vseeface
channel of @Virtual_Deat’s discord server.
The virtual camera can be used to use VSeeFace for teleconferences, Discord calls and similar. It can also be used in situations where using a game capture is not possible or very slow, due to specific laptop hardware setups.
To use the virtual camera, you have to enable it in the General settings
. For performance reasons, it is disabled again after closing the program. Starting with version 1.13.27, the virtual camera will always provide a clean (no UI) image, even while the UI of VSeeFace is not hidden using the small ※ button in the lower right corner.
When using it for the first time, you first have to install the camera driver by clicking the installation button in the virtual camera section of the General settings
. This should open an UAC prompt asking for permission to make changes to your computer, which is required to set up the virtual camera. If no such prompt appears and the installation fails, starting VSeeFace with administrator permissions may fix this, but it is not generally recommended. After a successful installation, the button will change to an uninstall button that allows you to remove the virtual camera from your system.
After installation, it should appear as a regular webcam. The virtual camera only supports the resolution 1280x720. Changing the window size will most likely lead to undesirable results, so it is recommended that the Allow window resizing
option be disabled while using the virtual camera.
The virtual camera supports loading background images, which can be useful for vtuber collabs over discord calls, by setting a unicolored background.
Should you encounter strange issues with with the virtual camera and have previously used it with a version of VSeeFace earlier than 1.13.22, please try uninstalling it using the UninstallAll.bat
, which can be found in VSeeFace_Data\StreamingAssets\UnityCapture
. If the camera outputs a strange green/yellow pattern, please do this as well.
If supported by the capture program, the virtual camera can be used to output video with alpha transparency. To make use of this, a fully transparent PNG needs to be loaded as the background image. Starting with version 1.13.25, such an image can be found in VSeeFace_Data\StreamingAssets
. Partially transparent backgrounds are supported as well. Please note that using (partially) transparent background images with a capture program that do not support RGBA webcams can lead to color errors. OBS and Streamlabs OBS support ARGB video camera capture, but require some additional setup. Apparently, the Twitch video capturing app supports it by default.
To setup OBS or Streamlabs OBS to capture video from the virtual camera with transparency, please follow these settings. The important settings are:
As the virtual camera keeps running even while the UI is shown, using it instead of a game capture can be useful if you often make changes to settings during a stream.
It is possible to perform the face tracking on a separate PC. This can, for example, help reduce CPU load. This process is a bit advanced and requires some general knowledge about the use of commandline programs and batch files. To do this, copy either the whole VSeeFace folder or the VSeeFace_Data\StreamingAssets\Binary\
folder to the second PC, which should have the camera attached. Inside this folder is a file called run.bat
. Running this file will open first ask for some information to set up the camera and then run the tracker process that is usually run in the background of VSeeFace. If you entered the correct information, it will show an image of the camera feed with overlaid tracking points, so do not run it while streaming your desktop. This can also be useful to figure out issues with the camera or tracking in general. The tracker can be stopped with the q
, while the image display window is active.
In the following, the PC running VSeeFace will be called PC A, and the PC running the face tracker will be called PC B.
To use it for network tracking, edit the run.bat
file or create a new batch file with the following content:
%ECHO OFF
facetracker -l 1
echo Make sure that nothing is accessing your camera before you proceed.
set /p cameraNum=Select your camera from the list above and enter the corresponding number:
facetracker -a %cameraNum%
set /p dcaps=Select your camera mode or -1 for default settings:
set /p fps=Select the FPS:
set /p ip=Enter the LAN IP of the PC running VSeeFace:
facetracker -c %cameraNum% -F %fps% -D %dcaps% -v 3 -P 1 -i %ip% --discard-after 0 --scan-every 0 --no-3d-adapt 1 --max-feature-updates 900
pause
If you would like to disable the webcam image display, you can change -v 3
to -v 0
.
When starting this modified file, in addition to the camera information, you will also have to enter the local network IP address of the PC A. You can start and stop the tracker process on PC B and VSeeFace on PC A independently. When no tracker process is running, the avatar in VSeeFace will simply not move.
On the VSeeFace side, select [Network tracking]
in the camera dropdown menu of the starting screen. Also, enter this PC’s (PC A) local network IP address in the Listen IP
field. Do not enter the IP address of PC B or it will not work. Press the start button. PC A should now be able to receive tracking data from PC B, while the tracker is running on PC B. You can find PC A’s local network IP address by enabling the VMC protocol receiver in the General settings
and clicking on Show LAN IP
.
If you are sure that the camera number will not change and know a bit about batch files, you can also modify the batch file to remove the interactive input and just hard code the values.
If things don’t work as expected, check the following things:
run.bat
should open a window with black background and grey text. Make sure you entered the necessary information and pressed enter.Took 20ms
at the beginning should appear. While a face is in the view of the camera, lines with Confidence
should appear too. A second window should show the camera view and red and yellow tracking points overlaid on the face. If this is not the case, something is wrong on this side of the process.VSeeFace has special support for certain custom VRM blend shape clips:
Surprised
is supported by the simple and experimental expression detection features.Brows up
and Brows down
will be used for eyebrow tracking if present on a model.A
, I
, U
, E
and O
blend shapes: SIL
, CH
, DD
, FF
, KK
, NN
, PP
, RR
, SS
, TH
A
, I
, U
, E
and O
are mapped to aa
, ih
, ou
, E
and oh
respectively.A
, I
, U
, E
and O
shapes by itself, so setting up custom VRM blend shape clips would be unnecessary effort. In this case it is better to have only the standard A
, I
, U
, E
and O
VRM blend shape clips on the model.You can set up VSeeFace to recognize your facial expressions and automatically trigger VRM blendshape clips in response. There are two different modes that can be selected in the General settings
.
This mode is easy to use, but it is limited to the Fun
, Angry
and Surprised
expressions. Simply enable it and it should work. There are two sliders at the bottom of the General settings
that can be used to adjust how it works.
To trigger the Fun
expression, smile, moving the corners of your mouth upwards. To trigger the Angry
expression, do not smile and move your eyebrows down. To trigger the Surprised
expression, move your eyebrows up.
This mode supports the Fun
, Angry
, Joy
, Sorrow
and Surprised
VRM expressions. To use it, you first have to teach the program how your face will look for each expression, which can be tricky and take a bit of time. What kind of face you make for each of them is completely up to you, but it’s usually a good idea to enable the tracking point display in the General settings
, so you can see how well the tracking can recognize the face you are making. The following video will explain the process:
When the Calibrate
button is pressed, most of the recorded data is used to train a detection system. The rest of the data will be used to verify the accuracy. This will result in a number between 0 (everything was misdetected) and 1 (everything was detected correctly) and is displayed above the calibration button. A good rule of thumb is to aim for a value between 0.95 and 0.98. While this might be unexpected, a value of 1 or very close to 1 is not actually a good thing and usually indicates that you need to record more data. A value significantly below 0.95 indicates that, most likely, some mixup occurred during recording (e.g. your sorrow expression was recorded for your surprised expression). If this happens, either reload your last saved calibration or restart from the beginning.
It is also possible to set up only a few of the possible expressions. This usually improves detection accuracy. However, make sure to always set up the Neutral
expression. This expression should contain any kind of expression that should not as one of the other expressions. To remove an already set up expression, press the corresponding Clear
button and then Calibrate
.
Having an expression detection setup loaded can increase the startup time of VSeeFace even if expression detection is disabled or set to simple mode. To avoid this, press the Clear calibration
button, which will clear out all calibration data and preventing it from being loaded at startup. You can always load your detection setup again using the Load calibration
button.
VSeeFace both supports sending and receiving motion data (humanoid bone rotations, root offset, blendshape values) using the VMC protocol introduced by Virtual Motion Capture. If both sending and receiving are enabled, sending will be done after received data has been applied. In this case, make sure that VSeeFace is not sending data to itself, i.e. the ports for sending and receiving are different, otherwise very strange things may happen.
When receiving motion data, VSeeFace can additionally perform its own tracking and apply it. Track face features
will apply blendshapes, eye bone and jaw bone rotations according to VSeeFace’s tracking. If only Track fingers
and Track hands to shoulders
are enabled, the Leap Motion tracking will be applied, but camera tracking will remain disabled. If any of the other options are enabled, camera based tracking will be enabled and the selected parts of it will be applied to the avatar.
Please note that received blendshape data will not be used for expression detection and that, if received blendshapes are applied to a model, triggering expressions via hotkeys will not work.
You can find a list of applications with support for the VMC protocol here.
Using the prepared Unity project and scene, pose data will be sent over VMC protocol while the scene is being played. If an animator is added to the model in the scene, the animation will be transmitted, otherwise it can be posed manually as well. For best results, it is recommended to use the same models in both VSeeFace and the Unity scene.
Certain iPhone apps like Waidayo can send perfect sync blendshape information over the VMC protocol, which VSeeFace can receive, allowing you to use iPhone based face tracking. This requires an especially prepared avatar containing the necessary blendshapes. A list of these blendshapes can be found here. You can find an example avatar containing the necessary blendshapes here. An easy, but not free, way to apply these blendshapes to VRoid avatars is to use HANA Tool.
To combine iPhone tracking with Leap Motion tracking, enable the Track fingers
and Track hands to shoulders
options in VMC reception settings in VSeeFace. Enabling all over options except Track face features
as well, will apply the usual head tracking and body movements, which may allow more freedom of movement than just the iPhone tracking on its own.
Send Motion IP Address
to your PC’s LAN IP address. You can find it by clicking on Show LAN IP
at the beginning of the VMC protocol receiver settings in VSeeFace.If VSeeFace’s tracking should be disabled to reduce CPU usage, only enable “Track fingers” and “Track hands to shoulders” on the VMC protocol receiver. This should lead to VSeeFace’s tracking being disabled while leaving the Leap Motion operable. If the tracking remains on, this may be caused by expression detection being enabled. In this case, additionally set the expression detection setting to none.
A full Japanese guide can be found here. The following gives a short English language summary. To use HANA Tool to add perfect sync blendshapes to a VRoid model, you need to install Unity, create a new project and add the UniVRM package and then the VRM version of the HANA Tool package to your project. You can do this by dragging in the .unitypackage
files into the file section of the Unity project. Next, make sure that your VRoid VRM is exported from VRoid v0.11.3 without optimizing or decimating the mesh. Create a folder for your model in the Assets
folder of your Unity project and copy in the VRM file. It should now get imported.
Face
. (Screenshot)HANA_Tool
menu at the top, select Reader
. A new window should appear. Drag the Face
object into the SkinnedMeshRenderer
slot at the top of the new window. Select the VRoid version and type of your model. Make sure to select Add
at the bottom, then click Read BlendShapes
. (Screenshot)Reader
window.HANA_Tool
menu at the top, select AddBlendShapeClip
. A new window should appear. Drag the model from the hierarchy into the VRMBlendShapeProxy
slot at the top of the new window. Again, drag the Face
object into the SkinnedMeshRenderer
slot underneath. Select your model type, not Extra
and press the button at the bottom. (Screenshot)Scene
tab and select your model in the hierarchy. Scroll down in the inspector until you see a list of blend shapes. You should be able to move the sliders and see the face of your model change. Below the regular VRM and VRoid blendshapes, there should now be a bit more than 50 additional blendshapes for perfect sync use, such as one to puff your cheeks. (Screenshot)VRM
menu, select UniVRM
, then Export humanoid
. All the necessary details should already be filled in, so you can press export to save your new VRM file. (Screenshot)It is possible to stream Perception Neuron motion capture data into VSeeFace by using the VMC protocol. To do so, load this project into Unity 2019.4.16f1 and load the included scene in the Scenes
folder. Create a new folder for your VRM avatar inside the Avatars
folder and put in the VRM file. Unity should import it automatically. You can then delete the included Vita model from the the scene and add your own avatar by dragging it into the Hierarchy
section on the left.
You can now start the Neuron software and set it up for transmitting BVH data on port 7001. Once this is done, press play in Unity to play the scene. If no red text appears, the avatar should have been set up correctly and should be receiving tracking data from the Neuron software, while also sending the tracking data over VMC protocol.
Next, you can start VSeeFace and set up the VMC receiver according to the port listed in the message displayed in the game view of the running Unity scene. Once enabled, it should start applying the motion tracking data from the Neuron to the avatar in VSeeFace.
The provided project includes NeuronAnimator by Keijiro Takahashi and uses it to receive the tracking data from the Perception Neuron software and apply it to the avatar.
ThreeDPoseTracker allows webcam based full body tracking. While the ThreeDPoseTracker application can be used freely for non-commercial and commercial uses, the source code is for non-commercial use only.
It allows transmitting its pose data using the VMC protocol, so by enabling VMC receiving in VSeeFace, you can use its webcam based fully body tracking to animate your avatar. From what I saw, it is set up in such a way that the avatar will face away from the camera in VSeeFace, so you will most likely have to turn the lights and camera around. By enabling the Track face features
option, you can apply VSeeFace’s face tracking to the avatar.
If you are working on an avatar, it can be useful to get an accurate idea of how it will look in VSeeFace before exporting the VRM. You can load this example project into Unity 2019.4.16f1 and load the included preview scene to preview your model with VSeeFace like lighting settings. This project also allows posing an avatar and sending the pose to VSeeFace using the VMC protocol starting with VSeeFace v1.13.34b.
After loading the project in Unity, load the provided scene inside the Scenes folder. If you press play, it should show some instructions on how to use it.
If you prefer settings things up yourself, the following settings in Unity should allow you to get an accurate idea of how the avatar will look with default settings in VSeeFace:
Edit -> Project Settings... -> Player -> Other Settings -> Color Space: Linear
Edit -> Project Settings... -> Quality
, select Ultra
and set the anti-aliasing to 8xIf you enabled shadows in the VSeeFace light settings, set the shadow type on the directional light to soft.
To see the model with better light and shadow quality, use the Game
view. You can align the camera with the current scene view by pressing Ctrl+Shift+F
or using Game Object -> Align with view
from the menu.
It is possible to translate VSeeFace into different languages and I am happy to add contributed translations! To add a new language, first make a new entry in VSeeFace_Data\StreamingAssets\Strings\Languages.json
with a new language code and the name of the language in that language. The language code should usually be given in two lowercase letters, but can be longer in special cases. For a partial reference of language codes, you can refer to this list. Afterwards, make a copy of VSeeFace_Data\StreamingAssets\Strings\en.json
and rename it to match the language code of the new language. Now you can edit this new file and translate the "text"
parts of each entry into your language. The "comment"
might help you find where the text is used, so you can more easily understand the context, but it otherwise doesn’t matter.
New languages should automatically appear in the language selection menu in VSeeFace, so you can check how your translation looks inside the program. Note that a JSON syntax error might lead to your whole file not loading correctly. In this case, you may be able to find the position of the error, by looking into the Player.log
, which can be found by using the button all the way at the bottom of the general settings.
Generally, your translation has to be enclosed by doublequotes "like this"
. If double quotes occur in your text, put a \ in front, for example "like \"this\""
. Line breaks can be written as \n
.
Translations are coordinated on GitHub in the VSeeFaceTranslations repository, but you can also send me contributions over Twitter or Discord DM.
Some people have gotten VSeeFace to run on Linux through wine and it might be possible on Mac as well, but nobody tried, to my knowledge. However, reading webcams is not possible through wine. As a workaround, you can set the camera in VSeeFace to [Network tracking]
and run the facetracker.py
script from OpenSeeFace manually. To do this, you will need a Python 3.7 or newer installation. To set up everything for the facetracker.py
, you can try something like this on Debian based distributions:
sudo apt-get install python3 python3-pip python3-virtualenv git
git clone https://github.com/emilianavt/OpenSeeFace
cd OpenSeeFace
virtualenv -p python3 env
source env/bin/activate
pip3 install onnxruntime opencv-python pillow numpy
To run the tracker, first enter the OpenSeeFace
directory and activate the virtual environment for the current session:
source env/bin/activate
Then you can run the tracker:
python facetracker.py -c 0 -W 1280 -H 720 --discard-after 0 --scan-every 0 --no-3d-adapt 1 --max-feature-updates 900
Running this command, will send the tracking data to a UDP port on localhost, on which VSeeFace will listen to receive the tracking data. The -c
argument specifies which camera should be used, with the first being 0
, while -W
and -H
let you specify the resolution. To see the webcam image with tracking points overlaid on your face, you can add the arguments -v 3 -P 1
somewhere.
Notes on running wine: First make sure you have the Arial font installed. You can put Arial.ttf
in your wine prefix’s C:\Windows\Fonts
folder and it should work. Secondly, make sure you have the 64bit version of wine installed. It often comes in a package called wine64
. Also make sure that you are using a 64bit wine prefix. After installing wine64
, you can set one up using WINEARCH=win64 WINEPREFIX=~/.wine64 wine whatever
, then unzip VSeeFace in ~/.wine64/drive_c/VSeeFace
and run it with WINEARCH=win64 WINEPREFIX=~/.wine64 wine VSeeFace.exe
.
Starting with VSeeFace v1.13.33f, while running under wine --background-color '#00FF00'
can be used to set a window background color. To disable wine mode and make things work like on Windows, --disable-wine-mode
can be used.
This section lists common issues and possible solutions for them.
If the VSeeFace window remains black when starting and you have an AMD graphics card, please try disabling Radeon Image Sharpening
either globally or for VSeeFace. It reportedly can cause this type of issue.
If an error appears after pressing the Start
button, please confirm that the VSeeFace folder is correctly unpacked. Previous causes have included:
VSeeFace_Data\StreamingAssets\Binary\facetracker.exe
, which is necessary for the correct operation of VSeeFace. Please confirm that this file exists and, if not, check whether it has been removed by anti virus software.If no window with a graphical user interface appears, please confirm that you have downloaded VSeeFace and not OpenSeeFace, which is just a backend library.
If tracking doesn’t work, you can actually test what the camera sees by running the run.bat
in the VSeeFace_Data\StreamingAssets\Binary
folder. Before running it, make sure that no other program, including VSeeFace, is using the camera. After starting it, you will first see a list of cameras, each with a number in front of it. Enter the number of the camera you would like to check and press enter. Next, it will ask you to select your camera settings as well as a frame rate. You can enter -1 to use the camera defaults and 24 as the frame rate. Press enter after entering each value. After this, a second window should open, showing the image captured by your camera. If your face is visible on the image, you should see red and yellow tracking dots marked on your face. You can use this to make sure your camera is working as expected, your room has enough light, there is no strong light from the background messing up the image and so on. If the tracking points accurately track your face, the tracking should work in VSeeFace as well. To close the window, either press q
in the window showing the camera image or press Ctrl+C in the console window.
If you would like to see the camera image while your avatar is being animated, you can start VSeeFace while run.bat
is running and select [Network tracking]
in the camera option. It should receive the tracking data from the active run.bat
process.
If an error message about the tracker process appears, it may be necessary to restart the program and, on the first screen of the program, enter a different camera resolution and/or frame rate that is known to be supported by the camera. To figure out a good combination, you can try adding your webcam as a video source in OBS and play with the parameters (resolution and frame rate) to find something that works.
Should the tracking still not work, one possible workaround is to capture the actual webcam using OBS and then re-export it as a camera using OBS-VirtualCam.
If tracking randomly stops and you are using Streamlabs OBS, you could see if it works properly with regular OBS. Another issue could be that Windows is putting the webcam’s USB port to sleep. You can disable this behaviour as follow:
Control Panel
Device Manager
and open it+
at Universal Serial Bus Controllers
USB Root Hub
and select Properties
Power Management
tabAllow the computer to turn off this device to save power
and click OKAlternatively or in addition, you can try the following approach:
Control Panel
Power Options
and open themChange plan settings
for your currently selected planChange advcanced power settings
+
in front of USB settings
+
in front of USB selective suspend setting
Disabled
and click OKPlease note that this is not a guaranteed fix by far, but it might help. If you are using a laptop where battery life is important, I recommend only following the second set of steps and setting them up for a power plan that is only active while the laptop is charging.
If, after installing it from the General settings
, the virtual camera is still not listed as a webcam under the name VSeeFaceCamera
in other programs or if it displays an odd green and yellow pattern while VSeeFace is not running, run the UninstallAll.bat
inside the folder VSeeFace_Data\StreamingAssets\UnityCapture
as administrator. Afterwards, run the Install.bat
inside the same folder as administrator. After installing the virtual camera in this way, it may be necessary to restart other programs like Discord before they recognize the virtual camera.
If the virtual camera is listed, but only shows a black picture, make sure that VSeeFace is running and that the virtual camera is enabled in the General settings
. It automatically disables itself when closing VSeeFace to reduce its performance impact, so it has to be manually re-enabled the next time it is used.
First, make sure you have your microphone selected on the starting screen. You can also change it in the General settings
. Also make sure that the Mouth size reduction
slider in the General settings
is not turned up.
If you change your audio output device in Windows, the lipsync function may stop working. If this happens, it should be possible to get it working again by changing the selected microphone in the General settings
or toggling the lipsync option off and on.
Lipsync and mouth animation relies on the model having VRM blendshape clips for the A, I, U, E, O mouth shapes. Jaw bones are not supported and known to cause trouble during VRM export, so it is recommended to unassign them from Unity’s humanoid avatar configuration if present.
If a stereo audio device is used for recording, please make sure that the voice data is on the left channel. If the voice is only on the right channel, it will not be detected. In this case, software like Equalizer APO or Voicemeeter can be used to respectively either copy the right channel to the left channel or provide a mono device that can be used as a mic in VSeeFace. In my experience Equalizer APO can work with less delay and is more stable, but harder to set up.
If no microphones are displayed in the list, please check the Player.log
in the log folder. Look for FMOD
errors. They might list some information on how to fix the issue. This thread on the Unity forums might contain helpful information.
In one case, having a microphone with a 192kHz sample rate installed on the system could make lip sync fail, even when using a different microphone. In this case setting it to 48kHz allowed lip sync to work.
This is usually caused by laptops where OBS runs on the integrated graphics chip, while VSeeFace runs on a separate discrete one. Enabling the SLI/Crossfire Capture Mode
option may enable it to work, but is usually slow. Further information can be found here.
In one case, Streamlabs OBS could only capture VSeeFace when both Streamlabs OBS and VSeeFace where running with admin privileges, which is very odd and should not usually happen, but if you can’t get the game capture to work, you could give it a try.
Another workaround is to use the virtual camera with a fully transparent background image and an ARGB video capture source, as described above.
The VSeeFace settings are not stored within the VSeeFace folder, so you can easily delete it or overwrite it when a new version comes around. If you wish to access the settings file or any of the log files produced by VSeeFace, starting with version 1.13.32g, you can click the Show log and settings folder
button at the bottom of the General settings
. Otherwise, you can find them as follows:
%APPDATA%\..\LocalLow\Emiliana_vt\VSeeFace
The settings file is called settings.ini
. If you performed a factory reset, the settings before the last factory reset can be found in a file called settings.factoryreset
. There are also some other files in this directory:
avatarList.ini
: Starting with VSeeFace 1.13.25, this file contains the list of VRM files listed in the avatar switcher.error.txt
: This contains error output from the facetracker.exe
background process.output.txt
: This contains additional output from the facetracker.exe
background process.Player.log
: This contains the Unity player log of VSeeFace.Player-prev.log
: This contains the Unity player log of VSeeFace from the previous run.This section contains some suggestions on how you can improve the performance of VSeeFace.
If VSeeFace becomes laggy while the window is in the background, you can try enabling the increased priority option from the General settings
, but this can impact the responsiveness of other programs running at the same time.
CPU usage is mainly caused by the separate face tracking process facetracker.exe
that runs alongside VSeeFace.
The first thing to try for performance tuning should be the Recommend Settings
button on the starting screen, which will run a system benchmark to adjust tracking quality and webcam frame rate automatically to a level that balances CPU usage with quality. This usually provides a reasonable starting point that you can adjust further to your needs.
One way to slightly reduce the face tracking process’s CPU usage is to turn on the synthetic gaze option in the General settings
which will cause the tracking process to skip running the gaze tracking model starting with version 1.13.31.
There are two other ways to reduce the amount of CPU used by the tracker. The first and most recommended way is to reduce the webcam frame rate on the starting screen of VSeeFace. Tracking at a frame rate of 15 should still give acceptable results. VSeeFace interpolates between tracking frames, so even low frame rates like 15 or 10 frames per second might look acceptable. The webcam resolution has almost no impact on CPU usage.
The tracking rate is the TR value given in the lower right corner. Please note that the tracking rate may already be lower than the webcam framerate entered on the starting screen. This can be either caused by the webcam slowing down due to insufficient lighting or hardware limitations, or because the CPU cannot keep up with the face tracking. Lowering the webcam frame rate on the starting screen will only lower CPU usage if it is set below the current tracking rate.
The second way is to use a lower quality tracking model. The tracking models can also be selected on the starting screen of VSeeFace. Please note you might not see a change in CPU usage, even if you reduce the tracking quality, if the tracking still runs slower than the webcam’s frame rate. For this reason, it is recommended to first reduce the frame rate until you can observe a reduction in CPU usage. At that point, you can reduce the tracking quality to further reduce CPU usage.
Here is a list of the different models:
High quality
: The default model with the best tracking and highest CPU utilization.Medium quality
: Slightly faster and slightly worse tracking quality.Barely okay quality
: Noticably faster than the first two models, but also noticably worse tracking. The worse tracking mainly results in worse eye blink and eyebrow tracking, as well as highly reduced expression detection performance. I recommend using auto blinking with this and the Low quality
model.Low quality
: Slightly faster and noticably worse tracking quality.Toaster
: This model is specifically intended for old PCs and is much faster than all the others, but it also offers noticably lower tracking quality. Eye blink and gaze tracking as well as expression detection are disabled when using this model.Certain models with a high number of meshes in them can cause significant slowdown. Starting with 1.23.25c, there is an option in the Advanced
section of the General settings
called Disable updates
. By turning on this option, this slowdown can be mostly prevented. However, while this option is enabled, parts of the avatar may disappear when looked at from certain angles. Only enable it when necessary.
In some cases it has been found that enabling this option and disabling it again mostly eliminates the slowdown as well, so give that a try if you encounter this issue. This should prevent any issues with disappearing avatar parts. However, in this case, enabling and disabling the checkbox has to be done each time after loading the model.
GPU usage is mainly dictated by frame rate and anti-aliasing. These options can be found in the General settings
.
If you find GPU usage is too high, first ensure that you do not have anti-aliasing set to Really nice
, because it can cause very heavy CPU load. Next, make sure that all effects in the effect settings are disabled. If it is still too high, make sure to disable the virtual camera and improved anti-aliasing. Finally, you can try reducing the regular anti-aliasing setting or reducing the framerate cap from 60 to something lower like 30 or 24.
Generally, rendering a single character should not be very hard on the GPU, but model optimization may still make a difference. You can use this cube model to test how much of your GPU utilization is related to the model. A model exported straight from VRoid with the hair meshes combined will probably still have a separate material for each strand of hair. Combined with the multiple passes of the MToon shader, this can easily lead to a few hundred draw calls, which are somewhat expensive. Merging materials and atlassing textures in Blender, then converting the model back to VRM in Unity can easily reduce the number of draw calls from a few hundred to around ten.
Some people with Nvidia GPUs who reported strange spikes in GPU load found that the issue went away after setting Prefer max performance
in the Nvidia power management settings and setting Texture Filtering - Quality
to High performance
in the Nvidia settings.
A surprising number of people have asked if it’s possible to support the development of VSeeFace, so I figured I’d add this section.
If you appreciate Deat’s contributions to VSeeFace, his amazing Tracking World or just him being him overall, you can buy him a Ko-fi, tip him through Streamlabs or subscribe to his Twitch channel.
I don’t really accept monetary donations, but getting fanart, you can find a reference here, makes me really, really happy and getting vtuber gift subs on Twitch is nice too, because it both helps the community and I get some cute emotes to use as well.
You really don’t have to at all, but if you really, really insist and happen to have Monero (XMR), you can send something to: 8AWmb7CTB6sMhvW4FVq6zh1yo7LeJdtGmR7tyofkcHYhPstQGaKEDpv1W2u1wokFGr7Q9RtbWXBmJZh7gAy6ouDDVqDev2t