The Character Made in Blender Was Broken in Three.js — A Retrospective on 3D Character Implementation

TL;DR

Understanding engine limitations: A perfect structure in Blender is not always the correct answer in Three.js.
Result-oriented rigging: Rigging is not just about "bringing in" but about the process of "converting to result data" for the engine.
Design centered: All criteria should focus on "how it operates efficiently at web runtime."

The portfolios of the two projects mentioned in this article can be viewed at Funda, Chaen.

Starting the Character Creation

During the FUNDA project, there was a need for a medium that visually reflects the core values of 'fun' and 'gamification' of the service. Not just decorations to adorn the screen, but characters were planned as UX components that emotionally connect with users and motivate learning.

Character Selection Process

Characters were selected based on three criteria.

1. Cuteness — The first impression that enhances accessibility to the service
2. Smart Image — Consistency with the identity of a learning service
3. Harmony with Brand Color — Visual consistency

Initially, we considered a panda, which sounds similar to 'Funda.' While pandas appealed for their cuteness, they seemed more fitting for a 'leisure' category than the 'intelligent' 'learning' category, so we held off.

Next, we thought about an otter, but considering an appearance closer to human form would be better for pondering animation, we kept it as a candidate.

Finally, the fox matched well with the smart image, developers, and learning, but the orange color clashed with the brand's colors, leading us to finally choose the arctic fox, which also belongs to the fox family.

Sketch/2D Works

Before starting full-scale 3D work, simple sketches and 2D implementations were done to decide the character's impression and dynamics. Although the 2D results did not seem to have major indicative issues for the service operation itself, feedback mentioned a slight lack of quality.

There was a lot of deliberation between the working time (Resource) and the quality of the outcomes, but it was concluded that the character was the key point of this project. If it's a crucial element that determines the project's success, investing more time to ensure solid quality was deemed right, thus beginning the challenge towards 3D characters.

Blender Work

Instead of indiscriminately increasing faces, I used the strategies of Weighted Normal and High Poly Normal Map Baking.

• Strategic Retopology: The total polygon count was kept below 10,000, with additional faces manually added only at sharp edges and silhouettes.
• Weighted Normal: Edge creases were smoothly corrected by adjusting face unit weights.
• High Poly Normal Map: High-poly model details were baked into the low-poly mesh data. Although the silhouette is low-poly, we tried to implement smooth highlights using the high-poly model.

2. Structural Simplification of the Eyeballs

The previous approach of applying a glass layer over the eyeballs was too heavy. This time, the eyeball mesh was unified, and details were baked into the texture. Refraction calculations were replaced by directly drawing highlights and depth in Photoshop. This choice aimed to reduce rendering load while maintaining visual satisfaction.

3. Texture Atlas and ORM Channel Packing

Modeling & Texturing

This was carried out based on the experience of implementing human characters twice. The deform was modified several times to capture the details. Most of the texturing was done using noise, but parts requiring detail were sourced from Poly Haven.

First Head	Second Deform	Third Deform

Particularly, the eyes received the most attention. To capture the mystical nature of the Arctic fox, a combination of purple and white was used, and the Transmission and Glass BSDF were utilized to create a glass-like refraction effect. However, this node structure later became a significant technical challenge in the Three.js environment. (to be discussed later)

Rigging & Shape Key

Rigging was done using the animal head + human body structure based on Rigify. The Damped Track was applied to add swaying to the scarf and tail. Facial expressions were controlled by assigning Shape Keys to the face mesh, eyelashes, eyebrows, tongue, and teeth.

body rigging	facial expression

In Blender, everything worked perfectly, and it just needed to be exported. The problems arose afterward.

🚨 Everything Broke When Transitioning to Three.js

1. Modifiers Not Applied

When uploaded to the web, it rendered without any modifiers applied. Three.js does not support real-time modifier calculation for each frame. Back in Blender, modifiers needed to be applied, but there was an error that meshes with Shape Keys cannot have Subdivision Modifier applied successfully.

Having recreated the Shape Keys more than five times, starting over was not an option due to time constraints. Eventually, I used an external plugin (SKKeeper) for forced application. Many error messages appeared stating that the Shape Key was missing, but upon transitioning to threejs, there seemed to be no issues on the screen, and the modifiers were applied well. However, since it wasn't the proper method, unease remained.

2. Constraints and Controllers Ignored

The Rigify-based IK and Constraint structures were completely ignored in Three.js. None of the intended movements were realized. It was expected when only the deform bone was exported, but even when exporting other MCH, ORG, DEF bones to recognize and control, they did not move at all. In Blender, real-time bone control through the Controller was possible. In contrast, in Three.js, the controller was entirely ignored.

The Chronicle of Trial and Error

Rigify bones were imported to be directly controlled, but did not behave as intended. Attempted to extract the bones multiple times, which took a long time, so I wrote five scripts to debug them in order.

1. Output Bone List — Verify if bones were extracted correctly
2. Bone Mapping Confirmation — Verify correct connection to arms, legs, and head
3. Dotting on Screen — Visualize if the position moves on the screen
4. Log Coordinate Values — Output to console to see if actual transform values change
5. Final Confirmation — Verify alignment of screen rendering and coordinate values

Result: Bones were all present, mapping was done, and coordinate values moved, but it was not reflected on the screen.

In conclusion, I realized that Rigify's structure is too complex to be directly controlled in Three.js. Only elements not directly connected to Rigify, such as eyeballs, eyebrows, and tongues parented to the head bone, could be moved.

Bonus — Redesigning Eyeball Materials in Three.js

When implementing the 'eyes' that determine a character's liveliness, encountered a technical bottleneck where the sophisticated node design in Blender did not work as intended in a web engine (Three.js).

1. Intended Design in Blender

• Structural Design: A hole was made in the center of the sclera mesh that wraps around to display the iris clearly.
• Texture Detail: It wasn't just an empty hole, but a clear membrane effect like the lens of an actual eyeball was provided by .

2. Limitations in Three.js

However, when exported to GLB and checked in Three.js, the intent in Blender was completely collapsed.

• Error in Alpha Interpretation: The basic material system of the web environment could not interpret the slight alpha channel value differences as 'glass-like physical texture'.
• Result: The iris part was displayed transparently as if completely open without the intended glass membrane, significantly reducing the liveliness.

3. Solution: Runtime Material Override

Determining that image baking alone could not replicate the physical glass texture on the web, we chose to inject MeshPhysicalMaterial directly at runtime.

MeshPhysicalMaterial supports the most physically accurate material expression in Three.js. By setting parameters such as IOR (refractive index), Transmission, and Coat, we were able to replicate results similar to Blender's Glass material at runtime. Although not identical, the intended mysterious eye expression was sufficiently achieved.

before	after

Solution: Completely Restructured

Animation Bake

Removed constraint-based rigging and baked all animations into frame-by-frame keyframe data. In the Dope Sheet and Action Editor, each action was organized and then pushed down to the NLA (Non-Linear Animation) Editor to manage tracks. Since clear clip names are necessary to trigger specific animations in the engine, all action names were refined and baked before extraction. This allowed animation control without confusion, such as actions['Greeting'].play() at the code level.

Role Separation

• Body → Plays baked animations
• Expression → Directly controlled by Shape Key (Morph Target) in Three.js

Interactions became possible after these two were separated.

useFixSkinnedMesh Custom Hook

New issues arose in Three.js after forced Subdivision application.

• The engine misrecognized the character's actual size/range
• Character suddenly disappearing during animations (Frustum Culling malfunction)
• Shape Key values not initializing, causing rendering instability

A custom hook was created to address these.

Why Not Bake All Animations?

As the project progressed, additional features like character eye tracking and click-triggered animations were added. The main concern was 'how much to bake'.

Baking all movements in Blender for implementation makes things much simpler. However, doing so equates to a non-interactive, merely 'play button' GIF. Ideally, various expression combinations would be reused flexibly according to the situation.

Therefore, a hybrid strategy was set as follows:

• Body: Complex calculations and structures were baked as keyframe data in Blender.
• Expression: For free combination and real-time control in different scenarios, Shape Key (Morph Target) control was managed in Three.js.

Additionally, a separate controller component and character experience page were built, allowing users to directly manipulate the controller to experience various animations and expression changes in real-time.

링크 정보를 불러오는 중...

⚠️ Key Logic Considerations During Implementation

Using the hybrid approach, issues arose where different control entities (animation data vs real-time mouse coordinates) conflicted, resolved by applying two exceptions.

1. Deactivate Eye Tracking During Animation Playback (Prevent Control Conflict)

If the gaze follows the mouse while certain animations (e.g., falling, greeting) play, the rotation value of the bones defined by animation data collides with real-time lookAt logic, causing screen jitter or unnatural movements. A flag was designed to temporarily block gaze tracking logic while active animations are executing.

2. Reset Previous Values When Switching Shape Key Groups (Prevent Value Overlap)

Expressions can overlap. If the 'smiling expression' weight is at 1 and 'angry expression' is set to 1, the mesh might distort grotesquely. To prevent this, a logic was implemented to precisely reset the Shape Key influences of the previous group to 0 before injecting new values when switching to a new expression group.

Mobile Optimization

Due to the structure where 3D characters are placed on many pages (battle, login, result screen, etc.), mobile performance was important. To prevent frame drops (FPS reduction) on low-end devices, we approached it from various angles.

Rendering Loop Optimization — requestAnimationFrame

The basic render loop of Three.js was configured based on requestAnimationFrame. This method executes callbacks right before the browser draws the next frame, naturally matching the monitor's refresh rate (60Hz, 120Hz, etc.). When the tab moves to the background, it automatically pauses, reducing unnecessary calculations.

Resolution Limiting renderer.setPixelRatio(Math.min(devicePixelRatio, 2))

The latest mobile devices can reach a pixel density (DPR) of 3 to 4. The higher the DPR, the square of the number of pixels the GPU needs to process increases. We limited the DPR to a maximum of 2 to secure performance without a significant visual difference.

Disabling Anti-aliasing antialias: false

Anti-aliasing reduces jagged edges but imposes a significant load on the GPU. In mobile environments, the high DPR already makes jagged edges less noticeable. Disabling it did not result in a major perceived quality difference, so we turned it off.

NLA Editor Unnecessary Track Cleanup

While working in Blender, temporary animation tracks like [Action Stash] can accumulate in the NLA Editor. If included when exporting to GLB, the file size unnecessarily increases, and Three.js wastes memory parsing all tracks during loading. Therefore, we deleted all unused tracks before exporting.

Texture Baking

Procedural shaders like Blender's Principled BSDF, Noise, and Gradient cannot be calculated in real-time in Three.js. By baking them into image textures before GLB export, Three.js only needs to apply the texture images. This greatly reduces the cost of real-time lighting and shader operations.

Baking targets: Color, Roughness, Normal, Alpha maps

Character Instancing SkeletonUtils.clone

Since the same character is used on multiple pages (battle, result screen, etc.), we reused it in an instancing manner instead of loading it anew each time. Simply using scene.clone() would mess up the bone structure, causing animations to apply redundantly to all instances. To address this, we used SkeletonUtils.clone to implement a structure where geometries are shared but skeletons are independently cloned.

Scene Componentization

Lighting settings and camera controls were not reconfigured for each page but were made into a single scene component for reuse. In the React structure, the same components share memory, reducing the cost of duplicate rendering.

Areas for Improvement

The mesh itself is high-polygon, which limited runtime optimization, and there was plenty of room for additional refinement at the Blender stage. If LOD (Level of Detail) had been applied, it would have automatically reduced polygon counts based on the distance from the camera, potentially boosting mobile performance. It's a feature I'd like to implement in the next project.

Second Character: Designed Differently from the Start

The performance bottlenecks experienced with the first character 'Pundi' taught me significant lessons. In a web environment, 3D models aren't just 'pretty models' but must be 'meticulously calculated assets.' For this human character project, the optimization pipeline was applied from the design stage.

1. Vertex Optimization

In the past, dozens of images were used per part, burdening the engine. This time, similar materials like face + torso, clothing parts, accessories, and interior, were integrated into three 2048px atlases.

In the PBR workflow, Occlusion, Roughness, and Metalness each require separate textures. However, three textures mean three times the GPU memory and network transmission.

Focusing on the fact that all three datasets are grayscale data with values from 0 to 1, they were packed into the R (AO), G (Roughness), B (Metallic) channels within a single image.

Instead of using a separate tool, I used Node.js and the sharp library to write a script for pipeline automation, which turned out to be much more efficient than expected.

Developer Tip: Creating ORM Maps with Node.js

Using sharp, you can batch-process dozens of textures. Especially for cases requiring an alpha channel, like hair, it can be managed by expanding to ORMA.

4. Grayscale Baking

Hair and outerwear, which frequently change color, were baked in black and white, then directly injected with material.color.set() in Three.js runtime to save memory and secure customization flexibility.

Grayscale	Color Mapping

5. Draco Compression

To prevent user drop-off, Draco compression in Blender was used to export and unpacked in Three.js, which reduced the model size by over 80%, significantly shortening network loading times.

In Conclusion

The conclusion from this work is that 3D character creation is not a modeling problem, but a structural design problem tailored to the execution environment.

A structure that works perfectly in Blender may be meaningless in Three.js. Conversely, even if it's slightly inconvenient in Blender, designing it in a way that can be controlled by the engine yields much better results. Thanks to the trial-and-error with the first character, the second character could be created properly from the start. The experience of working hard in the wrong direction ultimately established the correct design criteria.

• #Retrospect
• #Performance
• #Animation
• #Three.js
• #Draco Compression
• #Blender
• #Morph Target