Streamingle_URP/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Streamingle is a Unity-based virtual avatar streaming and control system for VTuber/avatar performance streaming. Built on Unity 6000.2.6f2 with Universal Render Pipeline (URP).

Key capabilities:

• Multi-camera control with Cinemachine presets
• Avatar outfit and item/prop management
• Event/animation triggering system
• Stream Deck integration via WebSocket server
• Real-time streaming output (Spout/NDI)
• Motion capture support (OptiTrack, iFacialMocap, Rokoko)

Build and Development Commands

Unity Editor

• Open the project in Unity 6000.2.6f2 or later
• The project uses URP, ensure Graphics settings point to Universal Render Pipeline asset
• Main scene is typically in Assets/Scenes/

Stream Deck Plugin Deployment

cd Streamdeck
deploy-plugin.bat

This requires administrator privileges and auto-deploys the Stream Deck plugin.

Testing Stream Deck Server

The WebSocket server runs on localhost:64211 by default. Test with any WebSocket client or the Stream Deck plugin.

UniCli (Unity Editor CLI Control)

UniCli lets you drive the Unity Editor from the terminal — useful for compiling, running tests, and inspecting/manipulating scenes without leaving the CLI (works well for AI agents).

Installed components:

• CLI binary: installed via Scoop (scoop install unicli), updatable with scoop update unicli
• Unity package: com.yucchiy.unicli-server (git URL in Packages/manifest.json)

Prerequisite: The Unity Editor must be open with this project — the server only runs while the editor is active. Verify with unicli check (expect Server: running).

Common commands (run from the project root):

unicli check                              # connection / version status
unicli commands                           # list all Unity-side commands (~119)
unicli exec Compile                       # trigger script compilation, return errors
unicli exec Console.GetLog                # read Editor console logs
unicli exec TestRunner.RunEditMode        # run EditMode tests
unicli exec GameObject.Find name=...      # query scene hierarchy
unicli exec Scene.GetActive               # active scene info
unicli exec PlayMode.Enter / PlayMode.Exit

Verifying changes: after editing scripts, prefer unicli exec Compile to confirm compilation succeeds and surface errors, rather than asking the user to check the Editor manually.

Core Architecture

IController Pattern

All major control systems implement the IController interface defined in Assets/Scripts/Streamingle/StreamingleControl/Controllers/IController.cs:

public interface IController
{
    void Set(int index);
    string GetControllerId();
    string GetControllerName();
    object GetControllerData();
    void ExecuteAction(string actionId, object parameters);
}

Key Controllers:

• CameraController: Multi-preset camera management with hotkeys
• ItemController: Toggle visibility/state of scene props/objects
• EventController: Trigger UnityEvents (animations, sounds, effects)
• AvatarOutfitController: Avatar appearance/clothing management
• SystemController: Motion capture, recording, screenshots

StreamDeckServerManager Central Hub

Assets/Scripts/Streamdeck/StreamDeckServerManager.cs orchestrates all controllers via WebSocket communication:

Message Flow:

Stream Deck Client (WebSocket)
    → StreamDeckServerManager (JSON protocol)
        → Dispatches to appropriate IController
            → Updates Unity scene/state
                → Broadcasts state changes back to clients

Thread Safety: Uses main thread action queuing for Unity API calls. When modifying server logic, always enqueue Unity operations via mainThreadActions.

Supported Message Types:

• Camera: switch_camera, get_camera_list, get_camera_state
• Items: set_item, toggle_item, get_item_list, get_item_state
• Events: execute_event, set_event, get_event_list, get_event_state
• Avatars: set_avatar_outfit, get_avatar_outfit_list, get_avatar_outfit_state
• System: start_motion_capture, stop_motion_capture, start_recording, etc.

Data Structures

Controllers export/import state via JSON-serializable data classes:

• *ListData: Full list of available items (cameras, items, events, outfits)
• *StateData: Current active state

Example: CameraListData, CameraStateData, ItemListData, ItemStateData

Camera Control System

Assets/Scripts/Streamingle/StreamingleControl/Controllers/CameraController.cs

Features:

• Cinemachine virtual camera presets with priority switching
• Hotkey recording using UnityRawInput
• Mouse-based camera control:
  • Right-click: Orbit rotation
  • Middle-click: Pan
  • Mouse wheel: Zoom
  • Alt+Left-click: Orbital rotation
  • Ctrl+Left-click: Precision zoom
  • Alt+Q: Restore initial state

Key Classes:

• CameraPreset: Stores virtual camera + hotkey + priority
• HotkeyCommand: Records and validates key combinations

Item/Prop System

Assets/Scripts/Streamingle/StreamingleControl/Controllers/ItemController.cs

Setup:

1. Create GameObjects representing props/items
2. Tag them as "ItemGroup" for auto-discovery
3. Assign hotkeys via Inspector
4. Items can be toggled on/off or set to specific states

Event System

Assets/Scripts/Streamingle/StreamingleControl/Controllers/EventController.cs

Usage:

• Events are NOT auto-discovered (unlike Items)
• Manually add EventGroups via Inspector
• Each EventGroup wraps a UnityEvent + hotkey
• Trigger animations, sounds, particle effects, etc.

Spout/NDI Streaming

Assets/Scripts/SpoutOutputScript/RenderStreamOutput.cs

Architecture:

• Custom URP Scriptable Render Pass (AlphaRecodingRenderPass)
• Dual output support: Spout + NDI simultaneously
• Alpha channel preservation for chroma-free compositing
• Auto-detects resolution changes

Important: Uses DestroyImmediate for texture cleanup. Be cautious when modifying render pass lifecycle.

External Dependencies

Critical Packages

• Universal Render Pipeline 17.2.0
• Cinemachine 3.1.4 (camera system)
• Animation Rigging 1.3.1
• Newtonsoft Json 3.2.1 (serialization)
• websocket-sharp (Stream Deck server)
• Klak.NDI 2.1.4 (NDI streaming)
• Klak.Spout (Spout output)

Major External Assets

• AmplifyShaderEditor: Visual shader creation
• lilToon/NiloToonURP: Anime/toon shaders
• MagicaCloth2: Cloth physics simulation
• FastSpringBone: Hair/cloth dynamics
• BioIK: Biological inverse kinematics
• OptiTrack Unity Plugin: Motion capture hardware
• Rokoko: Motion capture streaming
• iFacialMocap: iPhone-based facial mocap
• VRM Support: VRM avatar format handling

Scripting Define Symbols

The project uses several conditional compilation symbols (visible in ProjectSettings):

UNITY_URP
MAGICACLOTH2
DLSS_INSTALLED
TND_DLSS
AMPLIFY_SHADER_EDITOR
AURA_IN_SCENEVIEW
AURA_IN_PROJECT
VLB_URP

When adding features that depend on optional plugins, use #if directives to prevent compilation errors when the plugin is missing.

Editor Tools and Utilities

Camera Management

• CameraManagerEditor: Custom Inspector for CameraController with hotkey recording UI

Avatar Tools

• StreamingleAvatarExport: Export pipeline for avatar setups
• Tag-based setup automation in Assets/Scripts/Streamingle/StreamingleAvatarExport/

Animation/Pose Tools

• HumanPoseClipCreator: Record humanoid animation poses
• HumanPoseClipApplier: Apply saved poses to characters
• PoseRecorderWindow: Timeline-based pose recording

VRM Utilities (Assets/Scripts/Vrmtool/)

• ColliderAutoSetup: Auto-generate colliders for VRM models
• VRMSpringBoneMoveTool: Adjust spring bone dynamics
• RemoveInvalidVRMSpringBones: Clean up broken spring bones
• HumanBoneRenamer: Batch rename humanoid skeleton bones

Material/Rendering Tools

• MaterialAndTextureCollectorWindow: Asset collection and management
• NilotoonMaterialMatcapSetter: Auto-apply matcap textures to NiloToon materials

Debugging

• StreamingleDebugWindow: Runtime debug UI for monitoring system state
• BoneRendererContextMenu: Visualize bone hierarchies in scene

Common Modification Patterns

Adding a New Controller

1. Implement IController interface
2. Create corresponding *ListData and *StateData serializable classes
3. Register controller in StreamDeckServerManager
4. Add message handling in ProcessMessage() method
5. Update WebSocket protocol documentation

Adding Camera Control Features

1. Modify CameraController.cs
2. Update CameraPreset or HotkeyCommand classes if needed
3. Extend CameraListData/CameraStateData for new properties
4. Update CameraManagerEditor if Inspector UI changes needed

Extending Item/Event Systems

• Items: Tag GameObjects as "ItemGroup" for auto-discovery
• Events: Manually add via Inspector (no auto-discovery)
• Both use similar *Group wrapper pattern with hotkeys

Modifying Render Output

When changing RenderStreamOutput.cs:

• Always test both Spout and NDI outputs
• Verify alpha channel preservation
• Check texture lifecycle (creation/destruction)
• Test resolution changes

WebSocket Protocol Changes

When modifying StreamDeckServerManager:

1. Update message type handling in ProcessMessage()
2. Ensure thread-safe main thread dispatching via mainThreadActions
3. Update response JSON structure
4. Test with Stream Deck plugin
5. Document protocol changes

Performance Considerations

• GPU Skinning: Enabled globally (important for multi-avatar scenes)
• Graphics API: Direct3D 12 primary, Vulkan secondary
• Render Pipeline: Linear color space for accurate lighting
• Thread Safety: WebSocket server runs on background thread; Unity API calls must be queued to main thread

File Naming and Organization

Script Organization

Assets/Scripts/
├── Streamingle/              # Core framework (production code)
│   ├── StreamingleControl/   # Runtime control systems
│   └── StreamingleDebug/     # Debug utilities
├── Streamdeck/               # WebSocket server
├── SpoutOutputScript/        # Streaming output
├── KindRetargeting/          # Motion retargeting
├── HandTracking/             # Hand tracking
├── Vrmtool/                  # VRM utilities
├── YAMO_Scripts/             # Animation utilities
└── Editor/                   # Editor-only tools

Data Folder

Assets/StreamingleData/ contains runtime-generated JSON files for controller state persistence.

Resource Assets

Assets/ResourcesData/Character/ contains character models, animations, and avatar resources.

Important Notes

• Unity Version: Project requires Unity 6000.2.6f2 or later (URP compatibility)
• Platform Target: Primary platform is Windows Standalone; Android/iOS support exists but is secondary
• WebSocket Port: Default 64211 - ensure firewall allows localhost connections
• Motion Capture: Requires external hardware/software (OptiTrack, iPhone with iFacialMocap, Rokoko suit)
• Shader Compilation: First-time shader compilation may take significant time due to AmplifyShaderEditor and toon shaders

Testing

• Camera System: Test hotkey recording, preset switching, and mouse controls in Play mode
• Stream Deck: Test WebSocket connection using Stream Deck plugin or WebSocket client
• Streaming Output: Verify Spout/NDI output in OBS, vMix, or compatible software
• Item/Event Controllers: Ensure hotkeys trigger correctly and state synchronizes with Stream Deck
• Motion Capture: Test with actual mocap hardware when modifying SystemController

Git Workflow

Current branch: main

When committing:

• Avoid committing temporary .meta files for deleted assets
• Ignore Assets/StreamingleData/*.json (runtime state files)
• Be careful with ProjectSettings/ changes (often personal preferences)
• Shader template modifications in Assets/External/AmplifyShaderEditor/ should be reviewed carefully