Streamingle_URP/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

**Streamingle** is a Unity-based virtual avatar streaming and control system for VTuber/avatar performance streaming. Built on Unity 6000.2.6f2 with Universal Render Pipeline (URP).

**Key capabilities:**
- Multi-camera control with Cinemachine presets
- Avatar outfit and item/prop management
- Event/animation triggering system
- Stream Deck integration via WebSocket server
- Real-time streaming output (Spout/NDI)
- Motion capture support (OptiTrack, iFacialMocap, Rokoko)

## Build and Development Commands

### Unity Editor
- Open the project in Unity 6000.2.6f2 or later
- The project uses URP, ensure Graphics settings point to Universal Render Pipeline asset
- Main scene is typically in `Assets/Scenes/`

### Stream Deck Plugin Deployment
```batch
cd Streamdeck
deploy-plugin.bat
```
This requires administrator privileges and auto-deploys the Stream Deck plugin.

### Testing Stream Deck Server
The WebSocket server runs on `localhost:10701` by default. Test with any WebSocket client or the Stream Deck plugin.

## Core Architecture

### IController Pattern
All major control systems implement the `IController` interface defined in [Assets/Scripts/Streamingle/StreamingleControl/Controllers/IController.cs](Assets/Scripts/Streamingle/StreamingleControl/Controllers/IController.cs):

```csharp
public interface IController
{
    void Set(int index);
    string GetControllerId();
    string GetControllerName();
    object GetControllerData();
    void ExecuteAction(string actionId, object parameters);
}
```

**Key Controllers:**
- **CameraController**: Multi-preset camera management with hotkeys
- **ItemController**: Toggle visibility/state of scene props/objects
- **EventController**: Trigger UnityEvents (animations, sounds, effects)
- **AvatarOutfitController**: Avatar appearance/clothing management
- **SystemController**: Motion capture, recording, screenshots

### StreamDeckServerManager Central Hub
[Assets/Scripts/Streamdeck/StreamDeckServerManager.cs](Assets/Scripts/Streamdeck/StreamDeckServerManager.cs) orchestrates all controllers via WebSocket communication:

**Message Flow:**
```
Stream Deck Client (WebSocket)
    → StreamDeckServerManager (JSON protocol)
        → Dispatches to appropriate IController
            → Updates Unity scene/state
                → Broadcasts state changes back to clients
```

**Thread Safety:** Uses main thread action queuing for Unity API calls. When modifying server logic, always enqueue Unity operations via `mainThreadActions`.

**Supported Message Types:**
- Camera: `switch_camera`, `get_camera_list`, `get_camera_state`
- Items: `set_item`, `toggle_item`, `get_item_list`, `get_item_state`
- Events: `execute_event`, `set_event`, `get_event_list`, `get_event_state`
- Avatars: `set_avatar_outfit`, `get_avatar_outfit_list`, `get_avatar_outfit_state`
- System: `start_motion_capture`, `stop_motion_capture`, `start_recording`, etc.

### Data Structures
Controllers export/import state via JSON-serializable data classes:
- `*ListData`: Full list of available items (cameras, items, events, outfits)
- `*StateData`: Current active state

Example: `CameraListData`, `CameraStateData`, `ItemListData`, `ItemStateData`

### Camera Control System
[Assets/Scripts/Streamingle/StreamingleControl/Controllers/CameraController.cs](Assets/Scripts/Streamingle/StreamingleControl/Controllers/CameraController.cs)

**Features:**
- Cinemachine virtual camera presets with priority switching
- Hotkey recording using UnityRawInput
- Mouse-based camera control:
  - Right-click: Orbit rotation
  - Middle-click: Pan
  - Mouse wheel: Zoom
  - Alt+Left-click: Orbital rotation
  - Ctrl+Left-click: Precision zoom
  - Alt+Q: Restore initial state

**Key Classes:**
- `CameraPreset`: Stores virtual camera + hotkey + priority
- `HotkeyCommand`: Records and validates key combinations

### Item/Prop System
[Assets/Scripts/Streamingle/StreamingleControl/Controllers/ItemController.cs](Assets/Scripts/Streamingle/StreamingleControl/Controllers/ItemController.cs)

**Setup:**
1. Create GameObjects representing props/items
2. Tag them as "ItemGroup" for auto-discovery
3. Assign hotkeys via Inspector
4. Items can be toggled on/off or set to specific states

### Event System
[Assets/Scripts/Streamingle/StreamingleControl/Controllers/EventController.cs](Assets/Scripts/Streamingle/StreamingleControl/Controllers/EventController.cs)

**Usage:**
- Events are NOT auto-discovered (unlike Items)
- Manually add EventGroups via Inspector
- Each EventGroup wraps a UnityEvent + hotkey
- Trigger animations, sounds, particle effects, etc.

### Spout/NDI Streaming
[Assets/Scripts/SpoutOutputScript/RenderStreamOutput.cs](Assets/Scripts/SpoutOutputScript/RenderStreamOutput.cs)

**Architecture:**
- Custom URP Scriptable Render Pass (`AlphaRecodingRenderPass`)
- Dual output support: Spout + NDI simultaneously
- Alpha channel preservation for chroma-free compositing
- Auto-detects resolution changes

**Important:** Uses `DestroyImmediate` for texture cleanup. Be cautious when modifying render pass lifecycle.

## External Dependencies

### Critical Packages
- **Universal Render Pipeline** 17.2.0
- **Cinemachine** 3.1.4 (camera system)
- **Animation Rigging** 1.3.1
- **Newtonsoft Json** 3.2.1 (serialization)
- **websocket-sharp** (Stream Deck server)
- **Klak.NDI** 2.1.4 (NDI streaming)
- **Klak.Spout** (Spout output)

### Major External Assets
- **AmplifyShaderEditor**: Visual shader creation
- **lilToon/NiloToonURP**: Anime/toon shaders
- **MagicaCloth2**: Cloth physics simulation
- **FastSpringBone**: Hair/cloth dynamics
- **BioIK**: Biological inverse kinematics
- **OptiTrack Unity Plugin**: Motion capture hardware
- **Rokoko**: Motion capture streaming
- **iFacialMocap**: iPhone-based facial mocap
- **VRM Support**: VRM avatar format handling

## Scripting Define Symbols
The project uses several conditional compilation symbols (visible in ProjectSettings):
```
UNITY_URP
MAGICACLOTH2
DLSS_INSTALLED
TND_DLSS
AMPLIFY_SHADER_EDITOR
AURA_IN_SCENEVIEW
AURA_IN_PROJECT
VLB_URP
```

When adding features that depend on optional plugins, use `#if` directives to prevent compilation errors when the plugin is missing.

## Editor Tools and Utilities

### Camera Management
- **CameraManagerEditor**: Custom Inspector for CameraController with hotkey recording UI

### Avatar Tools
- **StreamingleAvatarExport**: Export pipeline for avatar setups
- **Tag-based setup automation** in [Assets/Scripts/Streamingle/StreamingleAvatarExport/](Assets/Scripts/Streamingle/StreamingleAvatarExport/)

### Animation/Pose Tools
- **HumanPoseClipCreator**: Record humanoid animation poses
- **HumanPoseClipApplier**: Apply saved poses to characters
- **PoseRecorderWindow**: Timeline-based pose recording

### VRM Utilities (Assets/Scripts/Vrmtool/)
- **ColliderAutoSetup**: Auto-generate colliders for VRM models
- **VRMSpringBoneMoveTool**: Adjust spring bone dynamics
- **RemoveInvalidVRMSpringBones**: Clean up broken spring bones
- **HumanBoneRenamer**: Batch rename humanoid skeleton bones

### Material/Rendering Tools
- **MaterialAndTextureCollectorWindow**: Asset collection and management
- **NilotoonMaterialMatcapSetter**: Auto-apply matcap textures to NiloToon materials

### Debugging
- **StreamingleDebugWindow**: Runtime debug UI for monitoring system state
- **BoneRendererContextMenu**: Visualize bone hierarchies in scene

## Common Modification Patterns

### Adding a New Controller
1. Implement `IController` interface
2. Create corresponding `*ListData` and `*StateData` serializable classes
3. Register controller in `StreamDeckServerManager`
4. Add message handling in `ProcessMessage()` method
5. Update WebSocket protocol documentation

### Adding Camera Control Features
1. Modify [CameraController.cs](Assets/Scripts/Streamingle/StreamingleControl/Controllers/CameraController.cs)
2. Update `CameraPreset` or `HotkeyCommand` classes if needed
3. Extend `CameraListData`/`CameraStateData` for new properties
4. Update `CameraManagerEditor` if Inspector UI changes needed

### Extending Item/Event Systems
- **Items**: Tag GameObjects as "ItemGroup" for auto-discovery
- **Events**: Manually add via Inspector (no auto-discovery)
- Both use similar `*Group` wrapper pattern with hotkeys

### Modifying Render Output
When changing [RenderStreamOutput.cs](Assets/Scripts/SpoutOutputScript/RenderStreamOutput.cs):
- Always test both Spout and NDI outputs
- Verify alpha channel preservation
- Check texture lifecycle (creation/destruction)
- Test resolution changes

### WebSocket Protocol Changes
When modifying `StreamDeckServerManager`:
1. Update message type handling in `ProcessMessage()`
2. Ensure thread-safe main thread dispatching via `mainThreadActions`
3. Update response JSON structure
4. Test with Stream Deck plugin
5. Document protocol changes

## Performance Considerations

- **GPU Skinning**: Enabled globally (important for multi-avatar scenes)
- **Graphics API**: Direct3D 12 primary, Vulkan secondary
- **Render Pipeline**: Linear color space for accurate lighting
- **Thread Safety**: WebSocket server runs on background thread; Unity API calls must be queued to main thread

## File Naming and Organization

### Script Organization
```
Assets/Scripts/
├── Streamingle/              # Core framework (production code)
│   ├── StreamingleControl/   # Runtime control systems
│   └── StreamingleDebug/     # Debug utilities
├── Streamdeck/               # WebSocket server
├── SpoutOutputScript/        # Streaming output
├── KindRetargeting/          # Motion retargeting
├── HandTracking/             # Hand tracking
├── Vrmtool/                  # VRM utilities
├── YAMO_Scripts/             # Animation utilities
└── Editor/                   # Editor-only tools
```

### Data Folder
`Assets/StreamingleData/` contains runtime-generated JSON files for controller state persistence.

### Resource Assets
`Assets/ResourcesData/Character/` contains character models, animations, and avatar resources.

## Important Notes

- **Unity Version**: Project requires Unity 6000.2.6f2 or later (URP compatibility)
- **Platform Target**: Primary platform is Windows Standalone; Android/iOS support exists but is secondary
- **WebSocket Port**: Default `10701` - ensure firewall allows localhost connections
- **Motion Capture**: Requires external hardware/software (OptiTrack, iPhone with iFacialMocap, Rokoko suit)
- **Shader Compilation**: First-time shader compilation may take significant time due to AmplifyShaderEditor and toon shaders

## Testing

- **Camera System**: Test hotkey recording, preset switching, and mouse controls in Play mode
- **Stream Deck**: Test WebSocket connection using Stream Deck plugin or WebSocket client
- **Streaming Output**: Verify Spout/NDI output in OBS, vMix, or compatible software
- **Item/Event Controllers**: Ensure hotkeys trigger correctly and state synchronizes with Stream Deck
- **Motion Capture**: Test with actual mocap hardware when modifying SystemController

## Git Workflow

Current branch: `main`

When committing:
- Avoid committing temporary `.meta` files for deleted assets
- Ignore `Assets/StreamingleData/*.json` (runtime state files)
- Be careful with `ProjectSettings/` changes (often personal preferences)
- Shader template modifications in `Assets/External/AmplifyShaderEditor/` should be reviewed carefully