# MuSS

**Repository Path**: pipieger/MuSS

## Basic Information

- **Project Name**: MuSS
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-06-25
- **Last Updated**: 2026-06-26

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<p align="center">
  <img src="assets/branding/muss-logo.png" alt="MuSS logo" width="620">
</p>

<h3 align="center">
  A Large-Scale Dataset and Cinematic Narrative Benchmark for <br>
  Multi-Shot Subject-to-Video Generation
</h3>

<p align="center">
  Haojie Zhang, Di Wu, Bingyan Liu, Linjie Zhong, Yuancheng Wei, Xingsong Ye, Nanqing Liu, Yaling Liang
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2604.23789">Paper</a> |
  <a href="https://zhang-haojie.github.io/project-pages/muss.html">Project Page</a> |
  <a href="#code">Code</a>
</p>

<p align="center">
  <a href="https://zhang-haojie.github.io/project-pages/muss.html"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a>
  <a href="https://arxiv.org/abs/2604.23789"><img src="https://img.shields.io/badge/arXiv-2604.23789-b31b1b.svg" alt="arXiv"></a>
  <img src="https://img.shields.io/badge/Code-Initial%20Release-blue" alt="Code">
</p>

<p align="center">
  <img src="assets/figures/teaser.png" alt="MuSS overview" width="100%">
</p>

## Overview

MuSS is a large-scale cinematic dataset and benchmark for **multi-shot video generation** and **Subject-to-Video generation**. It targets three limitations that are difficult to expose in single-shot settings:

- lack of authentic cinematic narrative logic;
- conflicts between global captions and local shot-level alignment;
- copy-paste shortcuts in Subject-to-Video models.

MuSS contains two complementary data settings: **Complex Cinematic Narrative**, which focuses on montage, shot transitions, and multi-character storytelling, and **Subject-Centric Narrative**, which focuses on preserving the same subject across disjoint shots and viewpoints.

## News

- **2026-04**: Paper released on arXiv.
- **Initial public release**: data construction code is released first. Dataset files, benchmark implementation, and model checkpoints are not included in this repository.

## Dataset

MuSS is sourced from over **3,000 movies** and contains more than **30,000 professionally captioned multi-shot clips**, totaling over **1,000 hours** of high-quality video content.

<p align="center">
  <img src="assets/figures/data_statistics.png" alt="MuSS dataset statistics" width="100%">
</p>

The construction pipeline first turns raw cinematic footage into high-quality physical shots with coherent captions, then builds cross-shot S2V pairs by sampling reference subjects from disjoint shot contexts.

<p align="center">
  <img src="assets/figures/dataset_pipeline.png" alt="MuSS dataset construction pipeline" width="100%">
</p>

### Data Examples

**Complex Cinematic Narrative.** Progressive multi-shot captions are aligned to physical shots, capturing shot transitions, scene changes, and multi-character narrative flow.

<p align="center">
  <img src="assets/examples/track1_data.png" alt="Track 1 data examples" width="88%">
</p>

**Subject-Centric Narrative.** A reference subject is extracted from a disjoint shot, while the target sequence preserves identity across different viewpoints and contexts.

<p align="center">
  <img src="assets/examples/track2_data.png" alt="Track 2 data examples" width="88%">
</p>

<details>
<summary>More raw cinematic examples</summary>

<p align="center">
  <img src="assets/examples/track1_raw.png" alt="Raw Track 1 cinematic transitions" width="49%">
  <img src="assets/examples/track2_raw.png" alt="Raw Track 2 subject-centric sequences" width="49%">
</p>

</details>

## Benchmark

The **Cinematic Narrative Benchmark** evaluates generated videos under realistic multi-shot storytelling conditions. It combines shot boundary parsing, expert perception models, and LMM-based visual-logic assessment.

<p align="center">
  <img src="assets/figures/benchmark_pipeline.png" alt="Cinematic Narrative Benchmark pipeline" width="100%">
</p>

| Track | Evaluation Goal | Metrics |
| --- | --- | --- |
| **Track 1: Narrative Effectiveness** | Shot-level alignment, transition precision, scene continuity, and visual logic. | `Txt.Align`, `Trans.Dev`, `Scene.Con`, `Con.Gap`, `Scene.Logic`, `Casting.Logic`, `Act.Logic`, `Spat.Logic` |
| **Track 2: Subject Consistency** | Cross-shot identity preservation, subject grounding, motion strength, and anti-copy-paste behavior. | `Subj.Recall`, `Ref-Sub.Con`, `Inter-Sub.Con`, `Act.Str`, `ACP-Var`, `CP-Rate` |

`ACP-Var` measures pose and structural diversity between the reference image and generated frames, explicitly penalizing rigid 2D reference copying.

<p align="center">
  <img src="assets/figures/qualitative.png" alt="Qualitative benchmark comparison" width="100%">
</p>

## Code

This repository currently releases the data-side code. The benchmark implementation is intentionally not included in this initial repository and will be released separately after further verification.

```text
MuSS/
├── download/        # Raw video acquisition from organized yearly YouTube lists
├── main-pipeline/   # Multi-shot video curation and progressive captioning
├── s2v-pipeline/    # Cross-shot subject extraction and S2V pair construction
├── assets/          # README figures converted from the paper and supplementary material
├── CITATION.cff
└── README.md
```

### Quick Start

```bash
git clone https://github.com/<your-org>/MuSS.git
cd MuSS
python3 download/download.py 2011
```

Module-level guides:

- [Download guide](download/README.md)
- [Multi-shot construction pipeline](main-pipeline/README.md)
- [Cross-shot S2V construction pipeline](s2v-pipeline/README.md)

## Citation

```bibtex
@article{zhang2026muss,
  title   = {MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation},
  author  = {Zhang, Haojie and Wu, Di and Liu, Bingyan and Zhong, Linjie and Wei, Yuancheng and Ye, Xingsong and Liu, Nanqing and Liang, Yaling},
  journal = {arXiv preprint arXiv:2604.23789},
  year    = {2026}
}
```

## License

The code and dataset license are being finalized. Please check this section before redistribution or commercial use.

## Acknowledgements

MuSS builds on open research infrastructure for video processing, perception, visual-language reasoning, and generative video evaluation. Please also follow the licenses of any third-party models, datasets, and tools used by the individual pipelines.