The HV-DMLP and HV-DPE Modules for the Semantic Segmentation of

# Transformers for Panoramic Semantic Segmentation (Trans4PASS & Trans4PASS+) **Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation**, CVPR 2022, [[PDF](https://arxiv.org/pdf/2203.01452.pdf)].  **Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation**, arxiv preprint, [[PDF](https://arxiv.org/pdf/2207.11860.pdf)].  [](https://paperswithcode.com/sota/semantic-segmentation-on-densepass?p=behind-every-domain-there-is-a-shift-adapting) [](https://paperswithcode.com/sota/semantic-segmentation-on-synpass?p=behind-every-domain-there-is-a-shift-adapting) [](https://paperswithcode.com/sota/semantic-segmentation-on-stanford2d3d-1?p=behind-every-domain-there-is-a-shift-adapting) [](https://paperswithcode.com/sota/semantic-segmentation-on-cityscapes-val?p=bending-reality-distortion-aware-transformers) ## Update - [08/2022], a new panoramic semantic segmentation benchmark **SynPASS** is released. - [08/2022], Trans4PASS+ is released. ## Environments ```bash conda create -n trans4pass python=3.8 conda activate trans4pass cd ~/path/to/trans4pass conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge pip install mmcv-full==1.3.9 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html pip install -r requirements.txt python setup.py develop --user # Optional: install apex follow: https://github.com/NVIDIA/apex ``` ## SynPASS dataset  SynPASS dataset contains **9080** panoramic images (1024x2048) and **22** categories. The scenes include cloudy, foggy, rainy, sunny, and day-/night-time conditions. ***The SynPASS dataset is now availabel at [GoogleDrive](https://drive.google.com/file/d/1u-5J13CD6MXpWB53apB-L6kZ3hK1JR77/view?usp=sharing).*** SynPASS statistic information: | | Cloudy | Foggy | Rainy | Sunny | ALL | |---|:---:|:---:|:---:|:---:|:---:| | **Split** | train/val/test | train/val/test | train/val/test | train/val/test | train/val/test | | **#Frames** | 1420/420/430 | 1420/430/420 | 1420/430/420 | 1440/410/420 | 5700/1690/1690 | | **Split** | day/night | day/night | day/night | day/night | day/night | | **#Frames** | 1980/290 | 1710/560 | 2040/230 | 1970/300 | 7700/1380 | | **Total** | 2270 | 2270 | 2270 | 2270 | 9080 | ## Data Preparation Prepare datasets: - [Cityscapes](https://www.cityscapes-dataset.com/) - [DensePASS](https://github.com/chma1024/DensePASS) - [SynPASS](https://drive.google.com/file/d/1u-5J13CD6MXpWB53apB-L6kZ3hK1JR77/view?usp=sharing) - [Stanford2D3D](https://arxiv.org/abs/1702.01105) - [Structured3D](https://structured3d-dataset.org/) ``` datasets/ ├── cityscapes │   ├── gtFine │   └── leftImg8bit ├── Stanford2D3D │   ├── area_1 │   ├── area_2 │   ├── area_3 │   ├── area_4 │   ├── area_5a │   ├── area_5b │   └── area_6 ├── Structured3D │   ├── scene_00000 │   ├── ... │   └── scene_00199 ├── SynPASS │   ├── img │   │   ├── cloud │   │   ├── fog │   │   ├── rain │   │   └── sun │   └── semantic │   ├── cloud │   ├── fog │   ├── rain │   └── sun ├── DensePASS │   ├── gtFine │   └── leftImg8bit ``` Prepare pretrained weights, which can be found in the public repository of [SegFormer](https://github.com/NVlabs/SegFormer). ``` pretrained/ ├── mit_b1.pth └── mit_b2.pth ``` ## Network Define The code of Network pipeline is in `segmentron/models/trans4pass.py`. The code of backbone is in `segmentron/models/backbones/trans4pass.py`. The code of DMLP decoder is in `segmentron/modules/dmlp.py`. The code of DMLPv2 decoder is in `segmentron/modules/dmlp2.py`. ## Train For example, to use 4 1080Ti GPUs to run the experiments: ```bash # Trans4PASS python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml # Trans4PASS+, please modify the version at segmentron/models/trans4pass.py python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml python -m torch.distributed.launch --nproc_per_node=4 tools/train_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml ``` ## Test Download the models from [GoogleDrive](https://drive.google.com/drive/folders/1snUdrg_Y9whnCQi9QNFLI6hRoYhGm3Qv?usp=sharing) and save in `./workdirs` folder as:

workdirs folder:

``` ./workdirs ├── cityscapes │   ├── trans4pass_plus_small_512x512 │   ├── trans4pass_plus_tiny_512x512 │   ├── trans4pass_small_512x512 │   └── trans4pass_tiny_512x512 ├── cityscapes13 │   ├── trans4pass_plus_small_512x512 │   ├── trans4pass_plus_tiny_512x512 │   ├── trans4pass_small_512x512 │   └── trans4pass_tiny_512x512 ├── stanford2d3d │   ├── trans4pass_plus_small_1080x1080 │   ├── trans4pass_plus_tiny_1080x1080 │   ├── trans4pass_small_1080x1080 │   └── trans4pass_tiny_1080x1080 ├── stanford2d3d8 │   ├── trans4pass_plus_small_1080x1080 │   ├── trans4pass_plus_tiny_1080x1080 │   ├── trans4pass_small_1080x1080 │   └── trans4pass_tiny_1080x1080 ├── stanford2d3d_pan │   ├── trans4pass_plus_small_1080x1080 │   ├── trans4pass_small_1080x1080 │   └── trans4pass_tiny_1080x1080 ├── structured3d8 │   ├── trans4pass_plus_small_512x512 │   ├── trans4pass_plus_tiny_512x512 │   ├── trans4pass_small_512x512 │   └── trans4pass_tiny_512x512 ├── synpass │   ├── trans4pass_plus_small_512x512 │   ├── trans4pass_plus_tiny_512x512 │   ├── trans4pass_small_512x512 │   └── trans4pass_tiny_512x512 └── synpass13 ├── trans4pass_plus_small_512x512 ├── trans4pass_plus_tiny_512x512 ├── trans4pass_small_512x512 └── trans4pass_tiny_512x512 ```

Some test examples: ```bash # Trans4PASS python tools/eval.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml # Trans4PASS+ python tools/eval.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml python tools/eval_dp13.py --config-file configs/synpass13/trans4pass_plus_tiny_512x512.yaml python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.ya