Skip to main content
SpringerLink
Log in

Combining Pixel-Level and Structure-Level Adaptation for Semantic Segmentation

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Domain adaptation for semantic segmentation requires pixel-level knowledge transfer from a labeled source domain to an unlabeled target domain. Existing approaches typically align the features of the source and target domains at different levels. However, they usually neglect the different adaptive complexities of different information flows within images. In this paper, we focus on combining two main information flows in semantic segmentation, ie., the pixel-level disparate information and image structure information. Specifically, we propose to combine two feature map-based prediction heads, which are thought to focus on pixel-level and structure-level information, to accommodate different complexities by adjusting the attention to adaptation functions of the target domain. We then align the outputs from the two heads through a consistency regularization to realize informative complementarity. The combined prediction head further enables regularizing the distance between different pixel representations of different classes, thereby mitigating the mis-adaptation problem of similar classes. The proposed method can achieve more competitive results than current state-of-the-art results on two publicly available benchmark datasets, ie., SYNTHIA \(\rightarrow \) Cityscapes and GTA5 \(\rightarrow \) Cityscapes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Chen C, Xie W, Huang W et al (2019) Progressive feature alignment for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 627–636

  2. Chen LC, Papandreou G, Kokkinos I et al (2017) DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  3. Chen W, Hu H (2020) Unsupervised domain adaptation via discriminative classes-center feature learning in adversarial network. Neural Process Lett 52(1):467–483

    Article  MathSciNet  Google Scholar 

  4. Chen Y (2022) Semantic image segmentation with feature fusion based on Laplacian pyramid. Neural Process Lett 54:4153–4170

    Article  Google Scholar 

  5. Chen YC, Lin YY, Yang MH et al (2019) CrDoCo: pixel-level domain transfer with cross-domain consistency. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1791–1800

  6. Choi J, Kim T, Kim C (2019) Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6830–6840

  7. Cordts M, Omran M, Ramos S et al (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  8. Dundar A, Liu MY, Wang TC et al (2018) Domain stylization: a strong, simple baseline for synthetic to real image domain adaptation. arXiv preprint arXiv:1807.09384

  9. Everingham M, Eslami SA, Van Gool L et al (2015) The pascal visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136

    Article  Google Scholar 

  10. Fu J, Liu J, Tian H et al (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3146–3154

  11. Ganin Y, Ustinova E, Ajakan H et al (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(59):1–35

    MathSciNet  MATH  Google Scholar 

  12. Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. Adv Neural Inf Process Syst 27:1–9

    Google Scholar 

  13. HassanPour Zonoozi M, Seydi V (2022) A survey on adversarial domain adaptation. Neural Process Lett. https://doi.org/10.1007/s11063-022-10977-5

  14. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  15. Hoffman J, Wang D, Yu F et al (2016) FCNs in the wild: pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649

  16. Hoffman J, Tzeng E, Park T et al (2018) CyCADA: cycle-consistent adversarial domain adaptation. In: International conference on machine learning, PMLR, pp 1989–1998

  17. Kim M, Byun H (2020) Learning texture invariant representation for domain adaptation of semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12,975–12,984

  18. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations, pp 1–11

  19. Kothandaraman D, Nambiar AM, Mittal A (2021) Domain adaptive knowledge distillation for driving scene semantic segmentation. In: WACV (Workshops), pp 134–143

  20. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  21. Lee CY, Batra T, Baig MH et al (2019) Sliced Wasserstein discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10,285–10,295

  22. Li S, Xie B, Zang B et al (2021) Semantic distribution-aware contrastive adaptation for semantic segmentation. arXiv preprint arXiv:2105.05013

  23. Li Y, Yuan L, Vasconcelos N (2019) Bidirectional learning for domain adaptation of semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6936–6945

  24. Liu S, Zhang H, Shao L et al (2021) Built-in depth-semantic coupled encoding for scene parsing, vehicle detection and road segmentation. IEEE Trans Intell Transp Syst 22(9):5520–5534

    Article  Google Scholar 

  25. Liu W, Li J, Liu B et al (2021) Unified cross-domain classification via geometric and statistical adaptations. Pattern Recognit 110(107):658

    Google Scholar 

  26. Liu Y, Deng J, Gao X et al (2021) BAPA-Net: boundary adaptation and prototype alignment for cross-domain semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8801–8811

  27. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  28. Luo Y, Liu P, Guan T et al (2019) Significance-aware information bottleneck for domain adaptive semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6778–6787

  29. Luo Y, Zheng L, Guan T et al (2019) Taking a closer look at domain shift: category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2507–2516

  30. Luo Y, Liu P, Zheng L et al (2021) Category-level adversarial adaptation for semantic segmentation using purified features. IEEE Trans Pattern Anal Mach Intell 44(8):3940–3956

    Google Scholar 

  31. Melas-Kyriazi L, Manrai AK (2021) PixMatch: unsupervised domain adaptation via pixelwise consistency training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12,435–12,445

  32. Murez Z, Kolouri S, Kriegman D et al (2018) Image to image translation for domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4500–4509

  33. Pan F, Shin I, Rameau F et al (2020) Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3764–3773

  34. Paul S, Tsai YH, Schulter S et al (2020) Domain adaptive semantic segmentation using weak labels. In: European conference on computer vision, Springer, pp 571–587

  35. Pizzati F, Charette Rd, Zaccaria M et al (2020) Domain bridge for unpaired image-to-image translation and unsupervised domain adaptation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2990–2998

  36. Reinhard E, Adhikhmin M, Gooch B et al (2001) Color transfer between images. IEEE Comput Gr Appl 21(5):34–41

    Article  Google Scholar 

  37. Richter SR, Vineet V, Roth S et al (2016) Playing for data: ground truth from computer games. In: European conference on computer vision, pp 102–118

  38. Ros G, Sellart L, Materzynska J et al (2016) The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3234–3243

  39. Saito K, Watanabe K, Ushiku Y et al (2018) Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3723–3732

  40. Sankaranarayanan S, Balaji Y, Jain A et al (2018) Learning from synthetic data: Addressing domain shift for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3752–3761

  41. Toldo M, Michieli U, Agresti G et al (2020) Unsupervised domain adaptation for mobile semantic segmentation based on cycle consistency and feature alignment. Image Vis Comput 95(103):889

    Google Scholar 

  42. Tsai YH, Hung WC, Schulter S et al (2018) Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7472–7481

  43. Tsai YH, Sohn K, Schulter S et al (2019) Domain adaptation for structured output via discriminative patch representations. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1456–1465

  44. Vu TH, Jain H, Bucher M et al (2019) ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2517–2526

  45. Weng L, Wang Y, Gao F (2022) Traffic scene perception based on joint object detection and semantic segmentation. Neural Process Lett 54:5333–5349

    Article  Google Scholar 

  46. Wu Z, Wang X, Gonzalez JE et al (2019) ACE: adapting to changing environments for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2121–2130

  47. Yang J, Xu R, Li R et al (2020) An adversarial perturbation oriented domain adaptation approach for semantic segmentation. In: Proceedings of the AAAI conference on artificial intelligence, pp 12,613–12,620

  48. Yang J, Zou H, Zhou Y et al (2020) Mind the discriminability: asymmetric adversarial domain adaptation. In: European conference on computer vision, pp 589–606

  49. Yang Y, Soatto S (2020) FDA: Fourier domain adaptation for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4085–4095

  50. Yu J, Tao D, Wang M (2012) Adaptive hypergraph learning and its application in image classification. IEEE Trans Image Process 21(7):3262–3272

    Article  MathSciNet  MATH  Google Scholar 

  51. Yu J, Yao J, Zhang J et al (2020) SPRNet: single-pixel reconstruction for one-stage instance segmentation. IEEE Trans Cybern 51(4):1731–1742

    Article  Google Scholar 

  52. Zhang H, Long Y, Liu L et al (2019) Adversarial unseen visual feature synthesis for zero-shot learning. Neurocomputing 329:12–20

    Article  Google Scholar 

  53. Zhang J, Yang J, Yu J et al (2022) Semisupervised image classification by mutual learning of multiple self-supervised models. Int J Intell Syst 37(5):3117–3141

    Article  Google Scholar 

  54. Zhang P, Zhang B, Zhang T et al (2021) Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12,414–12,424

  55. Zhang Q, Zhang J, Liu W et al (2019) Category anchor-guided unsupervised domain adaptation for semantic segmentation. Adv Neural Inf Process Syst 32:1–11

    Google Scholar 

  56. Zhang W, Zhang X, Lan L et al (2020) Maximum mean and covariance discrepancy for unsupervised domain adaptation. Neural Process Lett 51(1):347–366

    Article  Google Scholar 

  57. Zhang X, Chen Y, Zhang H et al (2021) When visual disparity generation meets semantic segmentation: a mutual encouragement approach. IEEE Trans Intell Transp Syst 22(3):1853–1867

    Article  Google Scholar 

  58. Zhang X, Zhang H, Lu J et al (2021) Target-targeted domain adaptation for unsupervised semantic segmentation. In: 2021 IEEE international conference on robotics and automation (ICRA), pp 13,560–13,566

  59. Zhang X, Chen Y, Shen Z et al (2022) Confidence-and-refinement adaptation model for cross-domain semantic segmentation. IEEE Trans Intell Transp Syst 23(7):9529–9542

    Article  Google Scholar 

  60. Zhang Y, Ye M, Gan Y et al (2020) Knowledge based domain adaptation for semantic segmentation. Knowl Based Syst 193(105):444

    Google Scholar 

  61. Zhao H, Shi J, Qi X et al (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  62. Zhou D, Fang J, Song X et al (2020a) Joint 3d instance segmentation and object detection for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1839–1849

  63. Zhou L, Zhang H, Long Y et al (2019) Depth embedded recurrent predictive parsing network for video scenes. IEEE Trans Intell Transp Syst 20(12):4643–4654

    Article  Google Scholar 

  64. Zhou W, Wang Y, Chu J et al (2020) Affinity space adaptation for semantic segmentation across domains. IEEE Trans Image Process 30:2549–2561

    Article  Google Scholar 

Download references

Acknowledgements

This work was in part supported by the National Natural Science Foundation of China (Nos. 61872187, 62072246), in part supported by the Natural Science Foundation of Jiangsu Province (No. BK20201306), and in part by the “111” Program under Grant No. B13022.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China

    Xiwen Bi, Dubing Chen & Haofeng Zhang

  2. Department of Data Link and Communication, Nanjing Research Institute of Electronic Engineering, Nanjing, 210007, China

    He Huang

  3. School of Engineering, Newcastle University, Newcastle Upon Tyne, NE17RU, UK

    Shidong Wang

Authors
  1. Xiwen Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dubing Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. He Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shidong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haofeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haofeng Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, X., Chen, D., Huang, H. et al. Combining Pixel-Level and Structure-Level Adaptation for Semantic Segmentation. Neural Process Lett 55, 9669–9684 (2023). https://doi.org/10.1007/s11063-023-11220-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11220-5

Keywords

Search

Navigation