Ze-Feng Gao / 高泽峰

I am a Postdoctoral at the Gaoling School of Artificial Intelligence at Renmin University of China. My Co-operative tutor is Prof. Ji-Rong Wen and Prof. Xin Zhao.

Currently, my primary research interests include model compression, data compression and pre-training language model based on tensor networks.

I have a Ph.D. of Science in the Department of Physics, Renmin University of China, advised by Prof. Zhong-Yi Lu.

Email: zfgao@ruc.edu.cn

GitHub  /  Google Scholar  /  DBLP  /  Zhihu

profile photo

Education

  • Ph.D. of Science, Renmin University of China, 2016-2021
  • B.Sc. of Physics, Renmin University of China, 2012-2016

    		•

    Work Experience

  • 2021-present, Renmin University of China, Postdoctoral Researcher. Supervisor: Prof. Ji-Rong Wen.
  • 2021-present, Renmin University of China, Beijing Key Laboratory of Physics Department, Associate Researcher

    		•



    Research Projects

  • Beijing Academy of Artificial Intelligence-Zhiyuan Foundation:Crosswise Tasks Program, Jan 2022 – Dec 2022,
    Lightweight Fine-tuning Strategy for Multimodal Pre-training Models based on Matrix Product Operator Method, Role: PI
  • National Natural Science Foundation – Youth Program(Nos. 62206299), Jan 2023 – Dec 2024
    Lightweight Fine-tuning and Model Scaling Approach for Large Scale Pre-trained Language Models, Role: PI
  • National Natural Science Foundation – General Program(Nos. 62276269), Jan 2023 – Dec 2025
    , Role: Participate
  • National Natural Science Foundation – General Program(Nos. 11874421), Jan 2019 – Dec 2022
    Some Issues in Multibody Localization, Role: Participate
  • National Natural Science Foundation – General Program(Nos. 11774422), Jan 2018 – Dec 2021
    Several Problems in Quantum Impurity Systems based on Natural Orbital Reformation Groups in Quantum Impurity Systems, Role: Participate

    		•



    Publications

    2022
    project image

    Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models


    Ze-Feng Gao*, Peiyu Liu*, Wayne Xin Zhao#, Zhong-Yi Lu, Ji-Rong Wen
    COLING 2022, Oral Presentation, 2022
    paper / arxiv / code /

    In this paper, we can reduce the parameters of the original MoE architecture by sharing a global central tensor across experts and keeping expert-specific auxiliary tensors. We also design the gradient mask strategy for the tensor structure of MPO to alleviate the overfitting problem.



    2021
    project image

    Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators


    Peiyu Liu*, Ze-Feng Gao*, Wayne Xin Zhao#, Z.Y. Xie, Zhong-Yi Lu#, Ji-Rong Wen
    ACL 2021 main conference, 2021
    paper / arxiv / code / slides / link /

    This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics.



    2020
    project image

    Compressing LSTM Networks by Matrix Product Operators


    Ze-Feng Gao*, Xingwei Sun*, Lan Gao, Junfeng Li#, Zhong-Yi Lu#
    Preprint, 2020
    paper / arxiv /

    We propose an alternative LSTM model to reduce the number of parameters significantly by representing the weight parameters based on matrix product operators (MPO), which are used to characterize the local correlation in quantum states in physics.
    project image

    A Model Compression Method With Matrix Product Operators for Speech Enhancement


    Xingwei Sun*, Ze-Feng Gao*, Zhong-Yi Lu#, Junfeng Li#, Yonghong Yan
    IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 2837-2847, 2020
    paper / arxiv / link /

    In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement.
    project image

    Compressing deep neural networks by matrix product operators


    Ze-Feng Gao*,Song Cheng*, Rong-Qiang He, Zhi-Yuan Xie#, Hui-Hai Zhao#, Zhong-Yi Lu#, Tao Xiang#
    Physical Review Research 2 (2), 023300, 2020
    paper / arxiv / code / link /

    In this paper, we show that neural network can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states.

    * Equal contribution           # Corresponding author




    Professional Services

  • Reviewer: WSDM2022, ACCV2022, COLING2022

    		•



    Selected Awards and Honors

  • Outstanding graduates, Renmin University of China, 2021
  • National Scholarship for Graduate Student, Ministry of Education of P.R.China, 2020
  • First class academic scholarship, Renmin University of China, 2020
  • Social Volunteer Service Scholarship,Renmin University of China, 2019
  • First class academic scholarship, Renmin University of China, 2019
  • First class academic scholarship, Renmin University of China, 2018
  • First class academic scholarship, Renmin University of China, 2016
  • Innovation experiment plan for college students, national level, 2013

    		•