Education
Ph.D. of Science, Renmin University of China, 2016-2021
B.Sc. of Physics, Renmin University of China, 2012-2016
|
Work Experience
2021-present, Renmin University of China, Postdoctoral Researcher. Supervisor: Prof. Ji-Rong Wen.
2021-present, Renmin University of China, Beijing Key Laboratory of Physics Department, Associate Researcher
|
Research Projects
Beijing Academy of Artificial Intelligence-Zhiyuan Foundation:Crosswise Tasks Program, Jan 2022 – Dec 2022,
Lightweight Fine-tuning Strategy for Multimodal Pre-training Models based on Matrix Product Operator Method, Role: PI
National Natural Science Foundation – Youth Program(Nos. 62206299), Jan 2023 – Dec 2024
Lightweight Fine-tuning and Model Scaling Approach for Large Scale Pre-trained Language Models, Role: PI
National Natural Science Foundation – General Program(Nos. 62276269), Jan 2023 – Dec 2025
, Role: Participate
National Natural Science Foundation – General Program(Nos. 11874421), Jan 2019 – Dec 2022
Some Issues in Multibody Localization, Role: Participate
National Natural Science Foundation – General Program(Nos. 11774422), Jan 2018 – Dec 2021
Several Problems in Quantum Impurity Systems based on Natural Orbital Reformation Groups in Quantum Impurity Systems, Role: Participate
|
|
Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models
Ze-Feng Gao*, Peiyu Liu*, Wayne Xin Zhao#, Zhong-Yi Lu, Ji-Rong Wen
COLING 2022, Oral Presentation, 2022
paper /
arxiv /
code /
In this paper, we can reduce the parameters of the original MoE architecture by sharing a global central tensor across experts and keeping expert-specific auxiliary tensors. We also design the gradient mask strategy for the tensor structure of MPO to alleviate the overfitting problem.
|
|
Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
Peiyu Liu*, Ze-Feng Gao*, Wayne Xin Zhao#, Z.Y. Xie, Zhong-Yi Lu#, Ji-Rong Wen
ACL 2021 main conference, 2021
paper /
arxiv /
code /
slides /
link /
This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics.
|
|
Compressing LSTM Networks by Matrix Product Operators
Ze-Feng Gao*, Xingwei Sun*, Lan Gao, Junfeng Li#, Zhong-Yi Lu#
Preprint, 2020
paper /
arxiv /
We propose an alternative LSTM model to reduce the number of parameters significantly by representing the weight parameters based on matrix product operators (MPO), which are used to characterize the local correlation in quantum states in physics.
|
|
A Model Compression Method With Matrix Product Operators for Speech Enhancement
Xingwei Sun*, Ze-Feng Gao*, Zhong-Yi Lu#, Junfeng Li#, Yonghong Yan
IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 2837-2847, 2020
paper /
arxiv /
link /
In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement.
|
|
Compressing deep neural networks by matrix product operators
Ze-Feng Gao*,Song Cheng*, Rong-Qiang He, Zhi-Yuan Xie#, Hui-Hai Zhao#, Zhong-Yi Lu#, Tao Xiang#
Physical Review Research 2 (2), 023300, 2020
paper /
arxiv /
code /
link /
In this paper, we show that neural network can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states.
|
* Equal contribution           # Corresponding author
Professional Services
Reviewer: WSDM2022, ACCV2022, COLING2022
|
Selected Awards and Honors
Outstanding graduates, Renmin University of China, 2021
National Scholarship for Graduate Student, Ministry of Education of P.R.China, 2020
First class academic scholarship, Renmin University of China, 2020
Social Volunteer Service Scholarship,Renmin University of China, 2019
First class academic scholarship, Renmin University of China, 2019
First class academic scholarship, Renmin University of China, 2018
First class academic scholarship, Renmin University of China, 2016
Innovation experiment plan for college students, national level, 2013
|
|