Article Details

Abstract

Aiming at the non-convex re-representation of the original convex problem in metric learning, ML_NSVR is proposed as a non-convex stochastic optimization algorithm with variance reduction strategy to solve non-convex metric learning problems more efficiently. Under appropriate initial selection, the algorithm can converge under restricted strongly convex conditions, and its convergence is proved theoretically. The effectiveness of the algorithm is verified by numerical experiments, which provides a new idea for solving non-convex metric learning problems. At the same time, this paper compares the optimization effect of the proposed algorithm with the traditional Stochastic Gradient Descent (SGD) algorithm and the Stochastic Variance Reduction Gradient (SVRG) algorithm.

Keywords

stochastic optimization; metric learning; variance reduction; non-convex optimization

References

1. [1] COLLET P，RENNARD J P. Stochastic Optimization Algorithms [M]//Intelligent information technologies：Concepts，methodologies，tools，and applications，2008：1121-1137.
2. [2] LI B，JIANG W S. A Novel Stochastic Optimization Algorithm [J].IEEE Transactions on Systems，Man，and Cybernetics，Part B （Cybernetics），2000，30（1）：193-198.
3. [3] FEYZMAHDAVIAN H R，AYTEKIN A，JOHANSSON M. An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization [J].IEEE Transactions on Automatic Control，2016，61（12）：3740-3754.
4. [4] ZENG J S，MA K，YAO Y. Finding Global Optima in Nonconvex Stochastic Semidefinite Optimization with Variance Reduction [J/OL].arXiv:1802.06232 [math.OC].[2024-09-28].https://arxiv.org/abs/1802.06232.
5. [5] KAYA M，BILGE H Ş. Deep Metric Learning: A Survey [J/OL].Symmetry，2019，11（9）：1066[2024-09-28].https://doi.org/10.3390/sym11091066.
6. [6] YANG L，JIN R. Distance Metric Learning: A Comprehensive Survey [J/OL].Michigan State Universiy，2006，2（2）：4[2024-09-28].https://www.cs.cmu.edu/afs/cs.cmu.edu/Web/People/liuy/frame_survey_v2.
7. [7] KULIS B. Metric Learning: A Survey [J].Foundations and Trends in Machine Learning，2012，5（4）：287-364.
8. [8] WANG R，WU X J，CHEN Z H，et al. SPD Manifold Deep Metric Learning for Image Set Classification [J].IEEE Transactions on Neural Networks and Learning Systems，2024，35（7）：8924-8938.
9. [9] ZHANG Q，ZHOU Y，PRATER-BENNETTE A，et al. Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization [J/OL].arXiv:2404.01200 [stat.ML].[2024-09-28].https://arxiv.org/abs/2404.01200?context=stat.ML.
10. [10] LI H C，QIAN J，TIAN Y，et al. Convex and Non-Convex Optimization under Generalized Smoothness [J/OL].arXiv:2306.01264 [math.OC].[2024-09-28].https://arxiv.org/abs/2306.01264.
11. [11] XIE C H，LI C X，ZHANG C W，et al. Trust Region Methods for Nonconvex Stochastic Optimization Beyond Lipschitz Smoothness [J/OL].arXiv:2310.17319 [math.OC].[2024-09-28].https://arxiv.org/abs/2310.17319.
12. [12] HU E L，KWOK J T. Low-Rank Matrix Learning Using Biconvex Surrogate inimization [J].IEEE Transactions on Neural Networks and Learning Systems，2019，30（11）：3517-3527.
13. [13] NESTEROV Y. Smooth Minimization of Non-Smooth Functions [J].Mathematical programming，2005，103（1）：127-152.
14. [14] ERDOGDU M A，OZDAGLAR A，PARRILO P A，et al. Convergence Rate of Block-Coordinate Maximization Burer–Monteiro Method for Solving Large SDPs [J/OL].arXiv:1807.04428[math.OC].[2024-09-28].https://arxiv.org/abs/1807.04428?context=cs.LG.

Funding

Key Laboratory of Modern Analytical Mathematics and Its Applications in Yunnan Province (202302AN360007); National Natural Science Project (62266055)

About the authors

Xu Zhongcheng (1999-), male, Han ethnicity, from Qujing, Yunnan, is a master's student with a research focus on machine learning.