Article Details

Abstract

In order to extract valuable information from Web pages efficiently and accurately, this paper proposes a Web content parsing method based on Deep Learning. This method aims to extract text information from complex Hyper Text Markup Language (HTML). This method combines the feature extraction ability of Deep Learning, Natural Language Processing technology and layout information in HTML documents to construct a Multi-Layer Neural Network model, so as to realize the recognition of Web content. The experimental results show that compared with the traditional Web content extraction method based on text density, this method has obvious advantages in accuracy, adaptability and robustness.

Keywords

Web content parsing; Deep Learning; Neural Network; adaptability

References

1. [1] BARBARESI A. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction [C]//In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations.Online：Association for Computational Linguistics，2021：122-131.
2. [2] Zheng Zhijian, Yu Furen, Wei Xiaowei, etc Design and Implementation of Job Website Crawling Based on Python [J]. Computer and Network, 2024, 50 (1): 24-27
3. [3] Chen Yingren, Guo Yingnan, Guo Xiang, etc Adaptive web wrapper based on feature similarity calculation [J]. Computer Science, 2021, 48 (S2): 218-224+257
4. [4] SONG R H，LIU H F，WEN J R，et al. Learning Block Importance Models for Web Pages [C]//Proceedings of the 13th international conference on World Wide Web.ACM，2004：203-211.
5. [5] Lv Fang Research on Similarity Calculation Technology of Phishing Websites Based on Visual Features [D]. Harbin: Harbin Institute of Technology, 2015
6. [6] Shen Yitao Research on Automatic Abstraction Technology for Chinese Web Pages Based on Visual Features and Text Structure Analysis [D]. Shanghai: East China Normal University, 2014
7. [7] Zhang Xin, Chen Mei, Wang Hanhu, etc Web Information Extraction Based on Visual Features and Domain Ontology [J]. Computer Technology and Development, 2011, 21 (2): 58-61+65
8. [8] Hong Honghui, Ding Shitao, Huang Ao, etc Web Text Extraction Method Based on Text and Symbol Density [J]. Electronic Design Engineering, 2019, 27 (8): 133-137
9. [9] Yang Dawei, Wang Shinian, Bao Liyan, etc Web Text Extraction Based on Text and HTML Tag Density [J]. Journal of Shenyang University of Technology, 2022, 41 (4): 14-19
10. [10] CONNEAU A，KHANDELWAL K，GOYAL N，et al. Unsupervised Cross-lingual Representation Learning at Scale [J/OL].arXiv:1911.02116 [cs.CL].[2024-10-18].https://arxiv.org/abs/1911.02116?file=1911.02116.

Funding

National Natural Science Foundation of China - Scene Intelligent Perception and Abnormal Behavior Warning for Public Safety (U20B2069)

About the authors

Yuan Gongping (1992-), male, Han nationality, from Rongjiang, Guizhou, engineer, master's student, research direction: data collection, deep learning, machine learning; Xie Hongtao (1992-), male, Han nationality, from Zunyi, Guizhou, engineer, master's student, research direction: machine learning; Shu Juelin (1992-), male, Han ethnicity, from Zhenyuan, Guizhou. He is an engineer and a master's student with research interests in big data and machine learning; Zhou Wei (1992-), male, Han ethnicity, born in Jingzhou, Hubei, is an engineer and a master's student. His research interests include statistical analysis, deep learning, and machine learning.