Unlocking Transitional Chinese: Word Segmentation in Modern Historical Texts - Régulations : marchés, populations, circulations Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Unlocking Transitional Chinese: Word Segmentation in Modern Historical Texts

Résumé

This research addresses Natural Language Processing (NLP) tokenization challenges for transitional Chinese, which lacks adequate digital resources. The project used a collection of articles from the Shenbao, a newspaper from this period, as their study base. They designed models tailored to transitional Chinese, with goals like historical information extraction, large-scale textual analysis, and creating new datasets for computational linguists. The team manually tokenized historical articles to understand the language’s linguistic patterns, syntactic structures, and lexical variations. They developed a custom model tailored to their dataset after evaluating various word segmentation tools. They also studied the impact of using pre-trained language models on historical data. The results showed that using language models aligned with the source languages resulted in superior performance. They assert that transitional Chinese they are processing is more related to ancient Chinese than contemporary Chinese, necessitating the training of language models specifically on their data. The study’s outcome is a model that achieves a performance of over 83% and an F-score that is 35% higher than using existing tokenization tools, signifying a substantial improvement. The availability of this new annotated dataset paves the way for refining the model’s performance in processing this type of data.
Fichier principal
Vignette du fichier
NLP4DH-2.pdf (135.64 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04618218 , version 1 (21-06-2024)

Identifiants

  • HAL Id : hal-04618218 , version 1

Citer

Baptiste Blouin, Hen-Hsen Huang, Christian Henriot, Cécile Armand. Unlocking Transitional Chinese: Word Segmentation in Modern Historical Texts. The Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages, Dec 2023, Tokyo (Japan), Japan. ⟨hal-04618218⟩
0 Consultations
0 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More