Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
An efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt, with a novel post-scraping block that removes boilerplate and line duplicates prior to filtering.
Gonçalo Vinagre, Rui Guerra, Pedro Gomes et al.
· 0 citations