Back to Projects

How Do Italian Varieties Behave in Embedding Spaces?

A benchmark of six embedding families across 17 language varieties. Character TF-IDF outperformed pretrained Transformers in recovering linguistic relationships (Spearman ρ = 0.905).

Python NLP Embeddings Transformers
View Code Paper

Research Question

Do text embeddings reflect linguistic relationships between Italian varieties? Developed for Bocconi University's Language Technology course, this project compares six model families across Friulian, Ligurian, Lombard, Sardinian, Sicilian, Venetian, and eleven standard languages.

Methods and Data

The experiments compare TF-IDF, FastText, and Word2Vec with the pretrained encoders XLM-R, CANINE, and LaBSE. Wikipedia text and OLDI parallel data support training; aligned FLORES+ sentences provide a shared evaluation set.

Experiments vary text normalization and model adaptation to examine how these choices affect the resulting representation of each variety.

Evaluation

  • Compare embedding distances with a lexicostatistical reference using Spearman correlation.
  • Estimate uncertainty with bootstrap confidence intervals and test distance-matrix relationships with Mantel permutations.
  • Inspect nearest neighbors, clustering, and visual projections.
  • Evaluate ChatGPT and Gemini dialect translations using BLEU, chrF, and paired bootstrap comparisons.

The repository provides data preparation scripts, experiment runners, and a common evaluation pipeline for reproducing the analyses.