feat(eda): capítulo text_distr (TEXTO/NLP) — primer capítulo de datos no tabulares
Añade el capítulo `text_distr` al motor AutomaticEDA: perfila columnas de texto libre largo (reseñas, descripciones, comentarios) que la distribución categórica no resume bien. Sigue el patrón de cat_distr/num_distr (build_text_distr(profile, ctx) -> Chapter | None) y se registra en CHAPTER_ORDER tras cat_distr. Activación en dos fases: gate barato desde el perfil (columna no numérica con len_mean >= 50 chars) + confirmación con muestra cruda (mediana de palabras >= 20). Un dataset sin texto largo (p.ej. titanic) devuelve None sin tocar el informe. Bloques por columna (Group con page_break): resumen (longitudes, vocabulario con TTR y % hapax, idioma dominante, % duplicados, legibilidad), histograma de longitudes, top términos (tabla + barras), bigramas/trigramas, idiomas detectados y nube de palabras opcional. Términos ttr/hapax enganchados al glosario clicable. Lógica delegada a 7 funciones nuevas del registry (datascience, tag eda), estilo dict-no-throw: - extract_text_sample (impura, push-down SQL DuckDB/Postgres) - compute_text_length_stats, compute_vocabulary_stats, compute_top_ngrams (puras, stdlib) - detect_corpus_language (langdetect opcional), compute_text_readability (textstat opcional), compute_text_duplicates (hash + datasketch opcional) Versión barata sin modelos pesados: las piezas que dependen de una librería opcional (langdetect, textstat, wordcloud, datasketch) degradan a omitidas sin lanzar. Añade langdetect y textstat (ligeras) al pyproject + uv.lock. Verificado: golden sobre dataset de reviews multi-idioma (capítulo presente en PDF+PPTX+MD con métricas reales), titanic sin capítulo (None), degradación sin libs, suite automatic_eda + pipeline verde (128 passed), fn index OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,559 @@
|
||||
"""Free-text / NLP distributions chapter (TEXT DISTR) for AutomaticEDA.
|
||||
|
||||
First chapter for **non-tabular** content: it profiles the linguistic content of
|
||||
any column holding long free text (reviews, descriptions, comments, tickets) that
|
||||
the categorical chapter cannot meaningfully summarize (high cardinality, many
|
||||
words per value). It is the cheap, model-free counterpart to ``cat_distr`` for
|
||||
columns that are prose rather than discrete labels.
|
||||
|
||||
Activation (returns ``None`` when it does not apply):
|
||||
|
||||
1. Cheap gate from the aggregated profile: at least one non-numeric column whose
|
||||
``categorical.len_mean`` (mean character length) is ``>= _MIN_LEN_CHARS``.
|
||||
A dataset whose only string columns are short labels (e.g. titanic's
|
||||
``Name``, ~27 chars) never passes this gate, so the chapter disappears with
|
||||
zero extra work and the existing report is untouched.
|
||||
2. Confirmation from a raw sample: each candidate column is sampled (push-down
|
||||
``extract_text_sample`` over ``ctx['db_path']``/``ctx['table']``, or an
|
||||
in-memory ``ctx['text_raw']`` for tests) and kept only if the **median word
|
||||
count is ``>= _MIN_WORDS``** — i.e. it is genuinely long text, not a long
|
||||
single token. If no column survives, the chapter returns ``None``.
|
||||
|
||||
Per surviving column the chapter emits, kept together on its own page/slide
|
||||
(``Group(page_break_before=...)``):
|
||||
|
||||
- a key/value summary (documents, length percentiles, vocabulary richness with
|
||||
**[[term:ttr]]TTR[[/term]]** and **[[term:hapax]]hapax legomena[[/term]]**,
|
||||
dominant language, exact-duplicate %, readability when available);
|
||||
- a word-count histogram figure;
|
||||
- a top-terms table + a horizontal bar figure;
|
||||
- bigram and trigram frequency tables;
|
||||
- a detected-language bar figure (when ``langdetect`` is available);
|
||||
- an optional word-cloud figure (only when ``wordcloud`` is installed);
|
||||
- a closing note on duplicates / readability degradation.
|
||||
|
||||
Every metric is delegated to pure ``eda`` registry functions
|
||||
(``compute_text_length_stats``, ``compute_vocabulary_stats``,
|
||||
``compute_top_ngrams``, ``detect_corpus_language``, ``compute_text_duplicates``,
|
||||
``compute_text_readability``) and the raw sample to ``extract_text_sample``; all
|
||||
are imported defensively so a missing function or optional library degrades that
|
||||
single piece to a note instead of aborting the chapter. Optional libraries
|
||||
(``langdetect``, ``textstat``, ``wordcloud``, ``datasketch``) are never required:
|
||||
the piece is silently omitted when they are absent.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "text_distr"
|
||||
CHAPTER_TITLE = "Texto libre (NLP)"
|
||||
|
||||
# Cheap activation gate (characters): a non-numeric column whose mean string
|
||||
# length reaches this is a candidate for "long text". Short labels (titanic's
|
||||
# Name ≈ 27 chars) stay below it, so the chapter does not fire on them.
|
||||
_MIN_LEN_CHARS = 50
|
||||
# Confirmation gate (words): a candidate is kept only if its median document has
|
||||
# at least this many words — genuine prose, not a long id/URL token.
|
||||
_MIN_WORDS = 20
|
||||
# Bound the document so very wide datasets stay readable.
|
||||
_MAX_TEXT_COLS = 5
|
||||
# Raw text rows to sample per column when the chapter must extract them itself.
|
||||
_SAMPLE_ROWS = 2000
|
||||
# Rows shown in the frequency tables.
|
||||
_TOP_TERMS = 15
|
||||
_TOP_NGRAMS = 10
|
||||
|
||||
# Glossary terms this chapter explains (registered in the shared collector and
|
||||
# marked clickable on first appearance — same mechanism as cat_distr's entropía).
|
||||
_TERMS = {
|
||||
"ttr": (
|
||||
"TTR (type-token ratio)",
|
||||
"Riqueza léxica de un texto: número de palabras distintas (tipos) "
|
||||
"dividido por el número total de palabras (tokens). Vale 1 cuando no se "
|
||||
"repite ninguna palabra (máxima variedad) y baja hacia 0 cuando el "
|
||||
"vocabulario se repite mucho. Depende de la longitud del corpus, así que "
|
||||
"compara mejor textos de tamaño parecido."),
|
||||
"hapax": (
|
||||
"Hapax legomena",
|
||||
"Palabras que aparecen una sola vez en todo el corpus. Un porcentaje "
|
||||
"alto de hapax indica vocabulario muy variado o, a veces, ruido "
|
||||
"(erratas, identificadores, tokens raros). Se expresa como porcentaje "
|
||||
"sobre el número de palabras distintas."),
|
||||
}
|
||||
|
||||
|
||||
def _fmt_int(value) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
return f"{int(value):,}".replace(",", ".")
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
|
||||
|
||||
def _fmt_num(value, decimals: int = 2) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return str(value)
|
||||
if isinstance(value, int):
|
||||
return f"{value:,}".replace(",", ".")
|
||||
if isinstance(value, float):
|
||||
if value != value: # NaN
|
||||
return "NaN"
|
||||
if value in (float("inf"), float("-inf")):
|
||||
return str(value)
|
||||
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
|
||||
return text if text else "0"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _fmt_pct(value, decimals: int = 1) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
return f"{float(value):.{decimals}f}%"
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
|
||||
|
||||
def _truncate(text, limit: int = 40) -> str:
|
||||
s = model._safe_str(text)
|
||||
return s if len(s) <= limit else s[: max(1, limit - 1)].rstrip() + "…"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Defensive wrappers around the registry functions: each returns the function's
|
||||
# output dict or a safe empty default, never raising and never importing at
|
||||
# module load (so the chapter stays importable even if a function is missing).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _length_stats(texts) -> dict:
|
||||
try:
|
||||
from datascience.compute_text_length_stats import compute_text_length_stats
|
||||
out = compute_text_length_stats(texts)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def _vocab_stats(texts) -> dict:
|
||||
try:
|
||||
from datascience.compute_vocabulary_stats import compute_vocabulary_stats
|
||||
out = compute_vocabulary_stats(texts, top_k=_TOP_TERMS)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def _ngrams(texts, n) -> list:
|
||||
try:
|
||||
from datascience.compute_top_ngrams import compute_top_ngrams
|
||||
out = compute_top_ngrams(texts, n=n, top_k=_TOP_NGRAMS)
|
||||
if isinstance(out, dict):
|
||||
return out.get("top") or []
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return []
|
||||
|
||||
|
||||
def _language(texts) -> dict:
|
||||
try:
|
||||
from datascience.detect_corpus_language import detect_corpus_language
|
||||
out = detect_corpus_language(texts)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return {"available": False, "distribution": [], "dominant": None}
|
||||
|
||||
|
||||
def _duplicates(texts) -> dict:
|
||||
try:
|
||||
from datascience.compute_text_duplicates import compute_text_duplicates
|
||||
out = compute_text_duplicates(texts)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def _readability(texts) -> dict:
|
||||
try:
|
||||
from datascience.compute_text_readability import compute_text_readability
|
||||
out = compute_text_readability(texts)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return {"available": False, "flesch": {}}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Candidate detection + raw sample acquisition.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _candidate_columns(profile: dict) -> list:
|
||||
"""Cheap gate: non-numeric columns whose mean char length reaches the
|
||||
threshold. Returns the list of column names (possibly empty)."""
|
||||
out = []
|
||||
for col in profile.get("columns") or []:
|
||||
if not isinstance(col, dict):
|
||||
continue
|
||||
if col.get("inferred_type") == "numeric":
|
||||
continue
|
||||
cat = col.get("categorical")
|
||||
if not isinstance(cat, dict):
|
||||
continue
|
||||
len_mean = cat.get("len_mean")
|
||||
if isinstance(len_mean, (int, float)) and not isinstance(len_mean, bool) \
|
||||
and len_mean >= _MIN_LEN_CHARS:
|
||||
name = col.get("name")
|
||||
if name:
|
||||
out.append(str(name))
|
||||
return out
|
||||
|
||||
|
||||
def _get_samples(profile: dict, ctx: dict, columns: list) -> dict:
|
||||
"""Return {col: [str, ...]} raw text samples for the candidate columns.
|
||||
|
||||
Prefers an in-memory ``ctx['text_raw']`` (used by tests); otherwise pushes a
|
||||
sample down to the database via ``extract_text_sample`` using ctx db_path /
|
||||
table. Never raises: returns {} when no sample can be obtained."""
|
||||
text_raw = ctx.get("text_raw")
|
||||
if isinstance(text_raw, dict) and text_raw:
|
||||
return {c: [str(v) for v in (text_raw.get(c) or []) if v is not None]
|
||||
for c in columns if text_raw.get(c)}
|
||||
|
||||
db_path = ctx.get("db_path")
|
||||
table = ctx.get("table")
|
||||
if not db_path or not table:
|
||||
return {}
|
||||
backend = ctx.get("backend") or "duckdb"
|
||||
sample = ctx.get("sample") or _SAMPLE_ROWS
|
||||
try:
|
||||
from datascience.extract_text_sample import extract_text_sample
|
||||
out = extract_text_sample(db_path, table, columns, backend=backend,
|
||||
sample=sample)
|
||||
if isinstance(out, dict) and out.get("status") == "ok":
|
||||
cols = out.get("columns")
|
||||
if isinstance(cols, dict):
|
||||
return {c: list(v) for c, v in cols.items() if v}
|
||||
except Exception: # noqa: BLE001 — dict-no-throw: no sample → chapter omits.
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def _confirm_long_text(samples: dict) -> dict:
|
||||
"""Keep only columns whose median word count reaches _MIN_WORDS. Returns
|
||||
{col: length_stats_dict} for the survivors, in input order."""
|
||||
survivors = {}
|
||||
for col, texts in samples.items():
|
||||
stats = _length_stats(texts)
|
||||
words = stats.get("words") if isinstance(stats, dict) else None
|
||||
median = words.get("p50") if isinstance(words, dict) else None
|
||||
if isinstance(median, (int, float)) and not isinstance(median, bool) \
|
||||
and median >= _MIN_WORDS:
|
||||
survivors[col] = stats
|
||||
return survivors
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Figures (lazy matplotlib, scaled by the renderers — same style as num_distr).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _hist_figure(name: str, length_stats: dict):
|
||||
def make():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
from matplotlib.figure import Figure
|
||||
fig = Figure(figsize=(6.2, 3.0))
|
||||
ax = fig.add_subplot(111)
|
||||
bins = (length_stats or {}).get("word_hist") or []
|
||||
drew = False
|
||||
for b in bins:
|
||||
if not isinstance(b, dict):
|
||||
continue
|
||||
lo, hi, count = b.get("lo"), b.get("hi"), b.get("count") or 0
|
||||
if lo is None or hi is None:
|
||||
continue
|
||||
width = (hi - lo) if hi > lo else max(abs(lo) * 1e-3, 1e-6)
|
||||
ax.bar(lo, count, width=width, align="edge", color="#9ec6df",
|
||||
edgecolor="#5b8aa6", linewidth=0.4)
|
||||
drew = True
|
||||
if not drew:
|
||||
ax.text(0.5, 0.5, "(sin datos de longitud)", ha="center",
|
||||
va="center", color="#8a8a8a", transform=ax.transAxes)
|
||||
ax.set_xlabel("palabras por documento", fontsize=8)
|
||||
ax.set_ylabel("nº de documentos", fontsize=8)
|
||||
ax.tick_params(labelsize=7)
|
||||
for spine in ("top", "right"):
|
||||
ax.spines[spine].set_visible(False)
|
||||
ax.set_title(f"Longitud de «{_truncate(name, 30)}»", fontsize=10,
|
||||
loc="left")
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
return make
|
||||
|
||||
|
||||
def _barh_figure(title: str, items: list, label_key: str, value_key: str,
|
||||
xlabel: str):
|
||||
"""Horizontal bar chart from [{label_key:..., value_key:...}, ...]."""
|
||||
def make():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
from matplotlib.figure import Figure
|
||||
rows = [it for it in (items or []) if isinstance(it, dict)
|
||||
and isinstance(it.get(value_key), (int, float))]
|
||||
rows = rows[:12]
|
||||
fig = Figure(figsize=(6.2, max(2.2, 0.32 * len(rows) + 0.8)))
|
||||
ax = fig.add_subplot(111)
|
||||
if not rows:
|
||||
ax.text(0.5, 0.5, "(sin datos)", ha="center", va="center",
|
||||
color="#8a8a8a", transform=ax.transAxes)
|
||||
ax.axis("off")
|
||||
return fig
|
||||
labels = [_truncate(r.get(label_key), 28) for r in rows][::-1]
|
||||
values = [float(r.get(value_key) or 0) for r in rows][::-1]
|
||||
ypos = range(len(rows))
|
||||
ax.barh(list(ypos), values, color="#9ec6df", edgecolor="#5b8aa6",
|
||||
linewidth=0.4)
|
||||
ax.set_yticks(list(ypos))
|
||||
ax.set_yticklabels(labels, fontsize=7)
|
||||
ax.set_xlabel(xlabel, fontsize=8)
|
||||
ax.tick_params(labelsize=7)
|
||||
for spine in ("top", "right"):
|
||||
ax.spines[spine].set_visible(False)
|
||||
ax.set_title(_truncate(title, 44), fontsize=10, loc="left")
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
return make
|
||||
|
||||
|
||||
def _wordcloud_figure(texts):
|
||||
"""Word-cloud figure callable, or None if wordcloud is not installed."""
|
||||
try:
|
||||
import wordcloud # noqa: F401
|
||||
except Exception: # noqa: BLE001 — optional dependency: omit the figure.
|
||||
return None
|
||||
|
||||
def make():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
from matplotlib.figure import Figure
|
||||
from wordcloud import WordCloud
|
||||
fig = Figure(figsize=(6.2, 3.2))
|
||||
ax = fig.add_subplot(111)
|
||||
joined = " ".join(t for t in texts if isinstance(t, str))
|
||||
try:
|
||||
wc = WordCloud(width=800, height=400, background_color="white",
|
||||
colormap="viridis").generate(joined)
|
||||
ax.imshow(wc, interpolation="bilinear")
|
||||
except Exception: # noqa: BLE001
|
||||
ax.text(0.5, 0.5, "(nube de palabras no disponible)", ha="center",
|
||||
va="center", color="#8a8a8a", transform=ax.transAxes)
|
||||
ax.axis("off")
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
return make
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Per-column block assembly.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _summary_kv(n_docs, length_stats, vocab, lang, dup, read):
|
||||
chars = (length_stats or {}).get("chars") or {}
|
||||
words = (length_stats or {}).get("words") or {}
|
||||
sents = (length_stats or {}).get("sentences") or {}
|
||||
rows = [
|
||||
("Documentos", _fmt_int(n_docs)),
|
||||
("Caracteres (media · p50 · p90 · p99)",
|
||||
f"{_fmt_num(chars.get('mean'))} · {_fmt_int(chars.get('p50'))} · "
|
||||
f"{_fmt_int(chars.get('p90'))} · {_fmt_int(chars.get('p99'))}"),
|
||||
("Palabras (media · p50 · p90 · p99)",
|
||||
f"{_fmt_num(words.get('mean'))} · {_fmt_int(words.get('p50'))} · "
|
||||
f"{_fmt_int(words.get('p90'))} · {_fmt_int(words.get('p99'))}"),
|
||||
("Frases (media · máx)",
|
||||
f"{_fmt_num(sents.get('mean'))} · {_fmt_int(sents.get('max'))}"),
|
||||
("Vocabulario (tokens · tipos · TTR)",
|
||||
f"{_fmt_int(vocab.get('n_tokens'))} · {_fmt_int(vocab.get('n_types'))} "
|
||||
f"· {_fmt_num(vocab.get('ttr'), 3)}"),
|
||||
("Hapax legomena",
|
||||
f"{_fmt_int(vocab.get('n_hapax'))} ({_fmt_pct(vocab.get('hapax_pct'))})"),
|
||||
]
|
||||
if isinstance(lang, dict) and lang.get("available"):
|
||||
dom = lang.get("dominant")
|
||||
n_langs = len(lang.get("distribution") or [])
|
||||
rows.append(("Idioma dominante · nº idiomas",
|
||||
f"{model._safe_str(dom) or '—'} · {_fmt_int(n_langs)}"))
|
||||
if isinstance(dup, dict) and dup.get("n_docs"):
|
||||
rows.append(("Duplicados exactos",
|
||||
f"{_fmt_int(dup.get('n_exact_dup'))} "
|
||||
f"({_fmt_pct(dup.get('exact_dup_pct'))})"))
|
||||
if isinstance(read, dict) and read.get("available"):
|
||||
flesch = read.get("flesch") or {}
|
||||
rows.append(("Legibilidad Flesch (media)",
|
||||
_fmt_num(flesch.get("mean"), 1)))
|
||||
return model.KVTable(rows=rows, title="Resumen del texto")
|
||||
|
||||
|
||||
def _terms_table(vocab) -> "model.DataTable | None":
|
||||
top = (vocab or {}).get("top_terms") or []
|
||||
rows = [[_truncate(t.get("term"), 32), _fmt_int(t.get("count")),
|
||||
_fmt_pct(t.get("pct"))]
|
||||
for t in top[:_TOP_TERMS] if isinstance(t, dict)]
|
||||
if not rows:
|
||||
return None
|
||||
return model.DataTable(header=["Término", "Conteo", "% tokens"], rows=rows,
|
||||
title="Términos más frecuentes",
|
||||
note="stopwords ES+EN eliminadas")
|
||||
|
||||
|
||||
def _ngram_table(items, n_label) -> "model.DataTable | None":
|
||||
rows = [[_truncate(it.get("ngram"), 40), _fmt_int(it.get("count"))]
|
||||
for it in (items or [])[:_TOP_NGRAMS] if isinstance(it, dict)]
|
||||
if not rows:
|
||||
return None
|
||||
return model.DataTable(header=[n_label, "Conteo"], rows=rows,
|
||||
title=f"{n_label} más frecuentes")
|
||||
|
||||
|
||||
def _dup_note(dup, lang, read) -> "model.Note | None":
|
||||
bits = []
|
||||
if isinstance(dup, dict):
|
||||
nd = dup.get("near_dup") or {}
|
||||
if nd.get("available"):
|
||||
bits.append(
|
||||
f"casi-duplicados detectados (MinHash, umbral "
|
||||
f"{_fmt_num(nd.get('threshold'))}): "
|
||||
f"{_fmt_int(nd.get('n_near_dup_docs'))} documentos")
|
||||
else:
|
||||
bits.append("near-duplicados no calculados (datasketch no instalado; "
|
||||
"se reportan solo los duplicados exactos por hash)")
|
||||
if isinstance(lang, dict) and not lang.get("available"):
|
||||
bits.append("detección de idioma omitida (langdetect no instalado)")
|
||||
if isinstance(read, dict) and not read.get("available"):
|
||||
bits.append("legibilidad omitida (textstat no instalado)")
|
||||
if not bits:
|
||||
return None
|
||||
return model.Note(" · ".join(bits))
|
||||
|
||||
|
||||
def _column_group(name, texts, length_stats, idx, mark_terms):
|
||||
vocab = _vocab_stats(texts)
|
||||
lang = _language(texts)
|
||||
dup = _duplicates(texts)
|
||||
read = _readability(texts)
|
||||
n_docs = (length_stats or {}).get("n_docs")
|
||||
|
||||
blocks = [
|
||||
model.Heading(text=str(name), level=2),
|
||||
_summary_kv(n_docs, length_stats, vocab, lang, dup, read),
|
||||
model.Figure(make=_hist_figure(name, length_stats),
|
||||
caption=f"Distribución de la longitud (palabras) de "
|
||||
f"«{_truncate(name, 30)}»."),
|
||||
]
|
||||
|
||||
terms_tbl = _terms_table(vocab)
|
||||
if terms_tbl is not None:
|
||||
blocks.append(terms_tbl)
|
||||
blocks.append(model.Figure(
|
||||
make=_barh_figure(f"Top términos de «{_truncate(name, 24)}»",
|
||||
vocab.get("top_terms"), "term", "count",
|
||||
"conteo"),
|
||||
caption="Términos más frecuentes (barras)."))
|
||||
|
||||
bi_tbl = _ngram_table(_ngrams(texts, 2), "Bigrama")
|
||||
if bi_tbl is not None:
|
||||
blocks.append(bi_tbl)
|
||||
tri_tbl = _ngram_table(_ngrams(texts, 3), "Trigrama")
|
||||
if tri_tbl is not None:
|
||||
blocks.append(tri_tbl)
|
||||
|
||||
if isinstance(lang, dict) and lang.get("available") \
|
||||
and lang.get("distribution"):
|
||||
blocks.append(model.Figure(
|
||||
make=_barh_figure(f"Idiomas detectados en «{_truncate(name, 24)}»",
|
||||
lang.get("distribution"), "lang", "count",
|
||||
"documentos"),
|
||||
caption="Distribución de idiomas detectados (langdetect)."))
|
||||
|
||||
wc = _wordcloud_figure(texts)
|
||||
if wc is not None:
|
||||
blocks.append(model.Figure(
|
||||
make=wc, caption=f"Nube de palabras de «{_truncate(name, 30)}»."))
|
||||
|
||||
note = _dup_note(dup, lang, read)
|
||||
if note is not None:
|
||||
blocks.append(note)
|
||||
|
||||
return model.Group(blocks=blocks, page_break_before=(idx > 0))
|
||||
|
||||
|
||||
def _intro_blocks(n_cols, mark_terms):
|
||||
ttr = ("[[term:ttr]]TTR[[/term]]" if mark_terms else "TTR")
|
||||
hapax = ("[[term:hapax]]hapax legomena[[/term]]" if mark_terms
|
||||
else "hapax legomena")
|
||||
text = (
|
||||
f"Este capítulo perfila las columnas de **texto libre largo** del "
|
||||
f"dataset (reseñas, descripciones, comentarios): contenido lingüístico "
|
||||
f"que la distribución categórica no resume bien. Para cada columna se "
|
||||
f"muestran la longitud de los documentos, la riqueza de vocabulario "
|
||||
f"(incluido el {ttr} y el porcentaje de {hapax}), los términos y "
|
||||
f"n-gramas más frecuentes, los idiomas detectados y el nivel de "
|
||||
f"duplicación. Las métricas son baratas y sin modelos pesados; las "
|
||||
f"piezas que dependen de una librería opcional se omiten si no está "
|
||||
f"instalada.")
|
||||
return [
|
||||
model.Heading(text=CHAPTER_TITLE, level=1),
|
||||
model.Markdown(text=text),
|
||||
]
|
||||
|
||||
|
||||
def build_text_distr(profile: dict, ctx: dict):
|
||||
"""Build the free-text Chapter, or None if no long-text column applies."""
|
||||
profile = profile or {}
|
||||
ctx = ctx or {}
|
||||
|
||||
# 1) Cheap gate from the profile (no DB access yet).
|
||||
candidates = _candidate_columns(profile)
|
||||
if not candidates:
|
||||
return None
|
||||
|
||||
# 2) Raw sample + 3) confirm genuine long text (median words >= threshold).
|
||||
samples = _get_samples(profile, ctx, candidates)
|
||||
if not samples:
|
||||
return None
|
||||
survivors = _confirm_long_text(samples)
|
||||
if not survivors:
|
||||
return None
|
||||
|
||||
# Register glossary terms (clickable) once we know the chapter applies.
|
||||
glossary = ctx.get("glossary")
|
||||
mark_terms = False
|
||||
if isinstance(glossary, model.GlossaryCollector):
|
||||
for key, (label, definition) in _TERMS.items():
|
||||
glossary.add(key, label, definition)
|
||||
mark_terms = True
|
||||
|
||||
blocks = list(_intro_blocks(len(survivors), mark_terms))
|
||||
|
||||
rendered = list(survivors.items())[:_MAX_TEXT_COLS]
|
||||
for idx, (name, length_stats) in enumerate(rendered):
|
||||
texts = samples.get(name) or []
|
||||
blocks.append(_column_group(name, texts, length_stats, idx, mark_terms))
|
||||
|
||||
if len(survivors) > len(rendered):
|
||||
omitted = len(survivors) - len(rendered)
|
||||
blocks.append(model.Note(
|
||||
f"Se muestran las primeras {len(rendered)} columnas de texto; "
|
||||
f"quedan {omitted} sin mostrar para mantener acotado el informe."))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Tests for the TEXT DISTR chapter — DoD: golden + edges + degradation.
|
||||
|
||||
Self-contained: builds synthetic TableProfiles and feeds the raw text sample
|
||||
in-memory through ``ctx['text_raw']`` (no DuckDB needed), so the suite is fast
|
||||
and deterministic. Verifies that ``build_text_distr``:
|
||||
|
||||
- GOLDEN: with a long-text column, emits the chapter with its key blocks
|
||||
(length summary, word histogram, top-terms table, n-gram tables, language
|
||||
bars) and registers the clickable glossary terms; and that it renders inside
|
||||
the full document to both PDF and PPTX showing that content.
|
||||
- EDGE (None): a dataset whose only string column is short labels (titanic-like
|
||||
``Name``) yields ``None`` without raising — the existing report is untouched.
|
||||
- EDGE (None): a column that passes the cheap char gate but whose documents are
|
||||
short (median words below the threshold) is rejected at the confirmation step.
|
||||
- DEGRADATION: with ``langdetect`` / ``textstat`` / ``wordcloud`` unavailable,
|
||||
the chapter still builds (those pieces are omitted) and never raises.
|
||||
"""
|
||||
|
||||
import builtins
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
from pptx import Presentation
|
||||
|
||||
from datascience.automatic_eda.model import (
|
||||
DataTable, Figure, GlossaryCollector, Group, Heading, KVTable, Markdown,
|
||||
Note,
|
||||
)
|
||||
from datascience.automatic_eda.chapters.text_distr import (
|
||||
CHAPTER_ID, CHAPTER_VERSION, build_text_distr,
|
||||
)
|
||||
from datascience.automatic_eda.chapters_registry import build_document
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Synthetic corpus + profiles.
|
||||
# --------------------------------------------------------------------------- #
|
||||
_ES = [
|
||||
"El producto llegó en perfecto estado y mucho antes de lo previsto por la tienda",
|
||||
"La calidad de los materiales es realmente excelente y se nota la diferencia al usarlo",
|
||||
"No me convenció del todo porque esperaba bastante más por el precio que pagué finalmente",
|
||||
"El servicio de atención al cliente fue rápido amable y resolvió mi problema sin demora",
|
||||
"Lo recomiendo totalmente ya que ha superado con creces todas mis expectativas iniciales",
|
||||
]
|
||||
_EN = [
|
||||
"The product arrived in perfect condition and much earlier than the store had promised me",
|
||||
"The build quality is genuinely outstanding and you can really feel the difference using it",
|
||||
"I was not fully convinced because I expected quite a lot more for the price i finally paid",
|
||||
"Customer support was fast friendly and solved my whole problem without any delay at all",
|
||||
"I highly recommend it since it has exceeded by far every one of my initial expectations",
|
||||
]
|
||||
|
||||
|
||||
def _long_reviews(n=40) -> list:
|
||||
"""A corpus of long multi-sentence reviews (>= 20 words each), mixing two
|
||||
languages and including a few exact duplicates."""
|
||||
out = []
|
||||
for i in range(n):
|
||||
base = _ES if i % 3 != 0 else _EN # mostly ES, some EN
|
||||
a = base[i % len(base)]
|
||||
b = base[(i + 2) % len(base)]
|
||||
out.append(f"{a}. {b}.")
|
||||
# Inject a couple of exact duplicates.
|
||||
out.append(out[0])
|
||||
out.append(out[1])
|
||||
return out
|
||||
|
||||
|
||||
def _text_profile() -> dict:
|
||||
"""Profile with a long free-text column (review) + a numeric + a short cat."""
|
||||
return {
|
||||
"table": "reviews",
|
||||
"source": "/data/reviews.duckdb",
|
||||
"profiled_at": "2026-06-30T10:00:00+00:00",
|
||||
"n_rows": 42,
|
||||
"n_cols": 3,
|
||||
"quality_score": 88.0,
|
||||
"columns": [
|
||||
{
|
||||
"name": "review",
|
||||
"inferred_type": "categorical",
|
||||
"categorical": {
|
||||
"top": [{"value": "x", "count": 2, "pct": 0.05}],
|
||||
"n_distinct": 40,
|
||||
"len_mean": 180.0,
|
||||
"len_min": 80,
|
||||
"len_max": 220,
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "rating",
|
||||
"inferred_type": "numeric",
|
||||
"numeric": {"mean": 3.1, "median": 3.0, "std": 1.2,
|
||||
"min": 1, "max": 5},
|
||||
},
|
||||
{
|
||||
"name": "product",
|
||||
"inferred_type": "categorical",
|
||||
"categorical": {
|
||||
"top": [{"value": "teclado", "count": 10, "pct": 0.25}],
|
||||
"n_distinct": 6,
|
||||
"len_mean": 7.0,
|
||||
"len_min": 5, "len_max": 11,
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _no_text_profile() -> dict:
|
||||
"""titanic-like: the only string column is short labels (Name ≈ 27 chars)."""
|
||||
return {
|
||||
"table": "titanic",
|
||||
"n_rows": 891,
|
||||
"n_cols": 3,
|
||||
"columns": [
|
||||
{"name": "Age", "inferred_type": "numeric",
|
||||
"numeric": {"mean": 29.7, "median": 28.0, "std": 14.5}},
|
||||
{"name": "Name", "inferred_type": "categorical",
|
||||
"categorical": {"top": [{"value": "Braund, Mr. Owen Harris",
|
||||
"count": 1, "pct": 0.001}],
|
||||
"n_distinct": 891, "len_mean": 27.0,
|
||||
"len_min": 12, "len_max": 82}},
|
||||
{"name": "Sex", "inferred_type": "categorical",
|
||||
"categorical": {"top": [{"value": "male", "count": 577,
|
||||
"pct": 0.65}],
|
||||
"n_distinct": 2, "len_mean": 4.6,
|
||||
"len_min": 4, "len_max": 6}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _flatten(blocks) -> list:
|
||||
"""Recursively flatten Group blocks so tests can inspect leaf blocks."""
|
||||
out = []
|
||||
for b in blocks:
|
||||
if isinstance(b, Group):
|
||||
out.extend(_flatten(b.blocks))
|
||||
else:
|
||||
out.append(b)
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Golden.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_golden_activa_con_texto():
|
||||
glossary = GlossaryCollector()
|
||||
ctx = {"text_raw": {"review": _long_reviews()}, "glossary": glossary}
|
||||
ch = build_text_distr(_text_profile(), ctx)
|
||||
|
||||
assert ch is not None, "el capítulo debe activarse con una columna de texto largo"
|
||||
assert ch.id == CHAPTER_ID
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
leaves = _flatten(ch.blocks)
|
||||
kinds = [b.kind for b in leaves]
|
||||
assert "heading" in kinds
|
||||
assert "kv_table" in kinds # summary
|
||||
assert "figure" in kinds # histogram / bars
|
||||
assert "data_table" in kinds # top terms + n-grams
|
||||
|
||||
# KV summary mentions vocabulary metrics.
|
||||
kv = next(b for b in leaves if isinstance(b, KVTable))
|
||||
labels = " ".join(str(r[0]) for r in kv.rows)
|
||||
assert "TTR" in labels
|
||||
assert "Hapax" in labels or "hapax" in labels
|
||||
|
||||
# There is a terms table and at least one n-gram table.
|
||||
titles = [getattr(b, "title", "") or "" for b in leaves
|
||||
if isinstance(b, DataTable)]
|
||||
assert any("Términos" in t for t in titles)
|
||||
assert any("Bigrama" in t for t in titles)
|
||||
|
||||
# Glossary terms were registered (clickable destinations).
|
||||
assert glossary.has("ttr")
|
||||
assert glossary.has("hapax")
|
||||
|
||||
|
||||
def test_golden_render_pdf_pptx():
|
||||
profile = _text_profile()
|
||||
ctx = {"text_raw": {"review": _long_reviews()},
|
||||
"dataset_name": "reviews"}
|
||||
chapters = build_document(profile, ctx)
|
||||
ids = [c.id for c in chapters]
|
||||
assert "text_distr" in ids, f"text_distr ausente en {ids}"
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "t.pdf")
|
||||
pptx = os.path.join(d, "t.pptx")
|
||||
rp = render_automatic_eda_pdf(profile, pdf, {"title": "EDA", "ctx": ctx})
|
||||
rx = render_automatic_eda_pptx(profile, pptx, {"title": "EDA", "ctx": ctx})
|
||||
assert rp.get("path") and os.path.exists(pdf)
|
||||
assert rx.get("path") and os.path.exists(pptx)
|
||||
|
||||
text = "\n".join(p.extract_text() or "" for p in PdfReader(pdf).pages)
|
||||
assert "Texto libre" in text or "TTR" in text
|
||||
|
||||
prs = Presentation(pptx)
|
||||
ptext = []
|
||||
for slide in prs.slides:
|
||||
for shp in slide.shapes:
|
||||
if shp.has_text_frame:
|
||||
ptext.append(shp.text_frame.text)
|
||||
joined = "\n".join(ptext)
|
||||
assert "Texto libre" in joined or "TTR" in joined
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Edges — None.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_edge_none_sin_texto_largo():
|
||||
# titanic-like: short labels only → chapter must not apply.
|
||||
assert build_text_distr(_no_text_profile(), {}) is None
|
||||
|
||||
|
||||
def test_edge_none_palabras_cortas():
|
||||
# Char gate passes (len_mean high) but documents are short → confirmation
|
||||
# rejects them (median words below threshold).
|
||||
profile = _text_profile()
|
||||
short = ["palabra " * 3] * 30 # 3 words each, < _MIN_WORDS
|
||||
ctx = {"text_raw": {"review": short}}
|
||||
assert build_text_distr(profile, ctx) is None
|
||||
|
||||
|
||||
def test_edge_none_empty_profile():
|
||||
assert build_text_distr({}, {}) is None
|
||||
assert build_text_distr(None, None) is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Degradation — optional libs absent.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_degradacion_sin_libs(monkeypatch):
|
||||
real_import = builtins.__import__
|
||||
blocked = ("langdetect", "textstat", "wordcloud", "datasketch")
|
||||
|
||||
def fake_import(name, *a, **k):
|
||||
if name in blocked or any(name.startswith(b + ".") for b in blocked):
|
||||
raise ImportError(f"simulado: {name}")
|
||||
return real_import(name, *a, **k)
|
||||
|
||||
monkeypatch.setattr(builtins, "__import__", fake_import)
|
||||
|
||||
ctx = {"text_raw": {"review": _long_reviews()}}
|
||||
ch = build_text_distr(_text_profile(), ctx)
|
||||
# Still builds (the cheap, stdlib-only pieces remain) and never raises.
|
||||
assert ch is not None
|
||||
leaves = _flatten(ch.blocks)
|
||||
assert any(isinstance(b, KVTable) for b in leaves)
|
||||
assert any(isinstance(b, DataTable) for b in leaves)
|
||||
# A degradation note is present mentioning the missing optional libs.
|
||||
notes = " ".join(b.text for b in leaves if isinstance(b, Note))
|
||||
assert "langdetect" in notes or "textstat" in notes or "datasketch" in notes
|
||||
Reference in New Issue
Block a user