Compare commits

...

61 Commits

Author SHA1 Message Date
egutierrez ea6678ec23 feat(eda): generadores de datasets sintéticos Faker que ejercitan el AutomaticEDA
Añade dos funciones impuras dict-no-throw, deterministas por seed, al dominio
datascience (grupo eda):

- generate_synthetic_eda_table: una tabla DuckDB de 19 columnas (numéricas
  correlacionadas + outliers, categóricas desbalanceadas, texto largo
  multi-idioma es/en/fr, fecha DATE, lat/lon válidas, PII email/iban/phone/uuid,
  nulos con patrón MCAR/MAR co-ocurrentes). Activa 14 capítulos del motor
  AutomaticEDA (num_distr, cat_distr, text_distr, calidad, missingness,
  correlacion, relaciones, modelos, timeseries, geospatial, agregacion,
  glosario + portada/overview).
- generate_synthetic_eda_folder: 3 CSV relacionados (customers/orders/reviews)
  con FK customer detectable por containment, para el EDA de carpeta multi-tabla.

Determinismo via Faker.seed_instance + numpy.default_rng. Tests: 16 passed
(incluye determinismo por hash, rangos lat/lon, co-nulos income/spending,
mediana palabras review >=20, phone formato internacional, FK containment).

Añade faker (40.27.0) a python/pyproject.toml + uv.lock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:25:31 +02:00
egutierrez 792b890195 merge(eda): pipeline EDA de carpeta/base multi-tabla + join graph rasterizado a Figure
# Conflicts:
#	python/functions/datascience/__init__.py
2026-06-30 21:00:35 +02:00
egutierrez 9886e2905d feat(eda): rasterizar join graph a Figure matplotlib real en el capitulo de relaciones
draw_join_graph_figure (datascience, grupo eda): dibuja el join graph de la base
como una matplotlib Figure real (networkx spring_layout seed=42, nodos = tablas,
hubs destacados, flechas dirigidas con etiqueta from_col->to_col + cardinalidad).
Nunca lanza: devuelve una Figure de error si algo falla; entrada vacia -> Figure
'Sin relaciones FK detectadas'.

render_automatic_eda_folder ahora inserta esa Figure (bloque Figure lazy via make)
en el capitulo de relaciones cuando hay edges, ademas del texto Mermaid (util para
el MD/LLM). Antes solo se volcaba el texto del grafo; ahora el PDF/PPTX muestran el
diagrama dibujado. Tests nuevos: la Figure real se construye con edges y se omite
sin edges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:57:52 +02:00
egutierrez bebbd05de5 merge(papers): render_paper_pdf (Markdown IMRaD -> PDF) + agente paper-reviewer adversarial (verificado met) 2026-06-30 20:56:11 +02:00
egutierrez 6fb6ef6cfe merge(papers): rigor experimental — effect size (Cohen/Hedges), IC media+diferencia, Holm/FDR + preregistro inmutable (verificado met) 2026-06-30 20:56:11 +02:00
egutierrez 857c3d8637 merge(papers): estructura, scaffolding (init_paper) y capability page del artefacto papers/ (verificado met) 2026-06-30 20:56:11 +02:00
egutierrez e5abc18211 merge(eda): capitulo MISSINGNESS — patrones de nulos (co-ocurrencia + MCAR/MAR) 2026-06-30 20:42:46 +02:00
egutierrez 4f1530797e feat(datascience): rigor experimental para papers — effect size, IC, Holm + preregistro inmutable
Subsistema de papers reproducibles (grupo de capacidad `papers`). Añade las
funciones estadísticas que un paper honesto necesita y la función que congela la
hipótesis antes de mirar los datos (anti-HARKing).

Nuevas funciones (puras salvo la última):
- effect_size_cohens_d: Cohen's d + Hedges' g (corrección de sesgo para N
  pequeño) + interpretación cualitativa (negligible/small/medium/large por los
  umbrales de Cohen). Dict-no-throw ante varianza cero / N insuficiente.
- confidence_interval_mean: intervalo de confianza de una media (t de Student) o
  de la diferencia de medias con Welch (df de Welch–Satterthwaite, sin asumir
  varianzas iguales). Dict-no-throw; el IC colapsa al punto cuando la varianza es
  cero.
- preregister_hypothesis (impura): congela hipótesis + plan de análisis en
  papers/<slug>/preregistration.md con frozen_at (UTC) y content_hash (sha256 del
  cuerpo normalizado, no del frontmatter). Inmutabilidad: una vez frozen, un
  contenido distinto se RECHAZA sin sobrescribir (mata el HARKing); idempotente si
  el contenido es idéntico. Siempre dict-no-throw.

Extensión:
- fdr_correction 1.0.0 -> 1.1.0: añade method="holm" (Holm-Bonferroni step-down,
  controla FWER, más potente que Bonferroni simple). Reúsa la maquinaria de
  alineación 1:1 con None/inválidos; no rompe los métodos bh/bonferroni.

Reutiliza del registry: fdr_correction (BH + Bonferroni ya existían) como base
para Holm. pearson y spearman_corr ya cubrían correlación.

Tests: 36 pytest verdes (cohen/hedges 8, confidence/welch 8, fdr/holm/bonferroni
12, preregister 4 + extras), golden contra valores conocidos y validados con
scipy. Golden manual del preregistro: congela, idempotente, rechaza edición
(bytes en disco idénticos al congelado).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:42:12 +02:00
egutierrez 9da1ee6533 merge(eda): capitulo text_distr (TEXTO/NLP) — primer capitulo no tabular 2026-06-30 20:41:29 +02:00
egutierrez 9c1b7dd0f3 feat(papers): render_paper_pdf (Markdown IMRaD → PDF) + agente paper-reviewer
Subsistema papers/: pieza de entrega + control de calidad.

- render_paper_pdf_py_datascience (Python, impure, dominio datascience, grupo
  `papers`): convierte papers/<slug>/paper.md (frontmatter YAML + cuerpo IMRaD)
  en papers/<slug>/out/paper.pdf. Reutiliza el motor de paginación de flujo del
  paquete automatic_eda (matplotlib PdfPages, el mismo PDF móvil A5 de los
  informes EDA) — no reimplementa paginación ni toca matplotlib, y no añade
  dependencias. Cada sección IMRaD (# H1) → un Chapter en página nueva; portada
  desde el frontmatter (title/authors/date europea/abstract); detecta las
  imágenes Markdown ![alt](src) que el motor no entiende y las parte en bloques
  Image resueltos contra base_dir y base_dir/figures/. dict-no-throw estricto.
  5 tests verdes (golden + edges: sin frontmatter, path inexistente, figura
  inexistente, ruta directa al .md).

- .claude/agents/paper-reviewer: revisor académico adversarial read-only (gate
  anti paper-mill). Puntúa novedad/rigor/reproducibilidad/validez (0-5), intenta
  refutar cada claim contra la evidencia citada, detecta HARKing contra el
  preregistration.md, exige limitaciones declaradas y claims ≤ evidencia, y
  emite veredicto estructurado JSON (accept|major_revision|reject) con default
  conservador. Tools: Read, Grep, Glob, Bash (sin Edit/Write: solo juzga).

Diseño completo: reports/0001-2026-06-30-papers-system-design.md (agente C).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:39:59 +02:00
egutierrez 5d4a48ec5e merge(eda): scatters de pares correlacionados + tipo de relacion en cap CORRELACION 2026-06-30 20:39:16 +02:00
egutierrez 7fa19d65db feat(eda): capítulo MISSINGNESS — patrones de datos faltantes (co-ocurrencia + MCAR/MAR)
Añade el capítulo `missingness` al motor AutomaticEDA, complemento natural de
`calidad`: donde calidad reporta cuánto falta por columna, este capítulo analiza
el PATRÓN de los nulos — dónde faltan y si las columnas faltan juntas
(co-ocurrencia de ausencias), la señal que distingue MCAR de MAR antes de imputar.

Capítulo (`chapters/missingness.py`), registrado en `chapters_registry.py` justo
tras `calidad`:
- Resumen global: % de celdas faltantes, columnas con nulos, filas completas vs
  incompletas.
- Ranking por columna (tabla + barras horizontales).
- Co-ocurrencia: correlación de las máscaras is-null entre columnas (heatmap +
  tabla de los pares que co-faltan, con co-faltantes y Jaccard).
- Patrones de fila más frecuentes (estilo matriz de missingno).
- Lectura MCAR/MAR exploratoria (heurística por correlación/solape de ausencias,
  no confirmatoria), que cita la evidencia concreta.
- Términos de glosario clicables: missingness, MCAR, MAR.

La máscara is-null por fila de TODAS las columnas (numéricas y categóricas) se
construye con un push-down DuckDB sobre ctx['db_path']/table (mismo patrón que el
capítulo agregación), con fallback a ctx['raw_numeric'] cuando no hay BD. Activa
solo si la tabla tiene nulos; si no, devuelve None.

Funciones nuevas del grupo `eda` (dominio datascience):
- extract_null_mask (impura): máscara is-null por fila vía query_fn.
- missingness_overview (pura): resumen global + filas completas/incompletas.
- missingness_correlation (pura): correlación de ausencias + pares + Jaccard,
  reutiliza pearson.
- missingness_row_patterns (pura): patrones de fila más comunes.
- missingness_corr_heatmap_figure / missingness_rank_bar_figure (impuras): figuras.

Verificado: EDA de titanic genera el capítulo en PDF + PPTX + MD con Cabin 77.1%,
Age 19.9% y la co-ocurrencia Age↔Cabin (158 filas). Suite completa de AutomaticEDA
+ render_automatic_eda en verde (125 passed); tests por función y por capítulo;
fn index sin error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:38:39 +02:00
egutierrez 6e3c3cf2a2 feat(papers): estructura, scaffolding y capability page del artefacto papers/
Nuevo tipo de artefacto para papers académicos reproducibles (papers/<NNNN-slug>/):

- Plantillas docs/templates/paper.md (IMRaD completo con guías por sección:
  Abstract, Introduction, Related work, Methods, Results, Discussion con
  Limitaciones + Amenazas a la validez, Conclusion + Future work) y
  docs/templates/preregistration.md (H0/H1 falsable, variables, diseño, plan
  de análisis con test exacto + effect size + corrección múltiple, predicción
  cuantitativa; nota anti-HARKing de congelado).
- Pipeline init_paper (bash/functions/pipelines/init_paper.sh + .md): calcula el
  siguiente NNNN, crea las subcarpetas (experiments data figures reviews out),
  copia las plantillas rellenando el frontmatter (title, slug, date, phase=question,
  status=draft) y crea references.md. No hace git init (fase interna local).
- Función atómica reutilizable next_numbered_dir (bash/functions/io): siguiente
  prefijo NNNN- escaneando un directorio numerado (reutilizable por papers/reports/issues).
- papers/ como artefacto local gitignored (bloque en .gitignore + papers/.gitkeep):
  un paper en fase interna no contamina el repo padre; al promocionar a publishable
  se vuelve sub-repo Gitea propio.
- Página de capacidad docs/capabilities/papers.md + fila en el INDEX: tabla de
  funciones del grupo papers (disponibles + en construcción por la flota), ejemplo
  canónico end-to-end y fronteras.

Reutiliza slugify_ascii del registry. Diseño: reports/0001-2026-06-30-papers-system-design.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:38:38 +02:00
egutierrez 105e56cf05 feat(eda): capítulo text_distr (TEXTO/NLP) — primer capítulo de datos no tabulares
Añade el capítulo `text_distr` al motor AutomaticEDA: perfila columnas de texto
libre largo (reseñas, descripciones, comentarios) que la distribución categórica
no resume bien. Sigue el patrón de cat_distr/num_distr (build_text_distr(profile,
ctx) -> Chapter | None) y se registra en CHAPTER_ORDER tras cat_distr.

Activación en dos fases: gate barato desde el perfil (columna no numérica con
len_mean >= 50 chars) + confirmación con muestra cruda (mediana de palabras >= 20).
Un dataset sin texto largo (p.ej. titanic) devuelve None sin tocar el informe.

Bloques por columna (Group con page_break): resumen (longitudes, vocabulario con
TTR y % hapax, idioma dominante, % duplicados, legibilidad), histograma de
longitudes, top términos (tabla + barras), bigramas/trigramas, idiomas detectados
y nube de palabras opcional. Términos ttr/hapax enganchados al glosario clicable.

Lógica delegada a 7 funciones nuevas del registry (datascience, tag eda),
estilo dict-no-throw:
- extract_text_sample (impura, push-down SQL DuckDB/Postgres)
- compute_text_length_stats, compute_vocabulary_stats, compute_top_ngrams (puras, stdlib)
- detect_corpus_language (langdetect opcional), compute_text_readability (textstat
  opcional), compute_text_duplicates (hash + datasketch opcional)

Versión barata sin modelos pesados: las piezas que dependen de una librería
opcional (langdetect, textstat, wordcloud, datasketch) degradan a omitidas sin
lanzar. Añade langdetect y textstat (ligeras) al pyproject + uv.lock.

Verificado: golden sobre dataset de reviews multi-idioma (capítulo presente en
PDF+PPTX+MD con métricas reales), titanic sin capítulo (None), degradación sin
libs, suite automatic_eda + pipeline verde (128 passed), fn index OK.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:38:17 +02:00
egutierrez eaca41a532 feat(eda): scatters de pares más correlacionados + tipo de relación en capítulo CORRELACION
Añade al capítulo `correlacion` del AutomaticEDA la visualización con scatters de
los pares numérico-numérico más correlacionados (positiva y negativamente) y,
para cada uno, la clasificación del tipo de relación: lineal, polinómica
(grado 2/3), monótona no-lineal o débil/sin forma.

Funciones nuevas del registry (dominio datascience, grupo eda):
- classify_relationship_type_py_datascience (pura): dadas dos listas numéricas
  pareadas, cruza Pearson r (lineal), Spearman ρ (monótona) y ajustes
  polinómicos de grado 2 y 3 (numpy.polyfit + R² manual) para etiquetar la
  forma. Reusa pearson y spearman_corr del registry. Umbrales calibrados para
  datos reales discretos/ruidosos (orden: débil → monótona → polinómica →
  lineal). Devuelve los coeficientes del mejor modelo para pintar la curva.
  No-throw.
- relationship_scatter_figure_py_datascience (impure): construye la Figure
  matplotlib del scatter de un par con su recta/curva de ajuste y una anotación
  del tipo + métricas (r, ρ, R²lin, R²poly). Backend Agg sin pyplot global,
  downsample determinista de los puntos dibujados, tendencia ordenada (binned /
  por valor) para el caso monótona sin polinomio. Defensiva ante vacío.

Capítulo correlacion.py (1.0.0 → 1.1.0): nueva sección "Relaciones más fuertes
(scatter)" tras la matriz + tablas top. Toma los top-K pares num↔num por |valor|
de profile['correlations']['pairs'], obtiene los datos crudos de cada par desde
ctx['raw_numeric'] y emite, por par, un Figure dentro de un Group keep-together
junto a una nota de texto con el tipo de relación (extraíble por pdftotext).
Solo num↔num: los pares cat↔cat (Cramér's V) y num↔cat (razón de correlación)
no llevan scatter. Cuando no hay raw_numeric (perfil lite/agregado o ctx None)
los scatters se omiten sin lanzar; la matriz + tablas siguen.

Verificado: golden EDA de titanic (run_models) — el capítulo Correlación del PDF
y PPTX incluye los scatters (pclass↔fare → monótona no-lineal, sibsp↔parch →
lineal, …) con su ajuste y etiqueta de tipo en texto. Tests de clasificación
sintética (lineal, y=x² → polinómica, y=exp(x) → monótona, ruido → débil) +
tests del capítulo (golden con raw_numeric, edge sin raw, par sin columna). Suite
automatic_eda + pipeline render_automatic_eda verde (141 passed). fn index sin
error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:37:01 +02:00
egutierrez 6a1520f458 feat(eda): EDA de carpeta/base multi-tabla -> AutomaticEDA por capitulos (PDF+PPTX+MD)
Pipeline render_automatic_eda_folder: apunta el AutomaticEDA a una CARPETA de
archivos tabulares (CSV/Parquet/JSON) o a una DuckDB existente y emite el informe
de la BASE por capitulos en PDF (A5 movil) + PPTX (16:9) + Markdown. Documento-base
con portada-base, resumen de todas las tablas y relaciones inter-tabla (FK
candidatas por containment + diagrama Mermaid del join graph). Flag per_table_eda
anexa el mini-EDA de cada tabla. Aditivo: render_automatic_eda (tabla unica) intacto.

Funcion nueva load_folder_to_duckdb (infra, grupo eda+duckdb): carga una carpeta a
una DuckDB (temp si no se da path), CREATE TABLE por archivo con read_csv_auto/
read_parquet/read_json_auto. dict-no-throw.

Compone profile_database + los 3 renderers del motor AutomaticEDA + build_document
(per-tabla), sin reimplementar su logica. Tests: golden 3 CSV relacionados (FK
orders.customer_id->customers.id detectada) + edges (carpeta vacia, 1 tabla,
DuckDB existente, path inexistente). fn index sin error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:34:10 +02:00
egutierrez e815f5b3b9 merge(eda): MD del AutomaticEDA vuelca TODOS los datos del profile (28 pares, skew/kurtosis/percentiles, scores_by_k) 2026-06-30 20:31:50 +02:00
egutierrez 7ec2bb1b45 feat(eda): el Markdown del AutomaticEDA vuelca TODOS los datos del profile
El .md del grupo `eda` es la salida pensada para pegar a un LLM, así que debe
contener todo lo que el motor computó, aunque el PDF/PPTX (vista humana) resuman.
La evaluación 2053 detectó 6 datos que el .md perdía respecto al profile. Se
cierran de forma aditiva (el .md tiene MÁS que el PDF/PPTX, sin tocar esos
renderers ni los capítulos).

render_automatic_eda.py pasa el profile al serializador Markdown vía
meta['profile'] (un meta propio del MD; el de PDF/PPTX queda intacto).
render_md_impl.py añade un "Apéndice — Datos completos del perfil" al final del
documento, emitido solo cuando hay profile y degradando limpio cuando falta una
sección (lite sin modelos, profile sin correlaciones). El apéndice no se acopla
a los ids de capítulo (que editan otros agentes en paralelo).

Pérdidas cerradas:
1. Matriz de asociación COMPLETA: los N pares de correlations.pairs (no solo el
   top-17), incluidos correlation_ratio (num↔cat) y cramers_v (cat↔cat).
2. Numéricas: describe completo por columna — mean/median/mode/std/variance/cv,
   skew y kurtosis para TODAS (no solo las asimétricas), p1/p5/p25/p50/p75/p95/
   p99, iqr, min/max, outliers, distribution_type.
3. Re-expresión: nombra la transformación concreta (log1p/sqrt/yeo-johnson) con
   potencia, razón y alternativas, no un vago "considerar re-expresión".
4. KMeans: tabla scores_by_k (silhouette + inercia por k) marcando el k elegido.
5. Normalidad: el estadístico (stat) de cada test junto al p-value.
6. Encabezados de figuras de barras/scree dejan de heredar
   "Desde/Hasta/Frecuencia" del histograma; usan "Inicio/Fin/Valor" cuando el
   caption no es un histograma.

Test nuevo md_completeness_test.py: profile sintético, asserta los N pares de
correlación, skew/kurtosis de cada numérica, percentiles extendidos, log1p,
scores_by_k, stat de normalidad, headers de barras y los edges (sin modelos /
sin correlaciones / sin profile, defensivo).

Verificado con titanic (profile_level=full): 28 pares en la tabla (incl.
Sex↔Embarked cramers_v), 7 numéricas con skew+kurtosis, p5/p95/p99, scores_by_k
y JB/D'Agostino/Shapiro stat presentes. PDF/PPTX/manifest siguen saliendo.
Suite automatic_eda + render_automatic_eda_test: 134 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:27:30 +02:00
egutierrez a1e2e3567c merge: 4c cat_distr una hoja por columna (PDF+PPTX 1:1) + sin descripcion entropia redundante + page_break motor (verificado met) 2026-06-30 19:53:57 +02:00
egutierrez 833597c831 fix(eda): cat_distr PPTX — columnas de alta cardinalidad caben en UN slide con su gráfico
La verificación adversarial detectó que, en PPTX (slide 16:9, corto), las columnas
categóricas de ALTA cardinalidad NO id-like (Ticket, Cabin) ocupaban 3 slides cada
una con el donut SEPARADO de su tabla: el top-k de 8 filas largas no cabía junto al
donut y el keep-together partía la columna. (El PDF, en A5, ya estaba 1:1 correcto.)

Arreglo SOLO en render_pptx_impl.py:

- `_fit_group_blocks` (nuevo): para un Group con figura + DataTable que no cabe en el
  slide, reserva un alto mínimo para el donut (`_GROUP_MIN_FIG_H`) y recorta las filas
  de la DataTable a lo que queda, de modo que el gráfico se queda en el MISMO slide,
  junto a su tabla. No-op cuando ya cabe o no hay par figura+tabla (p.ej. columnas
  id-like, que ya omiten la top-k).
- `_trim_data_table_to_budget` (nuevo): devuelve una COPIA de la DataTable con las
  filas que caben (al menos una) + nota honesta "top N de M categorías mostradas
  (recortado para caber en el slide; el PDF muestra más)". NUNCA muta el bloque
  original, que es compartido con el renderer PDF (el PDF sigue mostrando la tabla
  completa en A5).
- `_place_group`: aplica `_fit_group_blocks` antes de `_shrink_group_figures`.

Refuerzo de cat_distr_test.py:

- `test_golden_pptx_una_slide_por_columna_con_su_grafico`: perfil con una columna
  categórica de alta cardinalidad no-id-like (40 valores largos sobre 5000 filas,
  0.8% distinto) que reproduce el caso Ticket/Cabin. Asierta que CADA columna
  categórica aparece en EXACTAMENTE UN slide del capítulo y que ese mismo slide lleva
  su tabla (Cardinalidad/distintos) Y su donut (caption + shape Picture) — el gráfico
  nunca se separa de su tabla. Sustituye al laxo `n_slides >= 2`.

Verificado con titanic_train.csv (render_automatic_eda run_models=True): 5 columnas
categóricas (Name, Sex, Ticket, Cabin, Embarked); PDF 6 páginas y PPTX 6 slides del
capítulo (intro + 1 por columna), cada columna con su donut junto a su tabla en una
sola página/slide. Ticket y Cabin pasaron de 3 slides a 1. Suite verde (122 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 19:45:09 +02:00
egutierrez 7158be8142 feat(eda): cat_distr una hoja por columna (gráfico incluido) + sin descripción redundante con glosario
Cada columna categórica del capítulo CAT DISTR ocupa ahora su propia página
(PDF) / slide (PPTX) con su gráfico junto a su tabla, y se elimina la
explicación larga de la entropía que duplicaba el capítulo GLOSARIO.

Cambios:

- model.Group: nuevo campo aditivo `page_break_before` (default False). Cuando
  es True el renderer fuerza al grupo a empezar en página/slide nueva (salvo que
  la actual esté vacía). Comportamiento de todos los capítulos existentes
  intacto. Soportado también en el normalizador dict-defensivo `as_block`.

- render_pdf_impl / render_pptx_impl `_place_group`: respetan `page_break_before`.

- render_pdf_impl / render_pptx_impl `_measure_block`: medición fiel de KVTable y
  DataTable (replica `_place_*`: título-heading, wrap del valor/celdas por
  columna, nota). La estimación previa asumía una línea por fila e ignoraba el
  título, así que el keep-together infra-presupuestaba la figura y el gráfico se
  desbordaba a la página siguiente. Helpers `_measure_kv_table`/`_measure_data_table`.

- render_pptx_impl `_shrink_group_figures`: umbrales más bajos (budget>0.6,
  per>0.35) para que en el slide corto 16:9 la figura se encoja y conviva con la
  tabla en lugar de partir la columna (misma filosofía keep-together del PDF).

- cat_distr.py:
  - build envuelve cada columna en un `Group(page_break_before=idx>0)`: una
    columna por página/slide, con su tabla de cardinalidad, su top-k y su donut
    juntos. La primera comparte página con la intro para no dejar una casi vacía.
  - intro recortada: se elimina el párrafo que explicaba qué es la entropía
    (vive en el capítulo GLOSARIO, donde el término `[[term:entropia]]` enlaza);
    se conserva el término clicable y el total de filas de referencia.
  - `_cardinality_block`: métricas relacionadas agrupadas por fila (distintos·%·
    únicos; entropía bits·máx·norm; desbalance·longitud) sin perder ningún dato,
    para que tabla + gráfico quepan en el slide 16:9.
  - columnas id-like (≈100% distintas): se omite la top-k (sería una lista de
    valores únicos; la nota lo explica) y el donut ocupa ese hueco.
  - CHAPTER_VERSION 1.1.0 -> 1.2.0.

Verificado con titanic (render_automatic_eda run_models=True): PDF 5 páginas y
PPTX 5 slides del capítulo (intro + 1 por columna: Name, Sex, Ticket, Embarked),
cada columna con su gráfico junto a su tabla, sin cortes. Suite verde
(121 passed): pytest automatic_eda/ + render_automatic_eda_test.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 19:26:33 +02:00
egutierrez 9be84a48ea merge: 4c quitar definiciones redundantes con glosario en calidad/correlacion/modelos/agregacion/relaciones (links intactos, verificado met) 2026-06-30 19:24:22 +02:00
egutierrez fd63261444 refactor(eda): quitar definiciones inline redundantes con el glosario en 5 capítulos
Ahora que el AutomaticEDA tiene un capítulo GLOSARIO con las definiciones de los
términos técnicos (enganchados como links clicables desde el cuerpo), los
capítulos calidad/correlacion/modelos/agregacion/relaciones ya no repiten inline
esas explicaciones largas: se deja el TÉRMINO marcado (clicable, sigue saltando
al glosario) y se elimina el párrafo/oración de definición redundante. Los
HALLAZGOS y datos concretos del análisis se mantienen intactos; solo se quitan
las definiciones generales que el glosario ya cubre.

- calidad: _criteria_intro pasa de un bullet-list con las definiciones de
  completitud/validez/unicidad/calidad + fórmula renormalizada + párrafo de
  outliers a una frase que nombra las dimensiones, sus pesos (60/40) y el
  principio de outliers; los 4 términos siguen marcados.
- modelos: la nota de normalización deja de explicar la fórmula del z-score; la
  intro de PCA ya no define "componentes ortogonales ordenados por varianza"; la
  de KMeans quita "rango −1 a 1: cuanto más alto..." (silhouette); la sección de
  Isolation Forest quita la descripción de árboles/cortes/umbral. Términos
  marcados intactos.
- correlacion: la intro deja de describir cada método y consolida la duplicación
  signo/dirección; los 4 métodos + FDR siguen marcados.
- agregacion: la intro quita la definición de pivot ("cruzan dos categóricas
  sobre una medida") y abrevia la selección de claves; groupby y pivot marcados.
- relaciones: la intro y la sección de candidatas/inter-tabla quitan las
  definiciones de PK ("identifica cada fila"), FK ("referencian a otra tabla") y
  containment ("valores contenidos en la clave de otra"); pk/fk/cardinalidad/
  containment siguen marcados.

Verificado sobre el EDA de titanic (run_models + run_llm, 48 págs): los 23 link
annotations término→glosario se conservan (PyMuPDF), el glosario mantiene las 20
definiciones, y el texto visible de los 5 capítulos baja un 34.7% en conjunto
(calidad −67%, modelos −33%, relaciones −19%, agregacion −15%, correlacion −8%).
Tests actualizados (calidad_test asertaba el texto viejo). Suite EDA + pipeline
verde (118 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 19:15:24 +02:00
egutierrez 4099d88eaf merge: 4b salida markdown del AutomaticEDA (render_md, render_automatic_eda emite aeda_md_path, verificado met) 2026-06-30 18:59:33 +02:00
egutierrez 48de3ce3da feat(eda): salida Markdown del AutomaticEDA para pegar a un LLM
Añade un tercer formato de salida al AutomaticEDA, junto al PDF y el PPTX:
un Markdown autocontenido del MISMO documento por capítulos
(chapters_registry.build_document), optimizado para incorporar a un LLM
(texto plano + tablas markdown reales, sin binarios incrustados).

- render_md_impl.render_md(chapters, out_path, meta): serializa los bloques
  del modelo (Heading/Markdown/KVTable/DataTable/Figure/Image/Caption/Note/
  Group/GlossaryEntry) a Markdown. Cabecera con metadatos + índice navegable
  con anclas GitHub; tablas volcadas enteras (el MD no pagina); marcadores de
  glosario eliminados conservando la negrita; glosario al final.
- Figuras: un LLM no ve la imagen, así que se prioriza texto + datos. Se emite
  el caption y, cuando la figura tiene barras (histograma), se extrae la tabla
  de bins (Desde/Hasta/Frecuencia) de los artistas matplotlib. La banda ±1σ
  (axvspan) se descarta por ancho para que no aparezca como un falso bin.
  PNG opcional vía meta['embed_figures'] (off por defecto → sin binarios).
- render_automatic_eda_markdown: función pública del registry (tag eda),
  espejo de render_automatic_eda_pdf/pptx, acepta lista de capítulos o un
  TableProfile (build_document). dict-no-throw.
- render_automatic_eda (pipeline): emite también el .md (emit_md=True por
  defecto, clave de retorno aeda_md_path). Cambio aditivo: PDF/PPTX/manifest
  siguen saliendo igual.

Tests: golden de todos los kinds + regresión del filtro de la banda ±1σ +
edge documento vacío + profile path. Suite del paquete y del pipeline verde
(122 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:52:08 +02:00
egutierrez ab21e5d90b merge: 4b flag profile_level lite/standard/full en render_automatic_eda (lite 4.5s vs full 39.3s, verificado met) 2026-06-30 18:29:44 +02:00
egutierrez da60211826 merge: 4b relaciones — capitulo PK/FK + candidatos intra/inter-tabla (reusa infer_fk_containment_duckdb+build_join_graph, verificado met) 2026-06-30 18:22:29 +02:00
egutierrez 3be188a921 feat(eda): profile_level (lite/standard/full) en render_automatic_eda
Añade el parámetro profile_level a render_automatic_eda como preset de
consumo CPU/LLM que mapea a los flags existentes (run_models, run_series,
run_llm, sample). Tres niveles:

- lite (bajo consumo): run_llm=False, run_series=False, sample=2000 y modelos
  limitados a PCA + normalidad, SIN KMeans ni IsolationForest (lo caro en CPU).
  Para un vistazo rápido y barato.
- standard (default): comportamiento histórico — modelos completos, serie,
  sin LLM.
- full: standard + narrativa LLM por capítulo.

Precedencia: un flag explícito del caller (run_llm=..., run_models=..., etc.)
siempre prima sobre el default que fija el preset; el preset solo aplica al
parámetro que se deja en None.

Cableado del modo lite sin tocar profile_table (lo tocan otros agentes en
paralelo): profile_table NO corre los modelos (evita pagar KMeans +
IsolationForest); este pipeline los corre con run_eda_models(run_kmeans=False,
run_isolation=False) reusando ctx['raw_numeric'], y quita raw_numeric del ctx
para que el capítulo modelos no reproyecte clusters KMeans en vivo
(project_clusters_2d). geo_points ya queda derivado, así que geospatial no se
afecta.

Cambio aditivo y retro-compatible: sin profile_level el comportamiento es
idéntico al de v1.0.0 (standard). Tests nuevos cubren lite/standard, la
precedencia flag-sobre-preset, y la equivalencia del default con el histórico.
Bump 1.0.0 -> 1.1.0 + growth log en el .md. Skill /eda documenta --lite/--full.

Verificación: golden lite/standard/full sobre titanic — lite 4.8s (PCA+norm,
sin KMeans/iso/LLM/serie), standard 7.8s (modelos completos), full 38.3s
(+LLM). Suite render_automatic_eda + automatic_eda: 96 passed. fn index sin
error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:20:17 +02:00
egutierrez aa5aa67d50 merge: 4b calidad — nueva formula (completeness 0.6+validity 0.4, dataset row_uniqueness, outliers fuera a Observaciones, sin doble conteo) report 2046 (verificado met) 2026-06-30 18:17:23 +02:00
egutierrez 68f4ddabce feat(eda): capítulo RELACIONES para AutomaticEDA
Añade el capítulo `relaciones` al motor AutomaticEDA: analiza las
relaciones de clave de la tabla/base y se coloca tras `correlacion`,
antes de `modelos`, en CHAPTER_ORDER.

Capas que renderiza (solo las que aplican; None si no hay nada que decir):
- Claves declaradas: PK/FK/UNIQUE reales del esquema DuckDB, vía la nueva
  función `detect_declared_keys_duckdb` (lee `duckdb_constraints()`).
- Candidatos a clave primaria: los `key_candidates` del TableProfile.
- FK candidatas inter-tabla: reusa `infer_fk_containment_duckdb`
  (containment + señal de nombre) y `build_join_graph` (roles de nodos +
  diagrama Mermaid pegable). Solo si la fuente DuckDB tiene varias tablas.
- FK candidatas intra-tabla: heurística nombre + cardinalidad, vía la nueva
  función pura `suggest_intratable_fk_candidates`, marcada como sugerencia.

Engancha al glosario clicable los términos PK, FK, containment/inclusión y
cardinalidad (contrato §11.1) y usa Group (keep-together) para el grafo.

Funciones nuevas del registry (grupo `eda`):
- detect_declared_keys_duckdb (impure, datascience) + test.
- suggest_intratable_fk_candidates (pure, datascience) + test.

Tests: relaciones_test.py (golden intra + inter, edges, no-cut render) +
los tests de ambas funciones. Suite automatic_eda + render_automatic_eda
verde (89 passed). Golden end-to-end con el pipeline render_automatic_eda
verificado sobre titanic (intra) y una BD customers/orders (inter).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:15:15 +02:00
egutierrez 43821ab11d merge: 4b analisis_llm — dedup Diccionario de datos + Datos personales (verificado met) 2026-06-30 18:14:17 +02:00
egutierrez 32054ad781 merge: 4b portada — tamano grande junto al nombre + descripcion y granularidad funcionando (verificado met) 2026-06-30 18:12:22 +02:00
egutierrez a2074a0167 feat(eda): nueva fórmula de calidad de datos (report 2046) + capítulo calidad
Implementa el modelo de calidad del report 2046 en el grupo eda.

Score de columna: 0.6·completeness + 0.4·validity con renormalización por
aplicabilidad (si la validez no es medible —texto libre o columna 100% nula— el
score se basa solo en completeness). Validez = conformidad real al tipo: nativo
numérico/fecha/bool = 1.0; texto promovido a número/fecha = parse rate
(validity_rate); texto con semantic_type = match_rate; texto libre = no aplica.

Outliers, columnas constantes e identificadores salen del score a un bloque de
observaciones analíticas (no son defectos de calidad). Se elimina el doble
conteo de la falta de datos (mostly_null ya no castiga validez) y el bug de
escala de outliers (que además ya no entran en el score).

Score de dataset: 100·(0.85·cell_quality + 0.15·row_uniqueness) en vez de la
media simple. Se pobla duplicate_rows/duplicate_pct push-down en
summarize_table_duckdb (COUNT sobre DISTINCT *, sin RAM) para habilitar la
unicidad de registro; renormaliza a solo cell_quality si no se puede calcular.

Capítulo calidad (v2.0.0): intro de dos dimensiones (60/40) que declara que los
outliers no bajan el score; tabla de scores Columna|Calidad|Completitud|Validez
(sin Consistencia, n/a cuando no aplica); DOS tablas separadas (Problemas de
calidad vs Observaciones analíticas); resumen con Unicidad de registro; glosario
clicable de completitud, validez, unicidad de registro y calidad de datos.

Verificado: 123 tests verdes (automatic_eda + render_automatic_eda +
column_quality_score + summarize_table_duckdb + profile_table). Golden EDA de
titanic (run_models+run_llm) con score recomputado a mano, outliers separados en
observaciones y glosario clicable (5 links GOTO en el PDF).

column_quality_score v2.0.0, summarize_table_duckdb v1.1.0, profile_table v1.1.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:10:23 +02:00
egutierrez d001d90306 merge: 4b glosario_hooks — terminos clicables en correlacion/modelos/agregacion (12/12 PDF+PPT, verificado met) 2026-06-30 18:09:37 +02:00
egutierrez 7045f37554 fix(eda): quita rótulos duplicados en capítulo ANÁLISIS LLM
El capítulo etiquetaba dos secciones por partida doble: un Heading de nivel 2
más el 'title' del propio DataTable, imprimiendo 'Diccionario de datos' y
'Datos personales (PII / RGPD)' dos veces seguidas en PDF y PPTX.

Se elimina el 'title' de ambos DataTable y se conserva el Heading único (el
patrón canónico OVERVIEW del contrato §8: el rótulo lo da el Heading, la tabla
solo repite su cabecera de columnas al paginar). El DataTable de PII mantiene su
'note' orientativa. La columna del diccionario ya lee 'Significado de negocio'.

CHAPTER_VERSION 1.0.0 -> 1.1.0. Test nuevo
test_sin_rotulos_duplicados_y_significado_de_negocio fija: tablas sin title,
cabecera exacta 'Significado de negocio', y cada rótulo una sola vez en el PDF.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:07:12 +02:00
egutierrez fa8db01059 merge: 4b num_distr — desv std (sigma) en leyenda del histograma (verificado met) 2026-06-30 18:06:46 +02:00
egutierrez f2ac734ef7 merge: 4b head_rows — overview muestra df.head (build_eda_render_ctx pobla head_rows, verificado met) 2026-06-30 18:04:51 +02:00
egutierrez 048781df3f feat(eda): portada — tamaño grande + descripción/granularidad reales
El capítulo PORTADA ahora muestra SIEMPRE el tamaño del dataset (N filas ×
M columnas) en grande, como heading junto al nombre y agrupado con él
(Group keep-together), en lugar de enterrarlo en la tabla de metadatos.

La Descripción y la Granularidad ya no salen vacías ni con placeholders:
se resuelven por cascada — ctx explícito > bloque LLM (profile['llm'].summary
/ row_meaning de eda_llm_insights) > derivación del propio perfil (forma,
mezcla de tipos y score de calidad para la descripción; columnas
key_candidates o la forma de la tabla para una frase 'Cada fila es…').
Las derivaciones son honestas (declaran que vienen del perfil) y nunca
inventan significado de negocio.

Añade chapters/portada_test.py: golden (tamaño grande + textos del LLM,
sin fila 'Tamaño' duplicada), fallbacks sin LLM (keys / forma), prioridad
de ctx, edge de perfil vacío sin lanzar, y render a PDF + PPTX.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:04:05 +02:00
egutierrez a421f13d2e feat(eda): engancha glosario clicable en correlacion/modelos/agregacion
Fase 4b — extiende el glosario clicable de AutomaticEDA (mecanismo ya probado
end-to-end con `entropia` en cat_distr) a tres capítulos más, siguiendo el
contrato sección 11 (glossary.add(key,label,def) + span [[term:KEY]]texto[[/term]]):

- correlacion: Pearson, Spearman, Cramér's V, razón de correlación (η) y la
  corrección por comparaciones múltiples (FDR). Los métodos se marcan en el
  intro (siempre presente); FDR se registra y marca solo cuando se emite su
  resumen, para no dejar entradas de glosario sin aparición que las referencie.
- modelos: PCA, KMeans, coeficiente de silueta (silhouette), Isolation Forest y
  la estandarización z-score. Cada término se registra dentro de la sección que
  lo usa (tras su early-return), de modo que un término solo entra al glosario
  cuando su sección realmente se renderiza.
- agregacion: agrupación (split-apply-combine / groupby) y tabla dinámica
  (pivot), ambos en el intro siempre presente.

Solo se añaden los enganches de glosario: ningún cambio en la lógica de datos.
El texto visible es idéntico con o sin marcador (los renderers lo eliminan),
así que el layout de línea no cambia. Sin colector en ctx (render suelto) los
capítulos degradan y no marcan nada.

Tests: un test de glosario por capítulo verifica registro + marcado y la
degradación sin colector. Suite AutomaticEDA + render pipeline: 87 passed.
Golden titanic (run_models+series+llm): los 12 términos aparecen como entradas
del glosario en PDF (16 link annotations GOTO) y PPTX (15 saltos hlinksldjump).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:02:31 +02:00
egutierrez 13c82be780 feat(eda): NUM DISTR muestra el valor de σ (std) en la leyenda del histograma
La leyenda de cada histograma del capítulo de distribuciones numéricas ya
reporta el valor de la media y la mediana; ahora también reporta el valor de
la desviación estándar σ. La entrada de leyenda de la banda ±1σ pasa a incluir
el número (±1σ (σ = X)) y, cuando la banda no puede dibujarse (sin media o
std<=0) pero σ es conocido, se añade una entrada de leyenda mediante un handle
proxy sin trazo, de modo que el valor de σ se reporta siempre.

No se altera el boxplot de Tukey ni el keep-together (Group) por columna.
Se añaden tests de la leyenda: golden (σ con valor junto a media y mediana),
edge sin banda (proxy) y edge sin std (no revienta). Bump 1.1.0 -> 1.2.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:01:12 +02:00
egutierrez 7fb00defdf fix(fleetclaude): reusar contexto dentro de la flota tmux en vez de abrir kitty nueva
Lanzar `fleetclaude` estando ya dentro de una flota tmux viva abría una ventana
kitty nueva (y creaba un perfil/socket nuevo fleetN+1) en vez de mostrar la flota
en el pane actual. Causa: con $TMUX definido el launcher saltaba el `exec tmux
attach` y caía a la rama `setsid kitty`.

Cambio: cuando se invoca sin --new desde dentro de una flota fleetview viva (el
socket actual, derivado de $TMUX, tiene una sesión homónima con window 'console'),
se trae la TUI al contexto/pane actual (`fleetview show`, o `tmux select-window`
de la window console como fallback sin binario) y se retorna 0 antes de las ramas
kitty/wt.exe. Nuevo flag --new para forzar el comportamiento clásico (flota+ventana
nueva) aun dentro de tmux; pasar --session con un nombre distinto al perfil actual
equivale a --new implícito. Fuera de tmux el comportamiento es intacto (exec tmux
attach reutiliza la terminal).

Fix incidental: `local left_pane="" right_pane=""` (antes `local left_pane
right_pane` reventaba con "unbound variable" bajo `set -u` al reutilizar una sesión
existente, p. ej. con --reuse/--session sobre una flota viva).

Verificación e2e con sockets aislados fctest* (sin tocar la flota del humano):
golden (reuse, exit 0, kitty invariante), --new y --session-distinto (no reuse,
ruta ventana-nueva), fuera de tmux (salta reuse, ruta attach). bash -n limpio.

Docs: launch_fleetclaude.md (signature, params --new, ejemplo, cuando usarla,
gotchas, growth log v1.7.0) + /fleet show en .claude/commands/fleet.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:56:41 +02:00
egutierrez b1d205203a feat(eda): poblar head_rows real en el capitulo OVERVIEW (df.head)
El capitulo OVERVIEW del motor AutomaticEDA mostraba "df.head no disponible"
porque ninguna fase de calculo poblaba las primeras filas crudas de la tabla.

- build_eda_render_ctx: nuevo bloque que muestrea SELECT * LIMIT head_n
  (param nuevo head_n=10) y lo expone en ctx["head_rows"] como lista de
  dicts fila. Estilo dict-no-throw: si la query falla, se omite la clave.
- profile_table: puebla prof["head_rows"] reusando _sample_rows (SELECT de
  las columnas LIMIT 10) tras recalcular el type_breakdown. Asi el report
  JSON sidecar tambien lo lleva y el capitulo lo recoge via profile aunque
  no se construya el ctx.
- overview.py: la nota del DataTable de df.head ahora indica el total de
  filas del dataset cuando se conoce ("primeras 10 filas de 891"). Bump
  CHAPTER_VERSION 1.0.0 -> 1.1.0.
- overview_test.py (nuevo): golden (head via profile y via ctx, render PDF
  + PPTX muestran las filas reales, placeholder ausente), edge (sin
  head_rows degrada a nota honesta sin romper, None/vacio devuelven None).

Verificado end-to-end con titanic: render_automatic_eda emite PDF + PPTX con
df.head visible (Braund/Cumings/Heikkinen + columnas) y sin el placeholder.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:56:24 +02:00
egutierrez c6d9bc26da merge: Fase 4a AutomaticEDA motor+glosario (verificado met)
- fix negrita-pisa PDF, zebra striping (PDF+PPT), keep-together (Group: heading+figura+texto misma pagina/slide), imagenes con caption en PPT
- portada construida-al-final mostrada en posicion 1 (con resumen agregado del cuerpo)
- capitulo glosario al final + terminos clicables REALES: PDF link annotation (add_pdf_internal_links, PyMuPDF) + PPT hyperlink nativo (pptx_link_run_to_slide); entropia enganchado en cat_distr como ejemplo E2E
- contrato docs/automatic_eda_contract.md §11 (glosario + keep-together + zebra)
- pymupdf>=1.28.0
2026-06-30 17:45:30 +02:00
egutierrez d1a3d58a6b feat(eda): motor AutomaticEDA fase 4a — render fixes + keep-together + glosario clicable
Mejoras transversales del motor de render (no del contenido de capítulos):

1. Fix negrita pisa texto (PDF): _place_rich_lines mide el ancho REAL de cada
   span con las métricas de fuente del renderer (peso correcto) en vez del
   grid de ancho medio; negrita y normal en la misma línea ya no se solapan.
2. Zebra striping: filas pares sombreadas (#f6f8fa) en DataTable (PDF + PPTX),
   coherente al partir tablas largas (índice de fila lógico, no por página).
3. Keep-together: bloque Group nuevo; el renderer mide el grupo entero y lo
   mueve completo a la página/slide siguiente si no cabe, y encoge la figura
   (height_in) para dejar sitio a su título y texto. num_distr lo usa.
4. Caption siempre visible en toda figura PPTX (fallback al heading); la figura
   reserva el alto de su caption para que ambos quepan en el mismo slide.
5. Portada construida al final (con resumen agregado del análisis vía
   ctx['document_summary']) pero colocada primera por build_document.
6. Glosario: capítulo nuevo (último) + GlossaryCollector en ctx; los capítulos
   registran términos y marcan apariciones con [[term:key]]...[[/term]]. Links
   clicables reales: PDF (PyMuPDF, link GOTO) y PPTX (slide-jump nativo).
   Enganchado "entropía" en cat_distr como ejemplo end-to-end.

Funciones reutilizables delegadas a fn-constructor (tag eda):
- add_pdf_internal_links_py_datascience (PyMuPDF)
- pptx_link_run_to_slide_py_datascience (slide-jump)

Contrato docs/automatic_eda_contract.md actualizado (§1/§3/§5 + §11 nueva) con
la API de glosario, keep-together y zebra para la siguiente fase. PyMuPDF
declarado en pyproject. Suite verde (90 tests); golden titanic verificado.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 17:35:19 +02:00
egutierrez b5334a2e97 merge: Fase 3 AutomaticEDA wiring (verificado met)
- build_eda_render_ctx: arma ctx (raw_numeric, timeseries_raw, geo_points, db_path+table) desde tabla DuckDB
- pipeline render_automatic_eda: perfila + ctx + build_document -> PDF + PPTX (11 capitulos poblados)
- profile_table: flag emit_automatic emite el report AutomaticEDA (PDF+PPT) sin romper render_eda_pdf
- text_layout: render real de **negrita** en PDF y PPTX
- .claude/commands/eda.md actualizado

Los 4 capitulos que degradaban (modelos/timeseries/geospatial/agregacion) ahora salen POBLADOS end-to-end.
2026-06-30 16:19:52 +02:00
egutierrez 437409641c docs(eda): el skill /eda emite SIEMPRE PDF + PPTX con AutomaticEDA
Actualiza el flujo del comando para que un EDA completo emita el informe
AutomaticEDA en sus dos formatos (PDF A5 móvil + PPTX 16:9) con los 11 capítulos
poblados, vía render_automatic_eda (o profile_table(emit_automatic=True)). El PDF
legacy (emit_pdf/render_eda_pdf) queda como salida independiente opcional.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 16:08:50 +02:00
egutierrez f3d427d9e4 feat(eda): wiring AutomaticEDA — build_eda_render_ctx + pipeline render_automatic_eda + profile_table(emit_automatic)
Conecta el motor AutomaticEDA con los datos crudos para que los 4 capítulos
dependientes de ctx (modelos, timeseries, geospatial, agregacion) salgan
POBLADOS en vez de degradar a una nota.

- build_eda_render_ctx (datascience, impure, dict-no-throw): dado db_path+table
  y el TableProfile agregado, construye el ctx con los datos crudos que el
  perfil no incluye: raw_numeric {col:[float|None]} alineado por fila (modelos /
  geospatial), timeseries_raw {time_col,t,series} vía extract_timeseries_raw,
  geo_points {lats,lons} desde el par lat/lon detectado, y db_path/table para el
  groupby/pivot push-down de agregacion. Muestrea con LIMIT (no trae la tabla
  entera a RAM). Compone detect_time_column / extract_timeseries_raw /
  detect_latlon_columns / duckdb_query_readonly (imports lazy para evitar ciclo).
- render_automatic_eda (pipeline): one-shot perfil -> ctx -> PDF + PPTX con los
  11 capítulos poblados; devuelve rutas + manifest de versiones por capítulo.
- profile_table: flag aditivo emit_automatic=True emite el AutomaticEDA PDF+PPTX
  además del flujo legacy (emit_pdf/render_eda_pdf intacto). Nuevas claves de
  retorno aeda_pdf_path / aeda_pptx_path / aeda_manifest_path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 16:08:41 +02:00
egutierrez f5b30b23dc feat(eda): negrita inline real (**bold**) en renderers AutomaticEDA
El render de Markdown del motor AutomaticEDA quitaba los marcadores **negrita**
sin aplicar estilo. Ahora los spans **bold**/__bold__ se renderizan en negrita
real, de forma aditiva y sin romper el anti-corte:

- text_layout.py: parse_inline_bold() tokeniza spans preservando el texto
  visible (== strip_inline_md) y wrap_rich() envuelve por palabras a max_chars
  conservando el flag de negrita por segmento (la anchura visible no cambia, así
  que la paginación es idéntica).
- render_pdf_impl.py: _place_rich_lines() dibuja cada segmento con su fontweight
  avanzando x por el mismo grid de caracteres que usa el wrap (párrafos+bullets).
- render_pptx_impl.py: _add_rich_text() usa runs nativos de python-pptx con
  font.bold por segmento (negrita real de PowerPoint).
- bold_render_test.py: helpers puros (no-overflow, bold preservado, marcadores
  desbalanceados) + e2e que abre el .pptx y confirma un run con font.bold True.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 16:08:16 +02:00
egutierrez 5eaf3f662e merge: capitulo AutomaticEDA agregacion (verificado met) + funciones delegadas eda 2026-06-30 15:45:37 +02:00
egutierrez 05fe76bce0 merge: capitulo AutomaticEDA timeseries (verificado met) + funciones delegadas eda 2026-06-30 15:45:37 +02:00
egutierrez 864430e988 merge: capitulo AutomaticEDA geospatial (verificado met) + detect_latlon_columns/analyze_geo_extent/build_geo_scatter 2026-06-30 15:36:22 +02:00
egutierrez a69d14d38e feat(eda): capítulo TIMESERIES del AutomaticEDA (evolución + análisis de serie)
Capítulo nuevo build_timeseries(profile, ctx) -> Chapter|None del motor
AutomaticEDA. Cuando la tabla tiene columna de fecha/datetime, grafica la
evolución de cada columna numérica por periodo (valor agregado + conteo de filas)
y los paneles de descomposición STL y autocorrelación (ACF), con el análisis de
la serie: estacionariedad (ADF+KPSS), autocorrelación (Ljung-Box), fuerzas de
tendencia/estacionalidad (Hyndman) y la transformación sugerida (retornos o
diferencias) para evitar correlaciones espurias. Sin columna temporal devuelve
None. Consolida series OHLC casi idénticas en un único gráfico conservando el
análisis de cada columna.

La serie cruda llega por ctx['timeseries_raw'] (mismo patrón que modelos con
raw_numeric); las figuras son perezosas (Figure.make) y el paginador del núcleo
garantiza no-corte en PDF y PPTX. CHAPTER_VERSION 1.0.0.

Cubre los MUST del diseño (report 2043): MUST-9.1 (línea valor-vs-tiempo + conteo
por periodo), MUST-9.2 (paneles STL + ACF), MUST-9.3 (perfil datetime +
consolidación OHLC).

Funciones nuevas del registry (grupo eda), delegadas a fn-constructor, no inline:
- detect_time_column (pure): detecta la columna temporal y las numéricas
- profile_datetime (pure): rango/frecuencia/regularidad/huecos de la fecha
- resample_timeseries (pure): agrega la serie por periodo + conteo
- extract_timeseries_raw (impure): lee la serie cruda ordenada de DuckDB/PG

Verificación: 69 tests verdes (capítulo 9 + funciones 28 + núcleo/renderers);
golden real sobre seattle-weather (estacional) y aapl (OHLC) con PDF+PPTX sin
cortar nada (cols_cortadas=[]).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:35:42 +02:00
egutierrez fd59530751 feat(eda): capítulo AGREGACION del AutomaticEDA (groupby + pivot + barras)
Capítulo nuevo (siempre presente cuando hay categóricas agrupables) que analiza la
tabla por grupos: stats de numéricas por grupo, tablas dinámicas (pivot) y gráficos
de barras desde cero. Obtiene los datos por ctx['aggregations'] precomputado o en
vivo vía push-down (ctx['db_path']+table), siguiendo el patrón de chapters/modelos.py.
Degrada a None cuando no hay categóricas; emite los bloques del modelo (DataTable,
Markdown, Figure) para que el paginador del núcleo no corte nada en PDF ni PPTX.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:33:55 +02:00
egutierrez 96da9e3015 feat(eda): funciones de agregación/OLAP para AutomaticEDA (groupby/pivot push-down + selección LLM)
Cuatro funciones nuevas del grupo eda que nutren el capítulo AGREGACION:
- select_groupby_keys (pure): elige categóricas agrupables + numéricas medida desde el TableProfile.
- groupby_stats_duckdb (impure): GROUP BY push-down en DuckDB (count/mean/median/std/min/max por grupo).
- pivot_table_duckdb (impure): pivot A×B push-down, limitado a top filas/cols para no cortar.
- suggest_aggregations_llm (impure): el LLM elige las agregaciones interesantes con fallback determinista.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:33:55 +02:00
egutierrez 81b57f9acd merge: capitulo AutomaticEDA analisis_llm (verificado met) 2026-06-30 15:15:39 +02:00
egutierrez 02ee222dde merge: capitulo AutomaticEDA cat_distr (verificado met) 2026-06-30 15:15:39 +02:00
egutierrez ba162ab301 merge: capitulo AutomaticEDA correlacion (verificado met) 2026-06-30 15:15:39 +02:00
egutierrez 649de07d6b feat(eda): capítulo AutomaticEDA CAT DISTR + funciones cardinalidad/pie
Capítulo cat_distr del motor AutomaticEDA: distribuciones categóricas con
explicación de entropía de Shannon, métricas de cardinalidad por columna
(valores distintos, % distintos, total de filas, valores únicos, entropía y
su máximo log2(k) + normalizada), tabla top-k y un donut de las categorías
más comunes (top-k + «Otros»). Marca columnas id-like y dominadas.

Delegadas a fn-constructor (grupo eda):
- categorical_cardinality_block: deriva métricas de cardinalidad/entropía.
- categorical_top_pie_figure: figura donut top-k + «Otros», leyenda lateral.

Defensivo (dict-no-throw): None si no hay columnas categóricas; normaliza
mode_pct a escala 0-100 (summarize_categorical lo emite como fracción).
Tablas vía DataTable y figura perezosa: el paginador del núcleo garantiza
no-corte en PDF y PPTX. Tests: golden + edge (sin categóricas) + anti-corte
(label largo / muchas columnas) en ambos renderers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:04:10 +02:00
egutierrez af1dd9bcc2 test(eda): tests del capítulo ANÁLISIS LLM (golden + edges + anti-cortes)
Suite self-contained (perfil sintético + un golden, sin DuckDB):
- golden: build_analisis_llm devuelve el Chapter y el documento entero renderiza
  a PDF y PPTX con resumen, análisis sugeridos, limpieza y una columna del
  diccionario presentes.
- orden: el capítulo queda inmediatamente después de `overview`.
- edges: profile sin bloque `llm` (o None/{}/malformado/llm vacío) -> None sin
  lanzar; fallback a ctx['llm'].
- anti-cortes: diccionario de 40 filas + sugerencia de limpieza de ~150 chars se
  reparten en varias páginas/slides sin perder ninguna fila ni palabra.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:01:26 +02:00
egutierrez fc5bc334c8 feat(eda): capítulo ANÁLISIS LLM para AutomaticEDA, junto al overview
Nuevo capítulo `analisis_llm` del motor AutomaticEDA. Consume el bloque `llm`
que `eda_llm_insights` (grupo eda) ya deja en el TableProfile —no llama al LLM
ni recalcula— y lo convierte en bloques del modelo de documento para que se
renderice sin cortarse en PDF ni PPTX:

- Resumen de la tabla y significado de una fila -> bloques Markdown (el
  renderer los envuelve a líneas completas, nunca pierde texto).
- Diccionario de datos y PII -> DataTable (el paginador parte por filas
  repitiendo cabecera y envuelve celdas largas dentro de su columna).
- Análisis sugeridos y limpieza sugerida -> listas de viñetas Markdown; cada
  entrada es una línea completa que el renderer envuelve, nunca trunca.

Lectura defensiva (.get) en todo; devuelve None si el profile no trae bloque
`llm` (p.ej. profile_table sin run_llm) para omitir el capítulo.

MUST-3.2 (report 2043): se mueve `analisis_llm` en CHAPTER_ORDER a la posición
inmediatamente posterior a `overview`, como pidió el usuario ("va junto al
overview").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 15:01:26 +02:00
egutierrez 03f3dca823 feat(eda): capítulo CORRELACION de AutomaticEDA (matriz + top pares ±)
Implementa chapters/correlacion.py siguiendo el contrato de capítulos:
build_correlacion(profile, ctx) -> Chapter|None, CHAPTER_VERSION="1.0.0".

Consume profile['correlations'] (salida de association_matrix del grupo eda,
sin recalcular estadística) y emite, como bloques del modelo:

- Matriz de asociación (Figure/heatmap perezoso, RdBu_r, con signo en num-num
  y magnitud en métricas mixtas; etiquetas ordenadas por conectividad y
  recortadas a las 16 más conectadas para legibilidad).
- TOP de pares POSITIVOS y TOP de pares NEGATIVOS en dos DataTable separadas
  (los negativos son por construcción num-num, único método con signo), con
  método, valor, p-valor corregido (FDR) y significancia.
- Resumen FDR (multiple_testing) + leyenda de métodos.
- Aviso de espuriedad por niveles no estacionarios (Granger-Newbold) cuando el
  profile lo marca.

Lectura defensiva en todo (None si no hay pares; nunca lanza). Anti-cortes:
sólo bloques del modelo, el paginador parte tablas repitiendo cabecera y escala
la figura entera.

Test self-contained (5 casos): golden a nivel de bloques + golden render
PDF/PPTX, edge sin pares -> None, edge sólo positivos -> nota honesta, y
anti-corte con matriz ancha + etiquetas largas (dato íntegro a nivel de bloque,
ambos renderers sin reventar).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 14:59:50 +02:00
191 changed files with 28138 additions and 550 deletions
+141
View File
@@ -0,0 +1,141 @@
---
name: paper-reviewer
description: "Revisor académico adversarial (read-only) para los papers del subsistema `papers/`. Recibe el directorio de un paper (`papers/<slug>/`) y su `preregistration.md`, y lo juzga sin piedad: puntúa novedad, rigor, reproducibilidad y validez (0-5 cada uno), intenta REFUTAR cada claim contra la evidencia citada, detecta HARKing contra el pre-registro, y emite un veredicto estructurado (accept|major_revision|reject) con default conservador. Es el gate anti paper-mill: NO modifica el paper, solo lo evalúa."
model: opus
tools: Read, Grep, Glob, Bash
---
# Agente Paper-Reviewer — peer review adversarial
Eres un revisor académico **hostil pero justo**. Tu trabajo NO es ayudar al autor a sentirse bien: es proteger la integridad del registro científico. Asumes la posición de un revisor de conferencia top que ha visto cientos de papers inflados y sabe oler el humo. Por defecto **desconfías** de cada afirmación hasta que la evidencia citada la sostenga. Eres específico, citas líneas y archivos, y no rellenas con elogios.
Este agente es el **gate anti paper-mill** del subsistema `papers/`. El riesgo que combates: papers que *parecen* rigurosos (estructura IMRaD impecable, lenguaje académico, tablas bonitas) pero sin sustancia — hipótesis que no podían fallar, estadística de teatro, claims que exceden la evidencia, análisis inventados después de ver los datos. Si no hubo riesgo real de refutación, no es un paper.
---
## REGLA FUNDAMENTAL: read-only, solo juzgas
- **Lectura:** `paper.md`, `preregistration.md`, `references.md`/`.bib`, y todo lo que haya en `experiments/`, `data/`, `figures/`, `reviews/` del paper.
- **Escritura:** NINGUNA. No tienes Edit ni Write. No modificas el paper, no arreglas su prosa, no corriges sus tablas. Solo emites un veredicto.
- **Bash es read-only:** úsalo para inspeccionar evidencia (`ls`, `cat`, `head`, `wc`, `grep`, re-correr un script de análisis que YA exista en `experiments/` para verificar un número reportado, contar filas de un dataset, comprobar que una figura referenciada existe). NUNCA escribas archivos, NUNCA borres, NUNCA mutes estado externo (sin red con efectos, sin deploys).
---
## Input
Recibes el path de un directorio de paper:
- `paper_dir` (ej. `papers/0001-bucle-reactivo-calls`). Dentro esperas al menos `paper.md`; idealmente también `preregistration.md`, `experiments/`, `data/`, `figures/`.
Si falta `paper.md`, reporta que no hay paper que revisar y sal. Si falta `preregistration.md`, NO es excusa para aprobar: la ausencia de pre-registro es en sí misma una **amenaza grave a la validez** (no puedes distinguir análisis confirmatorios de exploratorios) y debe bajar el eje de rigor y reproducibilidad.
---
## Algoritmo de revisión
### 1. Lee todo el material primero
- `paper.md` completo (frontmatter + cuerpo IMRaD).
- `preregistration.md` (H0/H1, plan de análisis congelado, timestamp/hash si lo tiene).
- Inventaria la evidencia: `ls -R experiments/ data/ figures/`. Anota qué tablas, figuras, scripts y datasets existen REALMENTE en disco.
- Si hay `reviews/` previos, léelos para no repetir y para ver si el autor respondió a críticas anteriores.
No puntúes nada hasta haber leído el material. Una revisión sin abrir la evidencia es la enfermedad que combates.
### 2. Extrae y enumera los CLAIMS
Recorre Results y Discussion. Lista cada **afirmación de resultado** verificable (no las de contexto). Ejemplos de claim: "el método A reduce el error un 23%", "la diferencia es significativa (p<0.01)", "el efecto es grande (d=0.8)", "el patrón se mantiene en los 3 datasets". Para cada claim anota la evidencia que el paper cita (tabla X, figura Y, sección de `experiments/`).
### 3. Intenta REFUTAR cada claim
Para cada claim, posición de partida: **"no soportada"**. Solo lo marcas "soportada" si:
- La evidencia citada EXISTE en disco (la tabla/figura/dato está realmente ahí, no solo mencionada).
- El número del texto COINCIDE con el de la evidencia (si puedes re-derivarlo de un script o un CSV en `experiments/`/`data/`, hazlo con Bash y compáralo).
- La inferencia es válida: el claim no extrapola más allá de lo que el dato muestra (no confunde correlación con causalidad sin diseño que lo permita; no generaliza fuera de la población muestreada).
Si la evidencia no aparece, si el número no cuadra, o si no puedes reproducir el cálculo con lo descrito → claim **no soportada**. Apúntala en `claims_unsupported` con el motivo concreto (qué falta, qué no cuadra).
### 4. Puntúa los 4 ejes (0-5 cada uno)
Sé tacaño. 5 es excepcional y raro; 3 es "aceptable con reservas"; 0-2 es rechazo en ese eje. Justifica cada número con una frase concreta.
- **novelty (novedad):** ¿el paper aporta algo que no se sabía? ¿El gap está articulado y la contribución es explícita y real, o es un resultado obvio/ya conocido revestido de novedad? Related work honesto (reconoce lo que ya existe) sube; reinventar la rueda baja.
- **rigor:** método reproducible y estadística correcta. Exige: **effect size + intervalos de confianza**, no solo `p<0.05`; **corrección por comparaciones múltiples** (Holm-Bonferroni o similar) si se testean varias hipótesis; N justificado (no insuficiente); ausencia de p-hacking/cherry-picking. Estadística de teatro (p-valor suelto sin tamaño de efecto, "tendencia hacia la significancia", N=3 presentado como concluyente) hunde este eje.
- **reproducibility (reproducibilidad):** ¿otra persona puede re-correr el experimento con lo descrito? Exige protocolo, datos accesibles (o su descripción), código en `experiments/`, semillas/versiones. Si tú mismo no podrías reproducirlo con lo que hay, el eje es bajo. Pre-registro presente y seguido sube; ausente baja.
- **validity (validez):** las cuatro validez de Shadish/Cook/Campbell — **interna** (¿la causa es realmente la causa, o hay confusores?), **externa** (¿generaliza fuera de esta muestra?), **de constructo** (¿se mide lo que se dice medir?), **estadística** (¿las inferencias estadísticas son legítimas?). El paper debe DECLARAR sus amenazas a la validez. Amenazas no declaradas que tú detectas → bajan el eje y van a `gaps`.
### 5. Chequea coherencia con el pre-registro (HARKing)
Compara los análisis REPORTADOS en Results contra los PRE-REGISTRADOS en `preregistration.md`:
- ¿Los análisis confirmatorios presentados son exactamente los pre-registrados? Si aparecen análisis NO declarados presentados como si fueran confirmatorios → **HARKing** (Hypothesizing After Results are Known). Marca `harking_detected: true`.
- ¿Hay análisis pre-registrados que desaparecieron del paper (resultados incómodos enterrados)? Eso es cherry-picking — anótalo en `gaps`.
- Análisis exploratorios son legítimos SOLO si el paper los etiqueta honestamente como exploratorios (generan hipótesis, no las confirman). Presentar exploratorio como confirmatorio = HARKing.
- Si no hay `preregistration.md`, no puedes verificar esto: anótalo como amenaza grave y trata todos los resultados como potencialmente exploratorios.
### 6. Verifica honestidad: limitaciones y overclaiming
- ¿Hay una sección de **limitaciones / amenazas a la validez** declarada honestamente? Su ausencia es una bandera roja: ningún estudio real está libre de limitaciones.
- ¿Las **claims ≤ evidencia**? Compara el lenguaje de las conclusiones con lo que los datos permiten. "demostramos que X causa Y" sobre un diseño correlacional = **overclaiming**. "el método es superior" sobre un solo dataset = overclaiming. Lista cada overclaim en `gaps`.
### 7. Emite el veredicto
Default conservador. Reglas de decisión:
- **reject** si: hay claims no soportadas centrales al paper, O HARKing detectado, O rigor ≤ 2, O validez ≤ 2, O no hay riesgo real de refutación (la hipótesis no podía fallar).
- **major_revision** si: el núcleo es salvable pero hay gaps serios (evidencia incompleta, estadística mejorable, amenazas no declaradas, pre-registro ausente) — el caso por defecto cuando algo falta pero no es fraude.
- **accept** SOLO si: los 4 ejes ≥ 3, cero claims no soportadas centrales, sin HARKing, limitaciones declaradas, claims ≤ evidencia, reproducible. Es raro y hay que ganárselo.
Ante la duda, baja, no subas. Es preferible un major_revision injusto que dejar pasar un paper-mill.
---
## Output (formato obligatorio)
Devuelve un bloque JSON con EXACTAMENTE esta forma, seguido de un párrafo corto de justificación en prosa (crítico y específico, sin elogios de relleno):
```json
{
"scores": {
"novelty": 0,
"rigor": 0,
"reproducibility": 0,
"validity": 0
},
"claims_unsupported": [
"Claim '<texto>': <por qué no está soportada — evidencia ausente / número no cuadra / inferencia inválida>"
],
"harking_detected": false,
"gaps": [
"<amenaza a la validez no declarada / overclaim / estadística faltante / dato no reproducible>"
],
"verdict": "reject"
}
```
Reglas del output:
- `scores`: enteros 0-5. Tacaño por defecto.
- `claims_unsupported`: una entrada por claim que no superó la refutación, con el motivo concreto. Lista vacía solo si TODAS las claims se sostuvieron contra la evidencia.
- `harking_detected`: `true` en cuanto detectes un análisis confirmatorio no pre-registrado, o si la ausencia de pre-registro impide descartarlo (en ese caso explícalo en `gaps`).
- `gaps`: amenazas a la validez no declaradas, overclaims, estadística de teatro, datos no reproducibles. Concreto y accionable.
- `verdict`: `accept` | `major_revision` | `reject`. Default conservador según las reglas de la sección 7.
El párrafo de prosa que sigue al JSON resume el veredicto en lenguaje directo: qué hunde el paper o qué falta para subir de nivel. Sin "buen trabajo", sin "interesante contribución" de relleno — solo señal.
---
## Tono y anti-patrones
- **Crítico y específico.** "La tabla 2 reporta p=0.03 pero no da tamaño de efecto ni CI; con N=4 esto no sostiene el claim de la sección 4.2" — no "la estadística podría mejorarse".
- **Cita evidencia.** Siempre `archivo:línea` o `tabla/figura X`. Una crítica sin cita es ruido.
- **No inventes mérito.** Si el paper no aporta novedad, dilo. El sesgo de complacencia es el que alimenta los paper-mills.
- **No arregles el paper.** No es tu trabajo (no tienes Write). Tu trabajo es el veredicto. Sugiere QUÉ falta, no escribas el fix.
- **Default a fallar.** Evidencia ausente = claim no soportada. Pre-registro ausente = no se puede descartar HARKing. Duda = baja la nota.
## Relación con el ecosistema
- Es la materialización del **paso 9 (peer review)** del proceso de 10 pasos del subsistema `papers/` (ver `reports/0001-2026-06-30-papers-system-design.md`), heredando el patrón de **verificador adversarial** del modo orquestador (`.claude/rules/orchestration.md`): un juez independiente que por defecto refuta y solo aprueba con evidencia.
- Sus outputs se guardan en `papers/<slug>/reviews/` para trazar la evolución del paper entre revisiones.
- Complementa el `preregister_hypothesis` (rigor experimental, congela la hipótesis antes de los datos) y `render_paper_pdf` (entrega): este agente es el control de calidad que decide si el paper merece convertirse en PDF entregable o volver a revisión.
+22 -10
View File
@@ -25,9 +25,11 @@ Página madre del grupo: `docs/capabilities/eda.md` (léela primero para cargar
- `--models``run_models=True` (PCA/KMeans/IsolationForest/normalidad).
- `--llm``run_llm=True` (1 call LLM sobre el perfil agregado).
- `--series``run_series=True` (estacionariedad ADF+KPSS, ACF/PACF, STL, retornos por columna numérica).
- `--pdf``emit_pdf=True` (PDF A5 vertical legible en móvil).
- `--pdf``emit_pdf=True` (PDF A5 legacy de `render_eda_pdf`, legible en móvil).
- `--legacy-only` → emite SOLO el PDF legacy (sin AutomaticEDA), para casos en que solo se quiera el PDF rápido.
- `--lite` / `--bajo-consumo``render_automatic_eda(profile_level="lite")`: EDA barato y rápido (CI, vistazo previo, máquina sin GPU/red). Apaga LLM y serie temporal y limita los modelos a **PCA + normalidad** (sin KMeans ni IsolationForest, lo caro en CPU), con `sample` reducido. `--full``profile_level="full"` (standard + narrativa LLM). Por defecto `profile_level="standard"` (comportamiento histórico). Un flag explícito (`--llm`, `--models`, ...) prima sobre el preset.
Por defecto, para un EDA "completo" cuando el usuario no especifica, activa `run_models`, `run_series` y `emit_pdf`; deja `run_llm` para cuando lo pida o cuando interese la interpretación semántica (es la única parte que gasta tokens del modelo).
Por defecto, **un EDA completo emite SIEMPRE el informe AutomaticEDA en sus dos formatos: PDF (A5 móvil) Y PPTX (16:9 para compartir)** con los 11 capítulos poblados (portada, overview, distribuciones, calidad, correlaciones, modelos, series, geoespacial, agregación, interpretación LLM). Usa el pipeline `render_automatic_eda` (o `profile_table(emit_automatic=True)`), que activa `run_models` y `run_series` para que los capítulos de modelos/series/geoespacial/agregación salgan poblados. Deja `run_llm` para cuando el usuario lo pida o interese la interpretación semántica + narrativa por capítulo (es la única parte que gasta tokens del modelo).
## Reglas duras
@@ -35,7 +37,7 @@ Por defecto, para un EDA "completo" cuando el usuario no especifica, activa `run
2. **CSV/Parquet/Excel** entran cargándolos antes a DuckDB (`read_csv_auto`/`read_parquet`/`read_xlsx`) — DuckDB es el motor por defecto. No traigas la tabla entera a RAM.
3. **Secretos**: si la fuente es un DSN PostgreSQL con credenciales, NO las imprimas en los reports ni en el notebook; resuélvelas vía `resolve_pg_dsn`/`pass` cuando aplique.
4. **El report es un artefacto local**: vive en `reports/` (gitignored), no se sube a Gitea ni se versiona. Compartir = pasar la ruta (regla `reports.md`).
5. **Entrega las 4 salidas**: JSON sidecar + Markdown + **PDF móvil** + **notebook Jupyter colaborativo ejecutado en vivo**.
5. **Entrega las salidas**: el informe **AutomaticEDA PDF + PPTX** (siempre, con `render_automatic_eda` / `emit_automatic=True`) + (opcional) JSON sidecar + Markdown + PDF legacy + **notebook Jupyter colaborativo ejecutado en vivo**. Comparte las rutas de PDF y PPTX.
## Paso 1 — Perfilar y escribir los reports
@@ -43,18 +45,27 @@ Una tabla (caso normal):
```bash
PYTHONPATH=python/functions python/.venv/bin/python3 - <<'PYEOF'
from pipelines.profile_table import profile_table
r = profile_table(
from pipelines.render_automatic_eda import render_automatic_eda
# Informe AutomaticEDA COMPLETO one-shot: perfil + ctx (datos crudos) + PDF + PPTX
# con los 11 capítulos poblados (clusters pintados, evolución temporal, mapa,
# tablas de agregación). run_llm=True añade la narrativa LLM por capítulo.
r = render_automatic_eda(
"/ruta/datos.duckdb", "ventas",
run_models=True, run_series=True, emit_pdf=True, run_llm=False,
profile_level="standard", # "lite" = bajo consumo CPU/LLM; "full" = + narrativa LLM
out_dir="reports",
)
print("status:", r["status"])
print("md: ", r["report_md_path"])
print("json: ", r["report_json_path"])
print("pdf: ", r["pdf_path"])
print("pdf: ", r["pdf_path"], "(", r["n_pages"], "págs )")
print("pptx: ", r["pptx_path"], "(", r["n_slides"], "slides )")
print("manifest:", r["manifest_path"])
PYEOF
```
Si además quieres el report Markdown + JSON sidecar y/o el PDF legacy junto al
AutomaticEDA, usa `profile_table(emit_automatic=True, emit_pdf=True, write_report=True)`:
emite todo a la vez (`report_md_path`, `report_json_path`, `pdf_path` legacy,
`aeda_pdf_path`, `aeda_pptx_path`, `aeda_manifest_path`).
Una base entera (todas las tablas + relaciones FK):
```bash
@@ -90,6 +101,7 @@ Sigue la memoria `eda-workflow-registry` y la regla `notebook_collaboration.md`:
## Notas
- El `TableProfile` lleva ahora, además del perfilado base y las correlaciones con FDR: `series` (por columna numérica, con `run_series`), `reexpression` por columna numérica (escalera de Tukey) y `caveats` (siempre, avisos exploratorios). El Markdown y el PDF renderizan estas secciones automáticamente cuando están presentes.
- El PDF (`emit_pdf`) está pensado para leerse en el móvil (A5 vertical, tipografía grande, gráficos Tufte). Se escribe junto al Markdown en `reports/`.
- El informe **AutomaticEDA** (`render_automatic_eda` / `emit_automatic=True`) emite el MISMO documento por capítulos a **PDF (A5 móvil)** y **PPTX (16:9)** con garantía de no-corte (texto envuelto, tablas partidas repitiendo cabecera, figuras escaladas) y negrita real (`**texto**`). Escribe `automatic_eda_manifest.json` con la versión de cada capítulo. Los capítulos modelos/series/geoespacial/agregación se pueblan con los datos crudos que `build_eda_render_ctx` muestrea de la base (no se traen tablas enteras a RAM).
- El PDF legacy (`emit_pdf`, `render_eda_pdf`) sigue disponible y es independiente del AutomaticEDA (A5 vertical, gráficos Tufte). Se escribe junto al Markdown en `reports/`.
- `run_series` ordena por la primera columna datetime si existe; si no, por el orden físico de filas. Necesita ≥8 puntos válidos por columna.
- Fuentes: DuckDB (CSV/Parquet/Excel cargados antes) y PostgreSQL (`backend="postgres"`). `profile_database` (multi-tabla + FK) es solo DuckDB por ahora.
+26 -3
View File
@@ -1,6 +1,6 @@
---
description: Muestra la flota de Claudes vivos (sessionId + objetivo + estado) y, con argumento, salta con foco a esa conversación dentro de la sesión tmux fleet.
argument-hint: "[texto|sessionId|PID para saltar — vacío = listar la flota]"
description: Muestra la flota de Claudes vivos (sessionId + objetivo + estado) y, con argumento, salta con foco a esa conversación dentro de la sesión tmux fleet. `/fleet show` trae la TUI al contexto tmux actual.
argument-hint: "[show | texto|sessionId|PID para saltar — vacío = listar la flota]"
---
# /fleet — ver y navegar la flota de Claudes
@@ -33,9 +33,32 @@ cd "${FN_REGISTRY_ROOT:-$HOME/fn_registry}/apps/fleetview" && go build -o fleetv
- la sesión actual / orquestador si la puedes identificar (su `session_id` coincide con el de quien invoca).
4. Si la lista está vacía, indícalo y sugiere que el perfil fleet podría no estar activo (revisar `$FLEET_SOCKET` y que la sesión tmux exista).
### `show` → traer la TUI al contexto tmux actual
Si `$ARGUMENTS` es exactamente `show` (alias `open`/`attach`), el usuario quiere
volver a ver el panel FleetView en el contexto/pane actual sin abrir ninguna
ventana ni arrancar una flota nueva. Ejecuta:
```bash
"${FN_REGISTRY_ROOT:-$HOME/fn_registry}/apps/fleetview/fleetview" show
```
Comportamiento (decidido por la app, no abre terminal externa):
- **dentro de tmux con la flota viva** → `select-window` de la window `console`
del socket fleet (trae la TUI al frente; no abre nada).
- **fuera de tmux** → `attach` a la sesión fleet en la terminal actual (la reutiliza).
- **sin flota viva** → error claro, exit 1, no abre nada (sugiere arrancarla con
`fleetclaude`).
Es el equivalente del comportamiento de `fleetclaude` sin args invocado dentro de
una flota viva (reuse de contexto): úsalo cuando ya tengas una flota corriendo y
solo quieras recuperar la vista del panel. Para abrir una flota NUEVA aparte, usa
`fleetclaude --new` (no este comando).
### Con argumentos → saltar con foco
El usuario quiere que la interfaz tmux salte a una conversación concreta. `$ARGUMENTS` es el query: texto del objetivo, prefijo de `sessionId`, o PID.
El usuario quiere que la interfaz tmux salte a una conversación concreta. `$ARGUMENTS` es el query: texto del objetivo, prefijo de `sessionId`, o PID (cualquier valor que no sea `show`).
1. Ejecuta:
```bash
+7
View File
@@ -54,6 +54,13 @@ reports/*
!reports/.gitkeep
projects/*/reports/
# Papers — artefacto local: papers académicos reproducibles. En fase interna viven
# local y gitignored (como los reports); al promocionar a fase publishable se
# vuelven sub-repo Gitea propio (como apps/analyses). Solo el marcador .gitkeep se
# versiona. Convención: docs/capabilities/papers.md
papers/*
!papers/.gitkeep
# Node / pnpm
**/node_modules/
+73 -28
View File
@@ -3,10 +3,10 @@ name: launch_fleetclaude
kind: function
lang: bash
domain: infra
version: "1.6.0"
version: "1.7.0"
purity: impure
signature: "launch_fleetclaude [--cwd <dir>] [--bin <path>] [--session <name>] [--reuse] [--cols <n>]"
description: "Entrypoint de FleetView: abre una ventana de terminal con una sesion tmux (socket aislado por perfil) de dos panes (TUI fleetview a la izquierda, claude --dangerously-skip-permissions a la derecha) para centralizar la flota de Claudes. La terminal se AUTO-DETECTA sin config por PC: kitty si esta instalado y hay display ($DISPLAY/$WAYLAND_DISPLAY), si no Windows Terminal (wt.exe) en WSL adjuntando via wsl.exe. El pane de la TUI corre dentro del bucle supervisor supervise_fleetview_tui, que la relanza si muere (crash/panic/kill), asi el panel de control NUNCA se pierde. Soporta PERFILES multiples: sin --session/--reuse cada invocacion abre un perfil nuevo (fleet, fleet2, fleet3, ...) con su propia flota; inyecta FLEET_SOCKET/FLEET_SESSION a la TUI para que cada panel vea solo sus Claudes. Instala atajos alt+flechas/alt+enter/alt+n que controlan la TUI desde cualquier pane, y fija el ancho del sidebar con hooks."
signature: "launch_fleetclaude [--cwd <dir>] [--bin <path>] [--session <name>] [--reuse] [--new] [--cols <n>]"
description: "Entrypoint de FleetView: abre una ventana de terminal con una sesion tmux (socket aislado por perfil) de dos panes (TUI fleetview a la izquierda, claude --dangerously-skip-permissions a la derecha) para centralizar la flota de Claudes. REUSO DE CONTEXTO: si se invoca DENTRO de una flota tmux viva (su window 'console') sin --new, NO abre ventana ni crea un perfil nuevo; trae la TUI al pane/contexto actual (equivale a 'fleetview show'). El flag --new fuerza una flota+ventana nueva aunque estes en tmux. La terminal se AUTO-DETECTA sin config por PC: kitty si esta instalado y hay display ($DISPLAY/$WAYLAND_DISPLAY), si no Windows Terminal (wt.exe) en WSL adjuntando via wsl.exe. El pane de la TUI corre dentro del bucle supervisor supervise_fleetview_tui, que la relanza si muere (crash/panic/kill), asi el panel de control NUNCA se pierde. Soporta PERFILES multiples: fuera de tmux, o con --new, cada invocacion abre un perfil nuevo (fleet, fleet2, fleet3, ...) con su propia flota; inyecta FLEET_SOCKET/FLEET_SESSION a la TUI para que cada panel vea solo sus Claudes. Instala atajos alt+flechas/alt+enter/alt+n que controlan la TUI desde cualquier pane, y fija el ancho del sidebar con hooks."
tags: [claude-fleet, infra, kitty, tmux, claude, fleetview, launcher, wsl, windows-terminal]
params:
- name: --cwd
@@ -14,12 +14,14 @@ params:
- name: --bin
desc: "Ruta al binario de la TUI fleetview que corre en el pane izquierdo. Opcional. Default: <repo>/apps/fleetview/fleetview. Si no es ejecutable, el pane izquierdo muestra un mensaje de como compilarla y deja una shell viva."
- name: --session
desc: "Fija el perfil (socket+sesion tmux comparten nombre) por nombre exacto; reutiliza el existente si ya vive (idempotente sobre ese nombre). Opcional. Sin esta opcion, el perfil se elige automaticamente (primer nombre libre de la secuencia fleet, fleet2, ...)."
desc: "Fija el perfil (socket+sesion tmux comparten nombre) por nombre exacto; reutiliza el existente si ya vive (idempotente sobre ese nombre). Opcional. Sin esta opcion, el perfil se elige automaticamente (primer nombre libre de la secuencia fleet, fleet2, ...). Invocado DENTRO de tmux con un nombre DISTINTO al de la flota actual equivale a --new (pides otra flota: ventana nueva, sin reuse de contexto)."
- name: --reuse
desc: "Reattach al perfil principal 'fleet' en vez de abrir uno nuevo. Opcional. Recupera el comportamiento idempotente clasico (volver a invocar NO duplica la flota, reusa la existente)."
- name: --new
desc: "Fuerza una flota NUEVA en una ventana NUEVA (kitty/wt.exe) incluso estando dentro de una flota tmux. Opcional. Es la via explicita para abrir una FleetView aparte; sin este flag, invocado dentro de una flota viva se reusa el contexto actual (no abre ventana ni crea perfil)."
- name: --cols
desc: "Ancho en columnas del pane izquierdo (la TUI). Opcional. Default: 40."
output: "Crea/reutiliza una sesion tmux detached con dos panes y lanza una ventana de terminal 'FleetView' adjunta a ella (kitty o Windows Terminal segun auto-deteccion), desacoplada del shell padre. Imprime el estado por stdout. Sin valor de retorno; exit 0 en exito."
output: "Caso reuse de contexto (dentro de una flota tmux viva, sin --new): trae la TUI al pane/contexto actual con select-window de la window 'console' (o 'fleetview show' si el binario existe) y retorna 0, sin abrir nada. Caso ventana-nueva (fuera de tmux, o con --new): crea/reutiliza una sesion tmux detached con dos panes y lanza una ventana de terminal 'FleetView' adjunta (kitty o Windows Terminal segun auto-deteccion), desacoplada del shell padre. Imprime el estado por stdout. Sin valor de retorno; exit 0 en exito, !=0 con mensaje claro si no hay terminal ni contexto que reusar."
uses_functions:
- supervise_fleetview_tui_bash_infra
uses_types: []
@@ -36,32 +38,44 @@ file_path: "bash/functions/infra/launch_fleetclaude.sh"
## Ejemplo
```bash
# Via fn run (resuelve por nombre o ID):
fn run launch_fleetclaude
# DENTRO de una flota tmux viva (p. ej. en el pane del orquestador): reusa el
# contexto, trae la TUI al pane actual. NO abre ventana ni crea perfil nuevo.
fleetclaude
# Perfil nuevo automatico (fleet la 1a vez; fleet2, fleet3, ... si ya hay uno):
launch_fleetclaude
# FUERA de tmux: perfil nuevo automatico (fleet la 1a vez; fleet2, ... si ya hay
# uno) en una ventana de terminal nueva, reutilizando la terminal actual (attach):
fleetclaude
# Forzar una flota+ventana NUEVA aunque estes dentro de una flota tmux:
fleetclaude --new
# Reattach a la flota principal 'fleet' (comportamiento idempotente clasico):
launch_fleetclaude --reuse
fleetclaude --reuse
# Perfil con nombre fijo y ancho de pane personalizado:
launch_fleetclaude --session trabajo --cols 50
fleetclaude --session trabajo --cols 50
# Via fn run (resuelve por nombre o ID):
fn run launch_fleetclaude
```
Tras invocarlo aparece una ventana de terminal titulada `FleetView (<perfil>)` con dos
panes lado a lado: a la izquierda la TUI `fleetview`, a la derecha una sesion de
`claude --dangerously-skip-permissions`. Cada perfil es un socket+sesion tmux
aislados con su propia flota: puedes tener varias FleetView abiertas a la vez.
Por defecto, volver a invocarlo abre un perfil NUEVO (no reusa); usa `--reuse`
o `--session <nombre>` para volver a una flota concreta.
Dentro de una flota viva, `fleetclaude` sin args reusa el contexto (la window
`console` pasa al frente). Fuera de tmux (o con `--new`) aparece una ventana de
terminal titulada `FleetView (<perfil>)` con dos panes lado a lado: a la izquierda
la TUI `fleetview`, a la derecha una sesion de `claude --dangerously-skip-permissions`.
Cada perfil es un socket+sesion tmux aislados con su propia flota: puedes tener
varias FleetView abiertas a la vez con `--new`.
## Cuando usarla
Usala cuando quieras un unico punto de entrada a la flota de Claudes en vez de
N ventanas kitty sueltas: lanzas `fleetclaude` y tienes la TUI de control y un
Claude listo para trabajar en la misma ventana. Tipico al empezar la jornada o
al retomar el trabajo en el repo `fn_registry`.
al retomar el trabajo en el repo `fn_registry`. Si **ya estas dentro de una
flota** (en el pane del orquestador) y solo quieres volver a ver la TUI, lanza
`fleetclaude` sin args: trae el panel al contexto actual sin abrir otra ventana
ni arrancar una flota duplicada. Usa `--new` solo cuando quieras DELIBERADAMENTE
una segunda flota aparte.
## Gotchas
@@ -87,10 +101,27 @@ al retomar el trabajo en el repo `fn_registry`.
funciona en un PC con kitty y en otro WSL sin kitty, cada uno elige su
terminal. Causa raiz del sintoma "se lanza la flota pero no se ve": kitty no
instalado en WSL hacia que la sesion tmux se creara sin ventana que la mostrara.
- **Dentro de tmux abre ventana nueva**: si invocas `fleetclaude` desde dentro de
una sesion tmux (`$TMUX` definido), NO hace `attach` anidado (rompe / avisa de
nesting); cae a la ruta ventana-nueva (auto-deteccion de terminal). Fuera de
tmux y con TTY, reutiliza la terminal actual con `exec tmux attach`.
- **Dentro de una flota tmux viva: reuse de contexto (no ventana nueva)**: si
invocas `fleetclaude` sin `--new` desde dentro de una flota fleetview viva
(`$TMUX` definido y el socket actual tiene una sesion homonima con window
`console`), NO abre ventana ni crea un perfil `fleetN+1`: trae la TUI al pane
actual (`fleetview show`, o `tmux -L <perfil> select-window -t <perfil>:console`
si el binario no esta compilado) y retorna 0. El perfil de la flota actual se
deriva de `$TMUX` (basename del socket = nombre `-L`), senal fiable aunque
`$FLEET_SOCKET` venga vacio (ver `detect_fleet_context`). **`--new`** fuerza el
comportamiento clasico (flota+ventana nueva); pasar `--session <otro>` distinto
al perfil actual equivale a `--new` implicito. Fuera de tmux y con TTY, reutiliza
la terminal actual con `exec tmux attach` (nunca `attach` anidado dentro de
tmux). Sin TTY ni contexto que reusar (atajo de escritorio/cron) cae a la ruta
ventana-nueva. Antes de este fix (v1.6.0 y anteriores) cualquier `fleetclaude`
dentro de tmux abria una kitty nueva y un socket `fleetN+1` — el sintoma que
acumulaba 6+ sockets `fleet*`.
- **`local x` unbound bajo `set -u`**: el archivo corre con `set -euo pipefail`.
`local left_pane right_pane` dejaba esas vars *unbound* (no vacias), asi que la
rama "reutilizar sesion existente" (`--reuse`/`--session <vivo>`) reventaba con
`left_pane: unbound variable` al evaluar `[[ -z "$left_pane" ]]`. Se inicializan
explicitamente a `""` (`local left_pane="" right_pane=""`). Si tocas estas vars,
no vuelvas a declararlas sin valor.
- **kitty detached (setsid)**: la ventana kitty se lanza con `setsid ... &` para
sobrevivir al cierre de la terminal que la invoco. La ventana de Windows
Terminal (wt.exe) ya es un proceso Windows independiente del arbol Linux, asi
@@ -128,15 +159,29 @@ al retomar el trabajo en el repo `fn_registry`.
- **Ancho del sidebar via hooks**: `client-resized` y `window-layout-changed`
re-fijan el pane 0 (TUI) a `--cols` columnas, porque el `attach` de kitty y el
conmutar de Claude redistribuyen el espacio.
- **tmux siempre; terminal (kitty/wt.exe) solo sin TTY**: `tmux` es obligatorio
(aborta != 0 si falta). Una terminal nueva (kitty o Windows Terminal) solo se
necesita en la ruta sin-TTY (dentro de tmux, atajo de escritorio, cron, script),
donde abre una ventana nueva. Invocado desde una terminal interactiva fuera de
tmux (el caso normal del alias `fleetclaude`), reutiliza la terminal actual con
`exec tmux attach` y no necesita ni kitty ni wt.exe.
- **tmux siempre; terminal (kitty/wt.exe) solo en la ruta ventana-nueva**: `tmux`
es obligatorio (aborta != 0 si falta). Una terminal nueva (kitty o Windows
Terminal) solo se necesita en la ruta ventana-nueva: `--new`, o sin TTY ni flota
viva que reusar (atajo de escritorio, cron, script). Dentro de una flota viva sin
`--new` se reusa el contexto (ni kitty ni wt.exe). Invocado desde una terminal
interactiva fuera de tmux (el caso normal del alias `fleetclaude`), reutiliza la
terminal actual con `exec tmux attach` y tampoco necesita kitty ni wt.exe.
## Capability growth log
- v1.7.0 (2026-06-30) — **reuse de contexto dentro de la flota + flag `--new`**.
Invocado sin `--new` desde dentro de una flota tmux viva (su window `console`),
`fleetclaude` ya NO abre una kitty nueva ni crea un perfil `fleetN+1`: trae la
TUI al pane/contexto actual (`fleetview show`, o `tmux -L <perfil> select-window
-t <perfil>:console` como fallback sin binario) y retorna 0. El perfil actual se
deriva de `$TMUX` (basename del socket); pasar `--session <otro>` distinto al
actual equivale a `--new` implicito. Nuevo flag `--new` para forzar la ruta
clasica (flota+ventana nueva) aun dentro de tmux. Fuera de tmux el comportamiento
es intacto (`exec tmux attach` reutiliza la terminal). Arregla el sintoma de que
lanzar `fleetclaude` dentro de una flota abria ventana kitty + socket nuevo
(`fleet7`, `fleet8`, ...). Fix incidental: `local left_pane="" right_pane=""`
(antes `local left_pane right_pane` reventaba con `unbound variable` bajo
`set -u` al reutilizar una sesion existente).
- v1.6.0 (2026-06-29) — **auto-deteccion de terminal (kitty ↔ Windows Terminal)**.
La ruta ventana-nueva ya no asume kitty: elige terminal segun el host. kitty si
esta instalado y hay display (`$DISPLAY`/`$WAYLAND_DISPLAY`); si no, en WSL abre
+61 -2
View File
@@ -23,6 +23,7 @@ launch_fleetclaude() {
local cols=52
local explicit_session=0 # 1 si el usuario pasó --session <name> a mano
local reuse=0 # 1 si el usuario pidió --reuse (reattach al perfil principal)
local want_new=0 # 1 si el usuario pidió --new (forzar flota+ventana nueva)
local T="" # socket tmux aislado; se fija al resolver el perfil
# -----------------------------------------------------------------------
@@ -46,6 +47,9 @@ launch_fleetclaude() {
--reuse)
reuse=1
;;
--new)
want_new=1
;;
--cols)
shift
cols="${1:-40}"
@@ -62,6 +66,11 @@ Claudes). Sin --session ni --reuse, cada invocacion abre un perfil NUEVO: usa
el primer nombre libre de la secuencia fleet, fleet2, fleet3, ... Asi puedes
tener varias FleetView abiertas a la vez, cada una con su flota independiente.
REUSO DE CONTEXTO: si ya estas DENTRO de una flota tmux viva (p. ej. en el pane
del orquestador), 'fleetclaude' sin args NO abre una ventana ni crea un perfil
nuevo: trae la TUI al contexto/pane actual (equivale a 'fleetview show'). Para
abrir explicitamente una flota aparte en una ventana nueva, usa --new.
Opciones:
--cwd <dir> Directorio de trabajo de los panes.
Default: raiz del repo fn_registry (derivada dinamicamente).
@@ -69,13 +78,21 @@ Opciones:
Default: <repo>/apps/fleetview/fleetview
--session <name> Fija el perfil (socket+sesion) por nombre exacto; reutiliza
el existente si ya esta vivo. Sin esta opcion, perfil auto.
Si se invoca DENTRO de tmux con un nombre DISTINTO al de la
flota actual, equivale a --new (pides otra flota).
--reuse Reattach al perfil principal 'fleet' en vez de abrir uno
nuevo (vuelve al comportamiento idempotente clasico).
--new Fuerza una flota NUEVA en una ventana NUEVA (kitty/wt.exe),
incluso dentro de tmux. Es la via explicita para tener una
FleetView aparte; sin este flag, dentro de tmux se reusa el
contexto actual.
--cols <n> Ancho (columnas) del pane izquierdo. Default: 40.
-h, --help Muestra esta ayuda.
Ejemplos:
launch_fleetclaude # perfil nuevo (fleet, luego fleet2, ...)
launch_fleetclaude # dentro de la flota: reusa el contexto;
# fuera de tmux: perfil nuevo (fleet, ...)
launch_fleetclaude --new # flota+ventana nueva aunque estes en tmux
launch_fleetclaude --reuse # reattach a la flota principal 'fleet'
launch_fleetclaude --session trabajo # perfil con nombre fijo 'trabajo'
launch_fleetclaude --cwd ~/fn_registry --cols 50
@@ -127,6 +144,45 @@ USAGE
return 1
fi
# -----------------------------------------------------------------------
# REUSO DE CONTEXTO (sin --new): si ya estamos DENTRO de una flota tmux
# viva, 'fleetclaude' sin args NO abre una ventana/terminal nueva ni crea
# un perfil fleetN+1 — trae la TUI al contexto/pane actual, igual que
# 'fleetview show'. El flag --new fuerza el comportamiento clasico (flota
# nueva en ventana nueva); --reuse mantiene su semantica historica.
#
# El perfil de la flota actual se deriva de $TMUX (el basename del socket
# es el nombre -L; senal fiable aunque $FLEET_SOCKET venga vacio, ver
# detect_fleet_context). Si se paso --session con un nombre DISTINTO al
# actual, es pedir OTRA flota -> se trata como --new implicito (no reusa).
# "Flota viva" = el socket tiene una sesion homonima con una window
# 'console' (la firma de una FleetView), no un tmux cualquiera.
# -----------------------------------------------------------------------
if [[ "$want_new" -eq 0 && "$reuse" -eq 0 && -n "${TMUX:-}" ]]; then
local current_socket target_socket
current_socket="$(basename "${TMUX%%,*}")"
target_socket="$current_socket"
[[ "$explicit_session" -eq 1 ]] && target_socket="$session"
if [[ "$target_socket" == "$current_socket" ]] \
&& tmux -L "$current_socket" has-session -t "$current_socket" 2>/dev/null \
&& tmux -L "$current_socket" list-windows -t "$current_socket" \
-F '#{window_name}' 2>/dev/null | grep -qx console; then
# Traer la TUI al contexto actual sin abrir nada nuevo. Preferimos
# el binario (centraliza la politica en la app: 'fleetview show');
# si no esta compilado, caemos a 'select-window' directo, que es lo
# que 'show' hace por dentro dentro de tmux (cero dependencia).
if [[ -x "$bin" ]] \
&& FLEET_SOCKET="$current_socket" FLEET_SESSION="$current_socket" \
"$bin" show 2>/dev/null; then
return 0
fi
tmux -L "$current_socket" select-window -t "$current_socket":console
echo "launch_fleetclaude: flota '$current_socket' viva; TUI traida al contexto actual (sin ventana nueva)."
return 0
fi
fi
# -----------------------------------------------------------------------
# Resolver el PERFIL (socket+sesion tmux comparten nombre).
#
@@ -200,7 +256,10 @@ USAGE
# indice 1 y cualquier referencia a console.0 falla con
# "can't find pane: 0". Los pane ID son estables e inmunes al base-index.
# -----------------------------------------------------------------------
local left_pane right_pane
# Inicializadas a "" (no solo declaradas): bajo `set -u` una `local x` sin
# valor queda *unbound*, y al reutilizar una sesion existente el `[[ -z
# "$left_pane" ]]` de mas abajo reventaba con "unbound variable".
local left_pane="" right_pane=""
if $T has-session -t "$session" 2>/dev/null; then
echo "launch_fleetclaude: la sesion tmux '$session' ya existe; reutilizandola."
else
+58
View File
@@ -0,0 +1,58 @@
---
name: next_numbered_dir
kind: function
lang: bash
domain: io
version: "1.0.0"
purity: impure
signature: "next_numbered_dir(parent_dir: string, [width: int]) -> string"
description: "Calcula el siguiente prefijo numerico NNNN- para un directorio numerado incremental. Escanea los subdirectorios directos de parent_dir cuyo nombre empiece por NNNN- (4+ digitos seguidos de guion), toma el maximo, le suma 1 y lo imprime con zero-padding al ancho width (default 4). Si parent_dir no existe o no tiene subdirs que matcheen, imprime 0001."
tags: [papers, io, scaffold]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
params:
- name: parent_dir
desc: "directorio padre cuyos subdirectorios numerados (NNNN-...) se escanean; obligatorio"
- name: width
desc: "ancho del zero-padding del numero impreso (default 4); opcional"
output: "el siguiente numero como string con zero-padding a width digitos a stdout (ej. 0003); usage a stderr y exit 1 si falta parent_dir"
tested: false
tests: []
test_file_path: ""
file_path: "bash/functions/io/next_numbered_dir.sh"
---
## Ejemplo
```bash
source bash/functions/io/next_numbered_dir.sh
# Sobre un papers/ que ya contiene 0001-foo y 0002-bar
mkdir -p /tmp/papers/{0001-foo,0002-bar}
next_numbered_dir /tmp/papers
# -> 0003
# Directorio vacio o inexistente -> primer numero
next_numbered_dir /tmp/papers_nuevo
# -> 0001
# Ancho de padding distinto
next_numbered_dir /tmp/papers 6
# -> 000003
```
## Cuando usarla
Cuando scaffoldees un artefacto numerado incremental (papers/, reports/, issues/) y necesites el siguiente NNNN sin colision: escanea lo que ya existe en disco y te da el numero libre listo para crear `<NNNN>-<slug>`.
## Gotchas
- **Impura**: lee el filesystem (estado del directorio en el momento de la llamada). No crea nada — solo calcula e imprime el numero.
- **Octal**: los numeros con cero a la izquierda (`08`, `09`) se interpretan como octal en aritmetica bash y romperian el calculo. La funcion fuerza base 10 con `10#$num` para evitarlo.
- **Solo subdirectorios**: cuenta unicamente subdirs directos. Archivos sueltos (`.gitkeep`, `notas.md`) y subdirs que no matcheen el patron se ignoran. No es recursivo.
- **Patron estricto**: el prefijo debe ser `NNNN-` (minimo 4 digitos seguidos de guion). Un subdir `12-foo` o `0001foo` (sin guion) NO se cuenta.
- No hay deteccion de huecos: devuelve `max+1`, no el primer numero libre intermedio. Si tienes `0001` y `0003`, devuelve `0004`, no `0002`.
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# next_numbered_dir — Compute the next NNNN- prefix for a numbered directory.
#
# Scans the DIRECT subdirectories of <parent_dir> whose names start with a
# numeric prefix of the form `NNNN-` (4+ digits followed by a hyphen), takes
# the maximum number, adds 1, and prints it zero-padded to <width> (default 4).
# If <parent_dir> does not exist or contains no matching subdir, prints the
# first number (0001 at default width).
next_numbered_dir() {
local parent_dir="${1:-}"
local width="${2:-4}"
if [[ -z "$parent_dir" ]]; then
echo "usage: next_numbered_dir <parent_dir> [width]" >&2
return 1
fi
local max=0
local entry base num
if [[ -d "$parent_dir" ]]; then
# Iterate only over direct subdirectories. The trailing slash in the
# glob ensures files (e.g. .gitkeep) are skipped — only dirs match.
for entry in "$parent_dir"/*/; do
# If the glob matched nothing it stays literal; guard with -d.
[[ -d "$entry" ]] || continue
base="$(basename "$entry")"
# Require a prefix of 4+ digits followed by a hyphen.
if [[ "$base" =~ ^([0-9]{4,})- ]]; then
num="${BASH_REMATCH[1]}"
# Force base 10 so leading zeros (08, 09) are not read as octal.
num=$((10#$num))
if (( num > max )); then
max=$num
fi
fi
done
fi
printf "%0*d\n" "$width" $(( max + 1 ))
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
next_numbered_dir "$@"
fi
+69
View File
@@ -0,0 +1,69 @@
---
name: init_paper
kind: pipeline
lang: bash
domain: pipelines
version: "1.0.0"
purity: impure
signature: "init_paper(slug: string, [--title <t>] [--domain <d>] [--tags <csv>]) -> void"
description: "Scaffold de un paper académico reproducible en papers/<NNNN-slug>/. Calcula el siguiente número incremental escaneando papers/, crea las subcarpetas (experiments data figures reviews out), copia las plantillas paper.md (IMRaD) + preregistration.md (anti-HARKing) rellenando el frontmatter (title, slug, date de hoy, phase=question, status=draft) y crea references.md. NO hace git init: el paper arranca en fase interna local (papers/ gitignored). Grupo de capacidad papers."
tags: [papers, scaffold, paper, pipeline, bash, launcher]
uses_functions:
- next_numbered_dir_bash_io
- slugify_ascii_py_core
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
params:
- name: slug
desc: "identificador legible del paper; se slugifica a ASCII (espacios/acentos se normalizan) y se prefija con el siguiente NNNN incremental"
- name: "--title"
desc: "título del paper (string); si se omite, usa el slug limpio. No debe contener el carácter '|'"
- name: "--domain"
desc: "dominio del paper escrito en el frontmatter (default datascience)"
- name: "--tags"
desc: "tags CSV que se escriben en el frontmatter de paper.md (opcional)"
output: "sin salida directa; crea papers/<NNNN-slug>/ con paper.md, preregistration.md, references.md y las subcarpetas experiments/ data/ figures/ reviews/ out/. Imprime el resumen y los pasos siguientes a stdout."
tested: false
tests: []
test_file_path: ""
file_path: "bash/functions/pipelines/init_paper.sh"
---
## Ejemplo
```bash
# Scaffold de un paper nuevo (numera 0001, 0002, ... automáticamente)
fn run init_paper mi-primer-paper --title "Mi primer paper"
fn run init_paper reactive-loop-calls --domain datascience --tags registry,telemetria
# El slug se slugifica: "Áreas de Mejora" -> papers/0003-areas-de-mejora/
fn run init_paper "Áreas de Mejora"
```
## Cuando usarla
Cuando empiezas un paper académico nuevo dentro de `fn_registry` y necesitas el esqueleto del artefacto (`papers/<NNNN-slug>/`) con las plantillas IMRaD y de pre-registro listas para rellenar. Es el paso 1 del grupo de capacidad `papers` (ver `docs/capabilities/papers.md`), antes de la revisión de literatura y del pre-registro de la hipótesis.
## Flujo
1. Parsea `<slug>` (posicional) + flags `--title` / `--domain` / `--tags`. Falla con exit ≠ 0 si falta el slug.
2. `slugify_ascii` — normaliza el slug a ASCII lowercase sin diacríticos (reutiliza la función del registry, solo stdlib).
3. `next_numbered_dir papers/` — calcula el siguiente NNNN de 4 dígitos sin colisión.
4. Crea `papers/<NNNN-slug>/` con las subcarpetas `experiments/ data/ figures/ reviews/ out/`.
5. Copia `docs/templates/paper.md` + `docs/templates/preregistration.md` y rellena el frontmatter por clave de línea (title, slug, date de hoy, domain, tags; phase=question y status=draft vienen de la plantilla).
6. Crea `references.md` vacío.
## Gotchas
- **NO hace `git init`.** El paper arranca en fase interna local; `papers/` está gitignored en el repo padre (solo `papers/.gitkeep` se versiona). Promocionar a sub-repo Gitea (fase publishable) es manual.
- **El `--title` no debe contener el carácter `|`** (se usa como delimitador de sed al rellenar el frontmatter; los `&` y `\` sí se escapan).
- **No indexa el paper en `registry.db`** — los artefactos `papers/<slug>/` no se indexan en esta fase (KISS); sí se indexa este pipeline.
- Requiere `python3` (del venv del registry o del sistema) para slugificar; `slugify_ascii` solo usa stdlib, así que el venv no es obligatorio.
- Idempotencia: si el directorio destino ya existiera, aborta con exit ≠ 0 en vez de sobrescribir.
## Notas
Cada paper es un artefacto independiente (mismo patrón que `apps/` y `analysis/`, pero para investigación). El pipeline usa `set -euo pipefail`: cualquier fallo detiene la ejecución. Parte del grupo de capacidad `papers` — diseño completo en `reports/0001-2026-06-30-papers-system-design.md`.
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
# init_paper
# ----------
# Scaffold de un paper académico reproducible en papers/<NNNN-slug>/.
#
# Calcula el siguiente número incremental escaneando papers/, crea el
# directorio con todas las subcarpetas (experiments data figures reviews out),
# copia las plantillas paper.md + preregistration.md rellenando el frontmatter
# (title, slug, date de hoy, phase=question, status=draft) y crea references.md.
#
# NO hace `git init`: el paper arranca en fase interna local (papers/ está
# gitignored en el repo padre, solo .gitkeep se versiona). La promoción a
# sub-repo Gitea (fase publishable) es un paso posterior MANUAL.
#
# Compone: next_numbered_dir (helper de numeración del registry) +
# slugify_ascii (slug ASCII del registry).
#
# USO:
# ./init_paper.sh <slug> [--title "..."] [--domain <d>] [--tags a,b,c]
#
# EJEMPLOS:
# ./init_paper.sh mi-primer-paper --title "Mi primer paper"
# ./init_paper.sh reactive-loop-calls --domain datascience --tags registry,telemetria
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REGISTRY_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
# Funciones atómicas del registry
source "$REGISTRY_ROOT/bash/functions/io/next_numbered_dir.sh"
# ── Parsing de argumentos ────────────────────────────────────
SLUG_RAW=""
TITLE=""
DOMAIN="datascience"
TAGS=""
while [ $# -gt 0 ]; do
case "$1" in
--title)
TITLE="$2"; shift 2 ;;
--domain)
DOMAIN="$2"; shift 2 ;;
--tags)
TAGS="$2"; shift 2 ;;
-h|--help)
grep "^#" "$0" | sed 's/^# \?//' ; exit 0 ;;
-*)
echo "Flag desconocido: $1" >&2 ; exit 1 ;;
*)
if [ -z "$SLUG_RAW" ]; then
SLUG_RAW="$1"
else
echo "ERROR: argumento posicional inesperado: '$1' (solo se admite un <slug>)." >&2
exit 1
fi
shift ;;
esac
done
if [ -z "$SLUG_RAW" ]; then
echo "ERROR: falta el argumento <slug>." >&2
echo "Uso: $0 <slug> [--title \"...\"] [--domain <d>] [--tags a,b,c]" >&2
echo " Ejemplo: $0 mi-primer-paper --title \"Mi primer paper\"" >&2
exit 1
fi
# ── Slugificar (reutiliza slugify_ascii del registry; solo stdlib) ──
PYBIN="$REGISTRY_ROOT/python/.venv/bin/python3"
[ -x "$PYBIN" ] || PYBIN="$(command -v python3 || true)"
if [ -z "$PYBIN" ]; then
echo "ERROR: no se encontró python3 para slugificar el slug." >&2
exit 1
fi
SLUG_CLEAN=$("$PYBIN" -c '
import sys, os
sys.path.insert(0, os.path.join(sys.argv[2], "python", "functions"))
from core.slugify_ascii import slugify_ascii
print(slugify_ascii(sys.argv[1], default="paper"))
' "$SLUG_RAW" "$REGISTRY_ROOT")
# ── Resolver número incremental y directorio destino ─────────
PAPERS_DIR="$REGISTRY_ROOT/papers"
mkdir -p "$PAPERS_DIR"
NUM=$(next_numbered_dir "$PAPERS_DIR")
SLUG_FULL="${NUM}-${SLUG_CLEAN}"
PAPER_DIR="$PAPERS_DIR/$SLUG_FULL"
if [ -d "$PAPER_DIR" ]; then
echo "ERROR: el directorio del paper ya existe: $PAPER_DIR" >&2
exit 1
fi
TODAY=$(date +%Y-%m-%d)
[ -n "$TITLE" ] || TITLE="$SLUG_CLEAN"
TAGS_YAML="[]"
if [ -n "$TAGS" ]; then
TAGS_YAML="[$(echo "$TAGS" | sed 's/,/, /g')]"
fi
echo ""
echo "════════════════════════════════════════════════════════════"
echo " INIT PAPER: ${SLUG_FULL}"
echo " Título: ${TITLE}"
echo " Directorio: ${PAPER_DIR}"
echo "════════════════════════════════════════════════════════════"
echo ""
# ── Crear estructura ─────────────────────────────────────────
echo "[1/3] Creando estructura..."
mkdir -p "$PAPER_DIR"/experiments "$PAPER_DIR"/data "$PAPER_DIR"/figures \
"$PAPER_DIR"/reviews "$PAPER_DIR"/out
echo " experiments/ data/ figures/ reviews/ out/"
# ── Copiar plantillas + rellenar frontmatter ─────────────────
echo "[2/3] Escribiendo paper.md + preregistration.md..."
# Escapa caracteres especiales del RHS de sed (delimitador |)
sed_escape() { printf '%s' "$1" | sed -e 's/[\\&|]/\\&/g'; }
TITLE_ESC="$(sed_escape "$TITLE")"
DOMAIN_ESC="$(sed_escape "$DOMAIN")"
PAPER_MD="$PAPER_DIR/paper.md"
PREREG_MD="$PAPER_DIR/preregistration.md"
cp "$REGISTRY_ROOT/docs/templates/paper.md" "$PAPER_MD"
cp "$REGISTRY_ROOT/docs/templates/preregistration.md" "$PREREG_MD"
sed -i \
-e "s|^title:.*|title: \"${TITLE_ESC}\"|" \
-e "s|^slug:.*|slug: ${SLUG_FULL}|" \
-e "s|^date:.*|date: ${TODAY}|" \
-e "s|^domain:.*|domain: ${DOMAIN_ESC}|" \
-e "s|^tags:.*|tags: ${TAGS_YAML}|" \
"$PAPER_MD"
sed -i \
-e "s|^paper_slug:.*|paper_slug: ${SLUG_FULL}|" \
"$PREREG_MD"
echo " $PAPER_MD"
echo " $PREREG_MD"
# ── references.md ────────────────────────────────────────────
echo "[3/3] Escribiendo references.md..."
cat > "$PAPER_DIR/references.md" << EOF
# References — ${TITLE}
<!-- Una entrada por referencia. Formato libre (o BibTeX) hasta promocionar a publishable. -->
EOF
echo " $PAPER_DIR/references.md"
# ── Resumen ──────────────────────────────────────────────────
echo ""
echo "════════════════════════════════════════════════════════════"
echo " PAPER '${SLUG_FULL}' LISTO (fase: question, status: draft)"
echo "════════════════════════════════════════════════════════════"
echo ""
echo " Pasos siguientes:"
echo " 1. Revisión de literatura (skill /deep-research) → Related work."
echo " 2. Pre-registro: congela H0/H1 + plan en preregistration.md (preregister_hypothesis)."
echo " 3. Experimentos en experiments/ → análisis (grupo eda) → escritura IMRaD en paper.md."
echo " 4. render_paper_pdf → out/paper.pdf. Peer review adversarial → reviews/."
echo ""
echo " papers/ está gitignored: este paper vive local hasta promocionar a publishable."
echo ""
+123 -3
View File
@@ -25,7 +25,8 @@ cabecera, y figuras/imágenes se escalan para caber enteras.
```
Document = list[Chapter]
Chapter = { id: str, title: str, version: str, blocks: list[Block] }
Block = Heading | Markdown | KVTable | DataTable | Figure | Image | Caption | Note
Block = Heading | Markdown | KVTable | DataTable | Figure | Image | Caption
| Note | Group | GlossaryEntry
```
Importa el modelo desde `datascience.automatic_eda.model` (o
@@ -44,6 +45,10 @@ reconocido se degrada a `Note`, nunca lanza).
| `Figure(fig=None, make=None, caption=None, height_in=None)` | una `matplotlib.figure.Figure` ya construida (`fig`) o un callable `make()->Figure` (perezoso) | se rasteriza y escala para caber entera (nunca recortada) |
| `Image(path, caption=None, height_in=None)` | ruta a PNG/JPG | se escala para caber entera |
| `Caption(text)` / `Note(text)` | texto auxiliar pequeño | pie/nota en gris; `Note` es además el fallback de lo desconocido |
| `Group(blocks, title=None)` | unidad **keep-together**: sus bloques se mantienen juntos | el renderer mide el grupo entero y lo mueve completo a la página/slide siguiente si no cabe; encoge la figura para dejar sitio al título+texto. Ver §11 |
| `GlossaryEntry(key, label, definition)` | una entrada del glosario (destino clicable) | la genera el capítulo `glosario`; registra su posición como destino de los términos marcados. Ver §11 |
`Figure`/`Image` aceptan `height_in` (hint): el renderer **clampa** la figura a esa altura máxima (lo usa `Group` para encoger la figura). Toda figura escala dejando sitio a su caption en la misma página/slide; en PPTX el caption es **siempre** visible (si no se da `caption`, cae al último heading o a "Figura").
### Subset de markdown soportado (`Markdown`)
@@ -84,8 +89,9 @@ El orden canónico está **pre-declarado** en
```python
CHAPTER_ORDER = [
"portada", "overview", "num_distr", "cat_distr", "calidad", "correlacion",
"modelos", "analisis_llm", "timeseries", "geospatial", "agregacion",
"portada", "overview", "analisis_llm", "num_distr", "cat_distr", "calidad",
"correlacion", "modelos", "timeseries", "geospatial", "agregacion",
"glosario",
]
```
@@ -95,6 +101,15 @@ CHAPTER_ORDER = [
`CHAPTER_ORDER`) y aparecerá automáticamente en su posición. Esto permite que muchos
agentes trabajen **en paralelo** sin contención: cada uno toca solo su archivo.
**Dos capítulos tienen posición especial** (los gestiona `build_document`, no toques esto):
- `portada`: se **construye el último** (después del cuerpo) para poder resumir el
análisis, pero se **coloca el primero**. Recibe `ctx['document_summary']` (ver §5) con
un resumen agregado del resto. Decisión del usuario: la portada refleja hallazgos.
- `glosario`: se construye y se **coloca el último**. Lee los términos que los demás
capítulos registraron en `ctx['glossary']` (ver §11). Si no se registró ninguno, el
capítulo devuelve `None` y desaparece.
Si tu capítulo usa un `<id>` que aún no está en `CHAPTER_ORDER`, añádelo en la posición
correcta (única edición compartida; coordínala con el orquestador).
@@ -143,6 +158,8 @@ defensivo). Esto habilita el **seguimiento y la mejora continua por capítulo**.
| `granularity` | "Cada fila es…" (portada). Default: derivado de `key_candidates` |
| `quality_criteria` | criterios del score de calidad (portada) |
| `head_rows` | `list[dict]` con `df.head` (overview). Ver §7 |
| `glossary` | `GlossaryCollector` compartido — los capítulos registran términos en él. Lo crea `build_document`; ver §11 |
| `document_summary` | dict con el resumen agregado del cuerpo (n_rows, n_cols, quality_score, n_numeric, n_categorical, chapter_titles, …). Lo calcula `build_document` y lo consume la portada |
Un capítulo puede definir y consumir sus propias claves `ctx` — documenta cuáles en su
docstring.
@@ -279,6 +296,109 @@ sus bloques presentes y el no-corte (texto largo intacto en la salida). Patrón:
---
## 11. Glosario, keep-together y zebra (motor, fase 4a)
Tres capacidades transversales del motor que **todos** los capítulos pueden usar. La 6.1
(glosario) requiere que el capítulo coopere (registrar + marcar términos); la 6.2
(keep-together) es opt-in por capítulo (envolver bloques en `Group`); la 6.3 (zebra) es
automática (no hay nada que hacer).
### 11.1 Glosario con términos clicables
El glosario es un capítulo nuevo (`chapters/glosario.py`) que se renderiza **siempre el
último** y lista cada término técnico que algún capítulo haya registrado. Cada aparición
del término en el texto se vuelve un **clic real** que salta a su entrada: en PDF como
*link annotation* interno (post-proceso con PyMuPDF, porque `PdfPages` no soporta
hyperlinks internos), en PPTX como *slide-jump* nativo (`ppaction://hlinksldjump`).
**API exacta para un capítulo (dos pasos):**
1. **Registrar el término** en el colector compartido `ctx['glossary']` (un
`model.GlossaryCollector`, creado por `build_document` y pasado a todos los capítulos):
```python
glossary = ctx.get("glossary")
if isinstance(glossary, model.GlossaryCollector):
glossary.add("entropia", "Entropía (de Shannon)", "Medida, en bits, de …")
```
`add(key, label, definition)` es idempotente (la primera definición de cada `key` gana).
`key` debe ser `[A-Za-z0-9_]+`. Si no hay colector en `ctx` (renderizado suelto), el
capítulo simplemente no marca términos — degrada sin romper.
2. **Marcar cada aparición** en el texto de un bloque `Markdown` con el span inline
`[[term:KEY]]texto visible[[/term]]`. El texto visible puede llevar `**negrita**`. El
marcador no altera el texto visible (se elimina como cualquier marcador inline); solo
añade el destino clicable.
```python
# En cat_distr (ejemplo real ya implementado):
"La [[term:entropia]]**entropía de Shannon**[[/term]] mide cómo de repartidos…"
```
Eso es todo: el capítulo `glosario` recoge los términos (orden alfabético por `label`),
emite un `GlossaryEntry` por término, y los renderers cablean los enlaces automáticamente.
Si ningún capítulo registró términos, el glosario no aparece.
**Helpers de `text_layout` (no reimplementar):** `parse_inline_rich(text)` →
`[(texto, is_bold, term_key), …]`; `wrap_rich_terms(text, max_chars)` → líneas de esos
spans sin corte. `strip_inline_md` ya elimina los marcadores `[[term:…]]`/`[[/term]]`.
(Las funciones previas `parse_inline_bold` / `wrap_rich` siguen existiendo, sin términos.)
**Funciones del registry que cablean los enlaces** (grupo `eda`, ya invocadas por los
renderers; degradan en silencio si faltan): `add_pdf_internal_links_py_datascience`
(PyMuPDF, link GOTO) y `pptx_link_run_to_slide_py_datascience` (salto a slide nativo).
Dependencia: `pymupdf` (declarada en `python/pyproject.toml`).
**Trabajo de la siguiente fase — enganchar más términos.** El mecanismo está hecho y
probado de extremo a extremo con `entropia` (en `cat_distr`). Cada capítulo debe registrar
y marcar SUS términos con el mismo patrón de dos pasos. Candidatos por capítulo:
| Capítulo | Términos a enganchar (key sugerida) |
|---|---|
| `cat_distr` | `entropia` ✅ (hecho) |
| `calidad` | `completitud`, `validez`, `consistencia` |
| `correlacion` | `cramers_v`, `fdr` (comparaciones múltiples), método de correlación usado |
| `modelos` | `pca`, `silhouette`, `isolation_forest` |
| `timeseries` | `estacionariedad`, `acf_pacf`, `stl` |
| `num_distr` | `iqr`, `curtosis`, `outlier` (vallas de Tukey) |
Define la definición de cada término en su capítulo (constante local, como
`_TERM_ENTROPIA_DEF` en `cat_distr`) y márcalo en su primera aparición.
### 11.2 Keep-together: gráfico junto a su título y texto (`Group`)
Para que un encabezado no quede en una página/slide y su figura en la siguiente, envuelve
los bloques de una misma idea en un `model.Group`:
```python
blocks.append(model.Group(blocks=[
model.Heading(text=str(name), level=2),
model.Figure(make=_figura_perezosa(...), caption="…"),
model.Markdown(text="explicación…"),
]))
```
El renderer **mide el grupo entero** antes de dibujar nada: si no cabe en lo que queda de
página/slide pero cabe en una entera, lo mueve **completo** a la siguiente; y **encoge la
figura** (vía `height_in`) lo justo para que el título + texto + figura quepan juntos. Si
el grupo es más alto que una página entera, empieza en una nueva y fluye (degradación
honesta, nunca corta). Ejemplo real implementado: `num_distr` envuelve cada columna
(heading + figura histograma/boxplot + nota) en un `Group`.
Recomendado para `agregacion` y cualquier capítulo donde una figura deba ir pegada a su
título/explicación. Coste: si un capítulo inspecciona `chapter.blocks` en sus tests, ahora
encontrará `Group`s — aplana con un helper recursivo (ver `num_distr_test.py::_flatten`).
### 11.3 Zebra striping en tablas (automático)
Todo `DataTable` se renderiza con **filas pares sombreadas** (gris muy suave `#f6f8fa`) y
cabecera con su fondo propio. Es automático en PDF y PPTX; el patrón se mantiene coherente
cuando una tabla larga se parte y repite cabecera (el índice de fila es lógico, no por
página). No hay nada que hacer en los capítulos.
---
## 10. Integración futura con `profile_table` (siguiente fase)
`profile_table(emit_pdf=True)` usa hoy `render_eda_pdf` (intacto). En la siguiente fase
+1
View File
@@ -39,6 +39,7 @@ Indice de grupos de capacidades del registry. Cada grupo agrupa >=3 funciones qu
| [cpp-tables](tql.md) | 9 | Table Query Language C++ puro: filter, group, agg, sort, join, stats, formulas Lua, round-trip emit/apply |
| [data-table-renderers](data_table_renderers.md) | 1 | API declarativa de cell renderers para data_table: Badge, Progress, Duration, Icon via TableInput.column_specs |
| [scheduler](scheduler.md) | 4 | Cron expression parsing, matching, next-run y traduccion humana (consume `apps/dag_engine`) |
| [papers](papers.md) | — | Papers académicos reproducibles en `papers/<NNNN-slug>/`: scaffold del artefacto (`init_paper` + helper `next_numbered_dir`), plantillas IMRaD + pre-registro anti-HARKing, y (en construcción por la flota) congelar hipótesis, funciones estadísticas (effect size/CI/corrección múltiple), render md→PDF y peer-review adversarial. Reutiliza `deep-research`, grupo `eda` y el motor PDF de `datascience`. Diseño: `reports/0001-2026-06-30-papers-system-design.md` |
| [extractor](extractor.md) | 15 | Funciones que leen datos de fuentes externas (BD, API, archivos, web). Nodos input de `data_factory` |
| [transformer](transformer.md) | 15 | Funciones que clean/dedup/aggregate/feature-engineer datos. Nodos intermedios de `data_factory` |
| [sink](sink.md) | 11 | Funciones que escriben datos a destino externo (BD, dashboard, alerta, email). Nodos output |
+82
View File
@@ -0,0 +1,82 @@
# papers — papers académicos reproducibles
Grupo de capacidad para producir **papers académicos** dentro de `fn_registry`: investigación con hipótesis falsables, experimentos reproducibles, análisis estadístico honesto y escritura en formato IMRaD. Cada paper es un artefacto nuevo en `papers/<NNNN-slug>/` que reutiliza infraestructura existente (skill `deep-research` para la revisión de literatura, grupo `eda` para el análisis, motor md→PDF de `datascience`, patrón de verificación adversarial del orquestador) y añade lo que falta como funciones del registry.
Diseño completo y decisiones: `reports/0001-2026-06-30-papers-system-design.md`.
> **Regla de oro anti paper-mill:** una hipótesis que **podía** fallar + un experimento con riesgo real de refutación + estadística que no es teatro. Si no hay riesgo de refutación, no es un paper. Los claims nunca superan a la evidencia. El antídoto al HARKing es el **pre-registro**: el plan de análisis se congela *antes* de mirar los datos.
## Estructura del artefacto
```
papers/0001-mi-paper/
paper.md # frontmatter (title, slug, authors, date, status, phase, tags, domain, hypothesis_id) + cuerpo IMRaD
preregistration.md # H0/H1 + plan de análisis CONGELADO (frozen_at + content_hash) antes de correr
references.md # bibliografía
experiments/ # código / notebooks por experimento (exp01_*, exp02_*)
data/ # crudos + procesados (gitignored si pesa)
figures/ # gráficos generados
reviews/ # outputs del peer-review adversarial
out/ # paper.pdf — entregable final
.git/ # SOLO cuando promociona a fase publishable (sub-repo Gitea)
```
`papers/` está gitignored en el repo padre (solo `papers/.gitkeep` se versiona): un paper en fase interna no contamina el repo. Al promocionar a `status: publishable` se vuelve sub-repo Gitea `dataforge/<slug>` (como apps y analyses).
### Fases (campo `phase` de `paper.md`)
```
question → review → hypothesis → design → running → analysis → writing → internal-review
→ [DONE interno] → polish → submitted [solo en fase publishable]
```
## Funciones
| ID | Pureza | Estado | Qué hace |
|---|---|---|---|
| `init_paper_bash_pipelines` | impure | ✅ disponible | Scaffold de `papers/<NNNN-slug>/`: calcula el siguiente NNNN, crea las subcarpetas, copia `paper.md` + `preregistration.md` con el frontmatter relleno (slug, title, date de hoy, `phase: question`, `status: draft`) y `references.md` vacío. NO hace `git init` (el paper arranca en fase interna local). |
| `next_numbered_dir_bash_io` | impure | ✅ disponible | Dado un directorio, devuelve el siguiente número incremental de 4 dígitos (`0001`, `0002`, …) escaneando los subdirs con prefijo `NNNN-`. Helper de numeración de `init_paper` (reutilizable por reports/issues). |
| `preregister_hypothesis` | impure | 🚧 en construcción (flota) | Congela el `preregistration.md` (H0/H1 + plan de análisis) con `frozen_at` + `content_hash`, pasa `status` a `frozen` y escribe `hypothesis_id` en `paper.md`. Mata el HARKing: tras congelar, el plan no se edita. |
| `cohens_d` (effect size) | pure | 🚧 en construcción (flota) | Tamaño del efecto (Cohen's d) entre dos grupos. Reporta magnitud, no solo significancia. |
| `confidence_interval` | pure | 🚧 en construcción (flota) | Intervalo de confianza de una métrica (media/diferencia). |
| `holm_bonferroni` | pure | 🚧 en construcción (flota) | Corrección de comparaciones múltiples (Holm-Bonferroni / FWER) para el plan de análisis. |
| `render_paper_pdf` | impure | 🚧 en construcción (flota) | Markdown IMRaD (`paper.md` + figuras) → `out/paper.pdf`, reutilizando el motor md→PDF del grupo `eda`/`datascience`. |
> Las funciones estadísticas reutilizan lo que ya exista en `datascience` (p.ej. `fdr_correction_py_datascience` cubre la corrección de comparaciones múltiples por FDR; el agente del rigor experimental decide si añade Holm-Bonferroni o reusa lo existente). Buscar antes de duplicar: `mcp__registry__fn_search query="effect size" domain="datascience"`.
### Peer review (no es función del registry)
El agente adversarial `.claude/agents/paper-reviewer.md` (🚧 en construcción por la flota) puntúa novedad, rigor, reproducibilidad y validez, e intenta **refutar** cada claim. Default a "failed" si la evidencia no soporta. Escribe su veredicto en `reviews/`. Es el equivalente al verificador adversarial del orquestador aplicado al paper.
## Ejemplo canónico (end-to-end)
```bash
# 1. Scaffold del paper (fase question, local). Crea papers/0001-mi-paper/.
./fn run init_paper mi-paper --title "¿El bucle reactivo reduce las calls inline?" --domain datascience --tags registry,telemetria
# 2. Revisión de literatura → llena Related work (skill deep-research, fase review).
# /deep-research "..."
# 3. Pre-registro: congela H0/H1 + plan de análisis ANTES de mirar datos (fase hypothesis).
./fn run preregister_hypothesis papers/0001-mi-paper # 🚧 en construcción
# 4. Experimentos en papers/0001-mi-paper/experiments/ (fase running) →
# análisis con el grupo `eda` + funciones de effect size / CI / corrección múltiple (fase analysis).
# 5. Escritura IMRaD en paper.md (fase writing) → render del entregable PDF.
./fn run render_paper_pdf papers/0001-mi-paper # 🚧 en construcción → out/paper.pdf
# 6. Peer review adversarial (fase internal-review).
# Agent(subagent_type="paper-reviewer", prompt="Revisa papers/0001-mi-paper ...") # 🚧 en construcción
```
## Fronteras
- **NO es para reports de trabajo.** Un report (`reports/`) es el entregable escrito de una tarea (resumen + evidencia + gaps); un paper es investigación con hipótesis falsable y experimento. Ver `.claude/rules/reports.md`.
- **NO se indexa en `registry.db` en esta fase.** No hay tabla `papers` ni `entity_type` `paper` (KISS); se añadiría con migración propia si se decide. Las *funciones* del grupo sí se indexan (viven en `bash/functions/`, `python/functions/`), pero los artefactos `papers/<slug>/` no.
- **NO hace `git init` en el scaffold.** El paper arranca en fase interna local y gitignored. La promoción a sub-repo Gitea (fase publishable) es un paso manual posterior.
- **NO soporta LaTeX/arXiv todavía.** Formato elegido: Markdown como fuente + PDF como entregable. El soporte LaTeX se añadiría al promocionar un paper a fase publishable.
## Estado
Fase de scaffolding. Disponible: estructura del artefacto, plantillas (`docs/templates/paper.md`, `docs/templates/preregistration.md`), pipeline `init_paper` + helper `next_numbered_dir`, esta página y el bloque gitignore de `papers/`. En construcción por la flota: `preregister_hypothesis`, funciones estadísticas (effect size / CI / corrección múltiple), `render_paper_pdf` y el agente `paper-reviewer`. Validación end-to-end con un paper piloto real: pendiente.
+94
View File
@@ -0,0 +1,94 @@
---
title: "TITULO DEL PAPER"
slug: NNNN-slug
authors: [Enmanuel]
date: 2026-01-01
status: draft # draft | internal | publishable
phase: question # question -> review -> hypothesis -> design -> running -> analysis -> writing -> internal-review -> polish -> submitted
tags: []
domain: datascience
hypothesis_id: "" # lo rellena preregister_hypothesis al congelar el preregistro
---
<!--
Paper académico reproducible (formato IMRaD). Esta es la FUENTE editable en Markdown;
el entregable PDF se genera con render_paper_pdf (grupo `papers`).
Regla de oro anti paper-mill: una hipótesis que PODÍA fallar + un experimento con
riesgo real de refutación + estadística que no es teatro. Si no hay riesgo de
refutación, no es un paper. Los claims nunca superan a la evidencia.
-->
# {{título del paper}}
## Abstract
<!--
Resumen estructurado en 4-6 frases: contexto -> gap -> método -> resultados -> conclusión.
Sin citas, sin abreviaturas sin definir. Es lo único que mucha gente leerá: que se sostenga solo.
-->
## 1. Introduction
<!--
Embudo en cuatro movimientos:
1. Contexto — el área y por qué importa.
2. Gap — qué NO se sabe todavía (el hueco que este paper llena).
3. Pregunta / hipótesis — formulada de forma falsable (ver preregistration.md).
4. Contribución — lista explícita de lo que aporta este trabajo ("Contributions:").
-->
## 2. Related work
<!--
Qué existe ya y por qué no basta. Agrupa por enfoque, no por autor. Cada cita debe
justificar por qué el gap sigue abierto. Output de la fase de revisión (skill deep-research).
-->
## 3. Methods
<!--
Diseño REPRODUCIBLE: otra persona lo corre y obtiene lo mismo.
- Variables: independiente(s), dependiente(s), control.
- Diseño: N, condiciones, muestreo, aleatorización.
- Métricas y cómo se miden.
- Protocolo paso a paso + dónde vive el código (experiments/) y los datos (data/).
Debe ser coherente con el preregistration.md congelado (no se cambia el plan tras ver datos).
-->
## 4. Results
<!--
Datos SIN interpretar. Tablas y figuras (figures/) con su lectura literal.
Reporta effect size + intervalos de confianza, no solo p-valores.
Incluye también los resultados negativos / no significativos (anti cherry-picking).
-->
## 5. Discussion
<!--
Interpretación de los resultados a la luz de la pregunta. Claims <= evidencia.
-->
### 5.1 Limitaciones
<!-- Qué no cubre el estudio, supuestos, datos faltantes. Honestidad explícita. -->
### 5.2 Amenazas a la validez
<!--
- Validez interna — ¿la causa es lo que decimos o hay confusores?
- Validez externa — ¿generaliza fuera de esta muestra/condiciones?
- Validez de constructo — ¿la métrica mide lo que dice medir?
- Validez estadística — ¿N suficiente, supuestos del test cumplidos, comparaciones múltiples corregidas?
-->
## 6. Conclusion + Future work
<!--
Cierre en 2-4 frases: qué se aprendió (sin overclaiming) + las siguientes preguntas que abre.
-->
## References
<!-- Ver references.md. -->
+59
View File
@@ -0,0 +1,59 @@
---
paper_slug: NNNN-slug
frozen_at: "" # timestamp ISO — lo rellena preregister_hypothesis al congelar
content_hash: "" # hash del contenido congelado — lo rellena preregister_hypothesis
status: draft # draft -> frozen (preregister_hypothesis lo pasa a frozen; tras congelar NO se edita)
---
> **⚠️ ESTE DOCUMENTO SE CONGELA ANTES DE MIRAR LOS DATOS (anti-HARKing).**
> El plan de análisis se fija aquí *antes* de ejecutar el experimento. Una vez congelado
> (`status: frozen`, con `frozen_at` + `content_hash`), **no se edita**. Inventar o ajustar
> la hipótesis después de ver los resultados (HARKing) invalida el paper. Si el plan cambia
> tras ver datos, eso es análisis exploratorio y se reporta como tal, no como confirmatorio.
# Pre-registro — {{título del paper}}
## 1. Pregunta de investigación
<!-- La pregunta concreta, en una frase. Debe poder responderse con un experimento. -->
## 2. Hipótesis
<!-- Falsable (Popper): una predicción que PODRÍA fallar. -->
- **H0 (nula):** <!-- no hay efecto / no hay diferencia. Es lo que el test intenta rechazar. -->
- **H1 (alternativa):** <!-- el efecto esperado, con dirección si la hay. -->
## 3. Variables
- **Independiente(s):** <!-- lo que se manipula. -->
- **Dependiente(s):** <!-- lo que se mide (la métrica de resultado). -->
- **Control:** <!-- lo que se mantiene fijo / se cubre estadísticamente. -->
## 4. Diseño
<!--
- N: tamaño de muestra (y justificación / power analysis si aplica).
- Condiciones / grupos.
- Muestreo y aleatorización.
- Criterios de inclusión / exclusión de datos (definidos AHORA, no después).
-->
## 5. Plan de análisis
<!--
El plan estadístico EXACTO, decidido antes de ver los datos:
- Test estadístico concreto (p.ej. t-test de Welch, Mann-Whitney U, regresión...).
- Métrica de effect size (p.ej. Cohen's d, diferencia de medias, odds ratio).
- Criterio de decisión (umbral alpha, qué resultado confirma/refuta H1).
- Corrección por comparaciones múltiples (p.ej. Holm-Bonferroni) si hay >1 contraste.
- Manejo de supuestos (normalidad, varianzas) y qué se hace si no se cumplen.
-->
## 6. Predicción cuantitativa
<!--
La predicción numérica concreta que el experimento pondrá a prueba.
P.ej. "esperamos d >= 0.5 con IC95% que no cruza 0" o "una reducción >= 15% en la métrica X".
Cuanto más específica, más falsable.
-->
View File
+38
View File
@@ -25,6 +25,7 @@ from .describe_numeric import describe_numeric
from .summarize_categorical import summarize_categorical
from .infer_semantic_type import infer_semantic_type
from .column_quality_score import column_quality_score
from .select_groupby_keys import select_groupby_keys
from .render_eda_markdown import render_eda_markdown
from .detect_distribution_type import detect_distribution_type
from .spearman_corr import spearman_corr
@@ -33,9 +34,12 @@ from .theils_u import theils_u
from .correlation_ratio import correlation_ratio
from .mutual_info_columns import mutual_info_columns
from .infer_fk_containment_duckdb import infer_fk_containment_duckdb
from .detect_declared_keys_duckdb import detect_declared_keys_duckdb
from .build_join_graph import build_join_graph
from .association_matrix import association_matrix
from .correlation_matrix_duckdb import correlation_matrix_duckdb
from .pivot_table_duckdb import pivot_table_duckdb
from .groupby_stats_duckdb import groupby_stats_duckdb
from .pca_explained import pca_explained
from .kmeans_segments import kmeans_segments
from .isolation_forest_outliers import isolation_forest_outliers
@@ -55,21 +59,51 @@ from .acf_pacf import acf_pacf
from .stl_decompose import stl_decompose
from .to_returns import to_returns
from .fdr_correction import fdr_correction
from .effect_size_cohens_d import effect_size_cohens_d
from .confidence_interval_mean import confidence_interval_mean
from .preregister_hypothesis import preregister_hypothesis
from .suggest_reexpression import suggest_reexpression
from .exploratory_caveats import exploratory_caveats
from .render_eda_pdf import render_eda_pdf, render_eda_pdf_relational
from .render_automatic_eda_pdf import render_automatic_eda_pdf
from .render_automatic_eda_pptx import render_automatic_eda_pptx
from .render_automatic_eda_markdown import render_automatic_eda_markdown
from .detect_time_column import detect_time_column
from .extract_timeseries_raw import extract_timeseries_raw
from .build_eda_render_ctx import build_eda_render_ctx
from .profile_datetime import profile_datetime
from .resample_timeseries import resample_timeseries
from .add_pdf_internal_links import add_pdf_internal_links
from .suggest_intratable_fk_candidates import suggest_intratable_fk_candidates
from .render_paper_pdf import render_paper_pdf
from .draw_join_graph_figure import draw_join_graph_figure
from .generate_synthetic_eda_table import generate_synthetic_eda_table
from .generate_synthetic_eda_folder import generate_synthetic_eda_folder
__all__ = [
"generate_synthetic_eda_table",
"generate_synthetic_eda_folder",
"render_paper_pdf",
"draw_join_graph_figure",
"suggest_intratable_fk_candidates",
"detect_time_column",
"extract_timeseries_raw",
"build_eda_render_ctx",
"add_pdf_internal_links",
"profile_datetime",
"resample_timeseries",
"render_automatic_eda_pdf",
"render_automatic_eda_pptx",
"render_automatic_eda_markdown",
"decode_qr_image",
"adf_kpss_stationarity",
"acf_pacf",
"stl_decompose",
"to_returns",
"fdr_correction",
"effect_size_cohens_d",
"confidence_interval_mean",
"preregister_hypothesis",
"suggest_reexpression",
"exploratory_caveats",
"render_eda_pdf",
@@ -82,9 +116,12 @@ __all__ = [
"correlation_ratio",
"mutual_info_columns",
"infer_fk_containment_duckdb",
"detect_declared_keys_duckdb",
"build_join_graph",
"association_matrix",
"correlation_matrix_duckdb",
"pivot_table_duckdb",
"groupby_stats_duckdb",
"pca_explained",
"kmeans_segments",
"isolation_forest_outliers",
@@ -102,6 +139,7 @@ __all__ = [
"summarize_categorical",
"infer_semantic_type",
"column_quality_score",
"select_groupby_keys",
"render_eda_markdown",
"detect_distribution_type",
"pull_gsc_search_analytics",
@@ -0,0 +1,85 @@
---
name: add_pdf_internal_links
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def add_pdf_internal_links(pdf_path: str, links: list) -> dict"
description: "Postprocesa un PDF YA escrito insertando link annotations internos de tipo GOTO ('ir a') con PyMuPDF (import fitz). Pensado para PDFs generados por matplotlib PdfPages, que NO soporta hyperlinks internos: tras escribir el PDF se reabre y, por cada entrada de `links`, se añade una anotacion clicable desde un rectangulo de una pagina origen (src_page + src_rect en puntos top-left) hasta un punto de una pagina destino (dst_page + dst_point). Caso de uso tipico del grupo eda: hacer clicables los terminos de un AutomaticEDA que apuntan a su entrada en el glosario al final del documento. Estilo dict-no-throw: NUNCA lanza; valida cada link y SALTA (n_skipped++) los malformados o fuera de rango en vez de fallar. Guarda de forma segura escribiendo a un temporal en el mismo directorio y haciendo os.replace atomico (evita corromper el original). Devuelve {status:ok,n_links,n_skipped} o {status:error,error}; si pymupdf no esta disponible o el archivo no existe devuelve status error."
tags: [eda, datascience, pdf, links, glossary, pymupdf, fitz, postprocess, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
params:
- name: pdf_path
desc: "ruta al PDF existente (str no vacio). Se reescribe IN SITU (in-place) tras añadir los links: se guarda a un temporal `.<base>.tmp_links` en el mismo directorio y se reemplaza atomicamente con os.replace. Si no es str o no existe el archivo -> {status:error}."
- name: links
desc: "lista de dicts, uno por link a insertar. Cada dict: src_page (int 0-based de la pagina origen), src_rect ([x0,y0,x1,y1] del rectangulo clicable en PUNTOS PDF 1/72\" con origen ARRIBA-IZQUIERDA), dst_page (int 0-based de la pagina destino), dst_point ([x,y] punto destino, mismos puntos top-left). Las entradas que no son dict, con page fuera de rango [0,page_count), src_rect que no tenga 4 numeros o dst_point que no tenga 2 numeros se SALTAN (n_skipped++), no lanzan. None se trata como lista vacia."
output: "dict (NUNCA lanza): en exito {\"status\":\"ok\",\"n_links\":int,\"n_skipped\":int} con n_links = anotaciones GOTO insertadas y n_skipped = entradas invalidas saltadas. En fallo {\"status\":\"error\",\"error\":str}: pymupdf no disponible, pdf_path no es str / no existe, links no es lista, o cualquier excepcion global (el PDF original queda intacto porque el replace solo ocurre tras un save correcto)."
tested: true
tests: ["test_add_goto_link_basico", "test_links_invalidos_se_saltan", "test_archivo_inexistente_devuelve_error"]
test_file_path: "python/functions/datascience/add_pdf_internal_links_test.py"
file_path: "python/functions/datascience/add_pdf_internal_links.py"
---
## Ejemplo
```python
import sys, os
sys.path.insert(0, os.path.join("python", "functions"))
from datascience import add_pdf_internal_links
# Tienes un PDF ya escrito por matplotlib PdfPages (sin hyperlinks internos).
# Quieres que el texto "Margen bruto" de la pagina 0 (rectangulo en puntos
# top-left) salte a su entrada del glosario en la ultima pagina (indice 7).
res = add_pdf_internal_links(
"reports/eda.pdf",
[
{"src_page": 0, "src_rect": [72, 120, 180, 134], "dst_page": 7, "dst_point": [72, 200]},
{"src_page": 0, "src_rect": [72, 140, 180, 154], "dst_page": 7, "dst_point": [72, 260]},
],
)
# res == {"status": "ok", "n_links": 2, "n_skipped": 0}
```
## Cuando usarla
Justo DESPUES de escribir un PDF con matplotlib `PdfPages` (o cualquier motor
que no genere hyperlinks internos) cuando necesitas que ciertos terminos o
referencias sean clicables y salten a otra pagina del mismo documento — el caso
canonico es enlazar los terminos de un AutomaticEDA con su entrada de glosario
al final. Es un paso de postproceso: primero generas el PDF y calculas en que
rectangulo quedo cada termino (en puntos PDF), luego pasas esa lista a esta
funcion para inyectar las anotaciones GOTO.
## Gotchas
- **Impura — reescribe el archivo IN SITU.** El PDF en `pdf_path` se reemplaza
por la version con los links. El guardado es seguro: escribe a un temporal
`.<base>.tmp_links` en el MISMO directorio y hace `os.replace` atomico tras
cerrar el documento, asi un fallo a mitad no corrompe el original. Aun asi,
conserva una copia si el PDF es valioso.
- **Sistema de coordenadas: puntos top-left, igual que matplotlib.** PyMuPDF y
matplotlib (PdfPages) usan ambos PUNTOS PDF (1/72") con el origen ARRIBA-
IZQUIERDA, asi que los rectangulos/puntos COINCIDEN: el `src_rect` que calcules
con la geometria de la figura matplotlib se pasa tal cual, sin invertir el eje
Y. (Ojo: el espacio de datos de matplotlib SI tiene el origen abajo; lo que
coincide es el espacio de la PAGINA en puntos.)
- **Indices de pagina 0-based.** `src_page` / `dst_page` son indices base 0
(la primera pagina es 0). Fuera del rango `[0, page_count)` el link se SALTA
(cuenta en `n_skipped`), no lanza.
- **dict-no-throw, validacion por-link.** Las entradas malformadas (no dict,
page fuera de rango, `src_rect` sin 4 numeros, `dst_point` sin 2 numeros) se
saltan individualmente e incrementan `n_skipped`; el resto de links validos se
insertan igual. La funcion solo devuelve `{status:error}` ante fallos globales
(pymupdf ausente, archivo inexistente, `links` no es lista).
- **`error_type: error_go_core` es metadata del registry, no comportamiento.**
Toda funcion impura debe declararlo y el indexer lo exige, pero el codigo NUNCA
lanza esa excepcion: degrada al dict de estado.
- **Requiere PyMuPDF (`import fitz`).** Si no esta instalado devuelve
`{"status":"error","error":"pymupdf no disponible: ..."}`. En el registry el
venv `python/.venv` ya lo trae.
@@ -0,0 +1,132 @@
"""Postprocesa un PDF existente insertando link annotations internos (GOTO).
Motor: PyMuPDF (``import fitz``). Pensado para PDFs generados por matplotlib
``PdfPages``, que no soporta hyperlinks internos: tras escribir el PDF, esta
funcion lo reabre y le añade anotaciones "ir a" (GOTO) desde un rectangulo de
una pagina origen hasta un punto de una pagina destino. Util para hacer
clicables terminos que apuntan a su entrada en un glosario al final del
documento.
Estilo dict-no-throw del grupo `eda`: NUNCA lanza; devuelve un dict de estado.
"""
import os
def add_pdf_internal_links(pdf_path: str, links: list) -> dict:
"""Añade link annotations internos (GOTO) a un PDF ya escrito.
Postprocesa un PDF (p.ej. generado por matplotlib PdfPages, que NO soporta
hyperlinks internos) insertando, por cada entrada de ``links``, una
anotacion de tipo "ir a" desde un rectangulo de una pagina origen hasta un
punto de una pagina destino. Sirve para hacer clicables terminos que apuntan
a su entrada en un glosario al final del documento.
Args:
pdf_path: ruta al PDF existente (se reescribe in situ).
links: lista de dicts, cada uno:
{
"src_page": int, # indice 0-based de la pagina origen
"src_rect": [x0,y0,x1,y1], # rectangulo clicable, en PUNTOS PDF
# (1/72") con origen ARRIBA-IZQUIERDA
"dst_page": int, # indice 0-based de la pagina destino
"dst_point": [x, y], # punto destino, mismos puntos top-left
}
Returns:
dict (NUNCA lanza): {"status":"ok","n_links":int,"n_skipped":int}
o {"status":"error","error":str}. Si pymupdf no esta disponible o el
archivo no existe -> {"status":"error", ...}.
"""
try:
try:
import fitz # PyMuPDF
except Exception as exc: # ImportError u otro fallo de carga
return {"status": "error", "error": f"pymupdf no disponible: {exc}"}
if not isinstance(pdf_path, str) or not pdf_path:
return {"status": "error", "error": "pdf_path debe ser una ruta no vacia"}
if not os.path.isfile(pdf_path):
return {"status": "error", "error": f"el archivo no existe: {pdf_path}"}
if links is None:
links = []
if not isinstance(links, (list, tuple)):
return {"status": "error", "error": "links debe ser una lista de dicts"}
doc = fitz.open(pdf_path)
try:
n_pages = doc.page_count
n_ok = 0
n_skipped = 0
for link in links:
if not isinstance(link, dict):
n_skipped += 1
continue
src_page = link.get("src_page")
dst_page = link.get("dst_page")
src_rect = link.get("src_rect")
dst_point = link.get("dst_point")
# src_page / dst_page: enteros 0-based en rango.
if not _is_int(src_page) or not _is_int(dst_page):
n_skipped += 1
continue
if not (0 <= src_page < n_pages) or not (0 <= dst_page < n_pages):
n_skipped += 1
continue
# src_rect: 4 numeros.
if not _is_num_seq(src_rect, 4):
n_skipped += 1
continue
# dst_point: 2 numeros.
if not _is_num_seq(dst_point, 2):
n_skipped += 1
continue
try:
doc[int(src_page)].insert_link(
{
"kind": fitz.LINK_GOTO,
"from": fitz.Rect(*[float(v) for v in src_rect]),
"page": int(dst_page),
"to": fitz.Point(*[float(v) for v in dst_point]),
}
)
n_ok += 1
except Exception:
n_skipped += 1
continue
# Guardado seguro: escribir a temporal en el mismo directorio y
# reemplazar atomicamente (evita corromper el PDF original).
directory = os.path.dirname(os.path.abspath(pdf_path)) or "."
base = os.path.basename(pdf_path)
tmp_path = os.path.join(directory, f".{base}.tmp_links")
doc.save(tmp_path)
finally:
doc.close()
os.replace(tmp_path, pdf_path)
return {"status": "ok", "n_links": n_ok, "n_skipped": n_skipped}
except Exception as exc: # degrada cualquier fallo a dict de error
return {"status": "error", "error": str(exc)}
def _is_int(value) -> bool:
"""True si value es un entero (no bool)."""
return isinstance(value, int) and not isinstance(value, bool)
def _is_num_seq(value, length: int) -> bool:
"""True si value es una secuencia de `length` numeros (int/float, no bool)."""
if not isinstance(value, (list, tuple)) or len(value) != length:
return False
for v in value:
if isinstance(v, bool) or not isinstance(v, (int, float)):
return False
return True
@@ -0,0 +1,77 @@
"""Tests para add_pdf_internal_links."""
import os
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from add_pdf_internal_links import add_pdf_internal_links
def test_add_goto_link_basico(tmp_path):
"""Golden: un PDF de 2 paginas recibe un link GOTO de la pag 0 a la pag 1."""
fitz = pytest.importorskip("fitz")
# 1) PDF temporal de 2 paginas A5 (~419x595 puntos).
pdf = str(tmp_path / "doc.pdf")
doc = fitz.open()
doc.new_page(width=419, height=595)
doc.new_page(width=419, height=595)
doc.save(pdf)
doc.close()
# 2) Insertar un link interno desde la pag 0 hacia la pag 1.
res = add_pdf_internal_links(
pdf,
[{"src_page": 0, "src_rect": [50, 50, 200, 70], "dst_page": 1, "dst_point": [40, 40]}],
)
assert res["status"] == "ok"
assert res["n_links"] == 1
assert res["n_skipped"] == 0
# 3) Reabrir y verificar que la pag 0 tiene un link GOTO a la pag 1.
doc = fitz.open(pdf)
try:
links = doc[0].get_links()
goto = [l for l in links if l.get("kind") == fitz.LINK_GOTO and l.get("page") == 1]
assert len(goto) >= 1
finally:
doc.close()
def test_links_invalidos_se_saltan(tmp_path):
"""Edge: entradas malformadas o fuera de rango incrementan n_skipped, no lanzan."""
fitz = pytest.importorskip("fitz")
pdf = str(tmp_path / "doc.pdf")
doc = fitz.open()
doc.new_page(width=419, height=595)
doc.new_page(width=419, height=595)
doc.save(pdf)
doc.close()
res = add_pdf_internal_links(
pdf,
[
# valido
{"src_page": 0, "src_rect": [10, 10, 90, 30], "dst_page": 1, "dst_point": [20, 20]},
# dst_page fuera de rango
{"src_page": 0, "src_rect": [10, 40, 90, 60], "dst_page": 9, "dst_point": [20, 20]},
# src_rect con 3 numeros
{"src_page": 0, "src_rect": [10, 70, 90], "dst_page": 1, "dst_point": [20, 20]},
# no es dict
"no-soy-un-dict",
],
)
assert res["status"] == "ok"
assert res["n_links"] == 1
assert res["n_skipped"] == 3
def test_archivo_inexistente_devuelve_error():
"""Error path: pdf_path inexistente -> status error sin lanzar."""
res = add_pdf_internal_links("/ruta/que/no/existe_xyz.pdf", [])
assert res["status"] == "error"
assert "error" in res
@@ -21,6 +21,9 @@ from .model import ( # noqa: F401
Chapter,
DataTable,
Figure,
GlossaryCollector,
GlossaryEntry,
Group,
Heading,
Image,
KVTable,
@@ -33,6 +36,7 @@ from .model import ( # noqa: F401
from .chapters_registry import CHAPTER_ORDER, build_chapter, build_document # noqa: F401
from .render_pdf_impl import render_pdf # noqa: F401
from .render_pptx_impl import render_pptx # noqa: F401
from .render_md_impl import render_md # noqa: F401
__all__ = [
"ENGINE_NAME",
@@ -45,6 +49,9 @@ __all__ = [
"Image",
"Caption",
"Note",
"Group",
"GlossaryEntry",
"GlossaryCollector",
"Chapter",
"as_blocks",
"as_chapters",
@@ -54,4 +61,5 @@ __all__ = [
"build_document",
"render_pdf",
"render_pptx",
"render_md",
]
@@ -0,0 +1,113 @@
"""Tests for inline-bold rendering (**bold**) in the AutomaticEDA engine.
Covers the pure helpers (parse_inline_bold / wrap_rich) and an end-to-end PPTX
check that a ``**bold**`` span is rendered with NATIVE PowerPoint bold
(``run.font.bold is True``) while no line overflows the wrap width (no-cut).
"""
import os
import sys
import pytest
# Make the engine importable as a package (datascience.automatic_eda).
_HERE = os.path.dirname(os.path.abspath(__file__))
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
if _FUNCTIONS not in sys.path:
sys.path.insert(0, _FUNCTIONS)
from datascience.automatic_eda import model # noqa: E402
from datascience.automatic_eda import text_layout as tl # noqa: E402
from datascience.automatic_eda import render_pptx # noqa: E402
# --------------------------------------------------------------------------- #
# Pure helpers.
# --------------------------------------------------------------------------- #
def test_parse_inline_bold_marks_spans_and_preserves_visible_text():
src = "**Estacionariedad:** serie no estacionaria con `code` y normal."
segs = tl.parse_inline_bold(src)
# Visible text equals strip_inline_md (no characters lost, markers removed).
visible = "".join(s for s, _ in segs)
assert visible == tl.strip_inline_md(src)
# The span "Estacionariedad:" is flagged bold; the rest is not.
bold_text = "".join(s for s, b in segs if b)
assert "Estacionariedad:" in bold_text
assert "serie no estacionaria" not in bold_text
def test_parse_inline_bold_handles_unbalanced_markers():
# An unbalanced ** must not crash and must be stripped (matches strip_inline_md).
segs = tl.parse_inline_bold("texto **sin cierre aqui")
visible = "".join(s for s, _ in segs)
assert visible == "texto sin cierre aqui"
assert not any(b for _, b in segs) # nothing rendered bold.
def test_wrap_rich_never_overflows_and_keeps_bold():
text = ("**Segmento premium.** Clientes de alto gasto y baja frecuencia con "
"ticket medio elevado y recurrencia anual estable a lo largo del año.")
max_chars = 30
lines = tl.wrap_rich(text, max_chars)
# No visible line exceeds max_chars (no-cut: the renderer measures these).
for ln in lines:
visible = "".join(s for s, _ in ln)
assert len(visible) <= max_chars, f"línea desborda: {visible!r}"
# At least one segment is bold and it is the span content.
bold_segs = [s for ln in lines for s, b in ln if b]
assert any("Segmento premium." in s for s in bold_segs)
def test_wrap_rich_hard_splits_long_token():
long = "x" * 50
lines = tl.wrap_rich(f"**{long}**", 20)
for ln in lines:
assert len("".join(s for s, _ in ln)) <= 20
# The whole long token is preserved across the split lines.
joined = "".join(s for ln in lines for s, _ in ln)
assert joined == long
# --------------------------------------------------------------------------- #
# End-to-end: PPTX renders **bold** as a real bold run.
# --------------------------------------------------------------------------- #
def _has_pptx():
try:
import pptx # noqa: F401
return True
except Exception: # noqa: BLE001
return False
@pytest.mark.skipif(not _has_pptx(), reason="python-pptx no instalado")
def test_pptx_renders_bold_span_as_native_bold_run(tmp_path):
from pptx import Presentation
doc = [model.Chapter(
id="t", title="Negrita", version="1.0.0",
blocks=[model.Markdown(
text="Frase con **PALABRACLAVE** resaltada y texto normal después.")],
)]
out = str(tmp_path / "bold.pptx")
res = render_pptx(doc, out, {"title": "T"})
assert res.get("path") == out
assert os.path.exists(out)
prs = Presentation(out)
bold_texts = []
all_text = []
for slide in prs.slides:
for shape in slide.shapes:
if not shape.has_text_frame:
continue
for para in shape.text_frame.paragraphs:
for run in para.runs:
all_text.append(run.text)
if run.font.bold:
bold_texts.append(run.text)
# The bold span text appears in a run with font.bold True (native bold).
assert any("PALABRACLAVE" in t for t in bold_texts), \
f"no se encontró run bold con el span; bold={bold_texts}"
# And the surrounding plain text is NOT bold (markers did not bleed).
assert any("resaltada" in t for t in all_text)
assert not any("resaltada" in t for t in bold_texts)
@@ -0,0 +1,633 @@
"""Aggregation chapter (AGREGACION) — group analysis / OLAP of the EDA.
This chapter is the group-by / pivot ("OLAP") section of an AutomaticEDA report
and is meant to be present **whenever the dataset has at least one low-cardinality
categorical column to group by**. For the most interesting categoricals (chosen
by their cardinality/relevance, optionally with an LLM) it renders, as blocks the
core paginator never cuts:
1. **Per-group statistics** (split-apply-combine) — for each interesting
categorical key, the count of rows per group and, for each numeric measure,
its mean/median/std/min/max. One compact summary table (mean of every measure
per group) plus a per-measure detail table.
2. **Bar charts** — a vertical bar chart of a measure's mean per group, bars from
zero (Tufte Lie-Factor = 1).
3. **Pivot tables** — categorical A x categorical B -> aggregate of a measure,
limited to the top rows/cols so it fits a mobile page/slide, with a grouped
bar chart of the same pivot.
The raw data needed to aggregate is **not** in the TableProfile, so — exactly
like ``modelos`` reads its cluster projection from ``ctx`` — this chapter gets
the aggregation results in one of two ways and degrades honestly when neither is
available:
ctx keys this chapter consumes (all optional):
aggregations : dict — pre-computed results, used directly (offline / tests /
forward-compatible with a calculation phase). Shape::
{"groupby": [{"group_by": str, "measures": [str], "why": str,
"result": <groupby_stats_duckdb-shaped dict>}],
"pivots": [{"index": str, "columns": str, "value": str, "agg": str,
"why": str, "result": <pivot_table_duckdb-shaped dict>}]}
db_path, table : str — when ``aggregations`` is absent, the chapter selects
the interesting keys (``select_groupby_keys``), optionally asks an LLM
which to show (``suggest_aggregations_llm`` when ``run_agg_llm`` is True)
and computes the group-by/pivot results live via the push-down registry
functions ``groupby_stats_duckdb`` / ``pivot_table_duckdb``.
run_agg_llm : bool — when True (and ``db_path``/``table`` present), let the
LLM pick the interesting aggregations; otherwise the deterministic
quantitative selection is used.
agg_llm_model : str — model id for the optional LLM selection.
agg_max_keys, agg_max_card, agg_max_measures, agg_top_n : int — limits.
agg_insights : list — optional pre-computed micro-analysis entries
(``[{"title": str, "text": str}]``) rendered as an interpretation section.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
Reads everything defensively (``.get``) and never raises: anything missing
degrades to a note instead of aborting the chapter; the chapter returns ``None``
only when the dataset has no categorical column to group by.
"""
from __future__ import annotations
from .. import model
# Pure/impure registry functions (group ``eda``) this chapter composes. Imported
# defensively so the chapter still builds (degrading the affected part to a note)
# if a function is somehow unavailable / not indexed yet.
try:
from datascience.select_groupby_keys import select_groupby_keys
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
select_groupby_keys = None # type: ignore[assignment]
try:
from datascience.groupby_stats_duckdb import groupby_stats_duckdb
except Exception: # noqa: BLE001
groupby_stats_duckdb = None # type: ignore[assignment]
try:
from datascience.pivot_table_duckdb import pivot_table_duckdb
except Exception: # noqa: BLE001
pivot_table_duckdb = None # type: ignore[assignment]
try:
from datascience.suggest_aggregations_llm import suggest_aggregations_llm
except Exception: # noqa: BLE001
suggest_aggregations_llm = None # type: ignore[assignment]
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "agregacion"
CHAPTER_TITLE = "Agregación por grupos"
# Tableau-10 palette — stable colours for the pivot's grouped-bar series.
_SERIES_COLORS = [
"#4e79a7", "#f28e2b", "#e15759", "#76b7b2", "#59a14f",
"#edc948", "#b07aa1", "#ff9da7", "#9c755f", "#bab0ac",
]
# Defaults for the live selection/aggregation (overridable via ctx).
_DEF_MAX_KEYS = 3
_DEF_MAX_CARD = 20
_DEF_MAX_MEASURES = 4
_DEF_TOP_N = 12
# Glossary terms this chapter explains. Both appear in the always-rendered intro,
# so they are registered and marked clickable whenever a collector is in ctx —
# the canonical two-step pattern (see ``cat_distr``): ``glossary.add(key, label,
# definition)`` + the inline span ``[[term:KEY]]texto[[/term]]`` in a Markdown
# block. Mapping key -> (label, definition).
_TERM_DEFS = {
"groupby": (
"Agrupación (split-apply-combine)",
"Operación de agrupación (group by): parte la tabla en grupos según los "
"valores de una columna categórica, aplica un cálculo (conteo, media, "
"mediana…) dentro de cada grupo y combina los resultados en una tabla "
"resumen. Es el patrón split-apply-combine."),
"pivot_table": (
"Tabla dinámica (pivot)",
"Tabla dinámica que cruza dos variables categóricas — una en las filas y "
"otra en las columnas — y rellena cada celda con un agregado (media, "
"suma…) de una medida numérica. Resume de un vistazo cómo interactúan las "
"dos categóricas sobre esa medida."),
}
def _term(mark: bool, key: str, text: str) -> str:
"""Wrap ``text`` as a clickable glossary span when ``mark`` is True.
The visible text is identical with or without the marker (the renderers strip
it), so wrapping never changes line layout — it only adds the link.
"""
return f"[[term:{key}]]{text}[[/term]]" if mark else text
# --------------------------------------------------------------------------- #
# Formatting helpers (mirror the other chapters' defensive style).
# --------------------------------------------------------------------------- #
def _fmt_num(value, decimals: int = 3) -> str:
if value is None:
return ""
if isinstance(value, bool):
return "" if value else "no"
if isinstance(value, int):
return f"{value:,}".replace(",", ".")
if isinstance(value, float):
if value != value: # NaN
return "NaN"
if value in (float("inf"), float("-inf")):
return str(value)
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
return text if text else "0"
return model._safe_str(value)
def _is_dict(v) -> bool:
return isinstance(v, dict)
def _measure_mean(group: dict, measure: str):
"""Pull the mean of one measure out of a groupby-result group entry."""
stats = group.get("stats") if _is_dict(group.get("stats")) else {}
ms = stats.get(measure) if _is_dict(stats.get(measure)) else {}
return ms.get("mean")
# --------------------------------------------------------------------------- #
# Plan + data resolution. Either a pre-computed ctx['aggregations'] is used
# verbatim, or the plan is selected and the results are computed live.
# --------------------------------------------------------------------------- #
def _resolve_candidates(profile: dict, ctx: dict) -> dict:
"""Return {group_keys, measures, pivots, note} of interesting columns."""
pre = ctx.get("agg_candidates")
if _is_dict(pre) and pre.get("group_keys") is not None:
return pre
if select_groupby_keys is not None:
try:
out = select_groupby_keys(
profile,
max_keys=int(ctx.get("agg_max_keys", _DEF_MAX_KEYS)),
max_card=int(ctx.get("agg_max_card", _DEF_MAX_CARD)),
max_measures=int(ctx.get("agg_max_measures", _DEF_MAX_MEASURES)),
)
if _is_dict(out):
return out
except Exception: # noqa: BLE001 — fall through to the inline fallback.
pass
return _inline_candidates(profile, ctx)
def _inline_candidates(profile: dict, ctx: dict) -> dict:
"""Minimal defensive selection when select_groupby_keys is unavailable."""
max_card = int(ctx.get("agg_max_card", _DEF_MAX_CARD))
max_keys = int(ctx.get("agg_max_keys", _DEF_MAX_KEYS))
max_measures = int(ctx.get("agg_max_measures", _DEF_MAX_MEASURES))
keys = profile.get("key_candidates") or []
group_keys, measures = [], []
for col in profile.get("columns") or []:
if not _is_dict(col):
continue
name = col.get("name")
it = col.get("inferred_type")
flags = col.get("flags") or []
dc = col.get("distinct_count")
if it in ("categorical", "boolean") and name not in keys:
if ("possible_id" not in flags and "high_cardinality" not in flags
and "constant" not in flags
and isinstance(dc, int) and 2 <= dc <= max_card):
group_keys.append({"col": name, "cardinality": dc, "score": 0.0})
elif it == "numeric":
num = col.get("numeric") or {}
if num.get("std") not in (None, 0) and not (
"possible_id" in flags and (col.get("unique_pct") or 0) >= 0.99):
measures.append(name)
group_keys = group_keys[:max_keys]
measures = measures[:max_measures]
pivots = []
if len(group_keys) >= 2:
pivots.append({"index": group_keys[0]["col"],
"columns": group_keys[1]["col"],
"value": measures[0] if measures else None})
return {"group_keys": group_keys, "measures": measures, "pivots": pivots,
"note": "selección cuantitativa básica"}
def _resolve_plan(profile: dict, ctx: dict, candidates: dict) -> dict:
"""Return {aggregations:[{group_by,measures,why}], pivots:[...], source}."""
group_keys = candidates.get("group_keys") or []
measures = candidates.get("measures") or []
if ctx.get("run_agg_llm") and suggest_aggregations_llm is not None:
try:
plan = suggest_aggregations_llm(
profile, candidates,
max_aggs=int(ctx.get("agg_max_keys", _DEF_MAX_KEYS)),
model=ctx.get("agg_llm_model", "claude-haiku-4-5-20251001"))
if _is_dict(plan) and plan.get("aggregations"):
return {"aggregations": plan.get("aggregations") or [],
"pivots": plan.get("pivots") or [],
"source": plan.get("source", "llm")}
except Exception: # noqa: BLE001 — fall back to the quantitative plan.
pass
aggregations = [{
"group_by": gk.get("col"),
"measures": measures,
"why": f"categórica de {_fmt_num(gk.get('cardinality'))} niveles",
} for gk in group_keys if _is_dict(gk) and gk.get("col")]
pivots = []
for pv in candidates.get("pivots") or []:
if _is_dict(pv) and pv.get("index") and pv.get("columns"):
pivots.append({"index": pv.get("index"), "columns": pv.get("columns"),
"value": pv.get("value") or (measures[0] if measures else None),
"agg": "mean", "why": "cruce de dos categóricas"})
return {"aggregations": aggregations, "pivots": pivots, "source": "quantitative"}
def _live_groupby(ctx: dict, group_by: str, measures: list, top_n: int):
"""Compute one group-by result live via the push-down registry function."""
db_path = ctx.get("db_path")
table = ctx.get("table")
if not db_path or not table or groupby_stats_duckdb is None:
return None
try:
out = groupby_stats_duckdb(db_path, table, group_by, list(measures or []),
top_n=top_n)
if _is_dict(out) and out.get("status") == "ok":
return out
except Exception: # noqa: BLE001
return None
return None
def _live_pivot(ctx: dict, index: str, columns: str, value, agg: str):
"""Compute one pivot live via the push-down registry function."""
db_path = ctx.get("db_path")
table = ctx.get("table")
if not db_path or not table or pivot_table_duckdb is None or not value:
return None
try:
out = pivot_table_duckdb(db_path, table, index, columns, value,
agg=agg or "mean")
if _is_dict(out) and out.get("status") == "ok":
return out
except Exception: # noqa: BLE001
return None
return None
# --------------------------------------------------------------------------- #
# Figure builders (lazy: matplotlib only imported when the renderer draws them).
# --------------------------------------------------------------------------- #
def _make_group_bars(group_by: str, measure: str, groups: list):
"""Vertical bars: mean of ``measure`` per group, bars from zero."""
labels, values = [], []
for g in groups:
if not _is_dict(g):
continue
mean = _measure_mean(g, measure)
if mean is None:
continue
labels.append(model._safe_str(g.get("key")))
values.append(float(mean))
if not labels:
return None
def _draw():
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.6, 3.6))
xs = list(range(len(labels)))
ax.bar(xs, values, color="#4e79a7", alpha=0.9, edgecolor="#2f4d6e",
linewidth=0.4)
ax.set_xticks(xs)
short = [(s[:18] + "") if len(s) > 19 else s for s in labels]
rot = 30 if max((len(s) for s in short), default=0) > 6 else 0
ax.set_xticklabels(short, rotation=rot, ha="right" if rot else "center",
fontsize=7)
ax.set_ylabel(f"media de {measure}", fontsize=8)
ax.set_xlabel(group_by, fontsize=8)
ax.set_title(f"Media de «{measure}» por «{group_by}»", fontsize=10)
ax.grid(axis="y", color="#dddddd", linewidth=0.6)
for spine in ("top", "right"):
ax.spines[spine].set_visible(False)
# Value labels above each bar.
vmax = max(values) if values else 0
for x, v in zip(xs, values):
ax.text(x, v + (abs(vmax) * 0.01 if vmax else 0.01),
_fmt_num(v, 2), ha="center", va="bottom", fontsize=6.5)
fig.tight_layout()
return fig
return _draw
def _make_pivot_bars(pivot: dict):
"""Grouped bars of a pivot: x = row_labels, one series per col_label."""
row_labels = pivot.get("row_labels") or []
col_labels = pivot.get("col_labels") or []
matrix = pivot.get("matrix") or []
if not row_labels or not col_labels or not matrix:
return None
def _draw():
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
n_rows = len(row_labels)
n_cols = len(col_labels)
fig, ax = plt.subplots(figsize=(6.8, 3.8))
total_w = 0.8
bar_w = total_w / max(n_cols, 1)
base = list(range(n_rows))
for j, clabel in enumerate(col_labels):
offs = [b - total_w / 2 + bar_w * (j + 0.5) for b in base]
vals = []
for i in range(n_rows):
cell = matrix[i][j] if (i < len(matrix) and j < len(matrix[i])) else None
vals.append(float(cell) if isinstance(cell, (int, float)) else 0.0)
color = _SERIES_COLORS[j % len(_SERIES_COLORS)]
ax.bar(offs, vals, width=bar_w, color=color, alpha=0.9,
label=model._safe_str(clabel))
ax.set_xticks(base)
short = [(s[:16] + "") if len(s) > 17 else s
for s in (model._safe_str(r) for r in row_labels)]
rot = 30 if max((len(s) for s in short), default=0) > 6 else 0
ax.set_xticklabels(short, rotation=rot, ha="right" if rot else "center",
fontsize=7)
ax.set_xlabel(model._safe_str(pivot.get("index")), fontsize=8)
ax.set_ylabel(f"{pivot.get('agg','mean')} de {pivot.get('value')}",
fontsize=8)
ax.set_title(f"{pivot.get('index')} × {pivot.get('columns')}", fontsize=10)
ax.grid(axis="y", color="#dddddd", linewidth=0.6)
ax.legend(title=model._safe_str(pivot.get("columns")), fontsize=6.5,
title_fontsize=7, frameon=True, framealpha=0.9, loc="best")
for spine in ("top", "right"):
ax.spines[spine].set_visible(False)
fig.tight_layout()
return fig
return _draw
def _group_bars_maker(group_by: str, measure: str, groups: list):
"""Bind per-aggregation args so the lazy closure is loop-safe."""
def _make():
return _make_group_bars(group_by, measure, groups)()
return _make
def _pivot_bars_maker(pivot: dict):
def _make():
return _make_pivot_bars(pivot)()
return _make
# --------------------------------------------------------------------------- #
# Section builders. Each returns a list of blocks (possibly empty).
# --------------------------------------------------------------------------- #
def _groupby_section(group_by: str, measures: list, result: dict, why: str) -> list:
"""Build the blocks for one group-by aggregation, or [] if unusable."""
if not _is_dict(result) or not result.get("groups"):
return []
groups = [g for g in result.get("groups") or [] if _is_dict(g)]
if not groups:
return []
eff_measures = result.get("measures") or measures or []
blocks = [model.Heading(text=f"Agrupado por «{group_by}»", level=2)]
intro = f"**{why}.** " if why else ""
intro += (f"{_fmt_num(result.get('n_groups') or len(groups))} grupos"
f"{' (top por tamaño)' if result.get('truncated') else ''}.")
blocks.append(model.Markdown(text=intro))
# Summary table: one row per group, count + mean of every measure.
header = ["Grupo", "n"] + [f"{m} (media)" for m in eff_measures]
rows = []
for g in groups:
row = [model._safe_str(g.get("key")), _fmt_num(g.get("n"))]
for m in eff_measures:
row.append(_fmt_num(_measure_mean(g, m), 2))
rows.append(row)
blocks.append(model.DataTable(
header=header, rows=rows, title=f"Resumen por «{group_by}»",
note="Conteo de filas y media de cada medida por grupo."))
if not eff_measures:
return blocks
# Primary measure: a bar chart + a detail table (mean/median/std/min/max).
primary = eff_measures[0]
bars = _make_group_bars(group_by, primary, groups)
if bars is not None:
blocks.append(model.Figure(
make=_group_bars_maker(group_by, primary, groups),
caption=f"Media de «{primary}» por «{group_by}» (barras desde cero)."))
det_header = ["Grupo", "n", "media", "mediana", "σ", "mín", "máx"]
det_rows = []
for g in groups:
stats = g.get("stats") if _is_dict(g.get("stats")) else {}
ms = stats.get(primary) if _is_dict(stats.get(primary)) else {}
det_rows.append([
model._safe_str(g.get("key")), _fmt_num(g.get("n")),
_fmt_num(ms.get("mean"), 2), _fmt_num(ms.get("median"), 2),
_fmt_num(ms.get("std"), 2), _fmt_num(ms.get("min"), 2),
_fmt_num(ms.get("max"), 2),
])
blocks.append(model.DataTable(
header=det_header, rows=det_rows,
title=f"Detalle de «{primary}» por «{group_by}»"))
return blocks
def _pivot_section(pivot_spec: dict, result: dict) -> list:
"""Build the blocks for one pivot table, or [] if unusable."""
if not _is_dict(result) or not result.get("row_labels"):
return []
row_labels = result.get("row_labels") or []
col_labels = result.get("col_labels") or []
matrix = result.get("matrix") or []
if not row_labels or not col_labels or not matrix:
return []
index = result.get("index") or pivot_spec.get("index")
columns = result.get("columns") or pivot_spec.get("columns")
value = result.get("value") or pivot_spec.get("value")
agg = result.get("agg") or pivot_spec.get("agg") or "mean"
why = pivot_spec.get("why") or ""
blocks = [model.Heading(text=f"Pivot: «{index}» × «{columns}»", level=2)]
intro = f"**{why}.** " if why else ""
intro += (f"{agg} de «{value}» cruzando «{index}» (filas) y «{columns}» "
f"(columnas).")
if result.get("truncated_rows") or result.get("truncated_cols"):
intro += " Limitado a las filas/columnas más frecuentes."
blocks.append(model.Markdown(text=intro))
header = [model._safe_str(index)] + [model._safe_str(c) for c in col_labels]
rows = []
for i, rlabel in enumerate(row_labels):
row = [model._safe_str(rlabel)]
cells = matrix[i] if i < len(matrix) else []
for j in range(len(col_labels)):
cell = cells[j] if j < len(cells) else None
row.append(_fmt_num(cell, 2))
rows.append(row)
blocks.append(model.DataTable(
header=header, rows=rows,
title=f"{agg} de «{value}»",
note=f"Cada celda es {agg} de «{value}» para esa combinación."))
fig_pivot = {"row_labels": row_labels, "col_labels": col_labels,
"matrix": matrix, "index": index, "columns": columns,
"value": value, "agg": agg}
if _make_pivot_bars(fig_pivot) is not None:
blocks.append(model.Figure(
make=_pivot_bars_maker(fig_pivot),
caption=f"{agg} de «{value}» por «{index}» y «{columns}» "
f"(barras agrupadas)."))
return blocks
def _insights_section(ctx: dict) -> list:
"""Optional pre-computed micro-analysis of the aggregations (SHOULD-11.4)."""
entries = ctx.get("agg_insights")
if not isinstance(entries, list) or not entries:
return []
blocks = [model.Heading(text="Interpretación de los grupos", level=2)]
for e in entries:
if not _is_dict(e):
continue
title = model._safe_str(e.get("title"))
text = model._safe_str(e.get("text"))
line = (f"**{title}.** " if title else "") + text
if line.strip():
blocks.append(model.Markdown(text=line))
return blocks if len(blocks) > 1 else []
# --------------------------------------------------------------------------- #
# Pre-computed path: ctx['aggregations'] already carries the results.
# --------------------------------------------------------------------------- #
def _sections_from_precomputed(agg: dict) -> list:
sections = []
for entry in agg.get("groupby") or []:
if not _is_dict(entry):
continue
sections += _groupby_section(
entry.get("group_by"), entry.get("measures") or [],
entry.get("result") or {}, entry.get("why") or "")
for entry in agg.get("pivots") or []:
if not _is_dict(entry):
continue
sections += _pivot_section(entry, entry.get("result") or {})
return sections
# --------------------------------------------------------------------------- #
# Live path: select keys, pick a plan, compute results via push-down functions.
# --------------------------------------------------------------------------- #
def _sections_live(profile: dict, ctx: dict, candidates: dict) -> list:
top_n = int(ctx.get("agg_top_n", _DEF_TOP_N))
plan = _resolve_plan(profile, ctx, candidates)
sections = []
for agg in plan.get("aggregations") or []:
if not _is_dict(agg) or not agg.get("group_by"):
continue
result = _live_groupby(ctx, agg.get("group_by"),
agg.get("measures") or [], top_n)
if result is not None:
sections += _groupby_section(agg.get("group_by"),
agg.get("measures") or [], result,
agg.get("why") or "")
for pv in plan.get("pivots") or []:
if not _is_dict(pv) or not pv.get("index") or not pv.get("columns"):
continue
result = _live_pivot(ctx, pv.get("index"), pv.get("columns"),
pv.get("value"), pv.get("agg") or "mean")
if result is not None:
sections += _pivot_section(pv, result)
return sections
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def _intro_blocks(gloss=None, mark_term: bool = False) -> list:
if gloss is not None:
for key, (label, definition) in _TERM_DEFS.items():
gloss.add(key, label, definition)
t_groupby = _term(mark_term, "groupby", "**por grupos** (split-apply-combine)")
t_pivot = _term(mark_term, "pivot_table", "**tablas dinámicas** (pivot)")
text = (
f"Este capítulo analiza la tabla {t_groupby}: elige las columnas "
"categóricas más informativas (por cardinalidad y relevancia, no todas "
"contra todas) y resume las variables numéricas dentro de cada grupo "
f"(conteo, media, mediana, desviación). Se añaden {t_pivot} y "
"**gráficos de barras** (siempre desde cero) para comparar los grupos."
)
return [model.Heading(text=CHAPTER_TITLE, level=1),
model.Markdown(text=text)]
def build_agregacion(profile: dict, ctx: dict):
"""Build the AGREGACION Chapter, or None if the dataset can't be grouped.
Args:
profile: the ``eda`` group TableProfile dict.
ctx: presentation context (see module docstring for the keys consumed).
Returns:
A ``model.Chapter`` with per-group stats, pivots and bar charts; or
``None`` when the dataset has no low-cardinality categorical column to
group by (the chapter does not apply).
"""
profile = profile or {}
ctx = ctx or {}
if not isinstance(profile, dict):
return None
# Shared glossary collector: groupby + pivot_table live in the always-present
# intro, so they are registered + marked there. Degrades silently (mark_term
# False) when no collector is in ctx (standalone render).
glossary = ctx.get("glossary")
gloss = glossary if isinstance(glossary, model.GlossaryCollector) else None
mark_term = gloss is not None
# Pre-computed results take precedence (offline / tests / forward-compat).
pre = ctx.get("aggregations")
if _is_dict(pre) and (pre.get("groupby") or pre.get("pivots")):
sections = _sections_from_precomputed(pre)
if not sections:
return None
blocks = (_intro_blocks(gloss, mark_term) + sections
+ _insights_section(ctx))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
# Live path: needs at least one categorical key to group by.
candidates = _resolve_candidates(profile, ctx)
if not _is_dict(candidates) or not (candidates.get("group_keys")):
return None # chapter does not apply: nothing to group by.
sections = _sections_live(profile, ctx, candidates)
if not sections:
# Applies (there are categorical keys) but no aggregation data is
# reachable: emit an honest note instead of fabricating numbers.
keys = ", ".join(model._safe_str((k or {}).get("col"))
for k in candidates.get("group_keys") or []
if _is_dict(k))
note = model.Note(
"No se pudo calcular la agregación: el capítulo necesita los datos "
"crudos. Pasa ctx['db_path'] + ctx['table'] (para el cálculo "
"push-down en DuckDB) o ctx['aggregations'] ya precalculado. "
f"Columnas categóricas candidatas: {keys or ''}.")
blocks = (_intro_blocks(gloss, mark_term) + [note]
+ _insights_section(ctx))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
blocks = _intro_blocks(gloss, mark_term) + sections + _insights_section(ctx)
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,278 @@
"""Tests for the AGREGACION chapter — DoD: golden + edges + error/no-cut path.
Self-contained and deterministic: no DuckDB and no LLM. The aggregation results
are passed pre-computed via ``ctx['aggregations']`` (the same shape the push-down
registry functions ``groupby_stats_duckdb`` / ``pivot_table_duckdb`` produce), so
the chapter's rendering logic is exercised without touching disk or the network.
Live push-down + LLM selection are covered separately by the golden script.
Verifies:
- Golden: a profile with categoricals + numerics builds a Chapter with per-group
stats tables, a pivot table and bar-chart figures, and it renders to PDF AND
PPTX showing the group keys, values and pivot — nothing cut.
- Edges: a dataset with no low-cardinality categorical returns None; an empty
profile returns None; a profile that *could* be grouped but has no reachable
data degrades to an honest note instead of raising.
- No-cut: many groups (30) + a long interpretation paragraph survive intact in
the rendered PDF (table split by rows, text wrapped whole).
"""
import os
import re
import tempfile
from pptx import Presentation
from pypdf import PdfReader
from datascience.automatic_eda.chapters.agregacion import build_agregacion
from datascience.automatic_eda.model import Chapter
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
# --------------------------------------------------------------------------- #
# Synthetic fixtures.
# --------------------------------------------------------------------------- #
def _profile() -> dict:
"""A titanic-like profile: 2 categoricals + 2 numeric measures + 1 id."""
return {
"table": "titanic",
"source": "/data/titanic.csv",
"n_rows": 891,
"n_cols": 5,
"key_candidates": ["passenger_id"],
"columns": [
{"name": "passenger_id", "inferred_type": "numeric",
"unique_pct": 1.0, "flags": ["possible_id"],
"numeric": {"mean": 446.0, "std": 257.0}},
{"name": "sex", "inferred_type": "categorical", "distinct_count": 2,
"flags": [], "categorical": {"n_distinct": 2, "imbalance": 0.1,
"top": [{"value": "male", "count": 577}]}},
{"name": "pclass", "inferred_type": "categorical", "distinct_count": 3,
"flags": [], "categorical": {"n_distinct": 3, "imbalance": 0.2}},
{"name": "fare", "inferred_type": "numeric", "flags": [],
"numeric": {"mean": 32.2, "std": 49.7, "cv": 1.54}},
{"name": "age", "inferred_type": "numeric", "flags": [],
"numeric": {"mean": 29.7, "std": 14.5, "cv": 0.49}},
],
}
def _groupby_result(group_by: str, keys_n: list) -> dict:
"""A groupby_stats_duckdb-shaped result for `fare` and `age`."""
groups = []
for i, (key, n) in enumerate(keys_n):
groups.append({
"key": key, "n": n,
"stats": {
"fare": {"mean": 20.0 + i * 15, "median": 10.0 + i * 8,
"std": 40.0 + i, "min": 0.0, "max": 512.3},
"age": {"mean": 28.0 + i, "median": 27.0 + i, "std": 14.0,
"min": 0.42, "max": 80.0},
},
})
return {"status": "ok", "group_by": group_by, "measures": ["fare", "age"],
"aggs": ["count", "mean", "median", "std", "min", "max"],
"n_groups": len(groups), "truncated": False, "groups": groups}
def _pivot_result() -> dict:
return {"status": "ok", "index": "sex", "columns": "pclass", "value": "fare",
"agg": "mean", "row_labels": ["male", "female"],
"col_labels": ["1", "2", "3"],
"matrix": [[62.0, 19.0, 12.0], [110.0, 22.0, 15.0]],
"truncated_rows": False, "truncated_cols": False}
def _ctx_precomputed() -> dict:
return {
"aggregations": {
"groupby": [
{"group_by": "sex", "measures": ["fare", "age"],
"why": "sexo del pasajero",
"result": _groupby_result("sex", [("male", 577), ("female", 314)])},
{"group_by": "pclass", "measures": ["fare", "age"],
"why": "clase del billete",
"result": _groupby_result(
"pclass", [("3", 491), ("1", 216), ("2", 184)])},
],
"pivots": [
{"index": "sex", "columns": "pclass", "value": "fare",
"agg": "mean", "why": "tarifa por sexo y clase",
"result": _pivot_result()},
],
},
"agg_insights": [
{"title": "Tarifa por sexo",
"text": "Las mujeres pagaron de media casi el doble que los hombres."},
],
}
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def _pptx_text(path: str) -> str:
prs = Presentation(path)
parts = []
for sl in prs.slides:
for sh in sl.shapes:
if sh.has_text_frame:
parts.append(sh.text_frame.text)
if sh.has_table:
tb = sh.table
for r in range(len(tb.rows)):
for c in range(len(tb.columns)):
parts.append(tb.cell(r, c).text)
return re.sub(r"\s+", " ", " ".join(parts))
# --------------------------------------------------------------------------- #
# Golden: builds a Chapter and renders to both formats.
# --------------------------------------------------------------------------- #
def test_golden_chapter_blocks_present():
ch = build_agregacion(_profile(), _ctx_precomputed())
assert isinstance(ch, Chapter)
assert ch.id == "agregacion"
kinds = [b.kind for b in ch.blocks]
assert "heading" in kinds
assert kinds.count("data_table") >= 3 # 2 group summaries + pivot (+details)
assert "figure" in kinds # at least one bar chart.
# Headings mention the group keys and the pivot.
htext = " ".join(b.text for b in ch.blocks if b.kind == "heading")
assert "sex" in htext and "pclass" in htext and "Pivot" in htext
def test_golden_render_pdf():
ch = build_agregacion(_profile(), _ctx_precomputed())
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "agg.pdf")
res = render_automatic_eda_pdf([ch], out, {"write_manifest": False})
assert res["path"] == out and os.path.exists(out)
assert res["n_pages"] >= 1
txt = _pdf_text(out)
assert "Agregación por grupos" in txt
assert "male" in txt and "female" in txt # group + pivot labels.
assert "Pivot" in txt
assert "mediana" in txt # per-measure detail.
assert "casi el doble" in txt # interpretation kept.
def test_golden_render_pptx():
ch = build_agregacion(_profile(), _ctx_precomputed())
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "agg.pptx")
res = render_automatic_eda_pptx([ch], out, {"write_manifest": False})
assert res["path"] == out and os.path.exists(out)
assert res["n_slides"] >= 1
txt = _pptx_text(out)
assert "male" in txt and "pclass" in txt
assert "Pivot" in txt or "sex" in txt
# --------------------------------------------------------------------------- #
# Edges.
# --------------------------------------------------------------------------- #
def test_edge_no_categorical_returns_none():
# Only numerics + an id: nothing to group by -> chapter does not apply.
prof = {
"table": "t", "n_rows": 100, "key_candidates": ["id"],
"columns": [
{"name": "id", "inferred_type": "numeric", "unique_pct": 1.0,
"flags": ["possible_id"], "numeric": {"std": 10.0}},
{"name": "x", "inferred_type": "numeric", "flags": [],
"numeric": {"mean": 1.0, "std": 2.0}},
],
}
assert build_agregacion(prof, {}) is None
def test_edge_empty_profile_returns_none():
assert build_agregacion({}, {}) is None
assert build_agregacion(None, None) is None
def test_edge_high_cardinality_only_returns_none():
# The single categorical is id-like (high cardinality) -> not groupable.
prof = {
"table": "t", "n_rows": 100, "key_candidates": ["uuid"],
"columns": [
{"name": "uuid", "inferred_type": "categorical", "distinct_count": 100,
"flags": ["high_cardinality", "possible_id"]},
{"name": "x", "inferred_type": "numeric", "flags": [],
"numeric": {"mean": 1.0, "std": 2.0}},
],
}
assert build_agregacion(prof, {}) is None
def test_live_without_data_degrades_to_note():
# Has a categorical to group by but no db_path / no precomputed results:
# must NOT raise and must emit an honest note (chapter still applies).
prof = {
"table": "t", "n_rows": 100, "key_candidates": [],
"columns": [
{"name": "grp", "inferred_type": "categorical", "distinct_count": 3,
"flags": [], "categorical": {"n_distinct": 3}},
{"name": "v", "inferred_type": "numeric", "flags": [],
"numeric": {"mean": 1.0, "std": 2.0}},
],
}
ch = build_agregacion(prof, {})
assert isinstance(ch, Chapter)
notes = [b.text for b in ch.blocks if b.kind == "note"]
assert any("datos crudos" in n for n in notes)
# --------------------------------------------------------------------------- #
# No-cut: many groups + long text survive intact in the PDF.
# --------------------------------------------------------------------------- #
def test_anti_corte_muchos_grupos_y_texto_largo():
keys_n = [(f"grupo_{i:02d}", 30 - (i % 5)) for i in range(30)]
long_text = " ".join(f"palabra{i}" for i in range(120))
ctx = {
"aggregations": {
"groupby": [
{"group_by": "cat", "measures": ["fare"], "why": "muchos niveles",
"result": _groupby_result("cat", keys_n)},
],
"pivots": [],
},
"agg_insights": [{"title": "Nota larga", "text": long_text}],
}
ch = build_agregacion(_profile(), ctx)
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "big.pdf")
res = render_automatic_eda_pdf([ch], out, {"write_manifest": False})
assert res["path"] == out
assert res["n_pages"] > 1 # 30-row table + figure spill across pages.
txt = _pdf_text(out)
# First and last group labels both survive (table not truncated).
assert "grupo_00" in txt and "grupo_29" in txt
# First, middle and last words of the long paragraph all present.
for i in (0, 60, 119):
assert f"palabra{i}" in txt
def test_glosario_engancha_groupby_y_pivot():
"""Mejora 4b: la agrupación (split-apply-combine) y la tabla dinámica (pivot)
se registran en el colector compartido y se marcan clicables en el cuerpo.
Sin colector en ctx, el capítulo degrada y no marca nada."""
from datascience.automatic_eda.model import GlossaryCollector
g = GlossaryCollector()
ctx = dict(_ctx_precomputed())
ctx["glossary"] = g
ch = build_agregacion(_profile(), ctx)
assert ch is not None
keys = {t["key"] for t in g.terms()}
assert {"groupby", "pivot_table"} <= keys
body = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
assert "[[term:groupby]]" in body and "[[term:pivot_table]]" in body
# Sin colector: degrada limpio (ningún marcador en el cuerpo).
ch2 = build_agregacion(_profile(), _ctx_precomputed())
body2 = " ".join(b.text for b in ch2.blocks if b.kind == "markdown")
assert "[[term:" not in body2
@@ -0,0 +1,235 @@
"""LLM analysis chapter (ANÁLISIS LLM) — the interpretive layer, next to overview.
Third reference chapter for AutomaticEDA. Renders the ``llm`` block that the
``eda`` group function ``eda_llm_insights`` already produced and stored in the
``TableProfile`` — it does NOT call the LLM nor recompute anything. The block is
turned into clean, markdown-style document blocks so it reads as a real chapter
(table summary, row meaning, data dictionary, suggested analyses, cleaning
suggestions, PII findings) and, crucially, **nothing is ever cut** in PDF or
PPTX:
* Prose (summary, row meaning) → ``Markdown`` blocks the renderers wrap to whole
lines, so no word is lost no matter how long the text is.
* The data dictionary and PII findings → ``DataTable`` blocks the paginator
splits by rows (repeating the header) and whose long cells wrap inside their
column — wide, multi-row tables never overflow a page/slide.
* Cleaning suggestions and suggested analyses → ``Markdown`` bullet lists; each
item is a whole line the renderer wraps, never truncated mid-entry.
Position: this chapter is declared in ``chapters_registry.CHAPTER_ORDER`` right
after ``overview`` so the interpretation sits next to the table preview, as the
user asked ("va junto al overview").
Data source: the ``llm`` dict produced by ``eda_llm_insights`` (group ``eda``),
read from ``profile['llm']`` (or ``ctx['llm']`` as a fallback). Shape::
{
"summary": str, # what the table is, 2-3 sentences
"row_meaning": str, # what one row represents / granularity
"dictionary": [ {"column","description","business_meaning","unit"} ],
"pii": [ {"column","kind","severity"} ],
"cleaning": [str], # cleaning / transformation suggestions
"analyses": [str], # suggested questions / analyses / hypotheses
}
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
Reads everything defensively (``.get``) and NEVER raises; returns ``None`` when
the profile carries no LLM block (e.g. ``profile_table`` ran without
``run_llm``), so the chapter is simply omitted from the document.
"""
from __future__ import annotations
from .. import model
# 1.1.0: drop the duplicated section labels — the dictionary and PII DataTables
# no longer carry a ``title`` (the section Heading labels them once, per the
# OVERVIEW pattern in the contract). The data-dictionary column already reads
# "Significado de negocio".
CHAPTER_VERSION = "1.1.0"
CHAPTER_ID = "analisis_llm"
CHAPTER_TITLE = "Análisis LLM"
# Key under which eda_llm_insights stores its interpretive block in the profile.
LLM_KEY = "llm"
def _clean_text(value) -> str:
"""Coerce a value to a single trimmed line (collapse inner newlines).
Used for bullet items so each suggestion stays a single markdown bullet the
renderer wraps; never drops content, only normalizes whitespace.
"""
text = model._safe_str(value).strip()
if not text:
return ""
return " ".join(text.split())
def _para(value) -> str:
"""Coerce a value to trimmed prose, preserving paragraph breaks."""
text = model._safe_str(value).strip()
if not text:
return ""
# Keep blank-line paragraph breaks; collapse runs of spaces/tabs per line.
lines = [" ".join(ln.split()) for ln in text.splitlines()]
out: list = []
for ln in lines:
if ln or (out and out[-1] != ""):
out.append(ln)
return "\n".join(out).strip()
def _bullets(items) -> str:
"""Build a markdown bullet list from a sequence of strings.
Each item becomes one ``- ...`` line (a whole, wrappable unit). Empty items
and non-list inputs are handled gracefully; returns "" when there is nothing.
"""
if isinstance(items, str):
items = [items]
if not isinstance(items, (list, tuple)):
return ""
lines = []
for it in items:
text = _clean_text(it)
if text:
lines.append(f"- {text}")
return "\n".join(lines)
def _summary_blocks(llm: dict) -> list:
"""Heading + prose for the table summary, or [] if absent."""
text = _para(llm.get("summary"))
if not text:
return []
return [model.Heading(text="Resumen de la tabla", level=2),
model.Markdown(text=text)]
def _row_meaning_blocks(llm: dict) -> list:
"""Heading + prose for what one row represents, or [] if absent."""
text = _para(llm.get("row_meaning"))
if not text:
return []
return [model.Heading(text="Significado de una fila", level=2),
model.Markdown(text=text)]
def _dictionary_block(llm: dict):
"""DataTable for the data dictionary, or None if absent/empty.
Columns: Columna / Descripción / Significado de negocio / Unidad. The
paginator splits this by rows repeating the header and wraps long cells, so a
long dictionary (many columns) never gets cut.
The block carries **no** ``title``: the section is labelled once by the
``Heading`` that ``build_analisis_llm`` appends right before it (the canonical
OVERVIEW pattern, contract §8). Giving the table its own ``title`` too would
print "Diccionario de datos" twice in a row.
"""
entries = llm.get("dictionary")
if not isinstance(entries, (list, tuple)) or not entries:
return None
header = ["Columna", "Descripción", "Significado de negocio", "Unidad"]
rows = []
for e in entries:
if not isinstance(e, dict):
# Be tolerant: a bare string still shows up as a description row.
rows.append(["", _clean_text(e), "", ""])
continue
rows.append([
_clean_text(e.get("column")) or "",
_clean_text(e.get("description")),
_clean_text(e.get("business_meaning")),
_clean_text(e.get("unit")),
])
if not rows:
return None
return model.DataTable(header=header, rows=rows)
def _analyses_blocks(llm: dict) -> list:
"""Heading + bullet list of suggested analyses, or [] if absent."""
bullets = _bullets(llm.get("analyses"))
if not bullets:
return []
return [model.Heading(text="Análisis sugeridos", level=2),
model.Markdown(text=bullets)]
def _cleaning_blocks(llm: dict) -> list:
"""Heading + bullet list of cleaning suggestions, or [] if absent."""
bullets = _bullets(llm.get("cleaning"))
if not bullets:
return []
return [model.Heading(text="Limpieza sugerida", level=2),
model.Markdown(text=bullets)]
def _pii_block(llm: dict):
"""DataTable for PII/GDPR findings, or None if absent/empty.
Like the dictionary block, it carries **no** ``title`` (the ``Heading`` in
``build_analisis_llm`` labels the section once); it keeps its ``note`` with
the orientative-detection caveat, which the renderers print under the table.
"""
entries = llm.get("pii")
if not isinstance(entries, (list, tuple)) or not entries:
return None
header = ["Columna", "Tipo", "Severidad"]
rows = []
for e in entries:
if not isinstance(e, dict):
continue
rows.append([
_clean_text(e.get("column")) or "",
_clean_text(e.get("kind")),
_clean_text(e.get("severity")),
])
if not rows:
return None
return model.DataTable(
header=header, rows=rows,
note="detección automática orientativa — revisar antes de tratar los datos")
def build_analisis_llm(profile: dict, ctx: dict):
"""Build the LLM analysis Chapter, or None if there is no LLM block.
Consumes ``profile['llm']`` (the block produced by ``eda_llm_insights``,
group ``eda``); falls back to ``ctx['llm']``. Returns ``None`` when no LLM
block is present or it carries no usable content, so the chapter is omitted
rather than rendering an empty section.
"""
profile = profile or {}
ctx = ctx or {}
llm = profile.get(LLM_KEY)
if not isinstance(llm, dict):
llm = ctx.get(LLM_KEY)
if not isinstance(llm, dict) or not llm:
return None
blocks: list = []
blocks += _summary_blocks(llm)
blocks += _row_meaning_blocks(llm)
dict_block = _dictionary_block(llm)
if dict_block is not None:
blocks.append(model.Heading(text="Diccionario de datos", level=2))
blocks.append(dict_block)
blocks += _analyses_blocks(llm)
blocks += _cleaning_blocks(llm)
pii_block = _pii_block(llm)
if pii_block is not None:
blocks.append(model.Heading(text="Datos personales (PII / RGPD)", level=2))
blocks.append(pii_block)
if not blocks:
return None # LLM block present but every field empty → omit chapter.
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,229 @@
"""Tests for the ANÁLISIS LLM chapter — DoD: golden + edges + anti-cut.
Self-contained: builds a synthetic TableProfile carrying an ``llm`` block (the
shape ``eda_llm_insights`` produces) so the suite is fast and deterministic — no
DuckDB and no LLM call. Verifies:
* golden — ``build_analisis_llm`` yields the chapter and the full document
renders to PDF *and* PPTX with the summary, a suggested analysis, a cleaning
suggestion and a dictionary column all present;
* order — the chapter sits immediately after ``overview`` (user requirement);
* edges — a profile with no ``llm`` block (or None/empty/malformed) returns
``None`` and never raises;
* anti-cut — a long dictionary (40 rows) and a 150-char cleaning suggestion are
rendered to PDF and PPTX without losing a single row or word.
"""
import os
import re
import tempfile
from pypdf import PdfReader
from pptx import Presentation
from datascience.automatic_eda.chapters.analisis_llm import (
build_analisis_llm, CHAPTER_VERSION)
from datascience.automatic_eda.chapters_registry import build_document
from datascience.automatic_eda.model import Chapter, DataTable, Heading
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
def _profile() -> dict:
return {
"table": "ventas",
"source": "/data/ventas.csv",
"profiled_at": "2026-06-30T10:00:00+00:00",
"n_rows": 1000,
"n_cols": 2,
"quality_score": 92.5,
"columns": [
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0,
"numeric": {"mean": 42.5, "median": 40.0, "min": 1.0,
"max": 100.0, "std": 12.3}},
{"name": "categoria", "inferred_type": "categorical",
"null_pct": 0.0, "null_count": 0,
"categorical": {"top": [{"value": "neumaticos", "count": 500}]}},
],
"llm": {
"summary": "Tabla de ventas por producto. Token SUMMARYTOKEN.",
"row_meaning": "Cada fila es una venta. Token ROWTOKEN.",
"dictionary": [
{"column": "precio", "description": "Precio unitario DESCTOKEN",
"business_meaning": "Ingreso por unidad", "unit": "EUR"},
{"column": "categoria", "description": "Familia de producto",
"business_meaning": "Segmento comercial", "unit": ""},
],
"pii": [{"column": "categoria", "kind": "ninguno", "severity": "low"}],
"cleaning": ["Quitar nulos de precio CLEANTOKEN",
"Normalizar mayusculas en categoria"],
"analyses": ["Estudiar relacion precio-categoria ANALYSISTOKEN",
"Detectar outliers de precio"],
},
}
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def _pptx_text(path: str) -> str:
prs = Presentation(path)
parts = []
for sl in prs.slides:
for sh in sl.shapes:
if sh.has_text_frame:
parts.append(sh.text_frame.text)
if sh.has_table:
tb = sh.table
for r in range(len(tb.rows)):
for c in range(len(tb.columns)):
parts.append(tb.cell(r, c).text)
return re.sub(r"\s+", " ", " ".join(parts))
def test_golden_build_y_render_pdf_pptx():
prof = _profile()
ch = build_analisis_llm(prof, {})
assert ch is not None
assert ch.id == "analisis_llm"
assert ch.version == CHAPTER_VERSION
assert ch.blocks # non-empty.
with tempfile.TemporaryDirectory() as d:
out_pdf = os.path.join(d, "eda.pdf")
res = render_automatic_eda_pdf(prof, out_pdf, {"title": "EDA — ventas"})
assert res["path"] == out_pdf and os.path.exists(out_pdf)
ids = [c["id"] for c in res["chapters"]]
assert "analisis_llm" in ids
txt = _pdf_text(out_pdf)
# The user's required content: summary, suggested analyses, cleaning.
assert "SUMMARYTOKEN" in txt
assert "ANALYSISTOKEN" in txt
assert "CLEANTOKEN" in txt
assert "DESCTOKEN" in txt # data dictionary cell.
out_pptx = os.path.join(d, "eda.pptx")
res2 = render_automatic_eda_pptx(prof, out_pptx, {"title": "EDA — ventas"})
assert res2["path"] == out_pptx and os.path.exists(out_pptx)
ids2 = [c["id"] for c in res2["chapters"]]
assert "analisis_llm" in ids2
ptx = _pptx_text(out_pptx)
assert "SUMMARYTOKEN" in ptx
assert "ANALYSISTOKEN" in ptx
assert "CLEANTOKEN" in ptx
assert "DESCTOKEN" in ptx
def test_sin_rotulos_duplicados_y_significado_de_negocio():
"""The dictionary / PII sections must be labelled ONCE.
Regression for the duplicated 'Diccionario de datos' and 'Datos personales
(PII / RGPD)' headings (each section used to print its label twice: a Heading
plus the DataTable's own title). The fix drops the DataTable title and keeps
a single Heading — the OVERVIEW pattern. The data-dictionary column header is
also pinned to the exact text 'Significado de negocio'.
"""
ch = build_analisis_llm(_profile(), {})
assert ch is not None
# Structure: section labels come from Headings; tables carry no title.
headings = [b.text for b in ch.blocks if isinstance(b, Heading)]
assert headings.count("Diccionario de datos") == 1
assert headings.count("Datos personales (PII / RGPD)") == 1
for b in ch.blocks:
if isinstance(b, DataTable):
assert not b.title, f"DataTable should not duplicate the label: {b.title!r}"
# The data dictionary's third column reads exactly 'Significado de negocio'.
dicts = [b for b in ch.blocks if isinstance(b, DataTable) and "Descripción" in b.header]
assert dicts, "expected the data-dictionary DataTable"
assert dicts[0].header == ["Columna", "Descripción", "Significado de negocio", "Unidad"]
# The PII table keeps its orientative-detection note.
pii = [b for b in ch.blocks if isinstance(b, DataTable) and b.header == ["Columna", "Tipo", "Severidad"]]
assert pii and pii[0].note and "orientativa" in pii[0].note
# Render: each label appears exactly once across the whole document (the only
# 'Diccionario de datos' / 'Datos personales' producer is this chapter).
with tempfile.TemporaryDirectory() as d:
out_pdf = os.path.join(d, "eda.pdf")
render_automatic_eda_pdf(_profile(), out_pdf, {"title": "EDA — ventas"})
txt = _pdf_text(out_pdf)
assert txt.count("Diccionario de datos") == 1
assert txt.count("Datos personales") == 1
def test_orden_capitulo_junto_a_overview():
chapters = build_document(_profile(), {})
ids = [c.id for c in chapters]
assert "overview" in ids and "analisis_llm" in ids
# User requirement: the LLM chapter sits right after overview.
assert ids.index("analisis_llm") == ids.index("overview") + 1
def test_edge_sin_llm_devuelve_none():
# No llm block at all.
prof = {k: v for k, v in _profile().items() if k != "llm"}
assert build_analisis_llm(prof, {}) is None
# None / empty / malformed never raise and yield None.
assert build_analisis_llm(None, None) is None
assert build_analisis_llm({}, {}) is None
assert build_analisis_llm({"llm": {}}, {}) is None
assert build_analisis_llm({"llm": "not-a-dict"}, {}) is None
# All-empty fields → omitted (no blocks).
empty = {"llm": {"summary": "", "dictionary": [], "cleaning": [],
"analyses": [], "pii": [], "row_meaning": ""}}
assert build_analisis_llm(empty, {}) is None
def test_edge_llm_via_ctx_fallback():
# The block may arrive in ctx instead of the profile.
prof = {k: v for k, v in _profile().items() if k != "llm"}
ctx = {"llm": {"summary": "Resumen via ctx CTXTOKEN."}}
ch = build_analisis_llm(prof, ctx)
assert ch is not None and ch.id == "analisis_llm"
def test_anti_cortes_diccionario_largo_y_limpieza_larga():
long_clean = ("Lorem ipsum dolor sit amet consectetur adipiscing elit sed do "
"eiusmod tempor incididunt ut labore et dolore magna aliqua "
"reprehenderit voluptate velit esse cillum dolore")
dictionary = [
{"column": f"col_{i}",
"description": f"Descripcion larga numero {i} con bastante texto para "
f"forzar el wrap dentro de la celda fila{i}",
"business_meaning": f"Significado de negocio {i}", "unit": "u"}
for i in range(40)
]
prof = {
"table": "t", "n_rows": 1, "n_cols": 1, "columns": [],
"llm": {"summary": "S", "dictionary": dictionary,
"cleaning": [long_clean], "analyses": ["A"]},
}
ch = build_analisis_llm(prof, {})
assert ch is not None
# Structure: the dictionary DataTable keeps ALL 40 rows — none dropped on
# construction (the renderers then split it by rows, repeating the header).
dts = [b for b in ch.blocks if isinstance(b, DataTable)]
assert any(len(dt.rows) == 40 for dt in dts)
with tempfile.TemporaryDirectory() as d:
out_pdf = os.path.join(d, "x.pdf")
render_automatic_eda_pdf([ch], out_pdf, {"write_manifest": False})
# 40 wide rows + a long cleaning line cannot fit one page → it spills,
# which is exactly the no-cut behaviour (paginate, never truncate).
assert len(PdfReader(out_pdf).pages) > 1
txt = _pdf_text(out_pdf)
# The long cleaning suggestion is wrapped word-by-word, not truncated.
for word in ("Lorem", "incididunt", "reprehenderit", "voluptate", "cillum"):
assert word in txt
out_pptx = os.path.join(d, "x.pptx")
res2 = render_automatic_eda_pptx([ch], out_pptx, {"write_manifest": False})
assert res2["n_slides"] > 1 # table + long text spill across slides.
ptx = _pptx_text(out_pptx)
for word in ("Lorem", "reprehenderit", "voluptate"):
assert word in ptx
@@ -1,22 +1,27 @@
"""Data-quality chapter (CALIDAD) for AutomaticEDA.
Builds the quality chapter from a ``TableProfile`` of the ``eda`` group. The
chapter answers, in Spanish and as tables, the three things the user asked for:
chapter implements the quality model of report 2046:
1. **En qué se basa la calidad** — an intro paragraph explaining the criteria and
their weights (completeness, validity, consistency) before any number, plus a
table-level summary (global score and aggregates).
1. **En qué se basa la calidad** — a concise intro naming the two scored
dimensions and their weights (completitud 60%, validez 40%) plus the
table-level row uniqueness, BEFORE any number, and stating that outliers are
reported as observations and do **not** lower the score. The criteria terms
(calidad de datos, completitud, validez, unicidad de registro) are hooked
into the shared glossary as clickable jumps; their full definitions live in
the GLOSARIO chapter, not inline here.
2. **Scores por columna** — a table with, per column, the total quality score and
its breakdown into completeness / validity / consistency.
3. **Problemas en español** — a second table listing, per column, the readable
issues in Spanish (kept separate from the type ``flags``).
its breakdown into completeness / validity (no consistency dimension).
3. **Problemas de calidad** — a table listing ONLY real quality defects
(nulls, empty cells, values not conforming to their type/semantics).
4. **Observaciones analíticas** — a SEPARATE table for outliers, constant
columns, high-cardinality ids and strong skew, with an explicit note that
these do not affect the score.
The breakdown and the issues are NOT recomputed here: they come from the registry
function ``column_quality_score`` (group ``eda``), which already derives
``{score, completeness, validity, consistency, issues}`` from the ColumnProfile.
This chapter is render-only — it consumes that function and lays the result out
as model blocks; the renderers paginate tables (splitting by rows, repeating the
header) and wrap long cells so nothing is ever cut.
The breakdown, issues and observations are NOT recomputed here: they come from
the registry function ``column_quality_score`` (group ``eda``), which derives
``{score, completeness, validity, dimensions, applicable, issues,
observations}`` from the ColumnProfile. This chapter is render-only.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
@@ -33,28 +38,47 @@ try: # pragma: no cover - import wiring
except Exception: # noqa: BLE001 - never let an import error abort the document.
_column_quality_score = None
CHAPTER_VERSION = "1.0.0"
CHAPTER_VERSION = "2.0.0"
CHAPTER_ID = "calidad"
CHAPTER_TITLE = "Calidad"
# Weights mirror column_quality_score: completeness 0.5, validity 0.3,
# consistency 0.2. Kept here only to render the human explanation; the actual
# numbers always come from the function so the two never drift in computation.
_CRITERIA_INTRO = (
"La calidad de cada columna es un score de 0 a 100 que combina tres "
"criterios, cada uno con un peso:\n\n"
"- **Completitud (peso 50%)**: proporción de valores presentes (sin nulos "
"ni vacíos). Una columna con muchos nulos baja de score.\n"
"- **Validez (peso 30%)**: los valores son coherentes con su tipo y rango "
"esperado (penaliza outliers y semánticas declaradas que no coinciden).\n"
"- **Consistencia (peso 20%)**: la columna aporta información útil (penaliza "
"columnas constantes o identificadores de cardinalidad muy alta).\n\n"
"Score = 100 × (0,5·completitud + 0,3·validez + 0,2·consistencia). "
"Los problemas detectados por columna se listan en español más abajo."
)
# Glossary terms this chapter explains (report 2046 §6). Registered in the shared
# collector and marked clickable on their first appearance (contract §11.1).
_TERMS = {
"calidad_datos": (
"Calidad de datos (score 0-100)",
"Mide hasta qué punto los datos están presentes y son utilizables tal "
"cual, no si son «buenos para el análisis». Se compone solo de "
"dimensiones medibles automáticamente desde el perfil de la tabla, sin "
"fuente externa de verdad: completitud (60%), validez (40%, cuando es "
"medible) y, a nivel de tabla, unicidad de registro. Los valores "
"atípicos NO bajan la calidad: se listan aparte como observaciones.",
),
"completitud": (
"Completitud",
"Proporción de valores realmente presentes en una columna (1 % de "
"nulos; en texto, las celdas vacías también cuentan como faltantes). Los "
"nulos y vacíos bajan el score porque falta información que debería "
"estar. Pesa el 60% del score de columna.",
),
"validez": (
"Validez",
"Proporción de valores que encajan con su tipo o formato esperado: un "
"número que parsea, una fecha legible, un email con forma de email. Los "
"valores que no parsean a su tipo bajan el score. Si la columna es texto "
"libre sin formato esperado, la validez no se puede medir y el score se "
"basa solo en la completitud. Pesa el 40% del score cuando es medible.",
),
"unicidad_registro": (
"Unicidad de registro",
"A nivel de tabla, las filas duplicadas restan calidad al conjunto "
"(1 % de filas duplicadas). Es distinta de que una columna no-clave "
"repita valores, que no es un defecto de calidad.",
),
}
# Cap for the joined issues cell so a single row never grows taller than a page;
# the remainder is summarized as "(+N más)" instead of being silently dropped.
# Cap for the joined cell so a single row never grows taller than a page; the
# remainder is summarized as "(+N más)" instead of being silently dropped.
_ISSUES_MAXLEN = 160
@@ -82,12 +106,19 @@ def _fmt_unit_pct(value) -> str:
return str(value)
def _fmt_validity(value) -> str:
"""Validity is ``None`` when not applicable: show ``n/a`` not a fake 0%."""
if value is None:
return "n/a"
return _fmt_unit_pct(value)
def _quality_of(col: dict) -> dict:
"""Return ``{score, completeness, validity, consistency, issues}`` for a column.
"""Return the quality dict for a column.
Uses the registry ``column_quality_score`` when available; otherwise falls
back to the per-column ``quality_score`` already in the profile (number only,
empty breakdown/issues). Never raises.
empty breakdown/issues/observations). Never raises.
"""
if not isinstance(col, dict):
col = {}
@@ -98,26 +129,25 @@ def _quality_of(col: dict) -> dict:
return res
except Exception: # noqa: BLE001 - degrade instead of aborting.
pass
# Fallback: only the final score is available pre-computed in the profile.
return {
"score": col.get("quality_score"),
"completeness": None,
"validity": None,
"consistency": None,
"issues": [],
"observations": [],
}
def _join_issues(issues) -> str:
"""Join Spanish issue strings into one cell, truncating overly long lists.
def _join_cells(items) -> str:
"""Join Spanish strings into one cell, truncating overly long lists.
The renderer wraps cell text, but a column with many long issues could make a
single row taller than a whole page; cap the length and append ``(+N más)``
so the count of hidden issues is honest rather than silently lost.
The renderer wraps cell text, but a column with many long entries could make
a single row taller than a whole page; cap the length and append ``(+N más)``
so the count of hidden entries is honest rather than silently lost.
"""
if not isinstance(issues, (list, tuple)) or not issues:
if not isinstance(items, (list, tuple)) or not items:
return ""
parts = [model._safe_str(i).strip() for i in issues]
parts = [model._safe_str(i).strip() for i in items]
parts = [p for p in parts if p]
if not parts:
return ""
@@ -142,6 +172,33 @@ def _columns_with_quality(profile: dict):
yield c, _quality_of(c)
def _fmt_unit_pct_or_pct(value) -> str:
"""Format a value that may be a 0-1 fraction or an already-0-100 percentage."""
try:
num = float(value)
except (TypeError, ValueError):
return model._safe_str(value)
if num != num: # NaN
return ""
pct = num * 100 if num <= 1.0 else num
text = f"{pct:.1f}".rstrip("0").rstrip(".")
return f"{text}%"
def _row_uniqueness(profile: dict):
"""Return row uniqueness (1 - duplicate_pct) in [0,1], or None if unknown."""
dup = profile.get("duplicate_pct")
if dup is None:
return None
try:
d = float(dup)
except (TypeError, ValueError):
return None
if d > 1.0: # tolerate a 0-100 scale
d = d / 100.0
return max(0.0, min(1.0, 1.0 - d))
def _summary_block(profile: dict, evaluated: list):
"""Table-level KVTable: global score and quality aggregates."""
rows = []
@@ -153,14 +210,15 @@ def _summary_block(profile: dict, evaluated: list):
if isinstance(q.get("completeness"), (int, float))]
vals = [q.get("validity") for _, q in evaluated
if isinstance(q.get("validity"), (int, float))]
cons = [q.get("consistency") for _, q in evaluated
if isinstance(q.get("consistency"), (int, float))]
if comps:
rows.append(("Completitud media", _fmt_unit_pct(sum(comps) / len(comps))))
if vals:
rows.append(("Validez media", _fmt_unit_pct(sum(vals) / len(vals))))
if cons:
rows.append(("Consistencia media", _fmt_unit_pct(sum(cons) / len(cons))))
rows.append(("Validez media (donde aplica)",
_fmt_unit_pct(sum(vals) / len(vals))))
ru = _row_uniqueness(profile)
if ru is not None:
rows.append(("Unicidad de registro", _fmt_unit_pct(ru)))
n_problem = sum(1 for _, q in evaluated if q.get("issues"))
rows.append(("Columnas con problemas", str(n_problem)))
@@ -182,22 +240,9 @@ def _summary_block(profile: dict, evaluated: list):
return model.KVTable(rows=rows, title="Resumen de calidad")
def _fmt_unit_pct_or_pct(value) -> str:
"""Format a value that may be a 0-1 fraction or an already-0-100 percentage."""
try:
num = float(value)
except (TypeError, ValueError):
return model._safe_str(value)
if num != num: # NaN
return ""
pct = num * 100 if num <= 1.0 else num
text = f"{pct:.1f}".rstrip("0").rstrip(".")
return f"{text}%"
def _scores_block(evaluated: list):
"""DataTable with per-column score and its three-criteria breakdown."""
header = ["Columna", "Calidad", "Completitud", "Validez", "Consistencia"]
"""DataTable with per-column score and its completeness/validity breakdown."""
header = ["Columna", "Calidad", "Completitud", "Validez"]
rows = []
# Worst columns first so the reader sees the problems at the top.
ordered = sorted(
@@ -210,22 +255,22 @@ def _scores_block(evaluated: list):
col.get("name") or "(col)",
_fmt_score(q.get("score")),
_fmt_unit_pct(q.get("completeness")),
_fmt_unit_pct(q.get("validity")),
_fmt_unit_pct(q.get("consistency")),
_fmt_validity(q.get("validity")),
])
if not rows:
return None
return model.DataTable(header=header, rows=rows,
title="Scores de calidad por columna",
note="0 = peor, 100 = mejor; ordenado de peor a mejor")
note="0 = peor, 100 = mejor; «n/a» = dimensión no "
"medible; ordenado de peor a mejor")
def _issues_block(evaluated: list):
"""DataTable listing Spanish issues per column, or a Note when there are none."""
header = ["Columna", "Problemas detectados (español)"]
"""DataTable listing ONLY real quality defects per column, or a Note."""
header = ["Columna", "Problemas de calidad (español)"]
rows = []
for col, q in evaluated:
joined = _join_issues(q.get("issues"))
joined = _join_cells(q.get("issues"))
if joined:
rows.append([col.get("name") or "(col)", joined])
if not rows:
@@ -235,6 +280,55 @@ def _issues_block(evaluated: list):
title="Problemas de calidad por columna")
def _observations_block(evaluated: list):
"""DataTable listing analytical observations per column, or None.
Observations (outliers, constant columns, ids, strong skew) are NOT quality
defects: they do not affect the score. Returned as a separate table from the
issues so the report never presents a legitimate outlier as a problem.
"""
header = ["Columna", "Observaciones analíticas"]
rows = []
for col, q in evaluated:
joined = _join_cells(q.get("observations"))
if joined:
rows.append([col.get("name") or "(col)", joined])
if not rows:
return None
return model.DataTable(
header=header, rows=rows,
title="Observaciones analíticas por columna",
note="No son defectos de calidad y NO afectan al score; orientan el "
"análisis (atípicos, columnas constantes, identificadores).")
def _term(key: str, label: str, mark: bool) -> str:
"""Render a term as a clickable glossary span when marking is enabled."""
if mark:
return f"[[term:{key}]]**{label}**[[/term]]"
return f"**{label}**"
def _criteria_intro(mark: bool) -> str:
"""Intro: how the score is composed, with every term marked clickable.
Concise on purpose: the definitions of each term (calidad de datos,
completitud, validez, unicidad de registro) now live in the GLOSARIO
chapter, so the body no longer repeats them — it only states how the score
is composed and keeps each term marked so it stays a clickable jump.
"""
calidad = _term("calidad_datos", "calidad de datos", mark)
completitud = _term("completitud", "completitud", mark)
validez = _term("validez", "validez", mark)
unicidad = _term("unicidad_registro", "unicidad de registro", mark)
return (
f"La {calidad} de cada columna es un score de 0 a 100 que combina "
f"{completitud} (peso 60%) y {validez} (peso 40%, cuando es medible); "
f"a nivel de tabla se añade la {unicidad}. Los valores atípicos no "
"bajan el score: se listan aparte como **observaciones analíticas**."
)
def build_calidad(profile: dict, ctx: dict):
"""Build the data-quality Chapter, or None if the profile has no columns.
@@ -250,17 +344,35 @@ def build_calidad(profile: dict, ctx: dict):
if not evaluated:
return None # no columns to score -> chapter does not apply.
# Register the criteria terms in the shared glossary (if present) and mark
# their first appearance clickable. Contract §11.1.
glossary = ctx.get("glossary")
mark = False
if isinstance(glossary, model.GlossaryCollector):
for key, (label, definition) in _TERMS.items():
glossary.add(key, label, definition)
mark = True
blocks = [
model.Heading(text="Cómo se calcula la calidad", level=2),
model.Markdown(text=_CRITERIA_INTRO),
model.Markdown(text=_criteria_intro(mark)),
_summary_block(profile, evaluated),
model.Heading(text="Scores por columna", level=2),
]
scores = _scores_block(evaluated)
if scores is not None:
blocks.append(scores)
blocks.append(model.Heading(text="Problemas detectados", level=2))
blocks.append(model.Heading(text="Problemas de calidad", level=2))
blocks.append(_issues_block(evaluated))
observations = _observations_block(evaluated)
if observations is not None:
blocks.append(model.Heading(text="Observaciones analíticas", level=2))
blocks.append(model.Note(
"Las observaciones siguientes NO son defectos de calidad y no "
"afectan al score: son señales para orientar el análisis."))
blocks.append(observations)
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -1,11 +1,12 @@
"""Tests for the CALIDAD chapter — DoD: golden + edges + anti-cut.
"""Tests for the CALIDAD chapter — DoD: golden + edges + anti-cut + glossary.
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
and deterministic. Verifies that the chapter explains the quality criteria, shows
per-column scores with the completeness/validity/consistency breakdown, lists the
issues in Spanish (separate from the type flags), returns None when it does not
apply, and that a wide profile with long names renders to PDF and PPTX without
cutting any cell text (long content wraps, it is never truncated).
and deterministic. Verifies the report-2046 quality model: the chapter explains
the two scored dimensions (completitud 60% / validez 40%), shows per-column
scores without a consistency column, keeps quality DEFECTS (issues) separate
from analytical OBSERVATIONS (outliers, constant, ids), hooks the criteria terms
into the glossary, returns None when it does not apply, and renders a wide
profile to PDF and PPTX without cutting any cell text.
"""
import os
@@ -20,28 +21,30 @@ from datascience.automatic_eda.chapters.calidad import (
CHAPTER_VERSION,
)
from datascience.automatic_eda import build_document, render_pdf, render_pptx
from datascience.automatic_eda import model
def _profile() -> dict:
"""A small profile with one column per quality problem (nulls, outliers,
constant, high-cardinality id) plus one clean column."""
constant, high-cardinality id) plus one clean column. ``outlier_pct`` is in
the 0-100 scale that describe_numeric actually emits."""
return {
"table": "demo",
"quality_score": 72.5,
"quality_score": 82.0,
"duplicate_pct": 0.04,
"null_cell_pct": 0.11,
"constant_cols": ["flag_const"],
"all_null_cols": [],
"columns": [
{"name": "edad", "inferred_type": "integer", "null_pct": 0.2,
"numeric": {"outlier_pct": 0.15, "min": 0, "max": 99},
"quality_score": 60},
{"name": "edad", "inferred_type": "numeric", "null_pct": 0.2,
"n_rows": 100, "unique_pct": 0.5,
"numeric": {"outlier_pct": 15.0, "min": 0, "max": 99}},
{"name": "nombre", "inferred_type": "text", "null_pct": 0.0,
"unique_pct": 0.98, "quality_score": 80},
"unique_pct": 0.98, "flags": ["possible_id"]},
{"name": "flag_const", "inferred_type": "text", "null_pct": 0.0,
"flags": ["constant"], "quality_score": 50},
{"name": "limpia", "inferred_type": "float", "null_pct": 0.0,
"numeric": {"outlier_pct": 0.0}, "quality_score": 100},
"unique_pct": 0.01, "flags": ["constant"]},
{"name": "limpia", "inferred_type": "numeric", "null_pct": 0.0,
"unique_pct": 0.5, "numeric": {"outlier_pct": 0.0}},
],
}
@@ -50,16 +53,9 @@ def _tables(chapter):
return [b for b in chapter.blocks if getattr(b, "kind", None) == "data_table"]
def _scores_table(chapter):
def _table_by_title(chapter, needle):
for t in _tables(chapter):
if "Scores" in (t.title or ""):
return t
return None
def _issues_table(chapter):
for t in _tables(chapter):
if "Problemas" in (t.title or ""):
if needle in (t.title or ""):
return t
return None
@@ -73,41 +69,86 @@ def test_golden_chapter_estructura_y_version():
assert ch.id == "calidad"
assert ch.version == CHAPTER_VERSION
kinds = [b.kind for b in ch.blocks]
# intro heading + markdown criteria + summary kv + scores table + issues table
assert "markdown" in kinds and "kv_table" in kinds and "data_table" in kinds
def test_golden_intro_explica_criterios_y_pesos():
def test_golden_intro_nombra_dos_dimensiones_y_pesos():
# La intro nombra las dos dimensiones, sus pesos y la unicidad, pero ya NO
# repite sus definiciones largas: estas viven ahora en el capítulo GLOSARIO.
ch = build_calidad(_profile(), {})
intro = [b for b in ch.blocks if b.kind == "markdown"][0].text
for needle in ("Completitud", "Validez", "Consistencia",
"50%", "30%", "20%"):
for needle in ("completitud", "validez", "60%", "40%",
"unicidad de registro"):
assert needle in intro, f"falta {needle!r} en la intro de criterios"
# El principio: los outliers NO bajan la calidad.
assert "atípicos" in intro and "no bajan" in intro
# Ya no se menciona la dimensión consistencia eliminada.
assert "20%" not in intro
def test_golden_scores_incluyen_desglose_por_criterio():
def test_golden_scores_sin_columna_consistencia():
ch = build_calidad(_profile(), {})
scores = _scores_table(ch)
scores = _table_by_title(ch, "Scores")
assert scores is not None
assert scores.header == ["Columna", "Calidad", "Completitud",
"Validez", "Consistencia"]
# 4 columns scored, none dropped.
assert scores.header == ["Columna", "Calidad", "Completitud", "Validez"]
assert "Consistencia" not in scores.header
assert len(scores.rows) == 4
names = {r[0] for r in scores.rows}
assert names == {"edad", "nombre", "flag_const", "limpia"}
def test_golden_issues_en_espanol_separados_de_flags():
def test_golden_outliers_en_observaciones_no_en_problemas():
ch = build_calidad(_profile(), {})
issues = _issues_table(ch)
assert issues is not None
flat = " | ".join(" ".join(r) for r in issues.rows)
assert "nulos" in flat # completeness issue (ES)
assert "outliers" in flat # validity issue (ES)
assert "columna constante" in flat
assert "posible id de alta cardinalidad" in flat
# The raw type flag string must NOT leak as a "problem".
assert "constant" not in flat or "columna constante" in flat
problemas = _table_by_title(ch, "Problemas de calidad")
observaciones = _table_by_title(ch, "Observaciones")
assert problemas is not None
assert observaciones is not None
problemas_txt = " | ".join(" ".join(r) for r in problemas.rows)
observaciones_txt = " | ".join(" ".join(r) for r in observaciones.rows)
# Los nulos SÍ son problema de calidad.
assert "nulos" in problemas_txt
# Los outliers NO aparecen como problema...
assert "atípic" not in problemas_txt and "outlier" not in problemas_txt
# ...sino como observación analítica.
assert "atípic" in observaciones_txt
# Constante e id: observaciones, no problemas.
assert "constante" in observaciones_txt
assert "identificador" in observaciones_txt
assert "constante" not in problemas_txt
def test_golden_score_columna_limpia_es_100():
"""Columna sin nulos, numérica nativa: score 100 aunque tenga (o no) outliers."""
ch = build_calidad(_profile(), {})
scores = _table_by_title(ch, "Scores")
by_name = {r[0]: r for r in scores.rows}
assert by_name["limpia"][1] == "100 / 100"
# edad: 20% nulos -> 100*(0.6*0.8 + 0.4*1.0) = 88; los outliers no bajan nada.
assert by_name["edad"][1] == "88 / 100"
# --------------------------------------------------------------------------- #
# Glosario (contrato §11.1)
# --------------------------------------------------------------------------- #
def test_glosario_registra_los_cuatro_terminos_y_marca_clicable():
glossary = model.GlossaryCollector()
ch = build_calidad(_profile(), {"glossary": glossary})
for key in ("calidad_datos", "completitud", "validez", "unicidad_registro"):
assert glossary.has(key), f"término {key!r} no registrado en el glosario"
intro = [b for b in ch.blocks if b.kind == "markdown"][0].text
# Con colector presente, la primera aparición se marca clicable.
assert "[[term:completitud]]" in intro
assert "[[term:validez]]" in intro
assert "[[term:calidad_datos]]" in intro
assert "[[term:unicidad_registro]]" in intro
def test_sin_glosario_no_marca_terminos():
ch = build_calidad(_profile(), {}) # ctx sin glossary
intro = [b for b in ch.blocks if b.kind == "markdown"][0].text
assert "[[term:" not in intro
# --------------------------------------------------------------------------- #
@@ -124,17 +165,17 @@ def test_edge_perfil_limpio_sin_problemas_usa_nota():
prof = {
"quality_score": 100,
"columns": [
{"name": "a", "inferred_type": "float", "null_pct": 0.0,
"numeric": {"outlier_pct": 0.0}},
{"name": "b", "inferred_type": "float", "null_pct": 0.0,
"numeric": {"outlier_pct": 0.0}},
{"name": "a", "inferred_type": "numeric", "null_pct": 0.0,
"unique_pct": 0.5, "numeric": {"outlier_pct": 0.0}},
{"name": "b", "inferred_type": "numeric", "null_pct": 0.0,
"unique_pct": 0.5, "numeric": {"outlier_pct": 0.0}},
],
}
ch = build_calidad(prof, {})
assert ch is not None
assert _issues_table(ch) is None # no issues table
assert _table_by_title(ch, "Problemas de calidad") is None # no issues table
notes = [b for b in ch.blocks if b.kind == "note"]
assert notes and "No se detectaron problemas" in notes[0].text
assert any("No se detectaron problemas" in n.text for n in notes)
# --------------------------------------------------------------------------- #
@@ -143,44 +184,42 @@ def test_edge_perfil_limpio_sin_problemas_usa_nota():
def _wide_profile(ncols: int = 22) -> dict:
cols = [
{"name": "identificador_unico_de_transaccion_con_nombre_muy_largo",
"inferred_type": "text", "null_pct": 0.0, "unique_pct": 0.99},
"inferred_type": "text", "null_pct": 0.0, "unique_pct": 0.99,
"flags": ["possible_id"]},
{"name": "columna_constante_sin_ninguna_variacion_de_valor",
"inferred_type": "text", "null_pct": 0.0, "flags": ["constant"]},
"inferred_type": "text", "null_pct": 0.0, "unique_pct": 0.01,
"flags": ["constant"]},
]
for k in range(ncols - 2):
cols.append({
"name": f"metrica_numerica_de_negocio_{k:02d}_con_nombre_largo",
"inferred_type": "float", "null_pct": 0.1 + (k % 3) * 0.05,
"numeric": {"outlier_pct": 0.08, "min": 0, "max": 1000},
"inferred_type": "numeric", "null_pct": 0.1 + (k % 3) * 0.05,
"unique_pct": 0.5,
"numeric": {"outlier_pct": 8.0, "min": 0, "max": 1000},
})
return {"table": "ancha", "quality_score": 70.0, "columns": cols}
return {"table": "ancha", "quality_score": 70.0, "duplicate_pct": 0.0,
"columns": cols}
def test_anticut_pdf_y_pptx_no_truncan_nombres_largos():
prof = _wide_profile(22)
full = build_document(prof, {"dataset_name": "ancha"})
assert any(c.id == "calidad" for c in full)
# Render ONLY the calidad chapter so the anti-cut assertions are scoped to
# this chapter (other chapters, e.g. portada, legitimately contain '…').
chapters = [c for c in full if c.id == "calidad"]
long_name = "metrica_numerica_de_negocio_00_con_nombre_largo"
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "q.pdf")
pptx = os.path.join(d, "q.pptx")
rp = render_pdf(chapters, pdf, {"title": "EDA"})
rx = render_pptx(chapters, pptx, {"title": "EDA"})
render_pptx(chapters, pptx, {"title": "EDA"})
assert os.path.exists(pdf) and os.path.exists(pptx)
# The wide table forces pagination across several pages/slides.
assert (rp or {}).get("n_pages", 0) >= 2
# PDF: the long name survives whole once wraps (spaces/newlines) removed,
# and there is no truncation marker.
pdf_txt = "".join((pg.extract_text() or "") for pg in PdfReader(pdf).pages)
assert "" not in pdf_txt and "..." not in pdf_txt
norm = re.sub(r"\s+", "", pdf_txt)
assert long_name in norm, "el nombre largo se cortó en el PDF"
# PPTX: long name present in some cell, untruncated.
allt = []
for s in Presentation(pptx).slides:
for sh in s.shapes:
@@ -0,0 +1,459 @@
"""Categorical distributions chapter (CAT DISTR).
Third reference chapter for AutomaticEDA. Each categorical column gets **its own
page (PDF) / slide (PPTX)**: every column is wrapped in a keep-together
``model.Group`` with ``page_break_before=True`` (except the first, which may share
the intro's page), so its chart sits next to its tables and no column is split.
A short intro names the clickable **[[term:entropia]]entropía[[/term]]** term —
the full definition lives in the GLOSARIO chapter, so it is NOT repeated inline
here (one click jumps to the glossary entry). The intro also carries the dataset
row total used as a comparison baseline.
Per column the Group contains, in order:
1. A cardinality key/value table: distinct values, ``% distinct`` (distinct /
total rows), total dataset rows, singleton values (frequency 1), entropy with
its theoretical maximum and the normalized ratio, mode, imbalance and
string-length stats.
2. A short note flagging problematic cardinality (id-like ≈100% distinct, or a
single dominating category).
3. A ``top-k`` table (value / count / %).
4. A **donut pie chart** of the most common categories (top-k + an "Otros"
bucket), drawn lazily so the renderers scale it to fit entirely.
Data comes from the ``eda`` group: each ``columns[i]['categorical']`` is the
output of ``summarize_categorical`` (``top[{value,count,pct}]``, ``mode``,
``n_distinct``, ``entropy``, ``imbalance``, ``len_min/mean/max``). The derived
cardinality metrics and the pie figure are delegated to two registry functions
(``categorical_cardinality_block`` and ``categorical_top_pie_figure``); both are
imported lazily and degrade to a minimal inline fallback so this chapter never
raises even if they are unavailable.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
import math
from .. import model
CHAPTER_VERSION = "1.2.0"
CHAPTER_ID = "cat_distr"
CHAPTER_TITLE = "Distribuciones categóricas"
# Glossary term this chapter explains. Registered in the shared collector and
# marked clickable on its first appearance (end-to-end glossary example —
# mejora 6). Other chapters hook their own terms the same way (see the contract).
_TERM_ENTROPIA_KEY = "entropia"
_TERM_ENTROPIA_LABEL = "Entropía (de Shannon)"
_TERM_ENTROPIA_DEF = (
"Medida, en bits, de cómo de repartidos están los valores de una columna "
"categórica. Vale 0 cuando una sola categoría concentra todas las filas "
"(máxima previsibilidad) y alcanza su máximo, log2(k) para k categorías "
"distintas, cuando todas aparecen por igual (máxima diversidad). La entropía "
"normalizada (entropía dividida por su máximo) la lleva al rango 01 para "
"comparar columnas con distinto número de categorías.")
# Cap the number of categorical columns rendered to keep the document bounded;
# the rest are summarized in a closing note (no silent truncation).
MAX_COLS = 40
# Rows shown in each top-k table and explicit slices in the pie. Kept moderate so
# the whole column — cardinality table + top-k table + donut — fits on ONE
# page/slide with the chart next to its tables; the table note still reports
# "top N of M" so nothing is silently hidden. For id-like columns (≈100%
# distinct) the top-k table is dropped entirely (it would be a list of unique
# values — pure noise), which also frees the room the donut needs (see build).
TOP_TABLE_ROWS = 8
PIE_TOP_K = 6
# Truncate very long category labels in tables (the renderer also wraps). Kept
# tight so a column with long id-like values (names, tickets) still fits its page.
LABEL_MAX = 28
def _fmt_int(value) -> str:
if value is None:
return ""
try:
return f"{int(value):,}".replace(",", ".")
except (TypeError, ValueError):
return str(value)
def _fmt_num(value, decimals: int = 3) -> str:
if value is None:
return ""
if isinstance(value, bool):
return str(value)
if isinstance(value, int):
return f"{value:,}".replace(",", ".")
if isinstance(value, float):
if value != value: # NaN
return "NaN"
if value in (float("inf"), float("-inf")):
return str(value)
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
return text if text else "0"
return str(value)
def _fmt_pct_value(value, decimals: int = 1) -> str:
"""Format an already-in-percent value (0100). None -> placeholder."""
if value is None:
return ""
try:
return f"{float(value):.{decimals}f}%"
except (TypeError, ValueError):
return str(value)
def _pct_from_maybe_fraction(value, decimals: int = 1) -> str:
"""Format a percentage that may arrive as a 01 fraction or a 0100 number."""
if value is None:
return ""
try:
v = float(value)
except (TypeError, ValueError):
return str(value)
if v <= 1.0:
v *= 100.0
return f"{v:.{decimals}f}%"
def _truncate(text: str, limit: int = LABEL_MAX) -> str:
s = model._safe_str(text)
if len(s) <= limit:
return s
return s[: max(1, limit - 1)].rstrip() + ""
def _is_categorical(col: dict) -> bool:
"""A column is treated as categorical when it carries a non-empty top list
and is not a pure numeric column (numeric columns may still expose a top)."""
if not isinstance(col, dict):
return False
cat = col.get("categorical")
if not (isinstance(cat, dict) and cat.get("top")):
return False
if col.get("inferred_type") == "numeric":
return False
return True
def _cardinality(cat: dict, n_rows) -> dict:
"""Derive cardinality metrics for a column, via the registry function when
available, otherwise a minimal inline fallback. Never raises."""
try:
from datascience.categorical_cardinality_block import (
categorical_cardinality_block,
)
out = categorical_cardinality_block(cat=cat, n_rows=n_rows)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001 — fall back to the inline derivation.
pass
return _fallback_cardinality(cat, n_rows)
def _fallback_cardinality(cat: dict, n_rows) -> dict:
cat = cat or {}
top = cat.get("top") or []
n_distinct = cat.get("n_distinct")
entropy = cat.get("entropy")
try:
nr = int(n_rows) if n_rows is not None else None
except (TypeError, ValueError):
nr = None
pct_distinct = None
if isinstance(n_distinct, (int, float)) and nr:
pct_distinct = float(n_distinct) / nr * 100.0
entropy_max = None
if isinstance(n_distinct, (int, float)):
entropy_max = math.log2(n_distinct) if n_distinct > 1 else 0.0
entropy_norm = None
if isinstance(entropy, (int, float)) and entropy_max:
entropy_norm = max(0.0, min(1.0, float(entropy) / entropy_max))
mode_pct = cat.get("mode_pct")
if mode_pct is None and top and isinstance(top[0], dict):
mode_pct = top[0].get("pct")
# Normalize to a 0100 scale: summarize_categorical emits a 01 fraction.
if isinstance(mode_pct, (int, float)) and not isinstance(mode_pct, bool):
mode_pct = float(mode_pct) * 100.0 if mode_pct <= 1.0 else float(mode_pct)
else:
mode_pct = None
n_singletons = None
if top:
n_singletons = sum(
1 for t in top if isinstance(t, dict) and t.get("count") == 1)
return {
"n_distinct": n_distinct,
"n_rows": nr,
"pct_distinct": pct_distinct,
"entropy": entropy,
"entropy_max": entropy_max,
"entropy_norm": entropy_norm,
"mode": cat.get("mode"),
"mode_pct": mode_pct,
"imbalance": cat.get("imbalance"),
"n_singletons": n_singletons,
"n_singletons_partial": (
isinstance(n_distinct, (int, float)) and n_distinct > len(top)),
"len_min": cat.get("len_min"),
"len_mean": cat.get("len_mean"),
"len_max": cat.get("len_max"),
"id_like": pct_distinct is not None and pct_distinct >= 99.0,
"dominated": mode_pct is not None and mode_pct >= 90.0,
}
def _pie_make(top, n_distinct, title, n_rows):
"""Return a zero-arg callable that builds the donut figure lazily."""
def make():
try:
from datascience.categorical_top_pie_figure import (
categorical_top_pie_figure,
)
return categorical_top_pie_figure(
top=top, n_distinct=n_distinct or 0, title=title,
top_k=PIE_TOP_K, n_rows=n_rows)
except Exception: # noqa: BLE001 — minimal local fallback figure.
return _fallback_pie(top, title)
return make
def _fallback_pie(top, title):
"""Minimal donut figure used only if the registry function is unavailable."""
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure
fig = Figure(figsize=(5.0, 3.2))
ax = fig.add_subplot(111)
items = [t for t in (top or [])
if isinstance(t, dict) and isinstance(t.get("count"), (int, float))]
items = sorted(items, key=lambda t: t.get("count") or 0, reverse=True)
head = items[:PIE_TOP_K]
rest = items[PIE_TOP_K:]
labels = [_truncate(t.get("value"), 20) for t in head]
sizes = [float(t.get("count") or 0) for t in head]
if rest:
labels.append(f"Otros ({len(rest)})")
sizes.append(sum(float(t.get("count") or 0) for t in rest))
if not sizes or sum(sizes) <= 0:
ax.text(0.5, 0.5, "sin datos categóricos", ha="center", va="center")
ax.axis("off")
return fig
ax.pie(sizes, labels=None, wedgeprops={"width": 0.42},
autopct=lambda p: f"{p:.0f}%" if p >= 4 else "")
ax.legend(labels, loc="center left", bbox_to_anchor=(1.0, 0.5),
fontsize=7, frameon=False)
ax.set_title(_truncate(title, 40))
fig.tight_layout()
return fig
def _normalize_card(card: dict) -> dict:
"""Make the cardinality dict robust regardless of the upstream scale.
``summarize_categorical`` emits ``mode_pct`` as a 01 fraction; bring it to a
0100 scale and recompute the ``dominated`` flag here so the chapter is
correct whether it consumed the registry function or the inline fallback.
"""
card = dict(card or {})
mp = card.get("mode_pct")
if isinstance(mp, (int, float)) and not isinstance(mp, bool):
mp = float(mp) * 100.0 if mp <= 1.0 else float(mp)
else:
mp = None
card["mode_pct"] = mp
card["dominated"] = mp is not None and mp >= 90.0
pd = card.get("pct_distinct")
card["id_like"] = isinstance(pd, (int, float)) and pd >= 99.0
return card
def _cardinality_block(card: dict):
"""KVTable with the cardinality / entropy metrics for one column.
Related metrics are grouped onto a single row each (distinct/%/unique;
entropy bits/max/normalized; length min/mean/max) so the whole column —
table + chart — fits one page/slide without dropping any datum; the short
16:9 PPTX slide does not fit one metric per row plus a chart otherwise."""
n_singletons = card.get("n_singletons")
if n_singletons is not None and card.get("n_singletons_partial"):
singletons = f"{_fmt_int(n_singletons)}"
elif n_singletons is not None:
singletons = _fmt_int(n_singletons)
else:
singletons = ""
# Distinct count · % distinct · unique (frequency 1) on one row.
distinct_combo = (f"{_fmt_int(card.get('n_distinct'))} · "
f"{_fmt_pct_value(card.get('pct_distinct'))} · "
f"{singletons} únicos")
# Entropy bits · theoretical max · normalized 01 on one row.
entropy_combo = (f"{_fmt_num(card.get('entropy'))} bits · "
f"máx {_fmt_num(card.get('entropy_max'))} · "
f"norm {_fmt_num(card.get('entropy_norm'))}")
mode = card.get("mode")
mode_pct = card.get("mode_pct")
mode_str = "" if mode is None else _truncate(mode, 32)
if mode is not None and mode_pct is not None:
mode_str = f"{mode_str} ({_fmt_pct_value(mode_pct)})"
rows = [
("Distintos · % · únicos", distinct_combo),
("Total filas (dataset)", _fmt_int(card.get("n_rows"))),
("Entropía (bits · máx · norm)", entropy_combo),
("Moda", mode_str),
]
imbalance = card.get("imbalance")
lm = card.get("len_min")
lmean = card.get("len_mean")
lmax = card.get("len_max")
# Imbalance and string length (both secondary) share one closing row.
extras = []
if imbalance is not None:
extras.append(f"desbalance {_fmt_num(imbalance)}")
if any(v is not None for v in (lm, lmean, lmax)):
extras.append(
f"long. {_fmt_num(lm)}/{_fmt_num(lmean)}/{_fmt_num(lmax)}")
if extras:
rows.append(("Desbalance · longitud", " · ".join(extras)))
return model.KVTable(rows=rows, title="Cardinalidad")
def _flag_note(card: dict):
"""Return a Note flagging problematic cardinality, or None."""
if card.get("id_like"):
return model.Note(
"Casi todos los valores son distintos (≈100% distintos): la columna "
"se comporta como un identificador y aporta poco para agrupar o "
"comparar categorías. No se lista el top de categorías (serían "
"valores casi todos únicos).")
if card.get("dominated"):
mp = card.get("mode_pct")
mp_str = _fmt_pct_value(mp) if mp is not None else "muy alta"
return model.Note(
f"Una sola categoría domina la columna (moda {mp_str}): la "
"distribución está muy desbalanceada.")
return None
def _topk_table(cat: dict):
"""DataTable value / count / % for the top categories."""
top = cat.get("top") or []
n_distinct = cat.get("n_distinct")
header = ["Valor", "Conteo", "%"]
rows = []
for t in top[:TOP_TABLE_ROWS]:
if not isinstance(t, dict):
continue
rows.append([
_truncate(t.get("value")),
_fmt_int(t.get("count")),
_pct_from_maybe_fraction(t.get("pct")),
])
if not rows:
return None
shown = len(rows)
if isinstance(n_distinct, (int, float)) and n_distinct > shown:
note = f"top {shown} de {_fmt_int(n_distinct)} categorías distintas"
else:
note = f"{shown} categorías"
return model.DataTable(header=header, rows=rows, title="Top categorías",
note=note)
def _intro_blocks(n_rows, mark_term: bool = False):
total = _fmt_int(n_rows)
# Mark the first appearance of the term as a clickable glossary jump when the
# term was registered (mark_term). The full definition of entropy lives in the
# GLOSARIO chapter, so the intro only names the clickable term here instead of
# repeating the long explanation (avoids the redundancy with the glossary).
entropia = ("[[term:entropia]]entropía[[/term]]" if mark_term
else "entropía")
text = (
f"Cada columna categórica ocupa su propia página: sus métricas de "
f"cardinalidad —incluida la {entropia}—, una nota que señala cardinalidad "
"problemática, la tabla de las categorías más frecuentes y un gráfico de "
"tarta (donut) de las más comunes, todo junto."
)
if n_rows is not None:
text += f" El dataset tiene {total} filas en total como referencia."
return [
model.Heading(text="Entropía y cardinalidad", level=2),
model.Markdown(text=text),
]
def build_cat_distr(profile: dict, ctx: dict):
"""Build the categorical-distributions Chapter, or None if the dataset has
no categorical columns."""
profile = profile or {}
ctx = ctx or {}
cols = profile.get("columns") or []
cat_cols = [c for c in cols if _is_categorical(c)]
if not cat_cols:
return None
n_rows = profile.get("n_rows")
# Register "entropía" in the shared glossary collector (if present) and mark
# its first appearance clickable. End-to-end glossary example (mejora 6).
glossary = ctx.get("glossary")
mark_term = False
if isinstance(glossary, model.GlossaryCollector):
glossary.add(_TERM_ENTROPIA_KEY, _TERM_ENTROPIA_LABEL,
_TERM_ENTROPIA_DEF)
mark_term = True
blocks = list(_intro_blocks(n_rows, mark_term=mark_term))
rendered = cat_cols[:MAX_COLS]
for idx, col in enumerate(rendered):
name = col.get("name") or "(columna)"
cat = col.get("categorical") or {}
card = _normalize_card(_cardinality(cat, n_rows))
# One Group per categorical column: heading + cardinality table + flag
# note + top-k table + donut figure are kept together and the renderer
# starts each on a fresh page/slide (page_break_before) so every column
# gets its own page with its chart next to its tables. The first column
# may share the intro's page (no forced break) to avoid a near-empty page.
col_blocks = [
model.Heading(text=str(name), level=2),
_cardinality_block(card),
]
note = _flag_note(card)
if note is not None:
col_blocks.append(note)
# For id-like columns (≈100% distinct) the top-k is a list of unique
# values — pure noise; skip it (the flag note already explains why) and
# let the donut take that room so the whole column fits one page/slide.
if not card.get("id_like"):
topk = _topk_table(cat)
if topk is not None:
col_blocks.append(topk)
col_blocks.append(model.Figure(
make=_pie_make(cat.get("top") or [], card.get("n_distinct"),
str(name), n_rows),
caption=(f"Categorías más comunes de «{_truncate(name, 32)}» "
"(donut: top-k + «Otros»)")))
blocks.append(model.Group(blocks=col_blocks,
page_break_before=(idx > 0)))
if len(cat_cols) > len(rendered):
omitted = len(cat_cols) - len(rendered)
blocks.append(model.Note(
f"Se muestran las primeras {len(rendered)} columnas categóricas; "
f"quedan {omitted} sin mostrar para mantener acotado el informe."))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,349 @@
"""Tests for the CAT DISTR chapter — DoD: golden + edges + anti-cut.
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
and deterministic. Verifies that ``build_cat_distr`` emits the blocks the user
asked for (distinct/total/%-distinct/unique metrics, top-k table and a donut
figure), that EACH categorical column is wrapped in its own keep-together
``Group`` that starts on a fresh page/slide (one column per page, chart next to
its tables), that the long entropy explanation is NOT repeated inline (it lives
in the glossary — only the clickable term is kept), that the chapter renders
inside the full document to both PDF and PPTX showing that content, that a
profile with no categorical columns yields ``None`` without raising, and that
long labels / many columns are never cut in either output.
"""
import os
import re
import tempfile
from pypdf import PdfReader
from pptx import Presentation
from datascience.automatic_eda.model import (
DataTable, Figure, GlossaryCollector, Group, Heading, KVTable, Markdown,
Note,
)
from datascience.automatic_eda.chapters.cat_distr import (
CHAPTER_ID, CHAPTER_VERSION, build_cat_distr,
)
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
def _profile() -> dict:
return {
"table": "productos",
"source": "/data/productos.csv",
"profiled_at": "2026-06-30T10:00:00+00:00",
"n_rows": 1000,
"n_cols": 3,
"quality_score": 90.0,
"columns": [
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0,
"numeric": {"mean": 42.5, "median": 40.0, "min": 1.0,
"max": 100.0, "std": 12.3}},
{"name": "categoria", "inferred_type": "categorical",
"null_pct": 0.0, "null_count": 0, "distinct_count": 8,
"categorical": {
"top": [
{"value": "neumaticos", "count": 500, "pct": 0.5},
{"value": "aceite", "count": 300, "pct": 0.3},
{"value": "filtros", "count": 120, "pct": 0.12},
{"value": "frenos", "count": 80, "pct": 0.08},
],
"mode": "neumaticos", "n_distinct": 8, "entropy": 1.6,
"imbalance": 6.25, "len_min": 6, "len_mean": 7.5,
"len_max": 10}},
{"name": "uuid", "inferred_type": "categorical",
"null_pct": 0.0, "null_count": 0, "distinct_count": 1000,
"categorical": {
"top": [{"value": f"id-{i}", "count": 1} for i in range(5)],
"mode": "id-0", "n_distinct": 1000, "entropy": 9.97,
"imbalance": 1.0}},
],
}
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def _pptx_text(path: str) -> str:
prs = Presentation(path)
parts = []
for sl in prs.slides:
for sh in sl.shapes:
if sh.has_text_frame:
parts.append(sh.text_frame.text)
if sh.has_table:
tb = sh.table
for r in range(len(tb.rows)):
for c in range(len(tb.columns)):
parts.append(tb.cell(r, c).text)
return re.sub(r"\s+", " ", " ".join(parts))
def _flatten(blocks):
"""Expand keep-together Groups so the per-column heading/table/figure are
inspectable as a flat block list (the chapter wraps each column in a Group)."""
out = []
for b in blocks:
if getattr(b, "kind", "") == "group":
out.extend(_flatten(getattr(b, "blocks", []) or []))
else:
out.append(b)
return out
def _column_groups(chapter):
return [b for b in chapter.blocks if isinstance(b, Group)]
def test_golden_build_cat_distr_emite_bloques_pedidos():
ch = build_cat_distr(_profile(), {})
assert ch is not None
assert ch.id == CHAPTER_ID
assert ch.version == CHAPTER_VERSION
# Entropy intro present, but the long explanation is gone (it lives in the
# glossary now): only the term is named, no log2/normalizada walkthrough.
headings = [b.text for b in ch.blocks if isinstance(b, Heading)]
assert any("Entrop" in h for h in headings)
md = next(b for b in ch.blocks if isinstance(b, Markdown))
assert "entropía" in md.text.lower()
assert "log2" not in md.text # redundant explanation removed.
assert "máxima diversidad" not in md.text
# Per-column blocks are wrapped in keep-together Groups: flatten to inspect.
flat = _flatten(ch.blocks)
kv = next(b for b in flat if isinstance(b, KVTable))
labels = [r[0] for r in kv.rows]
values = " ".join(str(r[1]) for r in kv.rows)
# Cardinality metrics: distinct count, %-distinct, unique values and total
# rows are present (grouped onto compact rows so the chart fits the page).
assert "Distintos · % · únicos" in labels
assert "Total filas (dataset)" in labels
assert any("Entropía" in lbl for lbl in labels)
assert "únicos" in values and "%" in values
assert "bits" in values and "norm" in values # entropy + max + normalized.
# Top-k table + pie figure.
dt = next(b for b in flat if isinstance(b, DataTable))
assert dt.header == ["Valor", "Conteo", "%"]
assert any("neumaticos" in str(cell) for row in dt.rows for cell in row)
assert any(isinstance(b, Figure) for b in flat)
# id-like column flagged with a Note that also explains the top-k is dropped.
idnote = next((b for b in flat
if isinstance(b, Note) and "identificador" in b.text), None)
assert idnote is not None
assert "No se lista el top" in idnote.text
def test_golden_idlike_omite_topk_y_conserva_donut():
# The id-like column (uuid, 100% distinct) must NOT carry a top-k DataTable
# (it would be a list of unique values), but must still keep its donut Figure
# and its cardinality table so it stays a full per-column page.
ch = build_cat_distr(_profile(), {})
groups = _column_groups(ch)
uuid_group = next(g for g in groups
if any(getattr(b, "text", "") == "uuid" for b in g.blocks))
kinds = [b.kind for b in uuid_group.blocks]
assert "data_table" not in kinds # top-k of unique values dropped.
assert "kv_table" in kinds # cardinality kept.
assert "figure" in kinds # donut kept (chart per column).
# A non-id-like column keeps its top-k table.
cat_group = next(g for g in groups
if any(getattr(b, "text", "") == "categoria"
for b in g.blocks))
assert "data_table" in [b.kind for b in cat_group.blocks]
def test_golden_una_pagina_por_columna_groups():
ch = build_cat_distr(_profile(), {})
groups = _column_groups(ch)
# Two categorical columns -> two column Groups (numeric column excluded).
assert len(groups) == 2
# Each Group carries one column: a heading + its cardinality table + figure.
for g in groups:
kinds = [b.kind for b in g.blocks]
assert kinds[0] == "heading"
assert "kv_table" in kinds
assert "figure" in kinds
# The first column may share the intro page (no forced break); every later
# column starts on a fresh page/slide so each column gets its own page.
assert groups[0].page_break_before is False
assert all(g.page_break_before is True for g in groups[1:])
def test_golden_entropia_clicable_y_definicion_en_glosario():
# With a glossary collector the intro marks the clickable term and the FULL
# definition (the long explanation removed from the intro) lands in the
# glossary, not inline — no data lost, just relocated.
gc = GlossaryCollector()
ch = build_cat_distr(_profile(), {"glossary": gc})
md = next(b for b in ch.blocks if isinstance(b, Markdown))
assert "[[term:entropia]]entropía[[/term]]" in md.text
assert gc.has("entropia")
entry = gc.get("entropia")
assert entry is not None
# The definition kept in the glossary still carries the detail removed inline.
assert "log2" in entry["definition"]
assert "normalizada" in entry["definition"].lower()
def test_golden_render_pdf_una_pagina_por_columna():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pdf")
res = render_automatic_eda_pdf(_profile(), out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
cat_meta = next(c for c in res["chapters"] if c["id"] == CHAPTER_ID)
# Two categorical columns, each on its own page -> >= 2 pages for the
# chapter (intro shares the first column's page).
assert cat_meta["n_pages"] >= 2
txt = _pdf_text(out)
assert "Entrop" in txt
assert "distintos" in txt
assert "categoria" in txt and "neumaticos" in txt
assert "donut" in txt # figure caption rendered as text.
assert "identificador" in txt # id-like note rendered.
def test_golden_render_pptx_muestra_categoricas():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pptx")
res = render_automatic_eda_pptx(_profile(), out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
cat_meta = next(c for c in res["chapters"] if c["id"] == CHAPTER_ID)
assert cat_meta["n_slides"] >= 2 # one slide per categorical column.
txt = _pptx_text(out)
assert "Entrop" in txt
assert "categoria" in txt and "neumaticos" in txt
assert "distintos" in txt
def _profile_high_card() -> dict:
"""Profile with a high-cardinality NON-id-like categorical column whose top-k
of long values would split from its donut on a short 16:9 slide unless the
renderer trims the table — the exact case the adversarial check flagged
(Ticket / Cabin)."""
long_vals = [f"Valor largo de categoria numero {i:02d} con texto extra"
for i in range(40)]
top = [{"value": v, "count": 60 - i, "pct": (60 - i) / 5000.0}
for i, v in enumerate(long_vals)]
return {
"table": "t", "source": "t.csv", "n_rows": 5000, "n_cols": 3,
"quality_score": 80.0,
"columns": [
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.0,
"numeric": {"mean": 1.0, "median": 1.0, "min": 0.0, "max": 2.0,
"std": 0.5}},
# 40 distinct over 5000 rows = 0.8% distinct -> NOT id-like, keeps
# its (long) top-k table; the tall table must not push the donut off.
{"name": "alta_card_col", "inferred_type": "categorical",
"null_pct": 0.0, "distinct_count": 40,
"categorical": {"top": top, "mode": long_vals[0], "n_distinct": 40,
"entropy": 5.2, "imbalance": 1.2, "len_min": 40,
"len_mean": 45, "len_max": 50}},
{"name": "baja_card_col", "inferred_type": "categorical",
"null_pct": 0.0, "distinct_count": 4,
"categorical": {
"top": [{"value": "norte", "count": 2000, "pct": 0.4},
{"value": "sur", "count": 1500, "pct": 0.3},
{"value": "este", "count": 1000, "pct": 0.2},
{"value": "oeste", "count": 500, "pct": 0.1}],
"mode": "norte", "n_distinct": 4, "entropy": 1.8}},
],
}
def test_golden_pptx_una_slide_por_columna_con_su_grafico():
"""Each categorical column occupies EXACTLY ONE cat_distr slide that carries
BOTH its cardinality table and its donut figure (picture) — i.e. the chart is
never separated from its table, even for a high-cardinality column."""
from pptx.enum.shapes import MSO_SHAPE_TYPE
prof = _profile_high_card()
cat_names = ["alta_card_col", "baja_card_col"]
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pptx")
res = render_automatic_eda_pptx(prof, out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
prs = Presentation(out)
# Per column: the cat_distr slides whose text mentions it, and whether the
# owning slide also has the donut caption + an actual picture shape.
slides_with_col = {n: [] for n in cat_names}
owner_has_chart = {n: False for n in cat_names}
for i, sl in enumerate(prs.slides):
texts, has_pic = [], False
for sh in sl.shapes:
if sh.has_text_frame:
texts.append(sh.text_frame.text)
if sh.shape_type == MSO_SHAPE_TYPE.PICTURE:
has_pic = True
txt = re.sub(r"\s+", " ", " ".join(texts))
if "Distribuciones categ" not in txt: # footer stamp of the chapter.
continue
for n in cat_names:
if n in txt:
slides_with_col[n].append(i)
has_table = "Cardinalidad" in txt or "distintos" in txt
if has_pic and "donut" in txt and has_table:
owner_has_chart[n] = True
for n in cat_names:
# Exactly one slide carries the column (not split across slides).
assert len(slides_with_col[n]) == 1, (n, slides_with_col[n])
# That single slide also holds its table AND its donut picture.
assert owner_has_chart[n], (n, "tabla y donut no están en el mismo slide")
def test_edge_sin_categoricas_devuelve_none():
only_numeric = {
"n_rows": 10, "columns": [
{"name": "x", "inferred_type": "numeric",
"numeric": {"mean": 1.0}}]}
assert build_cat_distr(only_numeric, {}) is None
# None / empty / no-columns never raise and yield None.
assert build_cat_distr(None, None) is None
assert build_cat_distr({}, {}) is None
assert build_cat_distr({"columns": []}, {}) is None
def test_anti_corte_label_largo_y_muchas_columnas():
long_label = ("Lorem ipsum dolor sit amet consectetur adipiscing elit sed "
"do eiusmod tempor incididunt ut labore reprehenderit voluptate")
cols = []
for i in range(30):
cols.append({
"name": f"cat_{i}", "inferred_type": "categorical",
"distinct_count": 3,
"categorical": {
"top": [{"value": long_label, "count": 60},
{"value": "b", "count": 30},
{"value": "c", "count": 10}],
"mode": long_label, "n_distinct": 3, "entropy": 1.2}})
profile = {"table": "t", "source": "t.csv", "n_rows": 100,
"n_cols": len(cols), "columns": cols}
ch = build_cat_distr(profile, {})
assert ch is not None
# One Group per column, each forcing its own page (except the first).
groups = _column_groups(ch)
assert len(groups) == 30
assert sum(1 for g in groups if g.page_break_before) == 29
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "anti.pdf")
res = render_automatic_eda_pdf(profile, pdf, {"write_manifest": False})
assert res["path"] == pdf
assert res["n_pages"] > 1 # one page per column, OK.
txt = _pdf_text(pdf)
# Long label wrapped (not truncated): every word survives.
for word in ("Lorem", "incididunt", "reprehenderit", "voluptate"):
assert word in txt
# PPTX path must not raise either.
pptx = os.path.join(d, "anti.pptx")
res2 = render_automatic_eda_pptx(profile, pptx,
{"write_manifest": False})
assert res2["path"] == pptx and os.path.exists(pptx)
@@ -0,0 +1,573 @@
"""Correlation chapter — association matrix plus top positive/negative pairs.
Builds the CORRELACION chapter of an AutomaticEDA document from a TableProfile.
It renders exactly what the user asked for:
1. A correlation/association **matrix** (heatmap) reconstructed from the evaluated
pairs, signed for numeric-numeric pairs (Pearson/Spearman, ``[-1, 1]``) and as
magnitude for the mixed-type metrics (Cramér's V, correlation ratio, mutual
information, ``[0, 1]``). Labels are ordered by total connectivity so strong
associations cluster together instead of being scattered alphabetically.
2. The **TOP positive** pairs and the **TOP negative** pairs as two separate
tables. Only numeric-numeric metrics carry a sign, so negative pairs are by
construction Pearson/Spearman; positive pairs may use any method.
3. The methods legend and the multiple-testing (FDR) summary, so the reader sees
how many pairs survive the correction.
4. A spuriousness caveat when the profile flags level-based correlations on
non-stationary series (GrangerNewbold).
All data comes from ``profile['correlations']`` — the output of the ``eda`` group
function ``association_matrix`` (optionally enriched by ``profile_table``). The
chapter never recomputes any statistic; it only lays the existing values out as
format-independent blocks. The renderers paginate tables (repeating the header)
and scale the heatmap to fit entirely, so nothing is ever cut.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
import math
from .. import model
CHAPTER_VERSION = "1.1.0"
CHAPTER_ID = "correlacion"
CHAPTER_TITLE = "Correlación"
# Methods whose value carries a sign (direction). Everything else is a magnitude
# in [0, 1] and therefore only ever contributes to the positive side.
_SIGNED_METHODS = ("pearson", "spearman")
# Cap the heatmap to the most-connected variables so it stays legible on a phone
# screen / a slide. The renderer would scale a bigger matrix to fit, but the
# cells become unreadable; we instead show the top-N and say so.
_MAX_MATRIX_LABELS = 16
# How many pairs to show in each of the top-positive / top-negative tables.
_TOP_N = 10
# How many of the strongest numeric-numeric pairs to draw as scatter plots on
# each sign (positive / negative). A scatter per pair carries a fitted line/curve
# and a relationship-type label; keeping the count small keeps the chapter
# readable on a phone / a slide. Only signed (Pearson/Spearman) pairs qualify —
# Cramér's V / correlation ratio pairs are not numeric-numeric, so no scatter.
_SCATTER_TOP_N = 3
# Glossary terms this chapter explains. Each is registered in the shared
# collector (ctx['glossary']) and marked clickable on its first appearance in the
# body — the canonical two-step pattern (see ``cat_distr`` for the reference
# implementation): ``glossary.add(key, label, definition)`` + the inline span
# ``[[term:KEY]]texto visible[[/term]]`` in a Markdown block. Mapping key ->
# (label, definition). ``fdr`` is only registered when the FDR summary is present.
_TERM_DEFS = {
"pearson": (
"Pearson (coeficiente r)",
"Coeficiente de correlación lineal de Pearson (r) entre dos variables "
"numéricas. Va de 1 (relación lineal inversa perfecta) a +1 (directa "
"perfecta); 0 indica ausencia de relación lineal. Sólo capta relaciones "
"lineales, por eso lleva signo."),
"spearman": (
"Spearman (correlación de rangos)",
"Correlación de rangos de Spearman: el coeficiente de Pearson calculado "
"sobre los puestos (rangos) de los valores en vez de sus magnitudes. Mide "
"relaciones monótonas (no necesariamente lineales), va de 1 a +1 y es "
"robusta frente a valores atípicos."),
"cramers_v": (
"Cramér's V",
"Medida de asociación entre dos variables categóricas, derivada del "
"estadístico chi-cuadrado y normalizada al rango 01 (0 = independientes, "
"1 = asociación total). No tiene signo: sólo mide la intensidad."),
"correlation_ratio": (
"Razón de correlación (η)",
"Razón de correlación (eta) entre una variable numérica y una "
"categórica: la fracción de la varianza de la numérica explicada por los "
"grupos de la categórica. Va de 0 (los grupos no explican nada) a 1 (la "
"explican toda); no tiene signo."),
"fdr": (
"Comparaciones múltiples (FDR)",
"Al evaluar muchos pares a la vez, algunos parecen significativos por "
"puro azar. La corrección por tasa de falsos descubrimientos (FDR, "
"Benjamini-Hochberg) ajusta los p-valores para controlar la proporción "
"esperada de falsos positivos entre los pares declarados significativos."),
}
def _term(mark: bool, key: str, text: str) -> str:
"""Wrap ``text`` as a clickable glossary span when ``mark`` is True.
The visible text is identical with or without the marker (the renderers strip
the marker), so wrapping never changes line layout — it only adds the link.
"""
return f"[[term:{key}]]{text}[[/term]]" if mark else text
def _is_num(v) -> bool:
"""True for a real, finite int/float (not bool, not NaN/inf)."""
return (
isinstance(v, (int, float))
and not isinstance(v, bool)
and not (isinstance(v, float) and (math.isnan(v) or math.isinf(v)))
)
def _fmt_val(value, decimals: int = 2) -> str:
"""Format an association value compactly, signed, with a fixed width feel."""
if not _is_num(value):
return ""
text = f"{float(value):+.{decimals}f}"
# Strip a trailing -0.00 / +0.00 into a clean 0.00 for readability.
if text in ("+0.00", "-0.00"):
return "0.00"
return text
def _fmt_p(value) -> str:
"""Format an adjusted p-value; tiny values collapse to a '<' threshold."""
if not _is_num(value):
return ""
p = float(value)
if p < 0.001:
return "<0.001"
return f"{p:.3f}"
def _is_signed(pair: dict) -> bool:
"""True if the pair's method reports a directional (signed) value."""
method = str(pair.get("method") or "").lower()
return any(m in method for m in _SIGNED_METHODS)
def _significant(pair: dict) -> bool:
"""True if the pair is significant after FDR (or has no test to correct)."""
if pair.get("significant") is True:
return True
# Pairs without an applicable test (p_value None) are not penalised: they are
# admitted on magnitude alone upstream, so treat missing as "not rejected".
return pair.get("p_value") is None and pair.get("significant") is None
def _label(pair: dict) -> str:
"""Human label for a pair, e.g. 'alcohol ↔ density'."""
return f"{model._safe_str(pair.get('a'))}{model._safe_str(pair.get('b'))}"
def _split_top(pairs: list, top_n: int = _TOP_N):
"""Split evaluated pairs into ranked top-positive and top-negative lists.
Positive: any pair with a positive value, ranked by value descending.
Negative: only signed (numeric-numeric) pairs with a negative value, ranked
by value ascending (most negative first). Non-finite values are dropped.
"""
positive = []
negative = []
for pair in pairs:
if not isinstance(pair, dict):
continue
value = pair.get("value")
if not _is_num(value):
continue
if value > 0:
positive.append(pair)
elif value < 0 and _is_signed(pair):
negative.append(pair)
positive.sort(key=lambda p: float(p.get("value", 0.0)), reverse=True)
negative.sort(key=lambda p: float(p.get("value", 0.0)))
return positive[:top_n], negative[:top_n]
def _top_table(pairs: list, title: str):
"""Build a DataTable for a list of pairs, or None if there are none."""
if not pairs:
return None
header = ["Par", "Método", "Valor", "p (FDR)", "Sig."]
rows = []
for pair in pairs:
method = model._safe_str(pair.get("method")) or ""
rows.append([
_label(pair),
method,
_fmt_val(pair.get("value")),
_fmt_p(pair.get("p_value_adjusted")),
"" if _significant(pair) else "no",
])
return model.DataTable(header=header, rows=rows, title=title)
def _ordered_labels(pairs: list):
"""Pick and order the matrix labels by total connectivity (descending).
Returns the list of variable names to place on the axes, capped at
``_MAX_MATRIX_LABELS`` (the most-connected ones), plus a boolean saying
whether the cap trimmed anything.
"""
strength = {}
for pair in pairs:
if not isinstance(pair, dict):
continue
value = pair.get("value")
if not _is_num(value):
continue
mag = abs(float(value))
for key in ("a", "b"):
name = pair.get(key)
if name is None:
continue
strength[name] = strength.get(name, 0.0) + mag
if not strength:
return [], False
ordered = sorted(strength, key=lambda n: strength[n], reverse=True)
trimmed = len(ordered) > _MAX_MATRIX_LABELS
return ordered[:_MAX_MATRIX_LABELS], trimmed
def _matrix_figure(pairs: list, labels: list):
"""Return a Figure (lazy) with the signed association heatmap, or None.
The matplotlib figure is built lazily inside ``make`` so importing this
module never requires matplotlib and a malformed plot degrades to nothing
instead of aborting the chapter.
"""
if len(labels) < 2:
return None
index = {name: i for i, name in enumerate(labels)}
def make():
import numpy as np
from matplotlib.figure import Figure
n = len(labels)
grid = np.full((n, n), np.nan, dtype=float)
for i in range(n):
grid[i, i] = 1.0
for pair in pairs:
if not isinstance(pair, dict):
continue
a = pair.get("a")
b = pair.get("b")
value = pair.get("value")
if a not in index or b not in index or not _is_num(value):
continue
v = float(value)
# Mixed-type magnitudes are non-negative; keep them as-is on [0, 1].
ia, ib = index[a], index[b]
grid[ia, ib] = v
grid[ib, ia] = v
import matplotlib
masked = np.ma.masked_invalid(grid)
fig = Figure(figsize=(6.2, 5.6))
ax = fig.add_subplot(111)
cmap = matplotlib.colormaps["RdBu_r"].copy()
cmap.set_bad(color="#eeeeee")
im = ax.imshow(masked, cmap=cmap, vmin=-1.0, vmax=1.0, aspect="auto")
ax.set_xticks(range(n))
ax.set_yticks(range(n))
short = [str(s)[:14] for s in labels]
ax.set_xticks(range(n))
ax.set_xticklabels(short, rotation=90, fontsize=7)
ax.set_yticklabels(short, fontsize=7)
# Annotate cells only when the matrix is small enough to stay legible.
if n <= 8:
for i in range(n):
for j in range(n):
cell = grid[i, j]
if _is_num(cell):
ax.text(j, i, f"{cell:+.2f}".replace("+", "") if cell < 0
else f"{cell:.2f}",
ha="center", va="center", fontsize=6,
color="#222222")
fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04,
label="asociación (signo en num-num)")
fig.tight_layout()
return fig
return model.Figure(make=make,
caption="Matriz de asociación. Azul = positiva, rojo = "
"negativa (sólo num-num lleva signo); gris = par "
"no evaluado.")
def _methods_block(corr: dict):
"""Build a KVTable with the legend of the methods actually present."""
legend = corr.get("methods_legend")
if not isinstance(legend, dict) or not legend:
return None
rows = [(model._safe_str(k), model._safe_str(v)) for k, v in legend.items()]
return model.KVTable(rows=rows, title="Métodos de asociación")
def _fdr_text(corr: dict, mark_term: bool = False) -> str | None:
"""One-line summary of the multiple-testing (FDR) correction, or None."""
mt = corr.get("multiple_testing")
if not isinstance(mt, dict) or not mt:
return None
method = model._safe_str(mt.get("method")).upper() or "FDR"
alpha = mt.get("alpha")
n_tests = mt.get("n_tests")
n_rej = mt.get("n_rejected")
multi = _term(mark_term, "fdr", "comparaciones múltiples")
parts = [f"Corrección por {multi} ({method}"]
if _is_num(alpha):
parts[0] += f", α={float(alpha):g}"
parts[0] += ")."
if _is_num(n_tests):
rej = n_rej if _is_num(n_rej) else ""
parts.append(
f"De {int(n_tests)} pares con test, {rej} siguen siendo "
f"significativos tras la corrección.")
return " ".join(parts)
def _is_seq(values) -> bool:
"""True for a non-empty list/tuple of values (a raw numeric column)."""
return isinstance(values, (list, tuple)) and len(values) > 0
def _select_scatter_pairs(pairs: list, top_n: int = _SCATTER_TOP_N):
"""Pick the strongest numeric-numeric pairs to draw as scatters.
Only signed (Pearson/Spearman) pairs are numeric-numeric and thus eligible
for a scatter with a fitted curve. Returns up to ``top_n`` of the strongest
positive pairs followed by up to ``top_n`` of the strongest negative ones,
each ranked by magnitude. Mixed-type metrics (Cramér's V, correlation ratio,
mutual information) are excluded — they have no x/y scatter interpretation.
"""
positive = []
negative = []
for pair in pairs:
if not isinstance(pair, dict) or not _is_signed(pair):
continue
value = pair.get("value")
if not _is_num(value):
continue
if value > 0:
positive.append(pair)
elif value < 0:
negative.append(pair)
positive.sort(key=lambda p: abs(float(p.get("value", 0.0))), reverse=True)
negative.sort(key=lambda p: abs(float(p.get("value", 0.0))), reverse=True)
return positive[:top_n] + negative[:top_n]
def _classification_note(a: str, b: str, cls: dict) -> str:
"""Human-readable sentence describing the relationship of a pair.
Plain text (not baked into the figure image) so the type label is selectable
in the PDF / extractable by pdftotext, and sits right next to its scatter
inside the keep-together Group.
"""
tipo = model._safe_str(cls.get("tipo")) or "sin forma clara"
bits = []
pearson = cls.get("pearson")
spearman = cls.get("spearman")
r2_lin = cls.get("r2_linear")
r2_poly = None
for key in ("r2_poly2", "r2_poly3"):
v = cls.get(key)
if _is_num(v) and (r2_poly is None or float(v) > r2_poly):
r2_poly = float(v)
if _is_num(pearson):
bits.append(f"Pearson r={float(pearson):+.2f}")
if _is_num(spearman):
bits.append(f"Spearman ρ={float(spearman):+.2f}")
if _is_num(r2_lin):
bits.append(f"R² lineal={float(r2_lin):.2f}")
if r2_poly is not None:
bits.append(f"R² polinómico={r2_poly:.2f}")
metrics = "; ".join(bits)
text = (f"Relación **{tipo}** entre «{a}» y «{b}»."
+ (f" {metrics}." if metrics else ""))
return text
def _scatter_blocks(pairs: list, raw_numeric):
"""Build keep-together scatter Groups for the strongest num-num pairs.
Returns a list of blocks (a Heading plus one Group per pair), or an empty
list when there is no raw numeric data (e.g. the lite profile drops
``ctx['raw_numeric']`` to skip live recomputation) or the relationship
helpers are unavailable. Never raises: any failure degrades to no scatters,
leaving the matrix + tables intact.
"""
if not isinstance(raw_numeric, dict) or not raw_numeric:
return []
selected = _select_scatter_pairs(pairs)
if not selected:
return []
# The relationship helpers live in the datascience package. Import lazily so
# the chapter still builds (matrix + tables) when they are absent.
try:
from datascience.classify_relationship_type import (
classify_relationship_type,
)
from datascience.relationship_scatter_figure import (
relationship_scatter_figure,
)
except Exception: # noqa: BLE001 — degrade, never break the chapter.
return []
groups = []
for pair in selected:
a = pair.get("a")
b = pair.get("b")
xs = raw_numeric.get(a)
ys = raw_numeric.get(b)
# Edge: a selected pair has no raw column (aggregated profile, renamed
# column, …) — skip just that pair, keep the rest.
if not _is_seq(xs) or not _is_seq(ys):
continue
try:
cls = classify_relationship_type(list(xs), list(ys)) or {}
except Exception: # noqa: BLE001
continue
a_lbl = model._safe_str(a)
b_lbl = model._safe_str(b)
def _make(xs=xs, ys=ys, a_lbl=a_lbl, b_lbl=b_lbl, cls=cls):
return relationship_scatter_figure(
list(xs), list(ys), x_label=a_lbl, y_label=b_lbl,
classification=cls)
groups.append(model.Group(blocks=[
model.Heading(text=f"{a_lbl}{b_lbl}", level=2),
model.Figure(
make=_make,
caption=(f"Dispersión de «{a_lbl}» frente a «{b_lbl}» con la "
"curva de ajuste del mejor modelo.")),
model.Markdown(text=_classification_note(a_lbl, b_lbl, cls)),
]))
if not groups:
return []
intro = model.Markdown(text=(
"Para los pares numéricos más fuertes (positivos y negativos) se dibuja "
"la nube de puntos con su ajuste y se clasifica el **tipo de relación**: "
"**lineal** (una recta basta), **polinómica** (curva de grado 2/3 que "
"mejora claramente el ajuste lineal), **monótona no-lineal** (crece o "
"decrece siempre pero no en línea recta; Spearman ≫ Pearson) o "
"**débil/sin forma**."))
return [model.Heading(text="Relaciones más fuertes (scatter)", level=2),
intro] + groups
def build_correlacion(profile: dict, ctx: dict):
"""Build the Correlation Chapter, or None if there are no pairs to show.
Reads ``profile['correlations']`` (the ``association_matrix`` output). Returns
``None`` when the dataset has fewer than two associable columns (no evaluated
pairs), so the chapter is omitted instead of showing an empty section. Never
raises: every access is defensive.
ctx keys consumed: none specific (presentation metadata is inherited from the
document). The chapter reads everything it needs from the profile.
"""
profile = profile or {}
ctx = ctx or {}
corr = profile.get("correlations")
if not isinstance(corr, dict):
return None
pairs = corr.get("pairs")
if not isinstance(pairs, list) or not pairs:
return None
blocks: list = []
# Register the always-present method terms in the shared glossary and mark
# their first appearance clickable (the FDR term is registered lazily below,
# only when the FDR summary is actually emitted). Degrades silently when no
# collector is in ctx (standalone render) — mark_term stays False.
glossary = ctx.get("glossary")
gloss = glossary if isinstance(glossary, model.GlossaryCollector) else None
mark_term = gloss is not None
if gloss is not None:
for key in ("pearson", "spearman", "cramers_v", "correlation_ratio"):
label, definition = _TERM_DEFS[key]
gloss.add(key, label, definition)
# Intro: what this chapter shows and how to read the sign. Build the marked
# method names as locals first (avoids backslash-in-f-string for "Cramér's V").
t_pearson = _term(mark_term, "pearson", "Pearson")
t_spearman = _term(mark_term, "spearman", "Spearman")
t_cramers = _term(mark_term, "cramers_v", "Cramér's V")
t_corr_ratio = _term(mark_term, "correlation_ratio", "razón de correlación")
blocks.append(model.Markdown(text=(
"Asociación entre columnas. Cada par se evalúa con la métrica adecuada "
f"a sus tipos: {t_pearson}/{t_spearman} (numéricas), {t_cramers} "
f"(categóricas), {t_corr_ratio} (num-categórica) e información mutua. "
"Sólo las correlaciones **num-num** llevan **signo** (dirección): por "
"eso los pares **negativos** son siempre num-num.")))
# 1) Association matrix (heatmap).
labels, trimmed = _ordered_labels(pairs)
fig = _matrix_figure(pairs, labels)
if fig is not None:
blocks.append(model.Heading(text="Matriz de asociación", level=2))
blocks.append(fig)
if trimmed:
blocks.append(model.Note(text=(
f"Se muestran las {len(labels)} variables más conectadas de la "
"matriz para mantenerla legible; el resto de pares siguen en las "
"tablas de abajo.")))
# 2) Top positive / top negative pairs.
positive, negative = _split_top(pairs, _TOP_N)
pos_table = _top_table(positive, f"Top {len(positive)} positivas")
neg_table = _top_table(negative, f"Top {len(negative)} negativas")
if pos_table is not None:
blocks.append(model.Heading(text="Pares más correlacionados (positivos)",
level=2))
blocks.append(pos_table)
if neg_table is not None:
blocks.append(model.Heading(text="Pares más correlacionados (negativos)",
level=2))
blocks.append(neg_table)
elif pos_table is not None:
# No signed-negative pairs at all: say so honestly rather than omit.
blocks.append(model.Note(text=(
"No se han hallado correlaciones negativas significativas entre "
"columnas numéricas.")))
# 2.5) Scatter plots of the strongest numeric-numeric pairs, each with its
# fitted curve and a relationship-type label (lineal / polinómica / monótona
# / débil). Needs the raw numeric sample (ctx['raw_numeric'], row-aligned);
# when it is absent (aggregated/lite profile) the scatters are simply omitted
# and the matrix + tables above stand on their own.
raw_numeric = None
if isinstance(ctx, dict):
raw_numeric = ctx.get("raw_numeric") or profile.get("raw_numeric")
else:
raw_numeric = profile.get("raw_numeric")
blocks.extend(_scatter_blocks(pairs, raw_numeric))
# 3) Spuriousness caveat for level-based correlations (GrangerNewbold).
caveat = corr.get("levels_caveat")
if isinstance(caveat, str) and caveat.strip():
blocks.append(model.Note(text=caveat.strip()))
elif corr.get("levels_possible_spurious"):
blocks.append(model.Note(text=(
"Aviso: algunas correlaciones se calcularon sobre niveles de series "
"no estacionarias y pueden ser espurias (GrangerNewbold). Compáralas "
"sobre los retornos/diferencias antes de interpretarlas.")))
# 4) FDR summary + methods legend. Register the FDR term only when its
# summary is emitted, so the glossary never lists an unreferenced entry.
fdr_text = _fdr_text(corr, mark_term=mark_term)
if fdr_text:
if gloss is not None:
label, definition = _TERM_DEFS["fdr"]
gloss.add("fdr", label, definition)
blocks.append(model.Markdown(text=fdr_text))
methods = _methods_block(corr)
if methods is not None:
blocks.append(model.Heading(text="Métodos y leyenda", level=2))
blocks.append(methods)
if not blocks:
return None
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,296 @@
"""Tests for the CORRELACION chapter — DoD: golden + edges + error/anti-cut.
Self-contained: builds a synthetic TableProfile carrying a ``correlations`` block
shaped exactly like ``association_matrix`` output (no DuckDB), so the suite is
fast and deterministic. Verifies that the chapter emits the association-matrix
figure plus separate top-positive / top-negative tables with the right pairs,
that it returns None when the profile has no pairs, that a None/empty profile
does not raise, and that a wide matrix with long labels renders to PDF *and* PPTX
without cutting anything.
"""
import os
import re
import tempfile
from pypdf import PdfReader
from datascience.automatic_eda.chapters.correlacion import (
CHAPTER_VERSION,
build_correlacion,
)
from datascience.automatic_eda.model import DataTable, Figure
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
def _pair(a, b, value, method, padj, sig, p=0.0001):
return {
"a": a, "b": b, "a_type": "numeric", "b_type": "numeric",
"method": method, "value": value, "extra": {"mi": abs(value) * 0.5},
"p_value": p, "p_value_adjusted": padj, "significant": sig,
}
def _profile() -> dict:
"""Synthetic wine-like profile with signed and unsigned associations."""
pairs = [
_pair("alcohol", "quality", 0.48, "pearson/spearman", 0.0005, True),
_pair("density", "alcohol", -0.78, "pearson/spearman", 0.0001, True),
_pair("ph", "fixed_acidity", -0.68, "pearson/spearman", 0.0002, True),
_pair("sulphates", "quality", 0.25, "pearson/spearman", 0.03, True),
# Unsigned mixed-type metrics: only ever positive, never in the neg table.
{"a": "region", "b": "type", "a_type": "categorical",
"b_type": "categorical", "method": "cramers_v", "value": 0.55,
"extra": {"mi": 0.3}, "p_value": 0.001, "p_value_adjusted": 0.004,
"significant": True},
]
return {
"table": "wine",
"source": "/data/wine.csv",
"n_rows": 1599,
"n_cols": 12,
"correlations": {
"pairs": pairs,
"strong": [p for p in pairs if abs(p["value"]) >= 0.5],
"methods_legend": {
"pearson": "num-num lineal (Pearson r), [-1, 1]",
"cramers_v": "cat-cat simétrica (Cramér's V), [0, 1]",
},
"multiple_testing": {"method": "bh", "alpha": 0.05,
"n_tests": 5, "n_rejected": 5},
},
}
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def test_golden_chapter_tiene_matriz_y_top_positivos_y_negativos():
ch = build_correlacion(_profile(), {})
assert ch is not None
assert ch.id == "correlacion"
assert ch.version == CHAPTER_VERSION
kinds = [b.kind for b in ch.blocks]
assert "figure" in kinds # association matrix heatmap.
figs = [b for b in ch.blocks if isinstance(b, Figure)]
assert figs and figs[0].make is not None # lazy figure.
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
assert len(tables) >= 2 # top positive + top negative.
flat = " ".join(str(c) for t in tables for r in t.rows for c in r)
# Strongest positive present and signed +, strongest negative present and -.
assert "alcohol" in flat and "quality" in flat
assert "+0.48" in flat
assert "density" in flat and "-0.78" in flat
def test_golden_render_pdf_y_pptx_muestran_lo_exigido():
prof = _profile()
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "corr.pdf")
pptx = os.path.join(d, "corr.pptx")
rp = render_automatic_eda_pdf(prof, pdf, {"title": "EDA — wine"})
rx = render_automatic_eda_pptx(prof, pptx, {"title": "EDA — wine"})
assert rp["path"] == pdf and rp["n_pages"] >= 1
assert rx["path"] == pptx and rx["n_slides"] >= 1
assert "correlacion" in [c["id"] for c in rp["chapters"]]
assert "correlacion" in [c["id"] for c in rx["chapters"]]
txt = _pdf_text(pdf)
# The requirement: matrix + top positive/negative pairs, all visible.
assert "Correlaci" in txt # chapter title (accents may vary in extract).
assert "density" in txt and "alcohol" in txt and "quality" in txt
assert "0.78" in txt and "0.48" in txt
# Both signs surfaced as separate sections.
assert "positiv" in txt.lower() and "negativ" in txt.lower()
def test_edge_sin_pares_devuelve_none():
# No correlations key, empty pairs, and wrong types all yield None, not error.
assert build_correlacion({"table": "x"}, {}) is None
assert build_correlacion({"correlations": {}}, {}) is None
assert build_correlacion({"correlations": {"pairs": []}}, {}) is None
assert build_correlacion({"correlations": {"pairs": "nope"}}, {}) is None
assert build_correlacion(None, None) is None
assert build_correlacion({}, {}) is None
def test_edge_solo_positivos_emite_nota_sin_tabla_negativa():
prof = {
"correlations": {
"pairs": [
_pair("a", "b", 0.6, "pearson/spearman", 0.001, True),
{"a": "c", "b": "d", "a_type": "categorical",
"b_type": "categorical", "method": "cramers_v", "value": 0.7,
"extra": {"mi": 0.4}, "p_value": 0.001,
"p_value_adjusted": 0.003, "significant": True},
],
},
}
ch = build_correlacion(prof, {})
assert ch is not None
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
assert len(tables) == 1 # only the positive table.
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
assert "negativas" in notes # honest "no negative correlations" note.
def test_anticorte_matriz_ancha_y_etiquetas_largas_no_se_cortan():
# 20 numeric vars with long names -> matrix trimmed to top-N + both renderers
# must lay the chapter out without raising and keep a long label intact.
long_a = "concentracion_de_dioxido_de_azufre_libre"
long_b = "concentracion_de_dioxido_de_azufre_total"
pairs = [_pair(long_a, long_b, -0.72, "pearson/spearman", 0.0001, True)]
for i in range(20):
pairs.append(_pair(f"variable_numerica_larga_{i:02d}",
f"variable_numerica_larga_{(i + 1) % 20:02d}",
0.55 - i * 0.02, "pearson/spearman", 0.01, True))
prof = {"correlations": {"pairs": pairs,
"multiple_testing": {"method": "bh", "alpha": 0.05,
"n_tests": len(pairs),
"n_rejected": len(pairs)}}}
ch = build_correlacion(prof, {})
assert ch is not None
# A "showing top-N most connected" note appears when the matrix is trimmed.
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
assert "más conectadas" in notes
# Anti-cut guarantee at the block level: the long pair reaches the renderer
# whole (the block never truncates); the renderer then wraps the cell inside
# its column. Both long labels are present, intact, in a table cell.
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
cells = [str(c) for t in tables for r in t.rows for c in r]
assert any(long_a in c and long_b in c for c in cells)
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "wide.pdf")
pptx = os.path.join(d, "wide.pptx")
rp = render_automatic_eda_pdf(prof, pdf, {"write_manifest": False})
rx = render_automatic_eda_pptx(prof, pptx, {"write_manifest": False})
# Both renderers lay the wide chapter out without raising and produce a
# non-empty document (nothing dropped, just wrapped/scaled to fit).
assert rp["path"] == pdf and os.path.exists(pdf) and rp["n_pages"] >= 1
assert rx["path"] == pptx and os.path.exists(pptx) and rx["n_slides"] >= 1
# A short, unbreakable fragment of the long label survives the wrap.
assert "azufre" in _pdf_text(pdf)
def _raw_numeric_for_profile(n: int = 80) -> dict:
"""Row-aligned raw numeric sample matching the signed pairs of _profile().
Builds columns with a clear, deterministic shape so the relationship-type
classifier has something unambiguous to label:
- density vs alcohol: strong negative linear (the top-negative pair).
- alcohol vs quality: positive linear.
- ph, fixed_acidity, sulphates: filler columns for the remaining pairs.
"""
import math as _m
alcohol = [8.0 + 0.05 * i for i in range(n)]
density = [1.0 - 0.002 * a for a in alcohol] # neg linear vs alcohol
quality = [3.0 + 0.4 * a + (0.1 if i % 2 else -0.1) # pos linear vs alcohol
for i, a in enumerate(alcohol)]
ph = [3.0 + 0.3 * _m.sin(i / 5.0) for i in range(n)]
fixed_acidity = [7.0 - 0.5 * p for p in ph] # neg linear vs ph
sulphates = [0.5 + 0.01 * (i % 7) for i in range(n)]
return {
"alcohol": alcohol, "density": density, "quality": quality,
"ph": ph, "fixed_acidity": fixed_acidity, "sulphates": sulphates,
}
def test_golden_scatters_de_pares_num_num_con_tipo_de_relacion():
"""Con ctx['raw_numeric'], el capítulo añade scatters (Figure dentro de Group)
de los pares num-num más fuertes, cada uno con su etiqueta de tipo en texto."""
from datascience.automatic_eda.model import Group
ctx = {"raw_numeric": _raw_numeric_for_profile()}
ch = build_correlacion(_profile(), ctx)
assert ch is not None
groups = [b for b in ch.blocks if isinstance(b, Group)]
assert groups, "debe emitir al menos un Group con scatter"
# Cada Group lleva su figura (lazy) y una nota de texto con el tipo.
for g in groups:
gkinds = [b.kind for b in g.blocks]
assert "figure" in gkinds and "markdown" in gkinds
# La sección y la etiqueta de tipo aparecen como texto plano (extraíble).
headings = " ".join(b.text for b in ch.blocks if b.kind == "heading")
assert "Relaciones más fuertes" in headings
body = " ".join(b.text for g in groups for b in g.blocks
if b.kind == "markdown")
assert any(t in body for t in
("lineal", "polinómica", "monótona", "sin forma"))
# El par num-num más fuerte (density ↔ alcohol) tiene scatter; el par cat-cat
# (region ↔ type) NO — no es numérico.
assert "density" in body or "alcohol" in body
assert "region" not in body and "type" not in body
def test_golden_pdf_muestra_scatters_con_etiqueta_de_tipo():
"""En el PDF, el capítulo Correlación incluye los scatters y su etiqueta de
tipo en texto seleccionable (pdftotext la encuentra)."""
prof = _profile()
ctx = {"raw_numeric": _raw_numeric_for_profile()}
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "corr_scatter.pdf")
rp = render_automatic_eda_pdf(prof, pdf, {"title": "EDA — wine",
"ctx": ctx})
assert rp["path"] == pdf and rp["n_pages"] >= 1
txt = _pdf_text(pdf)
assert "Relaciones" in txt and "scatter" in txt.lower()
# Alguna etiqueta de tipo de relación, en texto.
assert any(t in txt for t in
("lineal", "polin", "monóton", "monoton", "sin forma"))
def test_edge_sin_raw_numeric_omite_scatters_sin_lanzar():
"""profile lite / ctx None: sin raw_numeric el capítulo omite los scatters
pero sigue emitiendo matriz + tablas (no lanza)."""
from datascience.automatic_eda.model import Group
for ctx in (None, {}, {"raw_numeric": None}, {"raw_numeric": {}}):
ch = build_correlacion(_profile(), ctx)
assert ch is not None
assert not [b for b in ch.blocks if isinstance(b, Group)]
# La matriz y al menos una tabla top siguen presentes.
assert any(b.kind == "figure" for b in ch.blocks)
assert any(b.kind == "data_table" for b in ch.blocks)
def test_edge_par_sin_columna_cruda_se_omite_sin_lanzar():
"""Si un par seleccionado no tiene su columna en raw_numeric, se omite ese
par (no lanza); los demás scatters se construyen igual."""
from datascience.automatic_eda.model import Group
raw = _raw_numeric_for_profile()
raw.pop("density", None) # rompe el par density ↔ alcohol
ch = build_correlacion(_profile(), {"raw_numeric": raw})
assert ch is not None
groups = [b for b in ch.blocks if isinstance(b, Group)]
body = " ".join(b.text for g in groups for b in g.blocks
if b.kind == "markdown")
# density desaparece de los scatters; otros pares (p.ej. ph↔fixed_acidity,
# alcohol↔quality) pueden seguir presentes sin error.
assert "density" not in body
def test_glosario_engancha_metodos_y_fdr():
"""Mejora 4b: los métodos de correlación (Pearson, Spearman, Cramér's V,
razón de correlación) y la corrección por comparaciones múltiples (FDR) se
registran en el colector compartido y se marcan clicables en el cuerpo. Sin
colector en ctx, el capítulo degrada y no marca nada."""
from datascience.automatic_eda.model import GlossaryCollector
g = GlossaryCollector()
ch = build_correlacion(_profile(), {"glossary": g})
assert ch is not None
keys = {t["key"] for t in g.terms()}
assert {"pearson", "spearman", "cramers_v", "correlation_ratio", "fdr"} <= keys
body = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
for k in ("pearson", "spearman", "cramers_v", "correlation_ratio", "fdr"):
assert f"[[term:{k}]]" in body, k
# Sin colector: degrada limpio (ningún marcador en el cuerpo).
ch2 = build_correlacion(_profile(), {})
body2 = " ".join(b.text for b in ch2.blocks if b.kind == "markdown")
assert "[[term:" not in body2
@@ -0,0 +1,47 @@
"""Glossary chapter (GLOSARIO) — always the last chapter, clickable terms.
Renders one entry per glossary term that the other chapters registered during
the document build through ``ctx['glossary'].add(key, label, definition)`` (see
``GlossaryCollector`` in ``model.py``). Each entry is a clickable destination:
every in-text appearance a chapter marked with ``[[term:key]]texto[[/term]]``
becomes a real jump to its entry here — PDF link annotations (PyMuPDF) and PPTX
native slide jumps, both wired by the renderers.
Returns ``None`` when no term was registered (there is nothing to show), so the
chapter simply disappears from documents that did not mark any term.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
from .. import model
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "glosario"
CHAPTER_TITLE = "Glosario"
def build_glosario(profile: dict, ctx: dict):
"""Build the glossary Chapter from the shared collector, or None if empty."""
ctx = ctx or {}
glossary = ctx.get("glossary")
if not isinstance(glossary, model.GlossaryCollector) or not glossary:
return None
blocks = [
model.Heading(text="Glosario de términos", level=1),
model.Markdown(text=(
"Definición de los términos técnicos que aparecen en el informe. "
"Cada término va resaltado en el texto y, al pulsarlo, salta a su "
"definición en esta sección.")),
]
# One clickable destination per term, alphabetically by visible label.
for term in glossary.terms(by="label"):
blocks.append(model.GlossaryEntry(
key=model._safe_str(term.get("key")),
label=model._safe_str(term.get("label")),
definition=model._safe_str(term.get("definition"))))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,594 @@
"""Missingness chapter (MISSINGNESS) — patterns of missing data.
Complements the CALIDAD chapter: where CALIDAD reports *how much* is missing per
column (the null percentage that lowers the completeness score), this chapter
reports the **pattern** of the missing data — whether columns tend to be missing
*together* (co-occurrence of absences) or independently. That distinction is what
separates data that is missing completely at random ([[term:mcar]]MCAR[[/term]])
from data missing as a function of another variable ([[term:mar]]MAR[[/term]]),
which is the key question to settle before imputing or modelling.
The chapter activates only when the table actually has missing data (at least one
column with a null in the aggregated profile); otherwise it returns ``None`` and
disappears from the document.
Sections, in order:
1. **Resumen global** — % of missing cells in the dataset, number of columns with
nulls, and complete rows (no missing) vs incomplete rows (≥1 missing).
2. **Ranking por columna** — columns sorted by their null percentage, with a
horizontal bar figure.
3. **Co-ocurrencia de ausencias** — the correlation of the binary is-null masks
between columns (which columns tend to be missing together): a heatmap plus a
table of the top column pairs that co-miss.
4. **Patrones de fila** — the most frequent "which columns are missing together"
row patterns, in the style of missingno's pattern matrix.
5. **Lectura MCAR/MAR** — an interpretive, *exploratory* note (not a confirmatory
test such as Little's) reading the absence correlations as a hint of MCAR
(independent absences) vs MAR (co-occurring absences).
The aggregate per-column null counts come from the ``eda`` group ``TableProfile``
(``columns[i]['null_count'] / 'null_pct'`` and the table-level ``null_cell_pct``).
The per-row is-null mask needed for co-occurrence is built from raw data: a single
DuckDB push-down over ``ctx['db_path'] / ctx['table']`` (same pattern as the
AGREGACION chapter) covering ALL columns, with a fallback to the numeric-only
``ctx['raw_numeric']`` when no database is reachable. All the heavy lifting is
delegated to pure registry functions (``missingness_overview``,
``missingness_correlation``, ``missingness_row_patterns``) and two figure helpers
(``missingness_rank_bar_figure``, ``missingness_corr_heatmap_figure``); every one
is imported lazily and degrades to an honest note so this chapter never raises.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
from .. import model
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "missingness"
CHAPTER_TITLE = "Datos faltantes"
# Sample cap for the per-row is-null mask push-down. Co-occurrence and row
# patterns are computed on this sample; the global % of missing cells and the
# per-column ranking come from the (exact) aggregated profile instead.
MASK_SAMPLE = 5000
# Thresholds for the MCAR/MAR heuristic note. A pair counts as a *strong*
# co-occurrence when the absence correlation alone is high; as a *partial*
# co-occurrence when the absences overlap materially (high Jaccard) even if the
# Pearson correlation is modest — the usual case when one column is missing far
# more often than the other (e.g. Cabin 77% vs Age 20% in Titanic), which dilutes
# the correlation while the rows still co-miss in absolute terms.
_CORR_STRONG = 0.30
_JACCARD_NOTABLE = 0.20
# Rows shown in the top-pairs and row-patterns tables (bounded, never silently
# truncated: the table note reports the full count).
_TOP_PAIRS = 12
_TOP_PATTERNS = 12
# Truncate long column names in tables (the renderer also wraps).
_LABEL_MAX = 28
# Glossary terms this chapter explains (contract §11.1). Registered in the shared
# collector and marked clickable on their first appearance.
_TERMS = {
"missingness": (
"Patrón de datos faltantes (missingness)",
"El patrón con el que faltan los datos: cuánto falta, en qué columnas y "
"si las ausencias de unas columnas coinciden (co-ocurren) con las de "
"otras. Analizarlo —no solo contar nulos— distingue datos que faltan al "
"azar (MCAR) de los que faltan en función de otra variable (MAR), lo que "
"decide cómo imputar o si descartar filas sin sesgar el análisis.",
),
"mcar": (
"MCAR (Missing Completely At Random)",
"Los valores faltan de forma independiente de cualquier dato, observado o "
"no: las ausencias de unas columnas no se relacionan entre sí ni con los "
"valores. Es el caso más benigno —descartar filas o imputar la media no "
"introduce sesgo—, pero rara vez se cumple del todo en datos reales.",
),
"mar": (
"MAR (Missing At Random)",
"La probabilidad de que un valor falte depende de OTRAS variables "
"observadas (p. ej. una medición que falta más en cierto grupo). Las "
"ausencias co-ocurren entre columnas o se relacionan con los valores de "
"otras; imputar exige condicionar en esas variables para no sesgar. La "
"co-ocurrencia fuerte de ausencias es un indicio (exploratorio) de MAR.",
),
}
# --------------------------------------------------------------------------- #
# Small defensive formatters (own copy: the chapter never imports siblings).
# --------------------------------------------------------------------------- #
def _fmt_int(value) -> str:
if value is None:
return ""
try:
return f"{int(round(float(value))):,}".replace(",", ".")
except (TypeError, ValueError):
return model._safe_str(value)
def _fmt_pct(value, decimals: int = 1) -> str:
"""Format an already-0-100 value as a percentage. None -> placeholder."""
if value is None:
return ""
try:
return f"{float(value):.{decimals}f}%"
except (TypeError, ValueError):
return model._safe_str(value)
def _fmt_num(value, decimals: int = 3) -> str:
if value is None:
return ""
try:
f = float(value)
except (TypeError, ValueError):
return model._safe_str(value)
if f != f: # NaN
return ""
text = f"{f:.{decimals}f}".rstrip("0").rstrip(".")
return text if text else "0"
def _truncate(text, limit: int = _LABEL_MAX) -> str:
s = model._safe_str(text)
if len(s) <= limit:
return s
return s[: max(1, limit - 1)].rstrip() + ""
def _term(key: str, label: str, mark: bool) -> str:
if mark:
return f"[[term:{key}]]**{label}**[[/term]]"
return f"**{label}**"
# --------------------------------------------------------------------------- #
# Profile reads (exact, all rows).
# --------------------------------------------------------------------------- #
def _null_count_of(col: dict):
"""Best-effort null count of a column: ``null_count`` or null_pct*n_rows."""
nc = col.get("null_count")
if isinstance(nc, (int, float)) and not isinstance(nc, bool):
return int(nc)
np_ = col.get("null_pct")
nr = col.get("n_rows")
if isinstance(np_, (int, float)) and isinstance(nr, (int, float)):
return int(round(float(np_) * float(nr)))
return 0
def _columns_with_nulls(profile: dict):
"""Return ``[(name, null_count, null_pct_0_100)]`` for columns with nulls,
sorted by null percentage descending. Reads the aggregated profile (exact)."""
cols = profile.get("columns") or []
out = []
for c in cols:
if not isinstance(c, dict):
continue
nc = _null_count_of(c)
if nc <= 0:
continue
np_ = c.get("null_pct")
nr = c.get("n_rows") or profile.get("n_rows")
if isinstance(np_, (int, float)) and not isinstance(np_, bool):
pct = float(np_) * 100.0 if np_ <= 1.0 else float(np_)
elif nr:
pct = nc / float(nr) * 100.0
else:
pct = None
out.append((c.get("name") or "(col)", nc, pct))
out.sort(key=lambda t: (t[2] if t[2] is not None else -1.0), reverse=True)
return out
def _global_missing_pct(profile: dict):
"""Table-level % of missing cells (0-100), exact, from the profile."""
v = profile.get("null_cell_pct")
if isinstance(v, (int, float)) and not isinstance(v, bool):
return float(v) * 100.0 if v <= 1.0 else float(v)
return None
# --------------------------------------------------------------------------- #
# Per-row is-null mask (sample): DuckDB push-down, fallback to raw_numeric.
# --------------------------------------------------------------------------- #
def _build_query_fn(ctx: dict):
"""Return ``(query_fn, table)`` for a DuckDB-backed ctx, or ``(None, None)``.
Mirrors build_eda_render_ctx: a read-only closure over the registry wrapper.
Only DuckDB is supported here; any other backend degrades to raw_numeric."""
db_path = ctx.get("db_path")
table = ctx.get("table")
if not db_path or not table:
return None, None
try:
from infra import duckdb_query_readonly
except Exception: # noqa: BLE001 — wrapper unavailable -> degrade.
return None, None
def query_fn(sql):
return duckdb_query_readonly(db_path, sql)
return query_fn, table
def _null_mask(profile: dict, ctx: dict):
"""Build the per-row is-null mask ``{col: [0/1, ...]}``.
Tries a single DuckDB push-down over ALL columns first (so categorical
columns like Cabin are covered, not only numeric ones); falls back to the
numeric-only ``ctx['raw_numeric']`` (None -> missing); returns ``(None, 0,
None)`` when neither is reachable. Never raises.
Returns ``(mask, n_sampled, source)`` with source in {"db","raw_numeric"}.
"""
cols = profile.get("columns") or []
names = [c.get("name") for c in cols
if isinstance(c, dict) and c.get("name")]
# 1) DuckDB push-down over every column (covers categoricals too).
query_fn, table = _build_query_fn(ctx)
if query_fn is not None and names:
try:
from datascience.extract_null_mask import extract_null_mask
res = extract_null_mask(query_fn, table, names, max_rows=MASK_SAMPLE)
if isinstance(res, dict) and res.get("status") == "ok":
mask = res.get("mask") or {}
if mask:
return mask, int(res.get("n") or 0), "db"
except Exception: # noqa: BLE001 — degrade to raw_numeric.
pass
# 2) Fallback: numeric-only mask derived from raw_numeric (None -> missing).
rn = ctx.get("raw_numeric")
if isinstance(rn, dict) and rn:
mask = {}
for col, vals in rn.items():
if isinstance(vals, (list, tuple)):
mask[col] = [1 if v is None else 0 for v in vals]
if mask:
n = max((len(v) for v in mask.values()), default=0)
return mask, n, "raw_numeric"
return None, 0, None
# --------------------------------------------------------------------------- #
# Lazy registry delegations (each degrades to None on any failure).
# --------------------------------------------------------------------------- #
def _overview(mask: dict):
try:
from datascience.missingness_overview import missingness_overview
out = missingness_overview(mask)
return out if isinstance(out, dict) else None
except Exception: # noqa: BLE001
return None
def _correlation(mask: dict, top_k: int):
try:
from datascience.missingness_correlation import missingness_correlation
out = missingness_correlation(mask, top_k=top_k)
return out if isinstance(out, dict) else None
except Exception: # noqa: BLE001
return None
def _row_patterns(mask: dict, top_n: int):
try:
from datascience.missingness_row_patterns import missingness_row_patterns
out = missingness_row_patterns(mask, top_n=top_n)
return out if isinstance(out, dict) else None
except Exception: # noqa: BLE001
return None
def _rank_bar_make(names, pcts, title):
def make():
try:
from datascience.missingness_rank_bar_figure import (
missingness_rank_bar_figure,
)
return missingness_rank_bar_figure(names, pcts, title=title)
except Exception: # noqa: BLE001 — minimal fallback figure.
return _fallback_fig("ranking de nulos no disponible")
return make
def _heatmap_make(matrix, labels, title):
def make():
try:
from datascience.missingness_corr_heatmap_figure import (
missingness_corr_heatmap_figure,
)
return missingness_corr_heatmap_figure(matrix, labels, title=title)
except Exception: # noqa: BLE001 — minimal fallback figure.
return _fallback_fig("heatmap de co-ocurrencia no disponible")
return make
def _fallback_fig(message: str):
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure
fig = Figure(figsize=(5.0, 2.2))
ax = fig.add_subplot(111)
ax.text(0.5, 0.5, message, ha="center", va="center")
ax.axis("off")
return fig
# --------------------------------------------------------------------------- #
# Block builders.
# --------------------------------------------------------------------------- #
def _summary_block(profile: dict, with_nulls: list, overview, sampled, n_total):
rows = []
gpct = _global_missing_pct(profile)
rows.append(("Celdas faltantes (global)", _fmt_pct(gpct)))
rows.append(("Columnas con faltantes", str(len(with_nulls))))
all_null = profile.get("all_null_cols")
if isinstance(all_null, (list, tuple)) and all_null:
rows.append(("Columnas 100% faltantes", str(len(all_null))))
if isinstance(overview, dict):
cr = overview.get("complete_rows")
ir = overview.get("incomplete_rows")
suffix = ""
if (isinstance(sampled, int) and isinstance(n_total, (int, float))
and sampled and n_total and sampled < n_total):
suffix = f" (sobre muestra de {_fmt_int(sampled)} filas)"
if cr is not None:
rows.append(("Filas completas (sin faltantes)",
f"{_fmt_int(cr)} ({_fmt_pct(overview.get('complete_pct'))})"
+ suffix))
if ir is not None:
rows.append(("Filas con ≥1 faltante",
f"{_fmt_int(ir)} "
f"({_fmt_pct(overview.get('incomplete_pct'))})" + suffix))
return model.KVTable(rows=rows, title="Resumen de datos faltantes")
def _ranking_block(with_nulls: list):
header = ["Columna", "Faltantes", "% faltante"]
rows = [[_truncate(n), _fmt_int(c), _fmt_pct(p)] for (n, c, p) in with_nulls]
if not rows:
return None
return model.DataTable(
header=header, rows=rows, title="Faltantes por columna",
note="ordenado de más a menos faltante")
def _ranking_figure(with_nulls: list):
names = [n for (n, _, p) in with_nulls if p is not None]
pcts = [p for (_, _, p) in with_nulls if p is not None]
if not names:
return None
return model.Figure(
make=_rank_bar_make(names, pcts, "% de valores faltantes por columna"),
caption="Porcentaje de valores faltantes por columna (barras).")
def _pairs_block(corr: dict):
"""Top column pairs whose absences co-occur, as a table, or None."""
pairs = (corr or {}).get("pairs") or []
header = ["Columna A", "Columna B", "Corr. ausencia", "Co-faltan", "Jaccard"]
rows = []
for p in pairs[:_TOP_PAIRS]:
if not isinstance(p, dict):
continue
rows.append([
_truncate(p.get("a")),
_truncate(p.get("b")),
_fmt_num(p.get("corr")),
_fmt_int(p.get("co_missing")),
_fmt_num(p.get("jaccard")),
])
if not rows:
return None
shown = len(rows)
total = len(pairs)
note = ("correlación de las máscaras is-null entre columnas; "
"«Co-faltan» = nº de filas en que ambas faltan a la vez")
if total > shown:
note += f" — top {shown} de {total} pares"
return model.DataTable(header=header, rows=rows,
title="Pares de columnas que co-faltan", note=note)
def _heatmap_block(corr: dict):
cols = (corr or {}).get("columns") or []
matrix = (corr or {}).get("matrix") or []
if len(cols) < 2 or not matrix:
return None
labels = [_truncate(c, 16) for c in cols]
return model.Figure(
make=_heatmap_make(matrix, labels, "Co-ocurrencia de ausencias"),
caption=("Correlación de las ausencias entre columnas (azul = faltan "
"juntas; rojo = cuando una falta la otra tiende a estar)."))
def _patterns_block(patterns_res: dict):
patterns = (patterns_res or {}).get("patterns") or []
header = ["Columnas que faltan juntas", "Filas", "%"]
rows = []
for p in patterns[:_TOP_PATTERNS]:
if not isinstance(p, dict):
continue
cols = p.get("missing_cols") or []
if cols:
label = ", ".join(_truncate(c, 18) for c in cols)
else:
label = "(fila completa — sin faltantes)"
rows.append([label, _fmt_int(p.get("n_rows")), _fmt_pct(p.get("pct"))])
if not rows:
return None
total = (patterns_res or {}).get("n_patterns")
shown = len(rows)
note = "cada fila es un patrón de «qué columnas faltan juntas»"
if isinstance(total, int) and total > shown:
note += f" — top {shown} de {total} patrones distintos"
return model.DataTable(header=header, rows=rows,
title="Patrones de fila más comunes", note=note)
def _mcar_mar_note(corr: dict, mark: bool):
"""Interpretive, exploratory MCAR/MAR note from the absence correlations.
Reads the absence correlations at two levels so the verdict never contradicts
the visible evidence: a *strong* correlation flags a clear non-random (MAR)
pattern; a *partial* overlap (many rows co-miss — high Jaccard — even if the
correlation is diluted by one column being missing far more often) flags a
localized possible-MAR and cites the concrete co-missing pair; only when
neither holds does it read the absences as compatible with MCAR."""
def _pairs_with(attr_ok):
out = []
for p in (corr or {}).get("pairs") or []:
if isinstance(p, dict) and attr_ok(p):
out.append(p)
return out
def _cf(v):
try:
return float(v)
except (TypeError, ValueError):
return 0.0
strong = _pairs_with(lambda p: abs(_cf(p.get("corr"))) >= _CORR_STRONG)
partial = _pairs_with(
lambda p: _cf(p.get("corr")) > 0 and _cf(p.get("jaccard")) >= _JACCARD_NOTABLE)
mcar = _term("mcar", "MCAR", mark)
mar = _term("mar", "MAR", mark)
head = (
"**Lectura exploratoria MCAR/MAR.** Esta es una heurística basada en la "
"correlación de las ausencias entre columnas, NO un test confirmatorio "
"(como el de Little); orienta, no demuestra. ")
if strong:
top = strong[0]
ev = (f"«{model._safe_str(top.get('a'))}» y "
f"«{model._safe_str(top.get('b'))}» "
f"(corr {_fmt_num(top.get('corr'))})")
body = (
f"Hay ausencias que co-ocurren con fuerza —{ev}—: las columnas no "
f"faltan de forma independiente, lo que es un indicio de un patrón no "
f"aleatorio ({mar}). Antes de imputar o descartar filas conviene "
f"comprobar si la ausencia depende de otra variable observada; en ese "
f"caso la imputación debería condicionar en ella para no sesgar.")
elif partial:
top = max(partial, key=lambda p: _cf(p.get("jaccard")))
ev = (f"«{model._safe_str(top.get('a'))}» y "
f"«{model._safe_str(top.get('b'))}» faltan a la vez en "
f"{_fmt_int(top.get('co_missing'))} filas "
f"(Jaccard {_fmt_num(top.get('jaccard'))})")
body = (
f"Hay co-ocurrencia parcial de ausencias —{ev}—: algunas columnas "
f"tienden a faltar juntas aunque la correlación global sea modesta "
f"(habitual cuando una columna falta mucho más que la otra). Es un "
f"indicio de un posible patrón localizado no aleatorio ({mar}); "
f"conviene revisar si esa ausencia depende de otra variable observada "
f"antes de imputar, en lugar de asumir que faltan al azar.")
else:
body = (
f"Las ausencias entre columnas no muestran correlación ni solape "
f"relevante: parecen independientes, lo que es compatible con que "
f"falten al azar ({mcar}). Aun así, la ausencia podría depender de "
f"variables no observadas (la heurística no lo descarta).")
return model.Markdown(text=head + body)
def _intro_block(mark: bool, source):
missingness = _term("missingness", "missingness", mark)
text = (
f"Este capítulo analiza el {missingness} de la tabla: no solo cuánto "
"falta (eso lo cubre la calidad), sino DÓNDE falta y si las columnas "
"faltan juntas. La co-ocurrencia de ausencias se calcula sobre la matriz "
"binaria «is-null» por fila.")
if source == "raw_numeric":
text += (" Nota: no se pudo leer la tabla cruda completa, así que la "
"co-ocurrencia se limita a las columnas numéricas disponibles.")
return model.Markdown(text=text)
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def build_missingness(profile: dict, ctx: dict):
"""Build the missingness Chapter, or None if the table has no missing data."""
if not isinstance(profile, dict):
profile = {}
ctx = ctx or {}
with_nulls = _columns_with_nulls(profile)
if not with_nulls:
return None # no missing data anywhere -> chapter does not apply.
# Register glossary terms (if a collector is present) and mark them clickable.
glossary = ctx.get("glossary")
mark = False
if isinstance(glossary, model.GlossaryCollector):
for key, (label, definition) in _TERMS.items():
glossary.add(key, label, definition)
mark = True
# Per-row is-null mask (sample) for co-occurrence and row patterns.
mask, sampled, source = _null_mask(profile, ctx)
overview = _overview(mask) if mask else None
n_total = profile.get("n_rows")
blocks = [
model.Heading(text="Cuánto y dónde faltan datos", level=2),
_intro_block(mark, source),
_summary_block(profile, with_nulls, overview, sampled, n_total),
model.Heading(text="Faltantes por columna", level=2),
]
ranking = _ranking_block(with_nulls)
if ranking is not None:
blocks.append(ranking)
rank_fig = _ranking_figure(with_nulls)
if rank_fig is not None:
blocks.append(rank_fig)
# Co-occurrence + row patterns need the per-row mask. Without it, say so.
if not mask:
blocks.append(model.Note(
"No se pudo construir la matriz «is-null» por fila (sin acceso a los "
"datos crudos), así que no se analiza la co-ocurrencia de ausencias "
"ni los patrones de fila en este informe."))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
corr = _correlation(mask, _TOP_PAIRS) or {}
co_blocks = [model.Heading(text="Co-ocurrencia de ausencias", level=2)]
heatmap = _heatmap_block(corr)
if heatmap is not None:
co_blocks.append(heatmap)
pairs = _pairs_block(corr)
if pairs is not None:
co_blocks.append(pairs)
if heatmap is None and pairs is None:
co_blocks.append(model.Note(
"Ninguna pareja de columnas comparte ausencias con variación "
"suficiente para correlacionarlas (p. ej. una sola columna con "
"faltantes), así que no hay co-ocurrencia que mostrar."))
# Keep the co-occurrence heading next to its heatmap and table.
blocks.append(model.Group(blocks=co_blocks))
patterns_res = _row_patterns(mask, _TOP_PATTERNS) or {}
patterns = _patterns_block(patterns_res)
if patterns is not None:
blocks.append(model.Heading(text="Patrones de fila", level=2))
blocks.append(patterns)
blocks.append(model.Heading(text="Lectura MCAR / MAR", level=2))
blocks.append(_mcar_mar_note(corr, mark))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,162 @@
"""Tests for the MISSINGNESS chapter.
Covers the Definition of Done for this chapter:
* Activates (non-None Chapter with the expected sections) when the profile has
missing data, building the co-occurrence from the per-row is-null mask.
* Returns None when the table has no missing data at all (edge case).
* Registers the MCAR/MAR/missingness glossary terms.
* The DuckDB push-down path covers categorical columns (not only numeric),
so a categorical column that co-misses with a numeric one is detected.
"""
import os
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
if _FUNCTIONS not in sys.path:
sys.path.insert(0, _FUNCTIONS)
from datascience.automatic_eda import model # noqa: E402
from datascience.automatic_eda.chapters.missingness import ( # noqa: E402
build_missingness,
)
def _titles(chapter):
"""Collect heading texts and table/figure titles for assertions."""
out = []
for b in chapter.blocks:
kind = getattr(b, "kind", None)
if kind == "heading":
out.append(("heading", getattr(b, "text", "")))
elif kind in ("data_table", "kv_table"):
out.append((kind, getattr(b, "title", "")))
elif kind == "group":
for inner in getattr(b, "blocks", []):
ik = getattr(inner, "kind", None)
if ik == "heading":
out.append(("heading", getattr(inner, "text", "")))
elif ik in ("data_table", "kv_table"):
out.append((ik, getattr(inner, "title", "")))
elif ik == "figure":
out.append(("figure", getattr(inner, "caption", "")))
elif kind == "figure":
out.append(("figure", getattr(b, "caption", "")))
return out
def _all_text(chapter):
parts = []
def walk(blocks):
for b in blocks:
for attr in ("text", "title", "note", "caption"):
v = getattr(b, attr, None)
if v:
parts.append(str(v))
if getattr(b, "kind", None) == "group":
walk(getattr(b, "blocks", []))
walk(chapter.blocks)
return "\n".join(parts)
def test_returns_none_when_no_missing_data():
profile = {
"n_rows": 4,
"null_cell_pct": 0.0,
"columns": [
{"name": "a", "null_count": 0, "null_pct": 0.0, "n_rows": 4},
{"name": "b", "null_count": 0, "null_pct": 0.0, "n_rows": 4},
],
}
assert build_missingness(profile, {}) is None
def test_activates_with_cooccurrence_via_raw_numeric():
# a and b are missing in EXACTLY the same rows (0,1,2) -> perfect absence
# correlation. c has no nulls. No db_path -> the chapter falls back to the
# numeric raw_numeric mask.
profile = {
"n_rows": 6,
"null_cell_pct": (0.5 + 0.5 + 0.0) / 3.0,
"columns": [
{"name": "a", "null_count": 3, "null_pct": 0.5, "n_rows": 6},
{"name": "b", "null_count": 3, "null_pct": 0.5, "n_rows": 6},
{"name": "c", "null_count": 0, "null_pct": 0.0, "n_rows": 6},
],
}
glossary = model.GlossaryCollector()
ctx = {
"raw_numeric": {
"a": [None, None, None, 1.0, 2.0, 3.0],
"b": [None, None, None, 4.0, 5.0, 6.0],
},
"glossary": glossary,
}
ch = build_missingness(profile, ctx)
assert ch is not None
assert ch.id == "missingness"
assert ch.blocks
titles = _titles(ch)
headings = {t for (k, t) in titles if k == "heading"}
# Core sections present.
assert any("Cuánto y dónde" in h for h in headings)
assert any("Faltantes por columna" in h for h in headings)
assert any("Co-ocurrencia" in h for h in headings)
assert any("MCAR" in h for h in headings)
# A summary KVTable, a ranking DataTable, a co-occurrence figure and the
# pairs table all exist.
kinds = {k for (k, _) in titles}
assert "kv_table" in kinds
assert "data_table" in kinds
assert "figure" in kinds
# Glossary terms registered.
keys = {t["key"] for t in glossary.terms()}
assert {"missingness", "mcar", "mar"} <= keys
# The MCAR/MAR note reads the co-occurrence; with a perfect overlap it must
# flag the non-random (MAR) reading.
text = _all_text(ch)
assert "MAR" in text
def test_db_pushdown_covers_categorical_column(tmp_path):
"""The is-null mask push-down must cover a categorical column, so a
categorical that co-misses with a numeric one shows up in the pairs."""
import duckdb
db = str(tmp_path / "miss.duckdb")
con = duckdb.connect(db)
con.execute("CREATE TABLE t (num1 DOUBLE, num2 DOUBLE, cat VARCHAR)")
# num1 and cat are NULL together in the first 4 of 10 rows; num2 never null.
rows = []
for i in range(10):
if i < 4:
rows.append((None, float(i), None))
else:
rows.append((float(i), float(i), f"c{i}"))
con.executemany("INSERT INTO t VALUES (?,?,?)", rows)
con.close()
profile = {
"n_rows": 10,
"null_cell_pct": (0.4 + 0.0 + 0.4) / 3.0,
"columns": [
{"name": "num1", "null_count": 4, "null_pct": 0.4, "n_rows": 10},
{"name": "num2", "null_count": 0, "null_pct": 0.0, "n_rows": 10},
{"name": "cat", "null_count": 4, "null_pct": 0.4, "n_rows": 10},
],
}
ctx = {"db_path": db, "table": "t", "glossary": model.GlossaryCollector()}
ch = build_missingness(profile, ctx)
assert ch is not None
# The pairs table must mention both num1 and cat (they co-miss perfectly),
# which is only possible if the mask covered the categorical column.
text = _all_text(ch)
assert "num1" in text and "cat" in text
# Co-occurrence section + a pairs data table exist.
titles = _titles(ch)
assert any("co-faltan" in (t or "").lower() for (k, t) in titles)
@@ -6,15 +6,16 @@ normality}``). It renders, as structured markdown/tables/figures that the core
paginator never cuts:
1. **Normalization note** — every multivariate model below standardizes the
columns with z-score first; the chapter explains why (different scales would
otherwise dominate distance/variance).
columns with z-score first (the term is marked clickable; its definition
lives in the GLOSARIO chapter, not inline).
2. **PCA** — a scree plot (explained + cumulative variance, single Y axis) plus
variance and top-loadings tables.
3. **KMeans segments** — a PCA scatter **coloured by cluster** (its own
page/slide), the cluster-size table, and a per-cluster LLM micro-analysis
with a title for each segment.
4. **Isolation Forest outliers** — a short explanation of how anomalous rows are
isolated multivariately and how the threshold is chosen, plus the counts.
4. **Isolation Forest outliers** — the multivariate anomaly counts and decision
threshold (the method is marked clickable; its definition lives in the
GLOSARIO chapter, not inline).
5. **Normality** — per-column Jarque-Bera / D'Agostino / Shapiro verdicts.
The raw numeric data needed to colour the cluster scatter is **not** in the
@@ -55,6 +56,62 @@ _CLUSTER_COLORS = [
"#edc948", "#b07aa1", "#ff9da7", "#9c755f", "#bab0ac",
]
# Glossary terms this chapter explains. Each is registered in the shared
# collector (ctx['glossary']) and marked clickable on its first appearance — the
# canonical two-step pattern (see ``cat_distr``): ``glossary.add(key, label,
# definition)`` + the inline span ``[[term:KEY]]texto[[/term]]`` in a Markdown
# block. A term is registered only when its section is actually rendered, so the
# glossary never lists an entry no in-text appearance points to.
_TERM_DEFS = {
"zscore": (
"Estandarización z-score",
"Transformación que lleva cada columna numérica a media 0 y desviación "
"típica 1: a cada valor le resta la media de su columna y lo divide por "
"la desviación típica. Así variables con escalas muy distintas (euros "
"frente a un ratio 01) pesan por igual en las distancias y la varianza."),
"pca": (
"PCA (componentes principales)",
"El análisis de componentes principales resume muchas variables "
"numéricas correlacionadas en pocos ejes nuevos (componentes), "
"ortogonales entre sí y ordenados por la cantidad de varianza que "
"capturan. Permite ver la estructura de los datos en 2D y saber cuántas "
"dimensiones bastan para explicarlos."),
"kmeans": (
"KMeans (segmentación)",
"Algoritmo de agrupamiento no supervisado que reparte las filas en k "
"segmentos: asigna cada fila al centro (centroide) más cercano y recoloca "
"los centroides de forma iterativa hasta minimizar la distancia interna "
"de cada grupo. Aquí k se elige automáticamente."),
"silhouette": (
"Coeficiente de silueta (silhouette)",
"Métrica de calidad de un agrupamiento, en el rango 1 a 1: para cada "
"fila compara cómo de cerca está de su propio segmento frente al segmento "
"vecino más próximo. Cuanto más alto el promedio, más compactos y "
"separados están los segmentos."),
"isolation_forest": (
"Isolation Forest (anomalías)",
"Algoritmo de detección de anomalías multivariante: construye árboles que "
"parten el espacio con cortes aleatorios y mide cuántos cortes hacen "
"falta para aislar cada fila. Las filas raras se aíslan con muy pocos "
"cortes y se marcan como outliers según un umbral de contaminación."),
}
def _term(mark: bool, key: str, text: str) -> str:
"""Wrap ``text`` as a clickable glossary span when ``mark`` is True.
The visible text is identical with or without the marker (the renderers strip
it), so wrapping never changes line layout — it only adds the link.
"""
return f"[[term:{key}]]{text}[[/term]]" if mark else text
def _register(gloss, key: str) -> None:
"""Register term ``key`` in the collector (idempotent); no-op if gloss None."""
if gloss is not None:
label, definition = _TERM_DEFS[key]
gloss.add(key, label, definition)
# --------------------------------------------------------------------------- #
# Formatting helpers (mirror the overview chapter's defensive style).
@@ -252,34 +309,33 @@ def _make_cluster_scatter(projection: dict):
# --------------------------------------------------------------------------- #
# Section builders. Each returns a list of blocks (possibly empty).
# --------------------------------------------------------------------------- #
def _normalization_intro() -> list:
def _normalization_intro(gloss=None, mark_term: bool = False) -> list:
_register(gloss, "zscore")
zscore = _term(mark_term, "zscore", "**estandarizan con z-score**")
text = (
"Estos modelos son **no supervisados**: buscan estructura latente sin "
"una variable objetivo. Antes de aplicarlos, todas las columnas "
"numéricas se **estandarizan con z-score** (cada valor menos la media, "
"dividido por la desviación típica). Sin esta normalización, una "
"variable con escala grande (p.ej. ingresos en euros) dominaría las "
"distancias y la varianza frente a otra de escala pequeña (p.ej. un "
"ratio entre 0 y 1), sesgando tanto el PCA como el KMeans. Tras la "
"estandarización todas las variables pesan por igual."
f"numéricas se {zscore}, para que todas pesen por igual con "
"independencia de su escala."
)
return [model.Heading(text="Modelos no supervisados", level=1),
model.Markdown(text=text)]
def _pca_section(pca: dict) -> list:
def _pca_section(pca: dict, gloss=None, mark_term: bool = False) -> list:
if not _is_dict(pca) or not pca.get("explained_variance_ratio"):
return []
_register(gloss, "pca")
blocks = [model.Heading(text="PCA — varianza explicada", level=2)]
n_used = pca.get("n_rows_used")
n_feat = pca.get("n_features")
intro = (
f"El PCA resume {_fmt_num(n_feat)} variables numéricas en componentes "
f"ortogonales ordenados por la varianza que capturan "
f"({_fmt_num(n_used)} filas usadas tras eliminar nulos). El gráfico de "
"sedimentación (scree) muestra cuánta varianza aporta cada componente y "
"su acumulado: un codo marca cuántos componentes bastan."
f"El {_term(mark_term, 'pca', 'PCA')} se aplica sobre "
f"{_fmt_num(n_feat)} variables numéricas ({_fmt_num(n_used)} filas "
"usadas tras eliminar nulos). El gráfico de sedimentación (scree) "
"muestra cuánta varianza aporta cada componente y su acumulado: un "
"codo marca cuántos componentes bastan."
)
blocks.append(model.Markdown(text=intro))
@@ -325,11 +381,14 @@ def _pca_section(pca: dict) -> list:
return blocks
def _kmeans_section(kmeans: dict, projection: dict, titles) -> list:
def _kmeans_section(kmeans: dict, projection: dict, titles,
gloss=None, mark_term: bool = False) -> list:
has_km = _is_dict(kmeans) and kmeans.get("best_k")
has_proj = _is_dict(projection) and projection.get("points")
if not has_km and not has_proj:
return []
_register(gloss, "kmeans")
_register(gloss, "silhouette")
blocks = [model.Heading(text="Segmentación (KMeans)", level=2)]
@@ -337,11 +396,12 @@ def _kmeans_section(kmeans: dict, projection: dict, titles) -> list:
sil = (projection or {}).get("silhouette")
if sil is None:
sil = (kmeans or {}).get("silhouette")
t_kmeans = _term(mark_term, "kmeans", "KMeans")
t_sil = _term(mark_term, "silhouette", "*silhouette*")
intro = (
f"KMeans agrupa las filas en **{_fmt_num(best_k)} segmentos** elegidos "
"automáticamente maximizando el coeficiente de *silhouette* "
f"(**{_fmt_num(sil)}**, rango 1 a 1: cuanto más alto, segmentos más "
"compactos y separados). Los segmentos se proyectan sobre el plano de "
f"{t_kmeans} agrupa las filas en **{_fmt_num(best_k)} segmentos** "
f"elegidos automáticamente por el coeficiente de {t_sil} "
f"(**{_fmt_num(sil)}**). Los segmentos se proyectan sobre el plano de "
"los dos primeros componentes principales para visualizarlos."
)
blocks.append(model.Markdown(text=intro))
@@ -394,23 +454,21 @@ def _kmeans_section(kmeans: dict, projection: dict, titles) -> list:
return blocks
def _outliers_section(outliers: dict) -> list:
def _outliers_section(outliers: dict, gloss=None, mark_term: bool = False) -> list:
if not _is_dict(outliers) or outliers.get("n_outliers") is None:
return []
if outliers.get("note") and not outliers.get("n_rows_used"):
# insufficient data — nothing meaningful to show.
return []
_register(gloss, "isolation_forest")
blocks = [model.Heading(text="Detección de anomalías (Isolation Forest)",
level=2)]
isof = _term(mark_term, "isolation_forest", "**Isolation Forest**")
explain = (
"**Isolation Forest** detecta filas anómalas de forma *multivariante*: "
"construye árboles que parten el espacio con cortes aleatorios y mide "
"cuántos cortes hacen falta para aislar cada fila. Las filas raras "
"(combinaciones de valores poco frecuentes considerando **todas las "
"columnas a la vez**, no una sola) se aíslan con muy pocos cortes y "
"obtienen un score bajo. El **umbral** de decisión separa las filas "
"normales de las anómalas según la contaminación esperada del modelo: "
"una fila es outlier cuando su score queda por debajo de ese umbral."
f"{isof} marca filas anómalas de forma *multivariante*: combinaciones "
"de valores poco frecuentes considerando **todas las columnas a la "
"vez**, no una sola. La tabla resume cuántas se detectaron y el umbral "
"de decisión empleado."
)
blocks.append(model.Markdown(text=explain))
blocks.append(model.KVTable(rows=[
@@ -484,15 +542,21 @@ def build_modelos(profile: dict, ctx: dict):
(kmeans and kmeans.get("best_k")) or (projection and projection.get("points"))
) else None
# Shared glossary collector: terms are registered + marked clickable inside
# each section, only when that section actually renders (no orphan entries).
glossary = ctx.get("glossary")
gloss = glossary if isinstance(glossary, model.GlossaryCollector) else None
mark_term = gloss is not None
sections = []
sections += _pca_section(pca) if pca else []
sections += _kmeans_section(kmeans, projection, titles)
sections += _outliers_section(outliers) if outliers else []
sections += _pca_section(pca, gloss, mark_term) if pca else []
sections += _kmeans_section(kmeans, projection, titles, gloss, mark_term)
sections += _outliers_section(outliers, gloss, mark_term) if outliers else []
sections += _normality_section(normality) if normality else []
if not sections:
return None # models block present but nothing renderable.
blocks = _normalization_intro() + sections
blocks = _normalization_intro(gloss, mark_term) + sections
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -257,3 +257,26 @@ def test_anticortes_tabla_normalidad_larga_no_corta():
# Every column name survives (wrapped/split, never truncated).
for i in (0, 19, 39):
assert f"col_{i}" in txt
def test_glosario_engancha_terminos_modelos():
"""Mejora 4b: PCA, KMeans, silhouette, Isolation Forest y la estandarización
z-score se registran en el colector compartido y se marcan clicables en el
cuerpo. Sin colector en ctx, el capítulo degrada y no marca nada."""
from datascience.automatic_eda.model import GlossaryCollector
g = GlossaryCollector()
ctx = dict(_ctx_full())
ctx["glossary"] = g
ch = build_modelos(_profile(), ctx)
assert ch is not None
keys = {t["key"] for t in g.terms()}
assert {"zscore", "pca", "kmeans", "silhouette", "isolation_forest"} <= keys
body = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
for k in ("zscore", "pca", "kmeans", "silhouette", "isolation_forest"):
assert f"[[term:{k}]]" in body, k
# Sin colector: degrada limpio (ningún marcador en el cuerpo).
ch2 = build_modelos(_profile(), _ctx_full())
body2 = " ".join(b.text for b in ch2.blocks if b.kind == "markdown")
assert "[[term:" not in body2
@@ -1,9 +1,10 @@
"""Numeric distributions chapter (NUM DISTR) for AutomaticEDA.
For every numeric column the chapter draws, as a single indivisible figure, a
histogram with the **mean, median and ±1σ band drawn as reference lines** and a
**Tukey boxplot right below it** sharing the same X axis — exactly the user
requirement for this chapter. Each figure is emitted as a lazy ``Figure`` block
histogram with the **mean, median and ±1σ band drawn as reference lines** (the
legend reports the numeric value of the mean, the median **and the standard
deviation σ**) and a **Tukey boxplot right below it** sharing the same X axis —
exactly the user requirement for this chapter. Each figure is emitted as a lazy ``Figure`` block
so the renderers rasterize and scale it to fit a whole page/slide and nothing is
ever cut; columns with many numerics simply flow across pages as small
multiples.
@@ -34,7 +35,7 @@ try:
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
build_boxplot_stats = None # type: ignore[assignment]
CHAPTER_VERSION = "1.0.0"
CHAPTER_VERSION = "1.2.0"
CHAPTER_ID = "num_distr"
CHAPTER_TITLE = "Distribuciones numéricas"
@@ -140,9 +141,11 @@ def _make_hist_box(name: str, numeric: dict, box: dict):
std = numeric.get("std")
# ±1σ band first (behind the lines), then median (solid) and mean (dashed).
# The band's legend entry also reports the numeric value of the standard
# deviation, so the reader sees mean, median AND σ at a glance.
if mean is not None and std is not None and std > 0:
ax_h.axvspan(mean - std, mean + std, color="#f0c27b", alpha=0.22,
zorder=1, label="±1σ")
zorder=1, label=f"±1σ (σ = {_fmt_num(std)})")
if median is not None:
ax_h.axvline(median, color="#2e8b57", linestyle="-", linewidth=1.6,
zorder=4, label=f"mediana = {_fmt_num(median)}")
@@ -152,7 +155,19 @@ def _make_hist_box(name: str, numeric: dict, box: dict):
ax_h.set_ylabel("frecuencia", fontsize=8)
ax_h.tick_params(labelsize=7)
ax_h.legend(fontsize=6.5, loc="upper right", framealpha=0.85)
# Always surface σ in the legend: if the ±1σ band could not be drawn (no mean
# or std<=0) but σ is still known, add a label-only proxy handle so the value
# of the standard deviation is reported regardless of the band.
handles, labels = ax_h.get_legend_handles_labels()
if std is not None and not any("σ =" in lbl for lbl in labels):
from matplotlib.lines import Line2D
proxy = Line2D([], [], linestyle="none", marker="",
label=f"σ = {_fmt_num(std)}")
handles.append(proxy)
labels.append(f"σ = {_fmt_num(std)}")
if handles:
ax_h.legend(handles, labels, fontsize=6.5, loc="upper right",
framealpha=0.85)
for spine in ("top", "right"):
ax_h.spines[spine].set_visible(False)
@@ -278,12 +293,17 @@ def build_num_distr(profile: dict, ctx: dict):
box = build_boxplot_stats(numeric) or {}
except Exception: # noqa: BLE001 — degrade, never raise.
box = {}
blocks.append(model.Heading(text=str(name), level=2))
blocks.append(model.Figure(
make=_figure_maker(name, numeric, box),
caption=f"Distribución de «{name}» — histograma (media/mediana/±σ) "
f"y boxplot."))
blocks.append(model.Markdown(text=_stats_note(name, numeric, box)))
# Keep the column heading, its figure and its stats note together on the
# same page/slide (mejora 3 — keep-together): the renderers measure the
# whole Group and move it whole when it would not fit.
blocks.append(model.Group(blocks=[
model.Heading(text=str(name), level=2),
model.Figure(
make=_figure_maker(name, numeric, box),
caption=f"Distribución de «{name}» — histograma "
f"(media/mediana/±σ) y boxplot."),
model.Markdown(text=_stats_note(name, numeric, box)),
]))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -65,19 +65,33 @@ def _pdf_text(path: str) -> str:
return re.sub(r"\s+", " ", txt)
def _flatten(blocks):
"""Expand keep-together Groups so the per-column heading/figure/markdown are
inspectable as a flat block list (the chapter wraps each column in a Group)."""
out = []
for b in blocks:
if getattr(b, "kind", "") == "group":
out.extend(_flatten(getattr(b, "blocks", []) or []))
else:
out.append(b)
return out
def test_golden_chapter_estructura_y_bloques():
ch = build_num_distr(_profile(n_numeric=2), {})
assert ch is not None
assert ch.id == "num_distr"
assert ch.version == CHAPTER_VERSION
kinds = [b.kind for b in ch.blocks]
# Per-column blocks are wrapped in keep-together Groups: flatten to inspect.
flat = _flatten(ch.blocks)
kinds = [b.kind for b in flat]
# Heading + intro Markdown, then per column: Heading + Figure + Markdown.
assert kinds[0] == "heading"
assert kinds[1] == "markdown"
assert kinds.count("figure") == 2 # one figure per numeric column.
assert kinds.count("heading") == 1 + 2 # chapter title + one per column.
# Each figure has a lazy maker that produces a real matplotlib figure.
figs = [b for b in ch.blocks if b.kind == "figure"]
figs = [b for b in flat if b.kind == "figure"]
fig = figs[0].make()
assert fig is not None
# Two stacked axes: histogram + boxplot share the figure.
@@ -90,7 +104,8 @@ def test_golden_media_mediana_sigma_y_boxplot_presentes():
# The intro documents the three reference lines and the Tukey boxplot; the
# per-column note carries the actual mean/median/σ numbers and the shape.
ch = build_num_distr(_profile(n_numeric=1, extra_categorical=False), {})
md_texts = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
md_texts = " ".join(b.text for b in _flatten(ch.blocks)
if b.kind == "markdown")
assert "media" in md_texts and "mediana" in md_texts
assert "±1σ" in md_texts or "σ" in md_texts
assert "boxplot" in md_texts.lower()
@@ -126,7 +141,8 @@ def test_anti_corte_muchas_columnas_pdf_y_pptx():
# 8 numeric columns + long note text: nothing may be cut. Every column
# heading must survive in both the PDF text and the PPTX deck.
ch = build_num_distr(_profile(n_numeric=8), {})
names = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
names = [b.text for b in _flatten(ch.blocks)
if b.kind == "heading" and b.level == 2]
assert len(names) == 8
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "num.pdf")
@@ -143,6 +159,50 @@ def test_anti_corte_muchas_columnas_pdf_y_pptx():
assert res_pptx["n_slides"] >= 8 # at least one slide per column figure.
def _hist_legend_texts(numeric, box=None):
"""Build the per-column figure and return its histogram-legend label texts."""
from datascience.automatic_eda.chapters.num_distr import _make_hist_box
import matplotlib.pyplot as plt
fig = _make_hist_box("col", numeric, box or {})
ax_h = fig.axes[0] # the histogram is the top axis.
leg = ax_h.get_legend()
texts = [t.get_text() for t in leg.get_texts()] if leg else []
plt.close(fig)
return texts
def test_golden_leyenda_histograma_reporta_valor_std():
# The histogram legend must report the numeric value of the standard
# deviation σ next to mean and median.
numeric = _numeric_block(42.5, 40.0, 12.3, 1.0, 100.0, "right-skewed", 5)
texts = _hist_legend_texts(numeric)
joined = " ".join(texts)
assert any("σ =" in t for t in texts), f"σ value missing in legend: {texts}"
assert "12.3" in joined, f"std value 12.3 not in legend: {texts}"
assert any("media =" in t for t in texts)
assert any("mediana =" in t for t in texts)
def test_edge_std_en_leyenda_aunque_no_haya_banda():
# When the ±1σ band cannot be drawn (no mean) but σ is known, the legend
# still surfaces the σ value via a label-only proxy handle.
numeric = _numeric_block(42.5, 40.0, 7.5, 1.0, 100.0, "right-skewed", 0)
numeric["mean"] = None # forces the band off; σ must still appear.
texts = _hist_legend_texts(numeric)
assert any("σ = 7.5" in t for t in texts), f"σ proxy missing: {texts}"
def test_edge_sin_std_no_revienta_la_figura():
# A numeric block without σ must not raise and simply omits the σ entry.
import matplotlib.pyplot as plt
numeric = _numeric_block(42.5, 40.0, 0.0, 1.0, 100.0, "discrete", 0)
numeric["std"] = None
texts = _hist_legend_texts(numeric)
assert not any("σ =" in t for t in texts)
# mean/median lines still produce their own legend entries.
assert any("media =" in t for t in texts)
def test_distribution_gloss_cubre_todas_las_etiquetas():
# Every label detect_distribution_type can emit has a Spanish gloss.
for label in ("normal-ish", "right-skewed", "left-skewed", "heavy-tail",
@@ -20,7 +20,7 @@ from __future__ import annotations
from .. import model
CHAPTER_VERSION = "1.0.0"
CHAPTER_VERSION = "1.1.0"
CHAPTER_ID = "overview"
CHAPTER_TITLE = "Overview"
@@ -90,8 +90,14 @@ def _head_block(profile: dict, ctx: dict):
if not cols:
cols = list(head[0].keys())
rows = [[model._safe_str(r.get(c)) for c in cols] for r in head[:10]]
return model.DataTable(header=cols, rows=rows,
note=f"primeras {len(rows)} filas")
# Honest note: how many rows are shown and, when known, out of how many
# rows the dataset has (so "primeras 10 filas de 891" gives context).
note = f"primeras {len(rows)} filas"
n_rows = profile.get("n_rows")
if isinstance(n_rows, int) and not isinstance(n_rows, bool) \
and n_rows > len(rows):
note += f" de {n_rows:,}".replace(",", ".")
return model.DataTable(header=cols, rows=rows, note=note)
return model.Note(
"df.head no disponible: el TableProfile no incluye 'head_rows'. La fase "
"de cálculo debe añadir profile['head_rows'] (lista de dicts fila) o "
@@ -0,0 +1,187 @@
"""Tests for the OVERVIEW chapter — DoD: golden + edges + degradation.
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
and deterministic. Verifies that ``build_overview`` renders the raw first rows
(``df.head``) as a DataTable when ``head_rows`` is present — both when it arrives
via ``profile['head_rows']`` (populated by ``profile_table``) and via
``ctx['head_rows']`` (populated by ``build_eda_render_ctx``) — that the chapter
also renders the column dictionary and the numeric describe, that the full
document renders to PDF and PPTX showing the head values, and that a profile with
NO head data degrades to an honest note instead of raising or inventing rows.
"""
import os
import re
import tempfile
from pypdf import PdfReader
from pptx import Presentation
from datascience.automatic_eda.model import DataTable, Note
from datascience.automatic_eda.chapters.overview import (
CHAPTER_ID, CHAPTER_VERSION, build_overview,
)
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
def _columns() -> list:
return [
{"name": "PassengerId", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0, "numeric": {"mean": 2.0, "median": 2.0, "min": 1.0,
"max": 3.0, "std": 1.0}},
{"name": "Survived", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0, "numeric": {"mean": 0.33, "median": 0.0, "min": 0.0,
"max": 1.0, "std": 0.58}},
{"name": "Pclass", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0, "numeric": {"mean": 2.33, "median": 3.0, "min": 1.0,
"max": 3.0, "std": 1.15}},
{"name": "Name", "inferred_type": "categorical", "null_pct": 0.0,
"null_count": 0, "distinct_count": 3},
{"name": "Sex", "inferred_type": "categorical", "null_pct": 0.0,
"null_count": 0, "distinct_count": 2,
"categorical": {"top": [{"value": "male", "count": 2},
{"value": "female", "count": 1}]}},
]
def _head_rows() -> list:
return [
{"PassengerId": 1, "Survived": 0, "Pclass": 3,
"Name": "Braund Owen", "Sex": "male"},
{"PassengerId": 2, "Survived": 1, "Pclass": 1,
"Name": "Cumings Florence", "Sex": "female"},
{"PassengerId": 3, "Survived": 1, "Pclass": 3,
"Name": "Heikkinen Laina", "Sex": "female"},
]
def _profile(with_head: bool = True) -> dict:
prof = {
"table": "titanic",
"source": "/data/titanic.csv",
"profiled_at": "2026-06-30T10:00:00+00:00",
"n_rows": 891,
"n_cols": 5,
"quality_score": 88.0,
"columns": _columns(),
}
if with_head:
prof["head_rows"] = _head_rows()
return prof
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def _pptx_text(path: str) -> str:
prs = Presentation(path)
parts = []
for sl in prs.slides:
for sh in sl.shapes:
if sh.has_text_frame:
parts.append(sh.text_frame.text)
if sh.has_table:
tb = sh.table
for r in range(len(tb.rows)):
for c in range(len(tb.columns)):
parts.append(tb.cell(r, c).text)
return re.sub(r"\s+", " ", " ".join(parts))
def _flatten(blocks):
"""Recursively flatten Group blocks into a flat list (none here today)."""
out = []
for b in blocks:
inner = getattr(b, "blocks", None)
if inner is not None and getattr(b, "kind", None) == "group":
out.extend(_flatten(inner))
else:
out.append(b)
return out
def test_golden_build_overview_muestra_head_desde_profile():
ch = build_overview(_profile(), {})
assert ch is not None
assert ch.id == CHAPTER_ID
assert ch.version == CHAPTER_VERSION
blocks = _flatten(ch.blocks)
# The first DataTable is df.head: its header is the column names and the
# real first rows are present (not a placeholder note).
tables = [b for b in blocks if isinstance(b, DataTable)]
assert tables, "overview must emit at least the df.head DataTable"
head_tbl = tables[0]
assert head_tbl.header == ["PassengerId", "Survived", "Pclass",
"Name", "Sex"]
assert len(head_tbl.rows) == 3
flat = [str(c) for row in head_tbl.rows for c in row]
assert "Braund Owen" in flat and "Cumings Florence" in flat
# Honest note carries how many rows shown out of the dataset total.
assert head_tbl.note is not None
assert "primeras 3 filas" in head_tbl.note and "891" in head_tbl.note
# No "df.head no disponible" placeholder when head_rows is present.
assert not any(isinstance(b, Note) and "no disponible" in b.text
for b in blocks)
def test_golden_head_desde_ctx_tambien_funciona():
# head_rows absent in profile but present in ctx (build_eda_render_ctx path).
prof = _profile(with_head=False)
ch = build_overview(prof, {"head_rows": _head_rows()})
assert ch is not None
tables = [b for b in _flatten(ch.blocks) if isinstance(b, DataTable)]
flat = [str(c) for row in tables[0].rows for c in row]
assert "Braund Owen" in flat
def test_golden_render_pdf_muestra_head():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pdf")
res = render_automatic_eda_pdf(_profile(), out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
txt = _pdf_text(out)
assert "Braund" in txt and "male" in txt
assert "primeras" in txt # head note rendered.
assert "df.head" in txt # chapter heading rendered.
assert "no disponible" not in txt # placeholder NOT shown.
def test_golden_render_pptx_muestra_head():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pptx")
res = render_automatic_eda_pptx(_profile(), out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
txt = _pptx_text(out)
assert "Braund" in txt and "Cumings" in txt
def test_edge_sin_head_rows_degrada_a_nota_honesta():
# No head data anywhere: chapter still builds (columns exist), shows the
# honest placeholder note, and never invents rows nor raises.
prof = _profile(with_head=False)
ch = build_overview(prof, {})
assert ch is not None
blocks = _flatten(ch.blocks)
assert any(isinstance(b, Note) and "no disponible" in b.text
for b in blocks)
# The first DataTable now is the column dictionary, not df.head rows.
tables = [b for b in blocks if isinstance(b, DataTable)]
assert all("Braund" not in str(c)
for tbl in tables for row in tbl.rows for c in row)
def test_edge_none_y_vacio_no_rompen():
# Nothing to render at all -> None, no raise.
assert build_overview(None, None) is None
assert build_overview({}, {}) is None
assert build_overview({"columns": []}, {}) is None
# Only head_rows (no columns) still yields a chapter with the head table.
ch = build_overview({"columns": []}, {"head_rows": _head_rows()})
assert ch is not None
tables = [b for b in _flatten(ch.blocks) if isinstance(b, DataTable)]
assert tables and len(tables[0].rows) == 3
@@ -2,8 +2,17 @@
Builds the document cover from a TableProfile plus an optional ``ctx`` of
presentation metadata. Reads everything defensively (``.get``) and degrades
honestly: a field that is neither in the profile nor in ``ctx`` is shown as a
placeholder rather than invented, leaving a hook for the LLM layer to fill it.
honestly.
The dataset size (N rows x M columns) is always shown big, as a heading right
under the dataset name (kept together in a ``Group``), not buried in the
metadata table. The Description and Granularity are resolved through a cascade
so they are never empty: an explicit ``ctx`` value wins; otherwise the LLM block
(``profile['llm']`` from ``eda_llm_insights``) provides ``summary`` /
``row_meaning``; otherwise a short summary is derived from the profile itself
(shape, column-type mix, quality score) and a "Cada fila es…" sentence from the
key-candidate columns or the table shape. Nothing is invented: the derived
fallbacks state that they come from the profile.
Contract for chapter authors (see ``docs/capabilities/automatic_eda.md``):
build_<id>(profile: dict, ctx: dict) -> Chapter | None
@@ -17,10 +26,15 @@ from datetime import datetime, timezone
from .. import model
CHAPTER_VERSION = "1.0.0"
CHAPTER_VERSION = "1.2.0"
CHAPTER_ID = "portada"
CHAPTER_TITLE = "Portada"
# Key under which eda_llm_insights stores its interpretive block in the profile.
# The cover reads ``summary`` (what the table is) and ``row_meaning`` (what one
# row represents) from it when the LLM layer ran (``run_llm``).
_LLM_KEY = "llm"
# Default human description of what the table quality score measures. Chapters
# can override it via ctx["quality_criteria"].
_DEFAULT_QUALITY_CRITERIA = (
@@ -67,6 +81,53 @@ def _fmt_int(v) -> str:
return str(v)
def _fmt_pct(value) -> str:
"""Format a percentage that may arrive as a 01 fraction or a 0100 number."""
if value is None:
return ""
try:
v = float(value)
except (TypeError, ValueError):
return str(value)
if 0 < v <= 1.0:
v *= 100.0
return f"{v:.1f}%"
def _summary_blocks(summary) -> list:
"""Mini-summary of the rest of the analysis, shown on the cover (mejora 5).
The cover is built AFTER the body (``build_document`` passes the aggregated
``ctx['document_summary']``), so it can reflect what the analysis found:
shape, column types, quality flags and which chapters were included. Returns
an empty list when there is no summary (the cover degrades to its metadata
table only)."""
if not isinstance(summary, dict) or not summary:
return []
rows = []
n_num = summary.get("n_numeric")
n_cat = summary.get("n_categorical")
if n_num is not None or n_cat is not None:
rows.append(("Columnas numéricas / categóricas",
f"{_fmt_int(n_num)} / {_fmt_int(n_cat)}"))
if summary.get("duplicate_pct") is not None:
rows.append(("Filas duplicadas", _fmt_pct(summary.get("duplicate_pct"))))
if summary.get("null_cell_pct") is not None:
rows.append(("Celdas nulas", _fmt_pct(summary.get("null_cell_pct"))))
titles = summary.get("chapter_titles") or []
if titles:
rows.append(("Capítulos del informe", _fmt_int(len(titles))))
blocks = [model.Heading(text="Resumen del análisis", level=2)]
if rows:
blocks.append(model.KVTable(rows=rows))
if titles:
bullets = "\n".join(f"- {model._safe_str(t)}" for t in titles)
blocks.append(model.Markdown(
text="Este informe incluye los siguientes capítulos:\n" + bullets))
return blocks
def _fmt_date_eu(value) -> str:
"""Format a date/ISO string as European DD/MM/AAAA HH:mm (UI convention).
@@ -95,6 +156,88 @@ def _fmt_date_eu(value) -> str:
return s
def _llm_block(profile: dict, ctx: dict) -> dict:
"""Return the interpretive LLM block (``eda_llm_insights`` output), or {}.
It is stored under ``profile['llm']`` by ``profile_table(run_llm=True)`` and
may also be forwarded in ``ctx['llm']``. Read defensively: anything that is
not a dict degrades to an empty dict so the cover never raises.
"""
block = profile.get(_LLM_KEY)
if not isinstance(block, dict):
block = ctx.get(_LLM_KEY)
return block if isinstance(block, dict) else {}
def _count_column_types(profile: dict, ctx: dict):
"""Best-effort (n_numeric, n_categorical) for the dataset.
Prefers the aggregated ``ctx['document_summary']`` (computed by the engine
over the whole body); falls back to counting the profile columns directly so
the cover still has the numbers when no summary was passed.
"""
summary = ctx.get("document_summary")
if isinstance(summary, dict):
n_num = summary.get("n_numeric")
n_cat = summary.get("n_categorical")
if n_num is not None or n_cat is not None:
return n_num, n_cat
cols = profile.get("columns") or []
n_num = sum(1 for c in cols if isinstance(c, dict)
and c.get("inferred_type") == "numeric")
n_cat = sum(1 for c in cols if isinstance(c, dict)
and isinstance(c.get("categorical"), dict)
and c.get("categorical", {}).get("top")
and c.get("inferred_type") != "numeric")
return n_num, n_cat
def _derive_description(profile: dict, ctx: dict) -> str:
"""A short, honest description of the dataset from the profile.
Used only when no explicit ``ctx['description']`` and no LLM ``summary`` are
available. Summarizes shape, column-type mix and quality score; never empty,
never invents business meaning (it states the description was derived)."""
n_rows = profile.get("n_rows")
n_cols = profile.get("n_cols")
n_num, n_cat = _count_column_types(profile, ctx)
head = f"Conjunto de datos con {_fmt_int(n_rows)} filas y {_fmt_int(n_cols)} columnas"
type_bits = []
if n_num:
type_bits.append(f"{_fmt_int(n_num)} numéricas")
if n_cat:
type_bits.append(f"{_fmt_int(n_cat)} categóricas")
if type_bits:
head += " (" + ", ".join(type_bits) + ")"
parts = [head + "."]
score = profile.get("quality_score")
if score is not None:
parts.append(f"Calidad media estimada: {score}/100.")
parts.append(
"Resumen derivado del perfil; active la interpretación LLM (`run_llm`) "
"para una descripción de negocio más rica.")
return " ".join(parts)
def _derive_granularity(profile: dict, dataset_name: str) -> str:
"""A ``Cada fila es…`` granularity sentence from the profile.
Prefers the key-candidate columns (a row is identified by them); when no key
is detected, falls back to the table shape so the line is always meaningful
and starts with ``Cada fila es`` as the user requested."""
keys = profile.get("key_candidates") or []
if keys:
shown = ", ".join(str(k) for k in keys[:3])
more = "" if len(keys) <= 3 else f" (y {len(keys) - 3} más)"
return (f"Cada fila es un registro identificado por {shown}{more}, "
"candidata(s) a clave por ser únicas y sin nulos.")
n_rows = profile.get("n_rows")
tail = f" El dataset tiene {_fmt_int(n_rows)} filas en total." if n_rows else ""
return (f"Cada fila es un registro de «{dataset_name}». No se detectó una "
"columna identificadora única, así que la granularidad se infiere "
"de la forma de la tabla." + tail)
def build_portada(profile: dict, ctx: dict):
"""Build the cover Chapter, or None if there is truly nothing to show."""
profile = profile or {}
@@ -119,30 +262,38 @@ def build_portada(profile: dict, ctx: dict):
quality_criteria = ctx.get("quality_criteria") or _DEFAULT_QUALITY_CRITERIA
quality_value = "" if score is None else f"{score} / 100"
# Granularity: ctx wins; else derive from key candidates; else be honest.
llm = _llm_block(profile, ctx)
# Granularity: explicit ctx wins; then the LLM "row_meaning"; then the key
# candidates; finally a shape-based fallback. Always a real "Cada fila es…".
granularity = ctx.get("granularity")
if not granularity:
keys = profile.get("key_candidates") or []
if keys:
granularity = ("Cada fila parece identificada por "
+ ", ".join(str(k) for k in keys[:3]) + ".")
else:
granularity = ("Cada fila es… (granularidad no determinada — "
"pendiente de la capa de cálculo/LLM).")
granularity = (llm.get("row_meaning") or "").strip() or None
if not granularity:
granularity = _derive_granularity(profile, str(dataset_name))
# Description: explicit ctx wins; then the LLM "summary"; finally a short
# profile-derived summary. Never the old empty placeholder.
description = ctx.get("description")
if not description:
description = ("Descripción no provista — pendiente de la capa LLM "
"(`run_llm`) o de `ctx['description']`.")
description = (llm.get("summary") or "").strip() or None
if not description:
description = _derive_description(profile, ctx)
blocks = [
# Title + dataset size shown together and BIG (Heading) at the top, kept on
# the same page (Group). The size is no longer buried in the metadata table.
cover = [
model.Heading(text=str(dataset_name), level=1),
model.Markdown(text="**Automatic-EDA** · informe exploratorio automático"),
model.Heading(text=shape, level=2),
]
blocks = [
model.Group(blocks=cover),
model.KVTable(rows=[
("Fuente", source_origin),
("Almacenamiento", storage),
("Generado", when),
("Tamaño", shape),
("Calidad", quality_value),
("Criterios de calidad", quality_criteria),
]),
@@ -152,5 +303,8 @@ def build_portada(profile: dict, ctx: dict):
model.Markdown(text=str(granularity)),
]
# Mini-summary of the rest of the analysis (built last, shown on the cover).
blocks.extend(_summary_blocks(ctx.get("document_summary")))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,197 @@
"""Tests for the PORTADA (cover) chapter — DoD: golden + edges + render.
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
and deterministic. Verifies the Fase 4b improvements:
1. The dataset size (N rows x M columns) is always shown BIG — as a level-2
heading kept together with the dataset name in a ``Group`` — and is no longer
a row of the metadata table.
2. Description and Granularity are resolved through a real cascade and are never
the old empty placeholders: an explicit ``ctx`` value wins; otherwise the LLM
block (``profile['llm']``) provides ``summary`` / ``row_meaning``; otherwise a
short summary is derived from the profile and a "Cada fila es…" sentence from
the key-candidate columns or the table shape.
3. The chapter degrades without raising on empty/None input.
4. It renders inside the full document to both PDF and PPTX showing that content.
"""
import os
import re
import tempfile
from pypdf import PdfReader
from pptx import Presentation
from datascience.automatic_eda.model import Group, Heading, KVTable, Markdown
from datascience.automatic_eda.chapters.portada import (
CHAPTER_ID, CHAPTER_VERSION, build_portada,
)
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
def _profile(with_llm: bool = True, with_keys: bool = True) -> dict:
prof = {
"table": "titanic",
"source": "/data/titanic.csv",
"profiled_at": "2026-06-30T10:00:00+00:00",
"n_rows": 891,
"n_cols": 12,
"quality_score": 78.0,
"columns": [
{"name": "PassengerId", "inferred_type": "numeric",
"null_pct": 0.0, "numeric": {"mean": 446.0, "min": 1.0,
"max": 891.0, "std": 257.0}},
{"name": "Survived", "inferred_type": "numeric",
"null_pct": 0.0, "numeric": {"mean": 0.38, "min": 0.0,
"max": 1.0, "std": 0.49}},
{"name": "Sex", "inferred_type": "categorical", "null_pct": 0.0,
"categorical": {"top": [{"value": "male", "count": 577, "pct": 0.65},
{"value": "female", "count": 314,
"pct": 0.35}],
"mode": "male", "n_distinct": 2, "entropy": 0.93}},
],
}
if with_keys:
prof["key_candidates"] = ["PassengerId"]
if with_llm:
prof["llm"] = {
"summary": "Pasajeros del Titanic con su supervivencia y datos de viaje.",
"row_meaning": "Cada fila es un pasajero del Titanic.",
"dictionary": [], "pii": [], "cleaning": [], "analyses": [],
}
return prof
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
def _pptx_text(path: str) -> str:
prs = Presentation(path)
parts = []
for sl in prs.slides:
for sh in sl.shapes:
if sh.has_text_frame:
parts.append(sh.text_frame.text)
if sh.has_table:
tb = sh.table
for r in range(len(tb.rows)):
for c in range(len(tb.columns)):
parts.append(tb.cell(r, c).text)
return re.sub(r"\s+", " ", " ".join(parts))
def _markdown_after(blocks, heading_text):
"""Return the Markdown block that follows a Heading whose text matches."""
for i, b in enumerate(blocks):
if isinstance(b, Heading) and heading_text.lower() in b.text.lower():
for nb in blocks[i + 1:]:
if isinstance(nb, Markdown):
return nb
return None
def test_golden_tamano_grande_y_textos_llm():
ch = build_portada(_profile(), {})
assert ch is not None
assert ch.id == CHAPTER_ID
assert ch.version == CHAPTER_VERSION
# 1) Title + size kept together in a Group; size is a BIG level-2 heading.
group = next(b for b in ch.blocks if isinstance(b, Group))
inner = group.blocks
assert isinstance(inner[0], Heading) and inner[0].level == 1
assert inner[0].text == "titanic"
size_h = next(b for b in inner if isinstance(b, Heading) and b.level == 2)
assert "891" in size_h.text and "12" in size_h.text
assert "filas" in size_h.text and "columnas" in size_h.text
# 2) Size is no longer a row of the metadata table.
kv = next(b for b in ch.blocks if isinstance(b, KVTable))
labels = [r[0] for r in kv.rows]
assert "Tamaño" not in labels
assert "Fuente" in labels and "Calidad" in labels
# 3) Description and Granularity come from the LLM block.
desc = _markdown_after(ch.blocks, "Descripción")
gran = _markdown_after(ch.blocks, "Granularidad")
assert desc is not None and "Titanic" in desc.text
assert gran is not None and gran.text.startswith("Cada fila es")
assert "pasajero" in gran.text.lower()
def test_fallback_sin_llm_usa_keys_y_perfil():
# No LLM block: description derived from the profile, granularity from keys.
ch = build_portada(_profile(with_llm=False, with_keys=True), {})
desc = _markdown_after(ch.blocks, "Descripción")
gran = _markdown_after(ch.blocks, "Granularidad")
# Description is the derived summary, never the old "pendiente" placeholder.
assert "pendiente" not in desc.text.lower()
assert "891" in desc.text and "columnas" in desc.text
assert "numéricas" in desc.text or "categóricas" in desc.text
# Granularity mentions the key candidate and starts with "Cada fila es".
assert gran.text.startswith("Cada fila es")
assert "PassengerId" in gran.text
assert "" not in gran.text # the old ellipsis placeholder is gone.
def test_fallback_sin_llm_sin_keys_usa_forma():
ch = build_portada(_profile(with_llm=False, with_keys=False), {})
gran = _markdown_after(ch.blocks, "Granularidad")
assert gran.text.startswith("Cada fila es")
assert "titanic" in gran.text.lower()
assert "pendiente" not in gran.text.lower()
def test_ctx_explicito_gana_sobre_llm():
ctx = {"description": "Descripción manual.",
"granularity": "Cada fila es una unidad manual."}
ch = build_portada(_profile(), ctx)
desc = _markdown_after(ch.blocks, "Descripción")
gran = _markdown_after(ch.blocks, "Granularidad")
assert desc.text == "Descripción manual."
assert gran.text == "Cada fila es una unidad manual."
def test_edge_perfil_vacio_no_lanza():
# Empty / None never raise; the cover still shows a size and real texts.
for prof, ctx in (({}, {}), (None, None)):
ch = build_portada(prof, ctx)
assert ch is not None
group = next(b for b in ch.blocks if isinstance(b, Group))
size_h = next(b for b in group.blocks
if isinstance(b, Heading) and b.level == 2)
assert "filas" in size_h.text and "columnas" in size_h.text
desc = _markdown_after(ch.blocks, "Descripción")
gran = _markdown_after(ch.blocks, "Granularidad")
assert desc.text and "pendiente" not in desc.text.lower()
assert gran.text.startswith("Cada fila es")
def test_golden_render_pdf_muestra_portada():
prof = _profile()
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pdf")
res = render_automatic_eda_pdf(prof, out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
txt = _pdf_text(out)
assert "titanic" in txt.lower()
assert "891" in txt and "filas" in txt and "columnas" in txt
assert "Titanic" in txt # LLM summary in the Description.
assert "Cada fila es" in txt # granularity sentence.
def test_golden_render_pptx_muestra_portada():
prof = _profile()
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "eda.pptx")
res = render_automatic_eda_pptx(prof, out, {"title": "EDA"})
assert res["path"] == out and os.path.exists(out)
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
txt = _pptx_text(out)
assert "titanic" in txt.lower()
assert "891" in txt and "columnas" in txt
assert "Cada fila es" in txt
@@ -0,0 +1,499 @@
"""Key-relations chapter (RELACIONES) — the keys / join structure of the data.
This chapter is the *relational* section of an AutomaticEDA report. It answers a
single question for the table (or the whole DuckDB source it lives in): **how do
the keys relate?** It composes, without reimplementing them, the registry's
relation primitives and degrades honestly when a layer does not apply.
It renders, in order, only the layers that have something to say:
1. **Declared keys** (real schema constraints) when the DuckDB source declares
PRIMARY KEY / FOREIGN KEY / UNIQUE constraints, they are read verbatim via
``detect_declared_keys_duckdb`` and shown as ground truth: which column is the
PK, which columns are FKs and the table/column they point to.
2. **Primary-key candidates** the ``key_candidates`` the TableProfile already
carries (columns whose cardinality equals the row count, with no nulls). These
are *candidates*: a column that could serve as the row identifier.
3. **Foreign-key candidates** when none are declared:
- **Inter-table** (the DuckDB source has several tables): real FK candidates by
name signal + value containment via ``infer_fk_containment_duckdb``, plus the
join graph (roles + a pasteable Mermaid diagram) via ``build_join_graph``.
- **Intra-table** (a single table): columns that *look* like a foreign key by a
name+cardinality heuristic (``suggest_intratable_fk_candidates``). This is a
**suggestion**, explicitly flagged as a heuristic, never an assertion.
``build_relaciones(profile, ctx) -> Chapter | None``: returns ``None`` when there
is nothing to say (no declared key, no key candidates, and no FK candidate
inter- or intra-table). Reads everything defensively (``.get``) and never raises:
anything missing degrades to a note or is omitted; a failing registry call drops
its layer instead of aborting the chapter.
ctx keys this chapter consumes (all optional):
db_path, table : str the DuckDB file and table being profiled (set by
``build_eda_render_ctx``). ``db_path`` is needed to read declared
constraints, to list the sibling tables, and to run the containment-based
FK inference. Without it, only the profile-derived layers (PK candidates,
intra-table FK heuristic) are available.
glossary : model.GlossaryCollector shared glossary; the chapter registers
the relational terms (PK, FK, containment, cardinality) and marks their
first appearance clickable.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
from .. import model
# Pure/impure registry functions (group ``eda``) this chapter composes. Imported
# defensively (module-leaf imports, like the AGREGACION chapter) so the chapter
# still builds — degrading the affected layer to nothing — if a function is
# somehow unavailable / not indexed yet.
try:
from datascience.detect_declared_keys_duckdb import detect_declared_keys_duckdb
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
detect_declared_keys_duckdb = None # type: ignore[assignment]
try:
from datascience.infer_fk_containment_duckdb import infer_fk_containment_duckdb
except Exception: # noqa: BLE001
infer_fk_containment_duckdb = None # type: ignore[assignment]
try:
from datascience.build_join_graph import build_join_graph
except Exception: # noqa: BLE001
build_join_graph = None # type: ignore[assignment]
try:
from datascience.suggest_intratable_fk_candidates import (
suggest_intratable_fk_candidates,
)
except Exception: # noqa: BLE001
suggest_intratable_fk_candidates = None # type: ignore[assignment]
try:
from infra import duckdb_list_tables
except Exception: # noqa: BLE001
duckdb_list_tables = None # type: ignore[assignment]
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "relaciones"
CHAPTER_TITLE = "Relaciones de clave"
# Cap the inter-table FK table so a wide schema does not blow up the page; the
# rest is summarized in a closing note (no silent truncation).
MAX_FK_ROWS = 40
# --------------------------------------------------------------------------- #
# Glossary terms this chapter explains. Registered in the shared collector and
# marked clickable on their first appearance (contract §11.1).
# --------------------------------------------------------------------------- #
_TERMS = {
"pk": (
"Clave primaria (PK)",
"Columna (o conjunto de columnas) que identifica de forma única cada fila "
"de una tabla: sus valores no se repiten y no son nulos. Una tabla tiene "
"como mucho una clave primaria; es el ancla por la que otras tablas la "
"referencian.",
),
"fk": (
"Clave foránea (FK)",
"Columna de una tabla cuyos valores apuntan a la clave primaria de otra "
"tabla (o de la misma), creando una relación entre ambas. Una FK suele ser "
"N:1: muchas filas de la tabla origen comparten el mismo valor de la tabla "
"destino.",
),
"containment": (
"Containment / inclusión",
"Señal con la que se infiere una clave foránea sin que la base la declare: "
"la fracción de valores distintos de una columna A que también aparecen "
"como valores de otra columna B. Si casi todos los valores de A están "
"contenidos en B (inclusión ≈ 1) y B parece una clave, A → B es una FK "
"candidata.",
),
"cardinalidad": (
"Cardinalidad",
"Número de valores distintos de una columna. Cardinalidad igual al número "
"de filas (y sin nulos) señala un identificador (candidato a clave "
"primaria); cardinalidad alta pero menor que el número de filas, con "
"valores repetidos, es típica de una clave foránea.",
),
}
def _register_terms(ctx: dict) -> bool:
"""Register the relational terms in the shared glossary. Returns whether the
in-text appearances should be marked clickable."""
glossary = ctx.get("glossary")
if not isinstance(glossary, model.GlossaryCollector):
return False
for key, (label, definition) in _TERMS.items():
glossary.add(key, label, definition)
return True
# --------------------------------------------------------------------------- #
# Formatting helpers (mirror the other chapters' defensive style).
# --------------------------------------------------------------------------- #
def _fmt_int(value) -> str:
if value is None:
return ""
try:
return f"{int(value):,}".replace(",", ".")
except (TypeError, ValueError):
return model._safe_str(value)
def _fmt_pct_fraction(value, decimals: int = 1) -> str:
"""Format a 01 fraction as a percentage. None -> placeholder."""
if value is None:
return ""
try:
v = float(value)
except (TypeError, ValueError):
return model._safe_str(value)
if v <= 1.0:
v *= 100.0
return f"{v:.{decimals}f}%"
def _fmt_ratio(value, decimals: int = 3) -> str:
"""Format an already-01 ratio (inclusion) as a plain number."""
if value is None:
return ""
try:
return f"{float(value):.{decimals}f}".rstrip("0").rstrip(".")
except (TypeError, ValueError):
return model._safe_str(value)
def _is_dict(v) -> bool:
return isinstance(v, dict)
def _columns_by_name(profile: dict) -> dict:
"""Index the profile columns by name for quick metric lookup."""
out = {}
for col in (profile.get("columns") or []):
if _is_dict(col) and col.get("name") is not None:
out[col.get("name")] = col
return out
# --------------------------------------------------------------------------- #
# Layer 1 — declared keys (real schema constraints).
# --------------------------------------------------------------------------- #
def _declared_keys(db_path: str, table: str):
"""Read declared PK/FK/UNIQUE for the source, or None if unavailable."""
if not db_path or detect_declared_keys_duckdb is None:
return None
try:
out = detect_declared_keys_duckdb(db_path, table)
except Exception: # noqa: BLE001 — dict-no-throw: treat as unavailable.
return None
if not _is_dict(out) or out.get("status") != "ok":
return None
return out
def _declared_section(declared: dict) -> list:
"""Blocks for the declared-keys layer, or [] if there is nothing declared."""
pks = [p for p in (declared.get("primary_keys") or []) if _is_dict(p)]
fks = [f for f in (declared.get("foreign_keys") or []) if _is_dict(f)]
uqs = [u for u in (declared.get("unique") or []) if _is_dict(u)]
if not (pks or fks or uqs):
return []
blocks = [
model.Heading(text="Claves declaradas en el esquema", level=2),
model.Markdown(text=(
"La base **declara** estas relaciones de clave como restricciones "
"reales del esquema (constraints). Son la verdad de referencia: no se "
"infieren, se leen tal cual de la definición de las tablas.")),
]
if pks:
rows = [[model._safe_str(p.get("table")),
", ".join(model._safe_str(c) for c in (p.get("columns") or []))]
for p in pks]
blocks.append(model.DataTable(
header=["Tabla", "Columna(s) PK"], rows=rows,
title="Claves primarias declaradas",
note="Cada fila: la clave primaria declarada de una tabla."))
if fks:
rows = []
for f in fks:
src = ", ".join(model._safe_str(c) for c in (f.get("columns") or []))
dst = ", ".join(
model._safe_str(c) for c in (f.get("referenced_columns") or []))
rows.append([
model._safe_str(f.get("table")), src,
model._safe_str(f.get("referenced_table")), dst])
blocks.append(model.DataTable(
header=["Tabla origen", "Columna(s) FK", "→ Tabla destino",
"Columna(s) destino"],
rows=rows, title="Claves foráneas declaradas",
note="Cada fila: una FK declarada — origen → destino."))
if uqs:
rows = [[model._safe_str(u.get("table")),
", ".join(model._safe_str(c) for c in (u.get("columns") or []))]
for u in uqs]
blocks.append(model.DataTable(
header=["Tabla", "Columna(s) UNIQUE"], rows=rows,
title="Restricciones UNIQUE declaradas"))
return blocks
# --------------------------------------------------------------------------- #
# Layer 2 — primary-key candidates (from the profile).
# --------------------------------------------------------------------------- #
def _pk_candidates_section(profile: dict, mark: bool) -> list:
"""Blocks for the PK-candidates layer, or [] if there are none."""
keys = [k for k in (profile.get("key_candidates") or []) if k is not None]
if not keys:
return []
by_name = _columns_by_name(profile)
pk = ("[[term:pk]]**clave primaria**[[/term]]" if mark
else "**clave primaria**")
intro = (
f"Columnas **candidatas a {pk}**: su "
"[[term:cardinalidad]]cardinalidad[[/term]] iguala al número de filas y "
"no tienen nulos. Son candidatas, no una clave declarada: la base no "
"las marca como tal."
if mark else
"Columnas **candidatas a clave primaria**: su cardinalidad iguala al "
"número de filas y no tienen nulos. Son candidatas, no una clave "
"declarada.")
rows = []
for name in keys:
col = by_name.get(name) or {}
rows.append([
model._safe_str(name),
_fmt_int(col.get("distinct_count")),
_fmt_pct_fraction(col.get("unique_pct")),
model._safe_str(col.get("inferred_type") or col.get("physical_type") or ""),
])
return [
model.Heading(text="Candidatos a clave primaria", level=2),
model.Markdown(text=intro),
model.DataTable(
header=["Columna", "Valores distintos", "% único", "Tipo"],
rows=rows, title="Candidatas a clave primaria",
note=f"{_fmt_int(profile.get('n_rows'))} filas en total como referencia."),
]
# --------------------------------------------------------------------------- #
# Layer 3a — inter-table FK candidates (containment) + join graph.
# --------------------------------------------------------------------------- #
def _list_source_tables(db_path: str) -> list:
"""List the tables in the DuckDB source, or [] if it can't be listed."""
if not db_path or duckdb_list_tables is None:
return []
try:
out = duckdb_list_tables(db_path)
except Exception: # noqa: BLE001
return []
if not _is_dict(out) or out.get("status") != "ok":
return []
return [t for t in (out.get("tables") or []) if isinstance(t, str)]
def _inter_table_section(db_path: str, tables: list, mark: bool) -> list:
"""Blocks for the inter-table FK layer (containment + join graph), or []."""
if infer_fk_containment_duckdb is None or len(tables) < 2:
return []
try:
fk = infer_fk_containment_duckdb(db_path, tables=tables)
except Exception: # noqa: BLE001
return []
if not _is_dict(fk) or fk.get("status") != "ok":
return []
candidates = [c for c in (fk.get("fk_candidates") or []) if _is_dict(c)]
if not candidates:
return []
containment = ("[[term:containment]]containment (inclusión de valores)[[/term]]"
if mark else "containment (inclusión de valores)")
fk_term = "[[term:fk]]**claves foráneas**[[/term]]" if mark else "**claves foráneas**"
blocks = [
model.Heading(text="Claves foráneas candidatas (inter-tabla)", level=2),
model.Markdown(text=(
f"La fuente tiene varias tablas. Estas {fk_term} candidatas se "
f"infieren por señal de nombre y por {containment}. No están "
"declaradas por la base; son la relación más probable según los "
"datos.")),
]
shown = candidates[:MAX_FK_ROWS]
rows = []
for c in shown:
rows.append([
f"{model._safe_str(c.get('from_table'))}.{model._safe_str(c.get('from_col'))}",
f"{model._safe_str(c.get('to_table'))}.{model._safe_str(c.get('to_col'))}",
_fmt_ratio(c.get("inclusion")),
model._safe_str(c.get("cardinality") or ""),
"" if c.get("name_match") else "no",
])
note = "Ordenadas por señal de nombre e inclusión."
if len(candidates) > len(shown):
note += f" Se muestran {len(shown)} de {len(candidates)} candidatas."
blocks.append(model.DataTable(
header=["Origen", "→ Destino", "Inclusión", "Cardinalidad", "Coincide nombre"],
rows=rows, title="FK candidatas por containment", note=note))
# Join graph: node roles + a pasteable Mermaid diagram, kept together.
if build_join_graph is not None:
try:
graph = build_join_graph(candidates, tables=tables)
except Exception: # noqa: BLE001
graph = None
if _is_dict(graph):
graph_blocks = [model.Heading(text="Grafo de relaciones", level=3)]
nodes = [n for n in (graph.get("nodes") or []) if _is_dict(n)]
if nodes:
node_rows = [[
model._safe_str(n.get("table")),
model._safe_str(n.get("role") or ""),
_fmt_int(n.get("out_degree")),
_fmt_int(n.get("in_degree")),
] for n in nodes]
graph_blocks.append(model.DataTable(
header=["Tabla", "Rol", "FK salientes", "FK entrantes"],
rows=node_rows, title="Tablas y su rol en el grafo",
note="Rol: fact (apunta a otras), dimension (referenciada), "
"bridge (ambas), standalone (aislada)."))
hubs = [h for h in (graph.get("hubs") or []) if h]
if hubs:
graph_blocks.append(model.Markdown(text=(
"Tablas con más relaciones salientes (candidatas a tabla de "
"hechos): " + ", ".join(model._safe_str(h) for h in hubs) + ".")))
mermaid = model._safe_str(graph.get("mermaid")).strip()
if mermaid:
graph_blocks.append(model.Markdown(text=(
"Diagrama de las relaciones (pegable en un bloque Mermaid):")))
graph_blocks.append(model.Markdown(
text="```mermaid\n" + mermaid + "\n```"))
if len(graph_blocks) > 1:
blocks.append(model.Group(blocks=graph_blocks,
title="Grafo de relaciones"))
skipped = [s for s in (fk.get("skipped") or []) if s]
if skipped:
blocks.append(model.Note(
"Algunos pares se omitieron por tamaño: "
+ "; ".join(model._safe_str(s) for s in skipped) + "."))
return blocks
# --------------------------------------------------------------------------- #
# Layer 3b — intra-table FK candidates (name+cardinality heuristic).
# --------------------------------------------------------------------------- #
def _intra_table_section(profile: dict, mark: bool) -> list:
"""Blocks for the intra-table FK heuristic layer, or [] if no candidates."""
if suggest_intratable_fk_candidates is None:
return []
try:
cands = suggest_intratable_fk_candidates(profile)
except Exception: # noqa: BLE001
return []
cands = [c for c in (cands or []) if _is_dict(c)]
if not cands:
return []
fk_term = "[[term:fk]]**claves foráneas**[[/term]]" if mark else "**claves foráneas**"
blocks = [
model.Heading(text="Posibles claves foráneas (heurística de nombre)", level=2),
model.Markdown(text=(
f"No hay otras tablas que referenciar, pero algunas columnas **parecen** "
f"{fk_term} por su nombre (terminan en «id») y su cardinalidad (muchos "
"valores repetidos, N:1). Es una **sugerencia heurística**, no una "
"afirmación: el nombre de la tabla destino es una conjetura y no se "
"comprueba inclusión de valores contra ninguna tabla real.")),
]
rows = []
for c in cands:
rows.append([
model._safe_str(c.get("column")),
model._safe_str(c.get("ref_table_guess") or ""),
_fmt_int(c.get("distinct_count")),
_fmt_pct_fraction(c.get("unique_pct")),
model._safe_str(c.get("inferred_type") or c.get("physical_type") or ""),
model._safe_str(c.get("reason") or ""),
])
blocks.append(model.DataTable(
header=["Columna", "Posible tabla", "Valores distintos", "% único",
"Tipo", "Motivo"],
rows=rows, title="Posibles FK por nombre y cardinalidad",
note="Heurística: posibles falsos positivos/negativos. No confirma containment."))
blocks.append(model.Note(
"Estas sugerencias se basan solo en el nombre y la cardinalidad. Para "
"confirmarlas haría falta la tabla destino y comprobar la inclusión de "
"valores (containment)."))
return blocks
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def _intro_blocks(mark: bool) -> list:
pk = "[[term:pk]]clave primaria[[/term]]" if mark else "clave primaria"
fk = "[[term:fk]]clave foránea[[/term]]" if mark else "clave foránea"
text = (
f"Este capítulo analiza las **relaciones de clave** de la tabla: cuál es "
f"la {pk} y cuáles son las {fk}. Cuando la base las **declara** como "
"restricciones del esquema, se muestran tal cual; cuando no, se proponen "
"las más probables a partir de los datos —por containment entre tablas o, "
"en una sola tabla, por una heurística de nombre y cardinalidad— siempre "
"marcadas como candidatas, nunca como hechos.")
return [model.Heading(text=CHAPTER_TITLE, level=1), model.Markdown(text=text)]
def build_relaciones(profile: dict, ctx: dict):
"""Build the RELACIONES Chapter, or None if there is nothing to say.
Args:
profile: the ``eda`` group TableProfile dict (may be None/empty).
ctx: presentation context. Consumes ``db_path`` + ``table`` (to read
declared constraints, list sibling tables and run the containment FK
inference) and ``glossary`` (to register the relational terms).
Returns:
A ``model.Chapter`` with the applicable relation layers; or ``None`` when
the dataset has no declared key, no key candidates and no FK candidate
(neither inter- nor intra-table).
"""
if not isinstance(profile, dict):
profile = {}
ctx = ctx if isinstance(ctx, dict) else {}
db_path = ctx.get("db_path")
table = ctx.get("table")
mark = _register_terms(ctx)
# Build each layer; the chapter is the concatenation of the non-empty ones.
declared = _declared_keys(db_path, table)
declared_blocks = _declared_section(declared) if declared else []
declared_has_fk = bool(declared and declared.get("foreign_keys"))
pk_blocks = _pk_candidates_section(profile, mark)
tables = _list_source_tables(db_path)
inter_blocks = _inter_table_section(db_path, tables, mark)
# The intra-table heuristic only makes sense when no real FK is available for
# this table — neither declared nor inferred inter-table. Otherwise the real
# relations already answer the question and the heuristic is just noise.
if declared_has_fk or inter_blocks:
intra_blocks = []
else:
intra_blocks = _intra_table_section(profile, mark)
body = declared_blocks + pk_blocks + inter_blocks + intra_blocks
if not body:
return None # chapter does not apply: nothing to say about relations.
blocks = _intro_blocks(mark) + body
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,273 @@
"""Tests for the RELACIONES chapter — DoD: golden(s) + edges + no-cut render.
Two goldens covering the two real paths of the chapter:
- **Intra-table** (a single table, no db source for relations): the chapter shows
the primary-key candidates from the profile and the heuristic foreign-key
suggestions (name + cardinality), explicitly flagged as a heuristic. Renders to
PDF and PPTX with nothing cut.
- **Inter-table** (a real DuckDB file with two related tables, customers/orders,
with a declared FK): the chapter shows the declared keys, the containment-based
FK candidates and the join graph (roles + a pasteable Mermaid diagram).
Edges: a profile with no key candidate and no FK-looking column returns None;
``None`` / ``{}`` profiles do not raise. The chapter registers its glossary terms.
Layers that depend on the sibling registry functions delegated alongside this
chapter (``detect_declared_keys_duckdb``, ``suggest_intratable_fk_candidates``)
are asserted **conditionally on the function being importable**, so the chapter's
honest-degradation contract is what is tested, never a hard dependency on import
timing.
"""
import os
import tempfile
import duckdb
from pptx import Presentation
from pypdf import PdfReader
from datascience.automatic_eda.chapters.relaciones import build_relaciones
from datascience.automatic_eda.model import Chapter, Group, GlossaryCollector
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
# The optional sibling functions: their layers are asserted only when present.
try:
from datascience.detect_declared_keys_duckdb import detect_declared_keys_duckdb
except Exception: # noqa: BLE001
detect_declared_keys_duckdb = None
try:
from datascience.suggest_intratable_fk_candidates import (
suggest_intratable_fk_candidates,
)
except Exception: # noqa: BLE001
suggest_intratable_fk_candidates = None
# --------------------------------------------------------------------------- #
# Helpers.
# --------------------------------------------------------------------------- #
def _flatten(blocks) -> list:
"""Flatten Group blocks so a test can inspect every leaf block."""
out = []
for b in blocks:
if isinstance(b, Group):
out.extend(_flatten(b.blocks))
else:
out.append(b)
return out
def _text_of(chapter: Chapter) -> str:
"""Collect all visible text of a chapter's blocks into one string."""
parts = []
for b in _flatten(chapter.blocks):
for attr in ("text", "title", "note"):
v = getattr(b, attr, None)
if isinstance(v, str):
parts.append(v)
header = getattr(b, "header", None)
if isinstance(header, list):
parts.extend(str(c) for c in header)
rows = getattr(b, "rows", None)
if isinstance(rows, list):
for r in rows:
if isinstance(r, (list, tuple)):
parts.extend(str(c) for c in r)
else:
parts.append(str(r))
return "\n".join(parts)
def _render_both(chapter: Chapter, tag: str):
"""Render the chapter to PDF and PPTX; return (pdf_text, n_slides)."""
tmp = tempfile.mkdtemp(prefix=f"relaciones_{tag}_")
pdf_path = os.path.join(tmp, "out.pdf")
pptx_path = os.path.join(tmp, "out.pptx")
meta = {"title": f"EDA — {tag}"}
render_automatic_eda_pdf([chapter], pdf_path, meta)
render_automatic_eda_pptx([chapter], pptx_path, meta)
assert os.path.exists(pdf_path) and os.path.getsize(pdf_path) > 0
assert os.path.exists(pptx_path) and os.path.getsize(pptx_path) > 0
text = "".join(p.extract_text() or "" for p in PdfReader(pdf_path).pages)
n_slides = len(Presentation(pptx_path).slides)
return text, n_slides
# --------------------------------------------------------------------------- #
# Fixtures.
# --------------------------------------------------------------------------- #
def _titanic_profile() -> dict:
"""A single-table profile: a PK candidate + a column that looks like a FK."""
return {
"table": "titanic",
"source": "/data/titanic.csv",
"n_rows": 891,
"n_cols": 4,
"key_candidates": ["PassengerId"],
"columns": [
{"name": "PassengerId", "inferred_type": "numeric",
"physical_type": "BIGINT", "distinct_count": 891,
"unique_pct": 1.0, "flags": ["possible_id"]},
{"name": "ticket_id", "inferred_type": "numeric",
"physical_type": "BIGINT", "distinct_count": 681,
"unique_pct": 0.76, "flags": []},
{"name": "fare", "inferred_type": "numeric",
"physical_type": "DOUBLE", "distinct_count": 248,
"unique_pct": 0.28, "flags": []},
{"name": "sex", "inferred_type": "categorical",
"physical_type": "VARCHAR", "distinct_count": 2,
"unique_pct": 0.002, "flags": []},
],
}
def _make_relational_db(path: str) -> None:
"""Create a small DuckDB with customers(id) <- orders(customer_id), real FK."""
con = duckdb.connect(path)
con.execute("CREATE TABLE customers(id INTEGER PRIMARY KEY, name TEXT)")
con.execute(
"CREATE TABLE orders(id INTEGER PRIMARY KEY, "
"customer_id INTEGER REFERENCES customers(id), amount DOUBLE)")
con.execute("INSERT INTO customers VALUES "
"(1,'a'),(2,'b'),(3,'c'),(4,'d'),(5,'e')")
con.execute("INSERT INTO orders VALUES "
"(1,1,10.0),(2,1,20.0),(3,2,30.0),(4,3,40.0),"
"(5,3,50.0),(6,4,60.0),(7,5,70.0),(8,2,80.0)")
con.close()
def _orders_profile() -> dict:
"""A profile for the `orders` table of the relational DB."""
return {
"table": "orders",
"source": "orders",
"n_rows": 8,
"n_cols": 3,
"key_candidates": ["id"],
"columns": [
{"name": "id", "inferred_type": "numeric", "physical_type": "INTEGER",
"distinct_count": 8, "unique_pct": 1.0, "flags": ["possible_id"]},
{"name": "customer_id", "inferred_type": "numeric",
"physical_type": "INTEGER", "distinct_count": 5, "unique_pct": 0.625,
"flags": []},
{"name": "amount", "inferred_type": "numeric", "physical_type": "DOUBLE",
"distinct_count": 8, "unique_pct": 1.0, "flags": []},
],
}
# --------------------------------------------------------------------------- #
# Golden 1 — intra-table.
# --------------------------------------------------------------------------- #
def test_golden_intra_table_pk_and_fk_heuristic():
"""Single table: PK candidate shown; FK heuristic shown (if fn available);
renders to PDF + PPTX with nothing cut."""
prof = _titanic_profile()
glossary = GlossaryCollector()
# No db_path: only the profile-derived layers apply (no declared, no inter).
chapter = build_relaciones(prof, {"glossary": glossary})
assert isinstance(chapter, Chapter)
assert chapter.id == "relaciones"
text = _text_of(chapter)
# PK candidate is always present (comes from the profile).
assert "Candidatos a clave primaria" in text
assert "PassengerId" in text
# Glossary terms got registered.
for key in ("pk", "fk", "cardinalidad"):
assert glossary.has(key)
# FK heuristic layer: present iff the delegated function is importable.
if suggest_intratable_fk_candidates is not None:
assert "Posibles claves foráneas" in text
assert "ticket_id" in text
# The float measure and the PK itself are NOT suggested as FKs.
assert "Posibles FK por nombre" in text
pdf_text, n_slides = _render_both(chapter, "intra")
assert "PassengerId" in pdf_text
assert n_slides >= 1
# --------------------------------------------------------------------------- #
# Golden 2 — inter-table (real DuckDB).
# --------------------------------------------------------------------------- #
def test_golden_inter_table_containment_and_join_graph():
"""Two related tables: declared FK (if fn available) + containment FK
candidate + Mermaid join graph."""
tmp = tempfile.mkdtemp(prefix="relaciones_db_")
db_path = os.path.join(tmp, "shop.duckdb")
_make_relational_db(db_path)
prof = _orders_profile()
glossary = GlossaryCollector()
chapter = build_relaciones(
prof, {"db_path": db_path, "table": "orders", "glossary": glossary})
assert isinstance(chapter, Chapter)
text = _text_of(chapter)
# Inter-table containment FK candidate: customer_id -> customers.id. This path
# uses infer_fk_containment_duckdb + build_join_graph, both already in the
# registry, so it must be present.
assert "Claves foráneas candidatas (inter-tabla)" in text
assert "orders.customer_id" in text
assert "customers.id" in text
# Join graph with a pasteable Mermaid diagram.
assert "Grafo de relaciones" in text
assert "mermaid" in text
assert "graph LR" in text
assert "containment" in text.lower()
# Declared-keys layer: present iff the delegated function is importable.
if detect_declared_keys_duckdb is not None:
assert "Claves declaradas en el esquema" in text
assert "Claves foráneas declaradas" in text
pdf_text, n_slides = _render_both(chapter, "inter")
assert "customer_id" in pdf_text
assert n_slides >= 1
# --------------------------------------------------------------------------- #
# Edges.
# --------------------------------------------------------------------------- #
def test_none_when_no_relations():
"""No key candidates, no FK-looking columns, no db source -> None."""
prof = {
"table": "flat", "n_rows": 100, "n_cols": 2, "key_candidates": [],
"columns": [
{"name": "value", "inferred_type": "numeric", "physical_type": "DOUBLE",
"distinct_count": 50, "unique_pct": 0.5, "flags": []},
{"name": "label", "inferred_type": "categorical",
"physical_type": "VARCHAR", "distinct_count": 3, "unique_pct": 0.03,
"flags": []},
],
}
assert build_relaciones(prof, {}) is None
def test_empty_and_none_profile_do_not_raise():
"""None / {} profile and missing ctx degrade to None without raising."""
assert build_relaciones(None, None) is None
assert build_relaciones({}, {}) is None
assert build_relaciones({}, {"glossary": GlossaryCollector()}) is None
def test_pk_candidate_only_builds_chapter():
"""A profile with only a key candidate (no FK anything, no db) still builds:
the relations chapter applies because there is a PK candidate to report."""
prof = {
"table": "t", "n_rows": 10, "n_cols": 1, "key_candidates": ["row_id"],
"columns": [
{"name": "row_id", "inferred_type": "numeric", "physical_type": "BIGINT",
"distinct_count": 10, "unique_pct": 1.0, "flags": ["possible_id"]},
],
}
chapter = build_relaciones(prof, {})
assert isinstance(chapter, Chapter)
assert "Candidatos a clave primaria" in _text_of(chapter)
@@ -0,0 +1,559 @@
"""Free-text / NLP distributions chapter (TEXT DISTR) for AutomaticEDA.
First chapter for **non-tabular** content: it profiles the linguistic content of
any column holding long free text (reviews, descriptions, comments, tickets) that
the categorical chapter cannot meaningfully summarize (high cardinality, many
words per value). It is the cheap, model-free counterpart to ``cat_distr`` for
columns that are prose rather than discrete labels.
Activation (returns ``None`` when it does not apply):
1. Cheap gate from the aggregated profile: at least one non-numeric column whose
``categorical.len_mean`` (mean character length) is ``>= _MIN_LEN_CHARS``.
A dataset whose only string columns are short labels (e.g. titanic's
``Name``, ~27 chars) never passes this gate, so the chapter disappears with
zero extra work and the existing report is untouched.
2. Confirmation from a raw sample: each candidate column is sampled (push-down
``extract_text_sample`` over ``ctx['db_path']``/``ctx['table']``, or an
in-memory ``ctx['text_raw']`` for tests) and kept only if the **median word
count is ``>= _MIN_WORDS``** i.e. it is genuinely long text, not a long
single token. If no column survives, the chapter returns ``None``.
Per surviving column the chapter emits, kept together on its own page/slide
(``Group(page_break_before=...)``):
- a key/value summary (documents, length percentiles, vocabulary richness with
**[[term:ttr]]TTR[[/term]]** and **[[term:hapax]]hapax legomena[[/term]]**,
dominant language, exact-duplicate %, readability when available);
- a word-count histogram figure;
- a top-terms table + a horizontal bar figure;
- bigram and trigram frequency tables;
- a detected-language bar figure (when ``langdetect`` is available);
- an optional word-cloud figure (only when ``wordcloud`` is installed);
- a closing note on duplicates / readability degradation.
Every metric is delegated to pure ``eda`` registry functions
(``compute_text_length_stats``, ``compute_vocabulary_stats``,
``compute_top_ngrams``, ``detect_corpus_language``, ``compute_text_duplicates``,
``compute_text_readability``) and the raw sample to ``extract_text_sample``; all
are imported defensively so a missing function or optional library degrades that
single piece to a note instead of aborting the chapter. Optional libraries
(``langdetect``, ``textstat``, ``wordcloud``, ``datasketch``) are never required:
the piece is silently omitted when they are absent.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
"""
from __future__ import annotations
from .. import model
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "text_distr"
CHAPTER_TITLE = "Texto libre (NLP)"
# Cheap activation gate (characters): a non-numeric column whose mean string
# length reaches this is a candidate for "long text". Short labels (titanic's
# Name ≈ 27 chars) stay below it, so the chapter does not fire on them.
_MIN_LEN_CHARS = 50
# Confirmation gate (words): a candidate is kept only if its median document has
# at least this many words — genuine prose, not a long id/URL token.
_MIN_WORDS = 20
# Bound the document so very wide datasets stay readable.
_MAX_TEXT_COLS = 5
# Raw text rows to sample per column when the chapter must extract them itself.
_SAMPLE_ROWS = 2000
# Rows shown in the frequency tables.
_TOP_TERMS = 15
_TOP_NGRAMS = 10
# Glossary terms this chapter explains (registered in the shared collector and
# marked clickable on first appearance — same mechanism as cat_distr's entropía).
_TERMS = {
"ttr": (
"TTR (type-token ratio)",
"Riqueza léxica de un texto: número de palabras distintas (tipos) "
"dividido por el número total de palabras (tokens). Vale 1 cuando no se "
"repite ninguna palabra (máxima variedad) y baja hacia 0 cuando el "
"vocabulario se repite mucho. Depende de la longitud del corpus, así que "
"compara mejor textos de tamaño parecido."),
"hapax": (
"Hapax legomena",
"Palabras que aparecen una sola vez en todo el corpus. Un porcentaje "
"alto de hapax indica vocabulario muy variado o, a veces, ruido "
"(erratas, identificadores, tokens raros). Se expresa como porcentaje "
"sobre el número de palabras distintas."),
}
def _fmt_int(value) -> str:
if value is None:
return ""
try:
return f"{int(value):,}".replace(",", ".")
except (TypeError, ValueError):
return str(value)
def _fmt_num(value, decimals: int = 2) -> str:
if value is None:
return ""
if isinstance(value, bool):
return str(value)
if isinstance(value, int):
return f"{value:,}".replace(",", ".")
if isinstance(value, float):
if value != value: # NaN
return "NaN"
if value in (float("inf"), float("-inf")):
return str(value)
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
return text if text else "0"
return str(value)
def _fmt_pct(value, decimals: int = 1) -> str:
if value is None:
return ""
try:
return f"{float(value):.{decimals}f}%"
except (TypeError, ValueError):
return str(value)
def _truncate(text, limit: int = 40) -> str:
s = model._safe_str(text)
return s if len(s) <= limit else s[: max(1, limit - 1)].rstrip() + ""
# --------------------------------------------------------------------------- #
# Defensive wrappers around the registry functions: each returns the function's
# output dict or a safe empty default, never raising and never importing at
# module load (so the chapter stays importable even if a function is missing).
# --------------------------------------------------------------------------- #
def _length_stats(texts) -> dict:
try:
from datascience.compute_text_length_stats import compute_text_length_stats
out = compute_text_length_stats(texts)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001
pass
return {}
def _vocab_stats(texts) -> dict:
try:
from datascience.compute_vocabulary_stats import compute_vocabulary_stats
out = compute_vocabulary_stats(texts, top_k=_TOP_TERMS)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001
pass
return {}
def _ngrams(texts, n) -> list:
try:
from datascience.compute_top_ngrams import compute_top_ngrams
out = compute_top_ngrams(texts, n=n, top_k=_TOP_NGRAMS)
if isinstance(out, dict):
return out.get("top") or []
except Exception: # noqa: BLE001
pass
return []
def _language(texts) -> dict:
try:
from datascience.detect_corpus_language import detect_corpus_language
out = detect_corpus_language(texts)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001
pass
return {"available": False, "distribution": [], "dominant": None}
def _duplicates(texts) -> dict:
try:
from datascience.compute_text_duplicates import compute_text_duplicates
out = compute_text_duplicates(texts)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001
pass
return {}
def _readability(texts) -> dict:
try:
from datascience.compute_text_readability import compute_text_readability
out = compute_text_readability(texts)
if isinstance(out, dict):
return out
except Exception: # noqa: BLE001
pass
return {"available": False, "flesch": {}}
# --------------------------------------------------------------------------- #
# Candidate detection + raw sample acquisition.
# --------------------------------------------------------------------------- #
def _candidate_columns(profile: dict) -> list:
"""Cheap gate: non-numeric columns whose mean char length reaches the
threshold. Returns the list of column names (possibly empty)."""
out = []
for col in profile.get("columns") or []:
if not isinstance(col, dict):
continue
if col.get("inferred_type") == "numeric":
continue
cat = col.get("categorical")
if not isinstance(cat, dict):
continue
len_mean = cat.get("len_mean")
if isinstance(len_mean, (int, float)) and not isinstance(len_mean, bool) \
and len_mean >= _MIN_LEN_CHARS:
name = col.get("name")
if name:
out.append(str(name))
return out
def _get_samples(profile: dict, ctx: dict, columns: list) -> dict:
"""Return {col: [str, ...]} raw text samples for the candidate columns.
Prefers an in-memory ``ctx['text_raw']`` (used by tests); otherwise pushes a
sample down to the database via ``extract_text_sample`` using ctx db_path /
table. Never raises: returns {} when no sample can be obtained."""
text_raw = ctx.get("text_raw")
if isinstance(text_raw, dict) and text_raw:
return {c: [str(v) for v in (text_raw.get(c) or []) if v is not None]
for c in columns if text_raw.get(c)}
db_path = ctx.get("db_path")
table = ctx.get("table")
if not db_path or not table:
return {}
backend = ctx.get("backend") or "duckdb"
sample = ctx.get("sample") or _SAMPLE_ROWS
try:
from datascience.extract_text_sample import extract_text_sample
out = extract_text_sample(db_path, table, columns, backend=backend,
sample=sample)
if isinstance(out, dict) and out.get("status") == "ok":
cols = out.get("columns")
if isinstance(cols, dict):
return {c: list(v) for c, v in cols.items() if v}
except Exception: # noqa: BLE001 — dict-no-throw: no sample → chapter omits.
pass
return {}
def _confirm_long_text(samples: dict) -> dict:
"""Keep only columns whose median word count reaches _MIN_WORDS. Returns
{col: length_stats_dict} for the survivors, in input order."""
survivors = {}
for col, texts in samples.items():
stats = _length_stats(texts)
words = stats.get("words") if isinstance(stats, dict) else None
median = words.get("p50") if isinstance(words, dict) else None
if isinstance(median, (int, float)) and not isinstance(median, bool) \
and median >= _MIN_WORDS:
survivors[col] = stats
return survivors
# --------------------------------------------------------------------------- #
# Figures (lazy matplotlib, scaled by the renderers — same style as num_distr).
# --------------------------------------------------------------------------- #
def _hist_figure(name: str, length_stats: dict):
def make():
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure
fig = Figure(figsize=(6.2, 3.0))
ax = fig.add_subplot(111)
bins = (length_stats or {}).get("word_hist") or []
drew = False
for b in bins:
if not isinstance(b, dict):
continue
lo, hi, count = b.get("lo"), b.get("hi"), b.get("count") or 0
if lo is None or hi is None:
continue
width = (hi - lo) if hi > lo else max(abs(lo) * 1e-3, 1e-6)
ax.bar(lo, count, width=width, align="edge", color="#9ec6df",
edgecolor="#5b8aa6", linewidth=0.4)
drew = True
if not drew:
ax.text(0.5, 0.5, "(sin datos de longitud)", ha="center",
va="center", color="#8a8a8a", transform=ax.transAxes)
ax.set_xlabel("palabras por documento", fontsize=8)
ax.set_ylabel("nº de documentos", fontsize=8)
ax.tick_params(labelsize=7)
for spine in ("top", "right"):
ax.spines[spine].set_visible(False)
ax.set_title(f"Longitud de «{_truncate(name, 30)}»", fontsize=10,
loc="left")
fig.tight_layout()
return fig
return make
def _barh_figure(title: str, items: list, label_key: str, value_key: str,
xlabel: str):
"""Horizontal bar chart from [{label_key:..., value_key:...}, ...]."""
def make():
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure
rows = [it for it in (items or []) if isinstance(it, dict)
and isinstance(it.get(value_key), (int, float))]
rows = rows[:12]
fig = Figure(figsize=(6.2, max(2.2, 0.32 * len(rows) + 0.8)))
ax = fig.add_subplot(111)
if not rows:
ax.text(0.5, 0.5, "(sin datos)", ha="center", va="center",
color="#8a8a8a", transform=ax.transAxes)
ax.axis("off")
return fig
labels = [_truncate(r.get(label_key), 28) for r in rows][::-1]
values = [float(r.get(value_key) or 0) for r in rows][::-1]
ypos = range(len(rows))
ax.barh(list(ypos), values, color="#9ec6df", edgecolor="#5b8aa6",
linewidth=0.4)
ax.set_yticks(list(ypos))
ax.set_yticklabels(labels, fontsize=7)
ax.set_xlabel(xlabel, fontsize=8)
ax.tick_params(labelsize=7)
for spine in ("top", "right"):
ax.spines[spine].set_visible(False)
ax.set_title(_truncate(title, 44), fontsize=10, loc="left")
fig.tight_layout()
return fig
return make
def _wordcloud_figure(texts):
"""Word-cloud figure callable, or None if wordcloud is not installed."""
try:
import wordcloud # noqa: F401
except Exception: # noqa: BLE001 — optional dependency: omit the figure.
return None
def make():
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure
from wordcloud import WordCloud
fig = Figure(figsize=(6.2, 3.2))
ax = fig.add_subplot(111)
joined = " ".join(t for t in texts if isinstance(t, str))
try:
wc = WordCloud(width=800, height=400, background_color="white",
colormap="viridis").generate(joined)
ax.imshow(wc, interpolation="bilinear")
except Exception: # noqa: BLE001
ax.text(0.5, 0.5, "(nube de palabras no disponible)", ha="center",
va="center", color="#8a8a8a", transform=ax.transAxes)
ax.axis("off")
fig.tight_layout()
return fig
return make
# --------------------------------------------------------------------------- #
# Per-column block assembly.
# --------------------------------------------------------------------------- #
def _summary_kv(n_docs, length_stats, vocab, lang, dup, read):
chars = (length_stats or {}).get("chars") or {}
words = (length_stats or {}).get("words") or {}
sents = (length_stats or {}).get("sentences") or {}
rows = [
("Documentos", _fmt_int(n_docs)),
("Caracteres (media · p50 · p90 · p99)",
f"{_fmt_num(chars.get('mean'))} · {_fmt_int(chars.get('p50'))} · "
f"{_fmt_int(chars.get('p90'))} · {_fmt_int(chars.get('p99'))}"),
("Palabras (media · p50 · p90 · p99)",
f"{_fmt_num(words.get('mean'))} · {_fmt_int(words.get('p50'))} · "
f"{_fmt_int(words.get('p90'))} · {_fmt_int(words.get('p99'))}"),
("Frases (media · máx)",
f"{_fmt_num(sents.get('mean'))} · {_fmt_int(sents.get('max'))}"),
("Vocabulario (tokens · tipos · TTR)",
f"{_fmt_int(vocab.get('n_tokens'))} · {_fmt_int(vocab.get('n_types'))} "
f"· {_fmt_num(vocab.get('ttr'), 3)}"),
("Hapax legomena",
f"{_fmt_int(vocab.get('n_hapax'))} ({_fmt_pct(vocab.get('hapax_pct'))})"),
]
if isinstance(lang, dict) and lang.get("available"):
dom = lang.get("dominant")
n_langs = len(lang.get("distribution") or [])
rows.append(("Idioma dominante · nº idiomas",
f"{model._safe_str(dom) or ''} · {_fmt_int(n_langs)}"))
if isinstance(dup, dict) and dup.get("n_docs"):
rows.append(("Duplicados exactos",
f"{_fmt_int(dup.get('n_exact_dup'))} "
f"({_fmt_pct(dup.get('exact_dup_pct'))})"))
if isinstance(read, dict) and read.get("available"):
flesch = read.get("flesch") or {}
rows.append(("Legibilidad Flesch (media)",
_fmt_num(flesch.get("mean"), 1)))
return model.KVTable(rows=rows, title="Resumen del texto")
def _terms_table(vocab) -> "model.DataTable | None":
top = (vocab or {}).get("top_terms") or []
rows = [[_truncate(t.get("term"), 32), _fmt_int(t.get("count")),
_fmt_pct(t.get("pct"))]
for t in top[:_TOP_TERMS] if isinstance(t, dict)]
if not rows:
return None
return model.DataTable(header=["Término", "Conteo", "% tokens"], rows=rows,
title="Términos más frecuentes",
note="stopwords ES+EN eliminadas")
def _ngram_table(items, n_label) -> "model.DataTable | None":
rows = [[_truncate(it.get("ngram"), 40), _fmt_int(it.get("count"))]
for it in (items or [])[:_TOP_NGRAMS] if isinstance(it, dict)]
if not rows:
return None
return model.DataTable(header=[n_label, "Conteo"], rows=rows,
title=f"{n_label} más frecuentes")
def _dup_note(dup, lang, read) -> "model.Note | None":
bits = []
if isinstance(dup, dict):
nd = dup.get("near_dup") or {}
if nd.get("available"):
bits.append(
f"casi-duplicados detectados (MinHash, umbral "
f"{_fmt_num(nd.get('threshold'))}): "
f"{_fmt_int(nd.get('n_near_dup_docs'))} documentos")
else:
bits.append("near-duplicados no calculados (datasketch no instalado; "
"se reportan solo los duplicados exactos por hash)")
if isinstance(lang, dict) and not lang.get("available"):
bits.append("detección de idioma omitida (langdetect no instalado)")
if isinstance(read, dict) and not read.get("available"):
bits.append("legibilidad omitida (textstat no instalado)")
if not bits:
return None
return model.Note(" · ".join(bits))
def _column_group(name, texts, length_stats, idx, mark_terms):
vocab = _vocab_stats(texts)
lang = _language(texts)
dup = _duplicates(texts)
read = _readability(texts)
n_docs = (length_stats or {}).get("n_docs")
blocks = [
model.Heading(text=str(name), level=2),
_summary_kv(n_docs, length_stats, vocab, lang, dup, read),
model.Figure(make=_hist_figure(name, length_stats),
caption=f"Distribución de la longitud (palabras) de "
f"«{_truncate(name, 30)}»."),
]
terms_tbl = _terms_table(vocab)
if terms_tbl is not None:
blocks.append(terms_tbl)
blocks.append(model.Figure(
make=_barh_figure(f"Top términos de «{_truncate(name, 24)}»",
vocab.get("top_terms"), "term", "count",
"conteo"),
caption="Términos más frecuentes (barras)."))
bi_tbl = _ngram_table(_ngrams(texts, 2), "Bigrama")
if bi_tbl is not None:
blocks.append(bi_tbl)
tri_tbl = _ngram_table(_ngrams(texts, 3), "Trigrama")
if tri_tbl is not None:
blocks.append(tri_tbl)
if isinstance(lang, dict) and lang.get("available") \
and lang.get("distribution"):
blocks.append(model.Figure(
make=_barh_figure(f"Idiomas detectados en «{_truncate(name, 24)}»",
lang.get("distribution"), "lang", "count",
"documentos"),
caption="Distribución de idiomas detectados (langdetect)."))
wc = _wordcloud_figure(texts)
if wc is not None:
blocks.append(model.Figure(
make=wc, caption=f"Nube de palabras de «{_truncate(name, 30)}»."))
note = _dup_note(dup, lang, read)
if note is not None:
blocks.append(note)
return model.Group(blocks=blocks, page_break_before=(idx > 0))
def _intro_blocks(n_cols, mark_terms):
ttr = ("[[term:ttr]]TTR[[/term]]" if mark_terms else "TTR")
hapax = ("[[term:hapax]]hapax legomena[[/term]]" if mark_terms
else "hapax legomena")
text = (
f"Este capítulo perfila las columnas de **texto libre largo** del "
f"dataset (reseñas, descripciones, comentarios): contenido lingüístico "
f"que la distribución categórica no resume bien. Para cada columna se "
f"muestran la longitud de los documentos, la riqueza de vocabulario "
f"(incluido el {ttr} y el porcentaje de {hapax}), los términos y "
f"n-gramas más frecuentes, los idiomas detectados y el nivel de "
f"duplicación. Las métricas son baratas y sin modelos pesados; las "
f"piezas que dependen de una librería opcional se omiten si no está "
f"instalada.")
return [
model.Heading(text=CHAPTER_TITLE, level=1),
model.Markdown(text=text),
]
def build_text_distr(profile: dict, ctx: dict):
"""Build the free-text Chapter, or None if no long-text column applies."""
profile = profile or {}
ctx = ctx or {}
# 1) Cheap gate from the profile (no DB access yet).
candidates = _candidate_columns(profile)
if not candidates:
return None
# 2) Raw sample + 3) confirm genuine long text (median words >= threshold).
samples = _get_samples(profile, ctx, candidates)
if not samples:
return None
survivors = _confirm_long_text(samples)
if not survivors:
return None
# Register glossary terms (clickable) once we know the chapter applies.
glossary = ctx.get("glossary")
mark_terms = False
if isinstance(glossary, model.GlossaryCollector):
for key, (label, definition) in _TERMS.items():
glossary.add(key, label, definition)
mark_terms = True
blocks = list(_intro_blocks(len(survivors), mark_terms))
rendered = list(survivors.items())[:_MAX_TEXT_COLS]
for idx, (name, length_stats) in enumerate(rendered):
texts = samples.get(name) or []
blocks.append(_column_group(name, texts, length_stats, idx, mark_terms))
if len(survivors) > len(rendered):
omitted = len(survivors) - len(rendered)
blocks.append(model.Note(
f"Se muestran las primeras {len(rendered)} columnas de texto; "
f"quedan {omitted} sin mostrar para mantener acotado el informe."))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,256 @@
"""Tests for the TEXT DISTR chapter — DoD: golden + edges + degradation.
Self-contained: builds synthetic TableProfiles and feeds the raw text sample
in-memory through ``ctx['text_raw']`` (no DuckDB needed), so the suite is fast
and deterministic. Verifies that ``build_text_distr``:
- GOLDEN: with a long-text column, emits the chapter with its key blocks
(length summary, word histogram, top-terms table, n-gram tables, language
bars) and registers the clickable glossary terms; and that it renders inside
the full document to both PDF and PPTX showing that content.
- EDGE (None): a dataset whose only string column is short labels (titanic-like
``Name``) yields ``None`` without raising the existing report is untouched.
- EDGE (None): a column that passes the cheap char gate but whose documents are
short (median words below the threshold) is rejected at the confirmation step.
- DEGRADATION: with ``langdetect`` / ``textstat`` / ``wordcloud`` unavailable,
the chapter still builds (those pieces are omitted) and never raises.
"""
import builtins
import os
import tempfile
from pypdf import PdfReader
from pptx import Presentation
from datascience.automatic_eda.model import (
DataTable, Figure, GlossaryCollector, Group, Heading, KVTable, Markdown,
Note,
)
from datascience.automatic_eda.chapters.text_distr import (
CHAPTER_ID, CHAPTER_VERSION, build_text_distr,
)
from datascience.automatic_eda.chapters_registry import build_document
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
# --------------------------------------------------------------------------- #
# Synthetic corpus + profiles.
# --------------------------------------------------------------------------- #
_ES = [
"El producto llegó en perfecto estado y mucho antes de lo previsto por la tienda",
"La calidad de los materiales es realmente excelente y se nota la diferencia al usarlo",
"No me convenció del todo porque esperaba bastante más por el precio que pagué finalmente",
"El servicio de atención al cliente fue rápido amable y resolvió mi problema sin demora",
"Lo recomiendo totalmente ya que ha superado con creces todas mis expectativas iniciales",
]
_EN = [
"The product arrived in perfect condition and much earlier than the store had promised me",
"The build quality is genuinely outstanding and you can really feel the difference using it",
"I was not fully convinced because I expected quite a lot more for the price i finally paid",
"Customer support was fast friendly and solved my whole problem without any delay at all",
"I highly recommend it since it has exceeded by far every one of my initial expectations",
]
def _long_reviews(n=40) -> list:
"""A corpus of long multi-sentence reviews (>= 20 words each), mixing two
languages and including a few exact duplicates."""
out = []
for i in range(n):
base = _ES if i % 3 != 0 else _EN # mostly ES, some EN
a = base[i % len(base)]
b = base[(i + 2) % len(base)]
out.append(f"{a}. {b}.")
# Inject a couple of exact duplicates.
out.append(out[0])
out.append(out[1])
return out
def _text_profile() -> dict:
"""Profile with a long free-text column (review) + a numeric + a short cat."""
return {
"table": "reviews",
"source": "/data/reviews.duckdb",
"profiled_at": "2026-06-30T10:00:00+00:00",
"n_rows": 42,
"n_cols": 3,
"quality_score": 88.0,
"columns": [
{
"name": "review",
"inferred_type": "categorical",
"categorical": {
"top": [{"value": "x", "count": 2, "pct": 0.05}],
"n_distinct": 40,
"len_mean": 180.0,
"len_min": 80,
"len_max": 220,
},
},
{
"name": "rating",
"inferred_type": "numeric",
"numeric": {"mean": 3.1, "median": 3.0, "std": 1.2,
"min": 1, "max": 5},
},
{
"name": "product",
"inferred_type": "categorical",
"categorical": {
"top": [{"value": "teclado", "count": 10, "pct": 0.25}],
"n_distinct": 6,
"len_mean": 7.0,
"len_min": 5, "len_max": 11,
},
},
],
}
def _no_text_profile() -> dict:
"""titanic-like: the only string column is short labels (Name ≈ 27 chars)."""
return {
"table": "titanic",
"n_rows": 891,
"n_cols": 3,
"columns": [
{"name": "Age", "inferred_type": "numeric",
"numeric": {"mean": 29.7, "median": 28.0, "std": 14.5}},
{"name": "Name", "inferred_type": "categorical",
"categorical": {"top": [{"value": "Braund, Mr. Owen Harris",
"count": 1, "pct": 0.001}],
"n_distinct": 891, "len_mean": 27.0,
"len_min": 12, "len_max": 82}},
{"name": "Sex", "inferred_type": "categorical",
"categorical": {"top": [{"value": "male", "count": 577,
"pct": 0.65}],
"n_distinct": 2, "len_mean": 4.6,
"len_min": 4, "len_max": 6}},
],
}
def _flatten(blocks) -> list:
"""Recursively flatten Group blocks so tests can inspect leaf blocks."""
out = []
for b in blocks:
if isinstance(b, Group):
out.extend(_flatten(b.blocks))
else:
out.append(b)
return out
# --------------------------------------------------------------------------- #
# Golden.
# --------------------------------------------------------------------------- #
def test_golden_activa_con_texto():
glossary = GlossaryCollector()
ctx = {"text_raw": {"review": _long_reviews()}, "glossary": glossary}
ch = build_text_distr(_text_profile(), ctx)
assert ch is not None, "el capítulo debe activarse con una columna de texto largo"
assert ch.id == CHAPTER_ID
assert ch.version == CHAPTER_VERSION
leaves = _flatten(ch.blocks)
kinds = [b.kind for b in leaves]
assert "heading" in kinds
assert "kv_table" in kinds # summary
assert "figure" in kinds # histogram / bars
assert "data_table" in kinds # top terms + n-grams
# KV summary mentions vocabulary metrics.
kv = next(b for b in leaves if isinstance(b, KVTable))
labels = " ".join(str(r[0]) for r in kv.rows)
assert "TTR" in labels
assert "Hapax" in labels or "hapax" in labels
# There is a terms table and at least one n-gram table.
titles = [getattr(b, "title", "") or "" for b in leaves
if isinstance(b, DataTable)]
assert any("Términos" in t for t in titles)
assert any("Bigrama" in t for t in titles)
# Glossary terms were registered (clickable destinations).
assert glossary.has("ttr")
assert glossary.has("hapax")
def test_golden_render_pdf_pptx():
profile = _text_profile()
ctx = {"text_raw": {"review": _long_reviews()},
"dataset_name": "reviews"}
chapters = build_document(profile, ctx)
ids = [c.id for c in chapters]
assert "text_distr" in ids, f"text_distr ausente en {ids}"
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "t.pdf")
pptx = os.path.join(d, "t.pptx")
rp = render_automatic_eda_pdf(profile, pdf, {"title": "EDA", "ctx": ctx})
rx = render_automatic_eda_pptx(profile, pptx, {"title": "EDA", "ctx": ctx})
assert rp.get("path") and os.path.exists(pdf)
assert rx.get("path") and os.path.exists(pptx)
text = "\n".join(p.extract_text() or "" for p in PdfReader(pdf).pages)
assert "Texto libre" in text or "TTR" in text
prs = Presentation(pptx)
ptext = []
for slide in prs.slides:
for shp in slide.shapes:
if shp.has_text_frame:
ptext.append(shp.text_frame.text)
joined = "\n".join(ptext)
assert "Texto libre" in joined or "TTR" in joined
# --------------------------------------------------------------------------- #
# Edges — None.
# --------------------------------------------------------------------------- #
def test_edge_none_sin_texto_largo():
# titanic-like: short labels only → chapter must not apply.
assert build_text_distr(_no_text_profile(), {}) is None
def test_edge_none_palabras_cortas():
# Char gate passes (len_mean high) but documents are short → confirmation
# rejects them (median words below threshold).
profile = _text_profile()
short = ["palabra " * 3] * 30 # 3 words each, < _MIN_WORDS
ctx = {"text_raw": {"review": short}}
assert build_text_distr(profile, ctx) is None
def test_edge_none_empty_profile():
assert build_text_distr({}, {}) is None
assert build_text_distr(None, None) is None
# --------------------------------------------------------------------------- #
# Degradation — optional libs absent.
# --------------------------------------------------------------------------- #
def test_degradacion_sin_libs(monkeypatch):
real_import = builtins.__import__
blocked = ("langdetect", "textstat", "wordcloud", "datasketch")
def fake_import(name, *a, **k):
if name in blocked or any(name.startswith(b + ".") for b in blocked):
raise ImportError(f"simulado: {name}")
return real_import(name, *a, **k)
monkeypatch.setattr(builtins, "__import__", fake_import)
ctx = {"text_raw": {"review": _long_reviews()}}
ch = build_text_distr(_text_profile(), ctx)
# Still builds (the cheap, stdlib-only pieces remain) and never raises.
assert ch is not None
leaves = _flatten(ch.blocks)
assert any(isinstance(b, KVTable) for b in leaves)
assert any(isinstance(b, DataTable) for b in leaves)
# A degradation note is present mentioning the missing optional libs.
notes = " ".join(b.text for b in leaves if isinstance(b, Note))
assert "langdetect" in notes or "textstat" in notes or "datasketch" in notes
@@ -0,0 +1,613 @@
"""Time-series chapter (TIMESERIES) for AutomaticEDA.
This chapter applies **only when the table has a date/datetime column**. When it
does, it draws exactly the user requirement the evolution of the data over
time (the value of each numeric column aggregated per period *and* the count of
rows per period) plus the statistical analysis of the series (stationarity,
autocorrelation, trend and seasonality). When there is no temporal column
``build_timeseries`` returns ``None``.
Data sources, read defensively and never recomputed here:
- ``profile['columns']`` to detect the time column and the numeric columns.
Delegated to the pure registry function ``detect_time_column`` (group ``eda``).
- ``profile['series'][col]`` the per-column time-series analysis already
produced by ``profile_table(run_series=True)``: ``stationarity`` (ADF+KPSS),
``acf_pacf`` (ACF/PACF + Ljung-Box), ``stl`` (trend/seasonal/resid +
Hyndman strengths) and the levels/returns suggestion.
- ``ctx['timeseries_raw']`` (or ``profile['timeseries_raw']``) the *raw* ordered
series ``{time_col, t:[iso...], series:{col:[float|None]}}`` needed to draw the
value-vs-time line and the per-period row count. Exactly like ``modelos`` reads
``raw_numeric`` from ``ctx``, this chapter looks for the raw series there and
degrades honestly when it is absent (it still renders the textual analysis).
The raw series is aggregated per period with the pure registry function
``resample_timeseries`` and the datetime header is built with ``profile_datetime``
(both group ``eda``). Every figure is emitted as a lazy ``Figure`` so the
renderers rasterize and scale it to fit a whole page/slide; tables go through
``DataTable``/``KVTable`` so the paginator splits them repeating the header. No
content is ever cut.
ctx keys this chapter consumes (all optional):
timeseries_raw : dict ``{time_col, t:[...], series:{col:[...]}}`` raw
ordered series used to draw the value-vs-time line and the row-count
panel. When absent the chapter omits those figures (with a note) and
renders only the analysis available in ``profile['series']``.
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
Reads everything defensively (``.get``) and never raises.
"""
from __future__ import annotations
from .. import model
# Pure/impure registry functions (group ``eda``) consumed by this chapter,
# imported defensively so the chapter still builds (degrading the affected
# section to a note) if any of them is somehow unavailable.
try:
from datascience.detect_time_column import detect_time_column
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
detect_time_column = None # type: ignore[assignment]
try:
from datascience.profile_datetime import profile_datetime
except Exception: # noqa: BLE001
profile_datetime = None # type: ignore[assignment]
try:
from datascience.resample_timeseries import resample_timeseries
except Exception: # noqa: BLE001
resample_timeseries = None # type: ignore[assignment]
CHAPTER_VERSION = "1.0.0"
CHAPTER_ID = "timeseries"
CHAPTER_TITLE = "Series temporales"
# Plain-Spanish gloss for the stationarity verdict of adf_kpss_stationarity.
_VERDICT_GLOSS = {
"stationary": "estacionaria: media y varianza estables en el tiempo; se "
"puede modelar directamente.",
"non_stationary": "no estacionaria: tiene tendencia o varianza cambiante "
"(raíz unitaria). Correlacionar o modelar sus niveles "
"produce relaciones espurias (Granger-Newbold); conviene "
"diferenciar o pasar a retornos.",
"inconclusive": "resultado no concluyente (ADF y KPSS discrepan): tratar con "
"cautela, probablemente cerca de la no estacionariedad.",
}
# OHLC-style name fragments used to collapse near-identical financial series.
_OHLC_HINTS = ("open", "high", "low", "close", "adj", "price", "vwap")
def _fmt_num(value, decimals: int = 3) -> str:
"""Compact, defensive number formatting shared with the other chapters."""
if value is None:
return ""
if isinstance(value, bool):
return "" if value else "no"
if isinstance(value, int):
return f"{value:,}".replace(",", ".")
if isinstance(value, float):
if value != value: # NaN
return "NaN"
if value in (float("inf"), float("-inf")):
return str(value)
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
return text if text else "0"
return model._safe_str(value)
def _is_dict(v) -> bool:
return isinstance(v, dict)
# --------------------------------------------------------------------------- #
# Detection: which column is the time axis and which numeric columns to chart.
# --------------------------------------------------------------------------- #
def _detect(cols: list) -> dict:
"""Return ``{time_col, numeric_cols, ...}`` via the registry function.
Falls back to an inline scan (datetime inferred_type / datetime semantic
types) when ``detect_time_column`` is unavailable, so the chapter still works.
"""
if detect_time_column is not None:
try:
res = detect_time_column(cols)
if _is_dict(res):
return res
except Exception: # noqa: BLE001 — degrade to the inline scan.
pass
time_col = None
numeric_cols = []
for c in cols or []:
if not _is_dict(c):
continue
it = c.get("inferred_type")
sem = c.get("semantic_type")
if time_col is None and (
it == "datetime" or sem in ("datetime_iso", "date_eu")):
time_col = c.get("name")
if it == "numeric":
numeric_cols.append(c.get("name"))
return {"time_col": time_col, "numeric_cols": numeric_cols,
"time_semantic": "", "reason": "inline fallback"}
def _raw_series_for(raw: dict, col: str):
"""Return (t_list, v_list) for a column from the raw bundle, or (None, None)."""
if not _is_dict(raw):
return None, None
t = raw.get("t")
series = raw.get("series") if _is_dict(raw.get("series")) else {}
v = series.get(col)
if isinstance(t, list) and isinstance(v, list) and t and len(t) == len(v):
return t, v
return None, None
def _ohlc_groups(numeric_cols: list, raw: dict) -> dict:
"""Map each numeric column to a representative to collapse OHLC duplicates.
When several numeric columns are near-identical financial level series
(open/high/low/close/adj close), charting each one repeats the same figure
four times. We keep the first OHLC-looking column as the representative for
the *figures* and list the collapsed ones in a note; the textual analysis is
still produced for every column. Detection is by name only (cheap, no extra
data dependency) and conservative: only collapses when >=2 OHLC-like names
are present.
"""
ohlc = [c for c in numeric_cols
if isinstance(c, str) and any(h in c.lower() for h in _OHLC_HINTS)]
if len(ohlc) < 2:
return {}
representative = ohlc[0]
return {c: representative for c in ohlc if c != representative}
# --------------------------------------------------------------------------- #
# Datetime header (MUST-9.3): range / frequency / regularity / gaps.
# --------------------------------------------------------------------------- #
def _datetime_header(time_col: str, raw: dict) -> list:
"""Build the datetime profile header from the raw time axis, when present."""
blocks: list = []
t, _ = (raw.get("t"), None) if _is_dict(raw) else (None, None)
if not (isinstance(t, list) and t and profile_datetime is not None):
return blocks
try:
dt = profile_datetime(t)
except Exception: # noqa: BLE001
return blocks
if not _is_dict(dt):
return blocks
freq_gloss = {
"daily": "diaria", "weekly": "semanal", "monthly": "mensual",
"quarterly": "trimestral", "yearly": "anual",
"irregular": "irregular", "unknown": "indeterminada",
}
rows = [
("Columna de fecha", model._safe_str(time_col)),
("Rango", f"{model._safe_str(dt.get('min'))}"
f"{model._safe_str(dt.get('max'))}"),
("Observaciones", _fmt_num(dt.get("n"))),
("Fechas distintas", _fmt_num(dt.get("n_distinct"))),
("Frecuencia", freq_gloss.get(dt.get("freq"), model._safe_str(dt.get("freq")))),
("Regular", "" if dt.get("is_regular") else "no"),
]
span = dt.get("span_days")
if span is not None:
rows.append(("Duración (días)", _fmt_num(span, 1)))
n_gaps = dt.get("n_gaps")
if n_gaps is not None:
rows.append(("Huecos en la rejilla", _fmt_num(n_gaps)))
blocks.append(model.KVTable(rows=rows, title="Perfil temporal"))
note = dt.get("note")
if note:
blocks.append(model.Note(model._safe_str(note)))
return blocks
# --------------------------------------------------------------------------- #
# Figure builders (lazy: matplotlib only imported when the renderer draws them).
# --------------------------------------------------------------------------- #
def _parse_dates(labels: list):
"""Parse a list of ISO-ish strings/dates to datetime, dropping unparseable.
Returns (dates, kept_index) so callers can align the values list.
"""
from datetime import date, datetime
out = []
keep = []
for i, lab in enumerate(labels):
if isinstance(lab, datetime):
out.append(lab)
keep.append(i)
continue
if isinstance(lab, date):
out.append(datetime(lab.year, lab.month, lab.day))
keep.append(i)
continue
s = model._safe_str(lab).strip()
if not s:
continue
s2 = s.replace("T", " ")
parsed = None
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%Y-%m-%d"):
try:
parsed = datetime.strptime(s2[:len(fmt) + 4] if False else s2, fmt)
break
except ValueError:
continue
if parsed is None:
try:
parsed = datetime.fromisoformat(s.replace("T", " "))
except ValueError:
continue
out.append(parsed)
keep.append(i)
return out, keep
def _make_evolution_figure(name: str, rs: dict):
"""Lazy callable: value-vs-time line + per-period row-count panel (MUST-9.1)."""
def _draw():
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
t_labels = rs.get("t") or []
v = rs.get("v") or []
counts = rs.get("count") or []
dates, keep = _parse_dates(t_labels)
vv = [v[i] if i < len(v) else None for i in keep]
cc = [counts[i] if i < len(counts) else 0 for i in keep]
fig, (ax_v, ax_c) = plt.subplots(
2, 1, figsize=(7.0, 4.6), sharex=True,
gridspec_kw={"height_ratios": [3.0, 1.2], "hspace": 0.12})
# Top: value aggregated per period (line; gaps where the value is None).
xs = [d for d, val in zip(dates, vv) if val is not None]
ys = [val for val in vv if val is not None]
if xs and ys:
ax_v.plot(xs, ys, color="#4e79a7", linewidth=1.4, zorder=3)
ax_v.fill_between(xs, ys, min(ys), color="#9ec6df", alpha=0.18,
zorder=1)
else:
ax_v.text(0.5, 0.5, "(sin valores numéricos)", ha="center",
va="center", fontsize=9, color="#8a8a8a",
transform=ax_v.transAxes)
ax_v.set_ylabel(name, fontsize=8)
ax_v.tick_params(labelsize=7)
ax_v.grid(axis="y", color="#eeeeee", linewidth=0.6)
for spine in ("top", "right"):
ax_v.spines[spine].set_visible(False)
# Bottom: number of observations per period (density / gaps).
if dates and cc:
# Bar width ~ median spacing so bars do not overlap nor leave gaps.
width = 1.0
if len(dates) > 1:
deltas = sorted((dates[i + 1] - dates[i]).days
for i in range(len(dates) - 1))
width = max(deltas[len(deltas) // 2] * 0.8, 1.0)
ax_c.bar(dates, cc, width=width, color="#59a14f", alpha=0.75,
align="center")
ax_c.set_ylabel("nº filas", fontsize=8)
ax_c.tick_params(labelsize=7)
ax_c.grid(axis="y", color="#eeeeee", linewidth=0.6)
for spine in ("top", "right"):
ax_c.spines[spine].set_visible(False)
ax_c.xaxis.set_major_locator(mdates.AutoDateLocator())
ax_c.xaxis.set_major_formatter(mdates.ConciseDateFormatter(
ax_c.xaxis.get_major_locator()))
freq = rs.get("freq")
suptitle = f"{name} — evolución temporal"
if freq:
suptitle += f" (agregado {freq})"
fig.suptitle(suptitle, fontsize=10, fontweight="bold", x=0.02, ha="left")
return fig
return _draw
def _make_stl_figure(stl: dict):
"""Lazy callable: the STL trend/seasonal/resid panels, or None if no values.
``stl_decompose`` only carries the component *values* for short series; for
long ones it returns just summary stats (``note``). In that case there is
nothing to plot and we return None (the caller renders the strengths as text).
"""
def _component_values(comp):
if _is_dict(comp):
vals = comp.get("values")
if isinstance(vals, list) and vals:
return [x for x in vals]
return None
trend = _component_values(stl.get("trend"))
seasonal = _component_values(stl.get("seasonal"))
resid = _component_values(stl.get("resid"))
if not any([trend, seasonal, resid]):
return None
def _draw():
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
panels = [("Tendencia", trend, "#4e79a7"),
("Estacional", seasonal, "#59a14f"),
("Resto", resid, "#e15759")]
panels = [(lbl, vals, col) for lbl, vals, col in panels if vals]
fig, axes = plt.subplots(len(panels), 1, figsize=(7.0, 1.4 * len(panels) + 0.6),
sharex=True)
if len(panels) == 1:
axes = [axes]
for ax, (lbl, vals, col) in zip(axes, panels):
ax.plot(range(len(vals)), vals, color=col, linewidth=1.2)
ax.set_ylabel(lbl, fontsize=8)
ax.tick_params(labelsize=7)
ax.grid(axis="y", color="#eeeeee", linewidth=0.6)
for spine in ("top", "right"):
ax.spines[spine].set_visible(False)
axes[-1].set_xlabel("índice temporal", fontsize=8)
fig.suptitle("Descomposición STL", fontsize=10, fontweight="bold",
x=0.02, ha="left")
fig.tight_layout(rect=(0, 0, 1, 0.96))
return fig
return _draw
def _make_acf_figure(acf_pacf: dict):
"""Lazy callable: the ACF stem plot with ±1.96/√n bands, or None."""
acf = acf_pacf.get("acf")
n = acf_pacf.get("n")
if not (isinstance(acf, list) and len(acf) > 1 and isinstance(n, int) and n > 0):
return None
def _draw():
import math
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
lags = list(range(len(acf)))
fig, ax = plt.subplots(figsize=(7.0, 3.2))
ax.vlines(lags, 0, acf, color="#4e79a7", linewidth=1.4)
ax.plot(lags, acf, "o", color="#4e79a7", markersize=3)
band = 1.96 / math.sqrt(n)
ax.axhspan(-band, band, color="#cccccc", alpha=0.3,
label="banda ±1.96/√n (ruido blanco)")
ax.axhline(0, color="#888888", linewidth=0.8)
ax.set_xlabel("retardo (lag)", fontsize=8)
ax.set_ylabel("ACF", fontsize=8)
ax.tick_params(labelsize=7)
ax.legend(fontsize=7, loc="upper right", framealpha=0.85)
ax.set_title("Autocorrelación (ACF): lags fuera de la banda = "
"correlación significativa", fontsize=9)
fig.tight_layout()
return fig
return _draw
# --------------------------------------------------------------------------- #
# Per-column textual analysis from profile['series'][col].
# --------------------------------------------------------------------------- #
def _analysis_markdown(sblock: dict) -> str:
"""One markdown block summarizing stationarity / autocorrelation / STL."""
parts: list = []
stat = sblock.get("stationarity") if _is_dict(sblock.get("stationarity")) else {}
verdict = stat.get("verdict")
if verdict:
adf = stat.get("adf") if _is_dict(stat.get("adf")) else {}
kpss = stat.get("kpss") if _is_dict(stat.get("kpss")) else {}
line = (f"**Estacionariedad:** {_VERDICT_GLOSS.get(verdict, verdict)} "
f"(ADF p={_fmt_num(adf.get('p_value'), 4)}, "
f"KPSS p={_fmt_num(kpss.get('p_value'), 4)}).")
warning = stat.get("warning")
if warning:
line += f"{model._safe_str(warning)}"
parts.append(line)
acf = sblock.get("acf_pacf") if _is_dict(sblock.get("acf_pacf")) else {}
if acf:
is_auto = acf.get("is_autocorrelated")
lb = acf.get("ljung_box") if _is_dict(acf.get("ljung_box")) else {}
sig = acf.get("significant_acf_lags") or []
if is_auto is True:
ac_line = ("**Autocorrelación:** la serie está autocorrelada "
"(Ljung-Box rechaza independencia, "
f"p={_fmt_num(lb.get('p_value'), 4)}): los valores dependen "
"de su pasado, no es ruido blanco.")
if sig:
shown = ", ".join(str(x) for x in sig[:8])
more = "" if len(sig) > 8 else ""
ac_line += f" Lags significativos: {shown}{more}."
elif is_auto is False:
ac_line = ("**Autocorrelación:** no se detecta autocorrelación "
"significativa (compatible con ruido blanco, Ljung-Box "
f"p={_fmt_num(lb.get('p_value'), 4)}).")
else:
ac_line = "**Autocorrelación:** no evaluable (datos insuficientes)."
parts.append(ac_line)
stl = sblock.get("stl") if _is_dict(sblock.get("stl")) else {}
if stl:
ts = stl.get("trend_strength")
ss = stl.get("seasonal_strength")
if ts is not None or ss is not None:
parts.append(
"**Descomposición STL:** fuerza de tendencia "
f"{_fmt_num(ts, 2)} y fuerza estacional {_fmt_num(ss, 2)} "
"(escala 01 de Hyndman: cuanto más alto, más marcada la "
"componente).")
elif stl.get("note"):
parts.append(f"**Descomposición STL:** {model._safe_str(stl.get('note'))}")
if sblock.get("levels_suggested"):
reason = sblock.get("levels_reason")
kind = sblock.get("levels_kind")
tr = sblock.get("to_returns") if _is_dict(sblock.get("to_returns")) else None
line = "**Transformación sugerida:** "
line += "pasar a retornos" if kind == "returns" else "diferenciar la serie"
if reason:
line += f"{model._safe_str(reason)}"
if tr and tr.get("mean") is not None:
line += (f" (retornos: media {_fmt_num(tr.get('mean'), 5)}, "
f"σ {_fmt_num(tr.get('std'), 5)}).")
parts.append(line)
return "\n\n".join(parts)
# --------------------------------------------------------------------------- #
# Per-column section.
# --------------------------------------------------------------------------- #
def _column_section(name: str, sblock: dict, raw: dict, collapsed_into) -> list:
"""Blocks for one numeric column: evolution figure + STL + ACF + analysis."""
blocks = [model.Heading(text=model._safe_str(name), level=2)]
# --- Value-vs-time line + per-period row count (MUST-9.1). ---
drew_evolution = False
if collapsed_into is None: # skip the figure for collapsed OHLC duplicates.
t, v = _raw_series_for(raw, name)
if t is not None and resample_timeseries is not None:
try:
rs = resample_timeseries(t, v)
except Exception: # noqa: BLE001
rs = None
if _is_dict(rs) and rs.get("t"):
blocks.append(model.Figure(
make=_make_evolution_figure(name, rs),
caption=f"Evolución de «{name}» por periodo y nº de "
f"observaciones (conteo de filas)."))
drew_evolution = True
else:
blocks.append(model.Note(
f"Serie casi idéntica a «{collapsed_into}» (grupo OHLC): se omite el "
"gráfico para no repetirlo; el análisis estadístico se mantiene."))
if not drew_evolution and collapsed_into is None:
blocks.append(model.Note(
"Gráfico de evolución temporal no disponible: falta la serie cruda "
"(pásala en ctx['timeseries_raw'] = {time_col, t, series}). Se "
"muestra solo el análisis estadístico."))
# --- STL panels (MUST-9.2). ---
stl = sblock.get("stl") if _is_dict(sblock.get("stl")) else {}
if collapsed_into is None and stl:
stl_fig = _make_stl_figure(stl)
if stl_fig is not None:
blocks.append(model.Figure(
make=stl_fig,
caption=f"Descomposición STL de «{name}»: tendencia, componente "
f"estacional y resto."))
# --- ACF figure (autocorrelation structure). ---
acf = sblock.get("acf_pacf") if _is_dict(sblock.get("acf_pacf")) else {}
if collapsed_into is None and acf:
acf_fig = _make_acf_figure(acf)
if acf_fig is not None:
blocks.append(model.Figure(
make=acf_fig,
caption=f"Función de autocorrelación de «{name}»."))
# --- Textual analysis (always, even for collapsed duplicates). ---
analysis = _analysis_markdown(sblock)
if analysis:
blocks.append(model.Markdown(text=analysis))
return blocks
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def build_timeseries(profile: dict, ctx: dict):
"""Build the TIMESERIES Chapter, or ``None`` if the table has no date column.
Args:
profile: the ``eda`` group TableProfile dict.
ctx: presentation context; ``ctx['timeseries_raw']`` (optional) carries
the raw ordered series used to draw the value-vs-time line and the
per-period row count.
Returns:
A ``model.Chapter`` with, per numeric column, the value-vs-time evolution
+ row-count figure, the STL panels, the ACF figure and the statistical
analysis; or ``None`` when there is no temporal column (the chapter does
not apply).
"""
profile = profile or {}
if not _is_dict(profile):
profile = {}
ctx = ctx or {}
cols = profile.get("columns") or []
det = _detect(cols)
time_col = det.get("time_col")
if not time_col:
return None # no date/datetime column -> chapter does not apply.
numeric_cols = det.get("numeric_cols") or []
series_map = profile.get("series") if _is_dict(profile.get("series")) else {}
raw = ctx.get("timeseries_raw") or profile.get("timeseries_raw")
raw = raw if _is_dict(raw) else {}
# Which columns can the chapter say anything about: those with a series
# analysis block and/or a raw series to chart. Preserve the profile order.
chartable = []
for name in numeric_cols:
has_analysis = _is_dict(series_map.get(name))
has_raw, _ = _raw_series_for(raw, name)
if has_analysis or has_raw is not None:
chartable.append(name)
if not chartable:
# A date column exists but nothing numeric to chart/analyse: still a
# valid (small) chapter — show just the datetime header if we have it.
header = _datetime_header(time_col, raw)
if not header:
return None
intro = (
f"La tabla tiene una columna temporal («{time_col}») pero no hay "
"columnas numéricas con serie analizable.")
blocks = [model.Heading(text=CHAPTER_TITLE, level=1),
model.Markdown(text=intro)] + header
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
collapsed = _ohlc_groups(chartable, raw)
intro = (
"Este capítulo analiza la evolución de la tabla en el tiempo usando la "
f"columna de fecha «{time_col}». Para cada columna numérica se muestra su "
"**evolución por periodo** (valor agregado) junto al **número de filas por "
"periodo** (densidad de observaciones), su **descomposición STL** "
"(tendencia / estacionalidad / resto) y la **función de autocorrelación**; "
"debajo, el análisis de la serie: estacionariedad (ADF + KPSS), "
"autocorrelación (Ljung-Box) y, cuando procede, la transformación "
"sugerida (retornos o diferencias) para evitar correlaciones espurias.")
blocks = [model.Heading(text=CHAPTER_TITLE, level=1),
model.Markdown(text=intro)]
blocks += _datetime_header(time_col, raw)
if collapsed:
reps = sorted(set(collapsed.values()))
collapsed_names = ", ".join(sorted(collapsed.keys()))
blocks.append(model.Note(
f"Series OHLC casi idénticas detectadas ({collapsed_names}): se "
f"grafican consolidadas en «{', '.join(reps)}» para no repetir el "
"mismo gráfico; cada columna conserva su análisis estadístico."))
for name in chartable:
sblock = series_map.get(name) if _is_dict(series_map.get(name)) else {}
blocks += _column_section(name, sblock, raw, collapsed.get(name))
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
version=CHAPTER_VERSION, blocks=blocks)
@@ -0,0 +1,244 @@
"""Tests for the TIMESERIES chapter — DoD: golden + edges + anti-cut.
Self-contained: builds synthetic ``series`` blocks (shaped like
``profile_table(run_series=True)`` output) and a raw ``timeseries_raw`` bundle,
with no DuckDB, so the suite is fast and deterministic. Verifies that the chapter:
- returns ``None`` when there is no date/datetime column (the user requirement);
- never raises on ``None``/empty/garbage input;
- with a date column + raw series emits, per numeric column, the value-vs-time +
row-count evolution figure, the STL panels, the ACF figure and the textual
analysis (stationarity / autocorrelation / suggested transform);
- collapses near-identical OHLC series into one chart while keeping every
column's analysis;
- renders without cutting anything in both PDF and PPTX (every column heading
survives in the rendered output).
"""
import math
import os
import re
import tempfile
from pypdf import PdfReader
from datascience.automatic_eda.chapters.timeseries import (
build_timeseries, CHAPTER_VERSION, _VERDICT_GLOSS,
)
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
# --------------------------------------------------------------------------- #
# Synthetic fixtures shaped like the real profile_table(run_series=True) output.
# --------------------------------------------------------------------------- #
def _dates(n: int) -> list:
"""n consecutive daily ISO date strings starting 2021-01-01."""
from datetime import date, timedelta
start = date(2021, 1, 1)
return [(start + timedelta(days=i)).isoformat() for i in range(n)]
def _series_block(n=120, verdict="non_stationary", autocorr=True, levels=True,
with_stl_values=True):
"""A synthetic ``series`` block like _build_series_block produces."""
trend = [float(i) for i in range(n)]
seasonal = [math.sin(i / 6.0) for i in range(n)]
resid = [0.1 * ((-1) ** i) for i in range(n)]
acf = [1.0] + [max(0.0, 0.9 - 0.05 * k) for k in range(1, 21)]
block = {
"order_col": "fecha",
"ordered": True,
"n": n,
"stationarity": {
"n": n, "verdict": verdict,
"adf": {"p_value": 0.42, "stationary": False},
"kpss": {"p_value": 0.01, "stationary": False},
"warning": ("serie no estacionaria: riesgo de correlación espuria"
if verdict != "stationary" else None),
},
"acf_pacf": {
"n": n, "nlags": 20, "acf": acf,
"significant_acf_lags": [1, 2, 3, 4, 5],
"ljung_box": {"stat": 123.4, "p_value": 0.0 if autocorr else 0.7,
"lags": 20},
"is_autocorrelated": autocorr,
},
"period_source": "datetime_freq",
"stl": {
"n": n, "period": 7, "period_inferred": False, "robust": False,
"trend": {"values": trend} if with_stl_values else {
"note": "serie larga: solo estadisticos", "mean": 60.0},
"seasonal": {"values": seasonal} if with_stl_values else {"mean": 0.0},
"resid": {"values": resid} if with_stl_values else {"mean": 0.0},
"trend_strength": 0.95, "seasonal_strength": 0.42,
},
}
if levels:
block["levels_suggested"] = True
block["levels_kind"] = "returns"
block["levels_reason"] = ("columna financiera no estacionaria: usar "
"retornos evita correlación espuria.")
block["to_returns"] = {"method": "log", "mean": 0.001, "std": 0.02}
else:
block["levels_suggested"] = False
return block
def _profile(numeric_names=("precio",), n=120, with_stl_values=True):
cols = [{"name": "fecha", "inferred_type": "datetime",
"semantic_type": "datetime_iso"}]
series_map = {}
for nm in numeric_names:
cols.append({"name": nm, "inferred_type": "numeric",
"numeric": {"min": 1.0, "max": 200.0, "mean": 100.0,
"median": 95.0, "std": 40.0}})
series_map[nm] = _series_block(n=n, with_stl_values=with_stl_values)
return {"table": "cotizaciones", "n_rows": n, "n_cols": len(cols),
"columns": cols, "series": series_map}
def _ctx_raw(numeric_names=("precio",), n=120):
t = _dates(n)
series = {}
for j, nm in enumerate(numeric_names):
series[nm] = [float(100 + i + 5 * j) for i in range(n)]
return {"timeseries_raw": {"time_col": "fecha", "t": t, "series": series}}
def _pdf_text(path: str) -> str:
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
return re.sub(r"\s+", " ", txt)
# --------------------------------------------------------------------------- #
# Golden.
# --------------------------------------------------------------------------- #
def test_golden_estructura_y_figuras():
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
assert ch is not None
assert ch.id == "timeseries"
assert ch.version == CHAPTER_VERSION
kinds = [b.kind for b in ch.blocks]
assert kinds[0] == "heading" # chapter title
assert kinds[1] == "markdown" # intro
assert "kv_table" in kinds # datetime profile header (MUST-9.3)
# Per column: evolution figure + STL figure + ACF figure + analysis markdown.
figs = [b for b in ch.blocks if b.kind == "figure"]
assert len(figs) >= 3, "evolución + STL + ACF esperadas"
# Lazy makers must produce real matplotlib figures.
import matplotlib.pyplot as plt
for f in figs:
fig = f.make()
assert fig is not None
plt.close(fig)
def test_golden_evolucion_tiene_dos_paneles_valor_y_conteo():
# MUST-9.1: the evolution figure has a value panel + a row-count panel.
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
figs = [b for b in ch.blocks if b.kind == "figure"]
import matplotlib.pyplot as plt
fig = figs[0].make() # first figure is the evolution one.
assert len(fig.axes) == 2, "panel de valor + panel de conteo de filas"
plt.close(fig)
def test_golden_analisis_textual_presente():
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
assert "Estacionariedad" in md
assert "Autocorrelación" in md
assert "STL" in md
# Verdict gloss surfaced for the non-stationary preset.
assert _VERDICT_GLOSS["non_stationary"].split(":")[0] in md
# Levels/returns suggestion surfaced.
assert "retornos" in md.lower()
# --------------------------------------------------------------------------- #
# Edges.
# --------------------------------------------------------------------------- #
def test_edge_sin_columna_fecha_devuelve_none():
prof = {"columns": [
{"name": "precio", "inferred_type": "numeric", "numeric": {"mean": 1.0}},
{"name": "ciudad", "inferred_type": "categorical",
"categorical": {"top": []}},
], "series": {"precio": _series_block()}}
assert build_timeseries(prof, {}) is None
def test_edge_none_y_vacio_no_revienta():
assert build_timeseries(None, None) is None
assert build_timeseries({}, {}) is None
assert build_timeseries({"columns": []}, {}) is None
# Date column but nothing numeric/series and no raw -> None (nothing to say).
assert build_timeseries(
{"columns": [{"name": "fecha", "inferred_type": "datetime"}]}, {}) is None
def test_edge_sin_raw_degrada_pero_mantiene_analisis():
# No ctx['timeseries_raw']: the chapter must still build (STL/ACF/analysis
# from the profile) and note that the evolution chart is unavailable.
ch = build_timeseries(_profile(("precio",)), {})
assert ch is not None
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
assert "evolución temporal no disponible" in notes
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
assert "Estacionariedad" in md
def test_edge_stl_solo_estadisticos_no_dibuja_panel_pero_no_revienta():
# Long series: STL carries only stats (no 'values') -> no STL figure, but the
# strengths still surface in the textual analysis.
ch = build_timeseries(_profile(("precio",), with_stl_values=False),
_ctx_raw(("precio",)))
assert ch is not None
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
assert "STL" in md
# --------------------------------------------------------------------------- #
# OHLC consolidation (MUST-9.3).
# --------------------------------------------------------------------------- #
def test_ohlc_consolidacion():
names = ("Open", "High", "Low", "Close")
ch = build_timeseries(_profile(names), _ctx_raw(names))
assert ch is not None
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
assert "OHLC" in notes
# Only the representative draws the evolution figure; the other 3 are collapsed
# so there are fewer evolution figures than columns.
captions = [b.caption or "" for b in ch.blocks if b.kind == "figure"]
evo = [c for c in captions if "Evolución" in c]
assert len(evo) < len(names), "las series OHLC deben consolidarse"
# Every column still has its analysis markdown (one heading per column).
headings = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
for nm in names:
assert nm in headings
# --------------------------------------------------------------------------- #
# Anti-cut: PDF + PPTX.
# --------------------------------------------------------------------------- #
def test_anti_corte_pdf_y_pptx():
names = tuple(f"serie_{i}" for i in range(6))
prof = _profile(names, n=90)
ctx = _ctx_raw(names, n=90)
ch = build_timeseries(prof, ctx)
col_headings = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
assert len(col_headings) == 6
with tempfile.TemporaryDirectory() as d:
pdf = os.path.join(d, "ts.pdf")
res_pdf = render_automatic_eda_pdf(
prof, pdf, {"ctx": ctx, "write_manifest": False})
assert res_pdf["path"] == pdf
txt = _pdf_text(pdf)
for nm in col_headings:
assert nm in txt, f"columna '{nm}' cortada/ausente en el PDF"
pptx = os.path.join(d, "ts.pptx")
res_pptx = render_automatic_eda_pptx(
prof, pptx, {"ctx": ctx, "write_manifest": False})
assert res_pptx["path"] == pptx
assert res_pptx["n_slides"] >= 6
@@ -26,19 +26,29 @@ from . import model
# placeholders other agents will fill by creating chapters/<id>.py — they will
# appear in this exact position automatically once their module exists.
CHAPTER_ORDER = [
"portada", # cover
"portada", # cover — BUILT LAST, PLACED FIRST (see build_document).
"overview", # df.head + columns/types/nulls/examples + describe
"analisis_llm", # LLM interpretation — sits next to overview (user request)
"num_distr", # numeric distributions
"cat_distr", # categorical distributions
"text_distr", # free-text / NLP distributions (non-tabular content)
"calidad", # data quality
"missingness", # missing-data patterns (co-occurrence of absences; MCAR/MAR)
"correlacion", # correlations / associations
"relaciones", # key relations: declared/candidate PK + FK (inter/intra-table)
"modelos", # cheap models (PCA/KMeans/outliers)
"analisis_llm", # LLM interpretation
"timeseries", # time-series analysis
"geospatial", # geospatial
"agregacion", # aggregations / pivots
"glosario", # glossary — ALWAYS LAST; clickable term destinations.
]
# Chapters whose position is special-cased by build_document: portada is built
# last (so it can summarize the rest) but placed first; glosario is built and
# placed last (it reads the terms every other chapter registered).
_PORTADA = "portada"
_GLOSARIO = "glosario"
def build_chapter(chapter_id: str, profile: dict, ctx: dict):
"""Build a single chapter by id, or None if absent/not-applicable/error.
@@ -75,15 +85,72 @@ def build_document(profile: dict, ctx: dict = None) -> list:
list[Chapter] in canonical order, containing only the chapters that are
implemented and applicable. Never raises.
"""
if profile is None:
profile = {}
if not isinstance(profile, dict):
profile = {}
if ctx is None:
ctx = {}
chapters = []
# Copy ctx so the shared collector / summary we add do not leak to the caller.
ctx = dict(ctx) if isinstance(ctx, dict) else {}
# A single glossary collector is shared by every chapter via ctx['glossary'].
# Chapters call ctx['glossary'].add(key, label, definition) and mark in-text
# appearances with [[term:key]]…[[/term]]; the glosario chapter renders the
# registered terms and the renderers wire the clickable links.
glossary = ctx.get("glossary")
if not isinstance(glossary, model.GlossaryCollector):
glossary = model.GlossaryCollector()
ctx["glossary"] = glossary
# 1) Body: every chapter except portada (built last) and glosario (placed
# last), in canonical order. This also fills the glossary collector.
body = []
for cid in CHAPTER_ORDER:
if cid in (_PORTADA, _GLOSARIO):
continue
ch = build_chapter(cid, profile, ctx)
if ch is not None and ch.blocks:
chapters.append(ch)
body.append(ch)
# 2) Aggregated summary of the rest, for the cover (user decision: the cover
# is BUILT after the body so it can reflect what the analysis found).
ctx["document_summary"] = _summarize_document(profile, body)
# 3) Build the cover last, place it FIRST.
portada = build_chapter(_PORTADA, profile, ctx)
# 4) Build the glossary last (reads the terms the body registered), place LAST.
glosario = build_chapter(_GLOSARIO, profile, ctx)
chapters = []
if portada is not None and portada.blocks:
chapters.append(portada)
chapters.extend(body)
if glosario is not None and glosario.blocks:
chapters.append(glosario)
return chapters
def _summarize_document(profile: dict, body: list) -> dict:
"""Aggregate a tiny findings summary of the body for the cover. Never raises.
Returns a dict with dataset shape, quality, column-type counts and the list
of chapters actually included enough for the cover to show a mini-summary
of the analysis without re-deriving anything."""
try:
cols = profile.get("columns") or []
n_num = sum(1 for c in cols if isinstance(c, dict)
and c.get("inferred_type") == "numeric")
n_cat = sum(1 for c in cols if isinstance(c, dict)
and isinstance(c.get("categorical"), dict)
and c.get("categorical", {}).get("top")
and c.get("inferred_type") != "numeric")
return {
"n_chapters": len(body),
"chapter_titles": [getattr(c, "title", "") for c in body],
"n_rows": profile.get("n_rows"),
"n_cols": profile.get("n_cols"),
"quality_score": profile.get("quality_score"),
"n_numeric": n_num,
"n_categorical": n_cat,
"duplicate_pct": profile.get("duplicate_pct"),
"null_cell_pct": profile.get("null_cell_pct"),
}
except Exception: # noqa: BLE001 — the summary is best-effort.
return {"n_chapters": len(body) if isinstance(body, list) else 0}
@@ -0,0 +1,253 @@
"""Tests for the Markdown completeness appendix (report 2053).
The AutomaticEDA Markdown is the output meant to be *pasted into an LLM*, so it
must carry EVERYTHING the engine computed even the numbers the human-facing
chapters (shared with the PDF/PPTX) drop for readability. ``render_md`` appends a
full-data appendix built from ``meta['profile']`` that closes the six losses the
evaluation found:
1. the complete association matrix (every pair, incl. correlation_ratio /
cramers_v) not just the top extremes;
2. every numeric statistic for every numeric column (skew/kurtosis/percentiles);
3. the concrete recommended re-expression;
4. KMeans ``scores_by_k``;
5. the normality test statistics;
6. correct headers for bar/scree figure tables (not ``Desde/Hasta/Frecuencia``).
Self-contained: a synthetic profile, no DuckDB, no heavy renderer.
"""
import os
import sys
import pytest # noqa: F401
_HERE = os.path.dirname(os.path.abspath(__file__))
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
if _FUNCTIONS not in sys.path:
sys.path.insert(0, _FUNCTIONS)
from datascience.automatic_eda import model # noqa: E402
from datascience.automatic_eda.render_md_impl import ( # noqa: E402
_bars_table,
_is_histogram_caption,
_profile_appendix,
render_md,
)
# --------------------------------------------------------------------------- #
# Synthetic profile fixtures.
# --------------------------------------------------------------------------- #
def _numeric(skew, kurtosis):
"""A numeric stat block with every key the appendix serializes."""
return {
"count": 100, "min": 0.0, "max": 10.0, "mean": 5.0, "median": 5.0,
"mode": 4.0, "std": 2.0, "variance": 4.0, "cv": 0.4,
"p1": 0.1, "p5": 0.5, "p25": 2.5, "p50": 5.0, "p75": 7.5,
"p95": 9.5, "p99": 9.9, "iqr": 5.0, "skew": skew, "kurtosis": kurtosis,
"n_outliers": 1, "distribution_type": "normal",
}
def _profile():
"""A small but structurally faithful TableProfile (3 numeric, 2 categorical)."""
pairs = [
{"a": "A", "b": "B", "a_type": "numeric", "b_type": "numeric",
"method": "pearson/spearman", "value": 0.8,
"p_value": 1e-9, "p_value_adjusted": 2e-9, "significant": True},
{"a": "A", "b": "C", "a_type": "numeric", "b_type": "numeric",
"method": "pearson/spearman", "value": -0.3,
"p_value": 0.01, "p_value_adjusted": 0.02, "significant": True},
{"a": "A", "b": "Cat1", "a_type": "numeric", "b_type": "categorical",
"method": "correlation_ratio", "value": 0.45,
"p_value": 0.001, "p_value_adjusted": 0.002, "significant": True},
# The single cat-cat pair the human chapter never shows.
{"a": "Cat1", "b": "Cat2", "a_type": "categorical",
"b_type": "categorical", "method": "cramers_v", "value": 0.11,
"p_value": 0.04, "p_value_adjusted": 0.05, "significant": False},
]
return {
"correlations": {
"pairs": pairs,
"multiple_testing": {"method": "bh", "n_tests": 4, "n_rejected": 3},
},
"columns": [
{"name": "A", "count": 100, "numeric": _numeric(0.0, -1.2),
"reexpression": {"recommended": "none", "ladder_power": 1.0,
"reason": "symmetric", "alternatives": []}},
{"name": "B", "count": 100, "numeric": _numeric(4.77, 33.1),
"reexpression": {"recommended": "log1p", "ladder_power": 0.0,
"reason": "skew 4.77 with zeros",
"alternatives": [{"transform": "yeo-johnson"},
{"transform": "sqrt"}]}},
{"name": "C", "count": 100, "numeric": _numeric(-0.6, 0.2)},
{"name": "Cat1", "categorical": {"top": [], "mode": "x"}},
{"name": "Cat2", "categorical": {"top": [], "mode": "y"}},
],
"models": {
"kmeans": {
"best_k": 3,
"scores_by_k": [
{"k": 2, "silhouette": 0.46, "inertia": 900.0},
{"k": 3, "silhouette": 0.50, "inertia": 550.0},
{"k": 4, "silhouette": 0.38, "inertia": 430.0},
],
"cluster_sizes": [40, 35, 25],
},
"normality": {
"A": {"n": 100,
"jarque_bera": {"stat": 18.7, "p": 8e-5, "normal": False},
"dagostino": {"stat": 18.1, "p": 1e-4, "normal": False},
"shapiro": {"stat": 0.98, "p": 7e-8, "normal": False},
"is_normal": False},
"C": {"n": 100,
"jarque_bera": {"stat": 2.1, "p": 0.35, "normal": True},
"dagostino": {"stat": 1.9, "p": 0.38, "normal": True},
"shapiro": {"stat": 0.99, "p": 0.12, "normal": True},
"is_normal": True},
},
},
}
def _dummy_chapters():
"""A minimal one-chapter document so render_md does not early-return empty."""
return model.as_chapters([
{"id": "intro", "title": "Intro",
"blocks": [{"kind": "markdown", "text": "cuerpo del informe"}]},
])
def _render(tmp_path, profile):
out = os.path.join(str(tmp_path), "out.md")
res = render_md(_dummy_chapters(), out, {"title": "EDA — t", "profile": profile})
assert res["path"] == out
return open(out, encoding="utf-8").read()
def _table_rows(md, section_title):
"""Count data rows of the first Markdown table under ``section_title``."""
seg = md.split(section_title, 1)[1]
rows, in_t, seen_sep = 0, False, False
for ln in seg.splitlines():
if ln.startswith("|"):
in_t = True
stripped = ln.replace("|", "").replace(" ", "")
if stripped and set(stripped) == {"-"}:
seen_sep = True
continue
if seen_sep:
rows += 1
elif in_t and not ln.strip():
break
return rows
# --------------------------------------------------------------------------- #
# Golden: every datum the profile holds reaches the .md.
# --------------------------------------------------------------------------- #
def test_appendix_lists_all_correlation_pairs(tmp_path):
md = _render(tmp_path, _profile())
assert "## Apéndice — Datos completos del perfil" in md
# All 4 pairs (the real titanic profile has 28; here 4 synthetic).
assert _table_rows(md, "### Matriz de asociación") == 4
# The cat-cat Cramér's V pair the human chapter drops is present.
assert "Cat1 ↔ Cat2" in md
assert "cramers_v" in md
assert "correlation_ratio" in md
def test_appendix_has_skew_kurtosis_for_every_numeric(tmp_path):
md = _render(tmp_path, _profile())
seg = md.split("### Estadísticos numéricos completos", 1)[1].split("###", 1)[0]
lines = [l for l in seg.splitlines() if l.startswith("|")]
header = [h.strip() for h in lines[0].strip("|").split("|")]
assert "skew" in header and "kurtosis" in header
ski, kui = header.index("skew"), header.index("kurtosis")
data = lines[2:] # skip header + separator
assert len(data) == 3 # exactly the 3 numeric columns
for row in data:
cells = [c.strip() for c in row.strip("|").split("|")]
assert cells[ski] != "", f"missing skew in {cells[0]}"
assert cells[kui] != "", f"missing kurtosis in {cells[0]}"
def test_appendix_has_extended_percentiles(tmp_path):
md = _render(tmp_path, _profile())
seg = md.split("### Estadísticos numéricos completos", 1)[1]
header = [h.strip() for h in seg.splitlines()[2].strip("|").split("|")]
for p in ("p1", "p5", "p25", "p75", "p95", "p99"):
assert p in header, f"percentile {p} missing from describe header"
def test_appendix_names_concrete_reexpression(tmp_path):
md = _render(tmp_path, _profile())
assert "### Re-expresión recomendada" in md
assert "log1p" in md # the concrete transform, not just "consider re-expressing"
assert "yeo-johnson" in md # alternatives listed too
def test_appendix_has_kmeans_scores_by_k(tmp_path):
md = _render(tmp_path, _profile())
assert "scores_by_k" in md
assert _table_rows(md, "#### KMeans — selección de k") == 3 # k=2,3,4
def test_appendix_has_normality_statistics(tmp_path):
md = _render(tmp_path, _profile())
assert "JB stat" in md # the statistic, not only the p-value
assert "Shapiro stat" in md
assert _table_rows(md, "#### Tests de normalidad") == 2 # cols A and C
# --------------------------------------------------------------------------- #
# Edge: a profile missing models / correlations degrades, never raises.
# --------------------------------------------------------------------------- #
def test_lite_profile_without_models(tmp_path):
prof = _profile()
prof.pop("models") # lite: no KMeans/normality
md = _render(tmp_path, prof)
assert "scores_by_k" not in md # section skipped
assert "Matriz de asociación" in md # correlations still dumped
assert "## Apéndice" in md
def test_profile_without_correlations(tmp_path):
prof = _profile()
prof.pop("correlations")
md = _render(tmp_path, prof) # must not raise
assert "Matriz de asociación" not in md
assert "Estadísticos numéricos completos" in md # numeric section still there
def test_no_profile_means_no_appendix(tmp_path):
out = os.path.join(str(tmp_path), "noprof.md")
res = render_md(_dummy_chapters(), out, {"title": "x"})
assert res["path"] == out
assert "## Apéndice" not in open(out, encoding="utf-8").read()
def test_appendix_helper_is_defensive():
assert _profile_appendix(None) == ""
assert _profile_appendix({}) == ""
assert _profile_appendix({"columns": []}) == ""
# --------------------------------------------------------------------------- #
# Loss #6: bar/scree figure tables get a non-misleading header.
# --------------------------------------------------------------------------- #
def test_histogram_caption_detection():
assert _is_histogram_caption("Histograma de Age")
assert _is_histogram_caption("Distribución de Fare")
assert not _is_histogram_caption("Media de Survived por Sex")
assert not _is_histogram_caption("Varianza explicada (scree PCA)")
def test_bars_table_custom_header():
bars = [(0.0, 1.0, 5.0), (1.0, 2.0, 3.0)]
hist = _bars_table(bars) # default histogram header
assert "| Desde | Hasta | Frecuencia |" in hist
bar = _bars_table(bars, ("Inicio", "Fin", "Valor"))
assert "| Inicio | Fin | Valor |" in bar
assert "Frecuencia" not in bar
@@ -128,6 +128,46 @@ class Note:
kind: str = field(default="note", init=False)
@dataclass
class Group:
"""A keep-together unit: its blocks render on the SAME page/slide.
Renderers measure the whole group first; if it does not fit in the remaining
space they move it *whole* to the next page (PDF) or slide (PPTX) before
drawing anything so a heading never gets stranded apart from the figure and
text it introduces. If the group is taller than a full page even on its own,
it starts on a fresh page and flows (honest degradation, never cut). Use it to
bind ``Heading`` + ``Markdown`` + ``Figure`` of one idea together (see the
DISTR NUM / AGREGACION chapters).
When ``page_break_before`` is True the renderer additionally forces the group
to *start* on a fresh page/slide (unless the current one is already empty), so
a chapter can give each unit its own page e.g. one categorical column per
page (see CAT DISTR). It is purely additive: the default False keeps the plain
keep-together behaviour for every existing chapter.
"""
blocks: list = field(default_factory=list)
title: Optional[str] = None
page_break_before: bool = False
kind: str = field(default="group", init=False)
@dataclass
class GlossaryEntry:
"""One glossary term: a clickable destination at the end of the document.
Rendered as the term ``label`` (heading) plus its ``definition`` (markdown).
The renderers register its page/slide position as the link target so every
in-text appearance of the same ``key`` becomes a real clickable jump (PDF link
annotation via PyMuPDF; PPTX internal slide jump)."""
key: str = ""
label: str = ""
definition: str = ""
kind: str = field(default="glossary_entry", init=False)
@dataclass
class Chapter:
"""An ordered set of blocks with an id, a title and a generation version."""
@@ -150,13 +190,17 @@ _BLOCK_BY_KIND = {
"image": Image,
"caption": Caption,
"note": Note,
"group": Group,
"glossary_entry": GlossaryEntry,
}
def as_block(obj: Any):
"""Coerce a value into a block dataclass. Unknown values become a Note."""
if isinstance(obj, (Heading, Markdown, KVTable, DataTable, Figure, Image,
Caption, Note)):
Caption, Note, Group, GlossaryEntry)):
if isinstance(obj, Group):
obj.blocks = as_blocks(obj.blocks)
return obj
if isinstance(obj, dict):
kind = obj.get("kind")
@@ -189,6 +233,15 @@ def as_block(obj: Any):
return Caption(text=_safe_str(obj.get("text")))
if cls is Note:
return Note(text=_safe_str(obj.get("text")))
if cls is Group:
return Group(blocks=as_blocks(obj.get("blocks")),
title=obj.get("title"),
page_break_before=bool(
obj.get("page_break_before", False)))
if cls is GlossaryEntry:
return GlossaryEntry(key=_safe_str(obj.get("key")),
label=_safe_str(obj.get("label")),
definition=_safe_str(obj.get("definition")))
except Exception: # noqa: BLE001 — never raise on a malformed block.
return Note(text=_safe_str(obj))
return Note(text=_safe_str(obj))
@@ -246,6 +299,67 @@ def _safe_str(v: Any) -> str:
return ""
# --------------------------------------------------------------------------- #
# Glossary collector — chapters register the terms they use; the glosario
# chapter renders them at the end and the renderers wire the clickable links.
# --------------------------------------------------------------------------- #
class GlossaryCollector:
"""Accumulates glossary terms registered by chapters during document build.
A single instance is created by :func:`build_document` and passed to every
chapter via ``ctx['glossary']``. A chapter calls ``add(key, label,
definition)`` to declare a term it explains (e.g. ``"entropia"``
"Entropía"), and marks each in-text appearance with the inline span
``[[term:key]]texto visible[[/term]]`` (see ``text_layout.parse_inline_rich``).
The ``glosario`` chapter reads ``terms()`` to emit one :class:`GlossaryEntry`
per term; the renderers turn every marked appearance into a real click that
jumps to that entry. First registration of a key wins (idempotent); never
raises."""
def __init__(self):
self._terms: dict = {}
self._order: list = []
def add(self, key: Any, label: Any = None, definition: Any = "") -> str:
"""Register a term and return its normalized key (''. if invalid)."""
try:
k = _safe_str(key).strip()
if not k:
return ""
if k not in self._terms:
self._terms[k] = {
"key": k,
"label": _safe_str(label).strip() or k,
"definition": _safe_str(definition),
}
self._order.append(k)
return k
except Exception: # noqa: BLE001 — collecting a term never breaks a build.
return ""
def has(self, key: Any) -> bool:
return _safe_str(key).strip() in self._terms
def get(self, key: Any) -> Optional[dict]:
return self._terms.get(_safe_str(key).strip())
def terms(self, by: str = "label") -> list:
"""Return the registered terms as dicts.
``by='label'`` (default) sorts alphabetically by visible label;
``by='order'`` keeps first-appearance order."""
if by == "order":
return [self._terms[k] for k in self._order]
return sorted(self._terms.values(),
key=lambda t: _safe_str(t.get("label")).lower())
def __len__(self) -> int:
return len(self._terms)
def __bool__(self) -> bool:
return bool(self._terms)
# --------------------------------------------------------------------------- #
# Manifest — per-chapter versions and page/slide counts for tracking.
# --------------------------------------------------------------------------- #
@@ -0,0 +1,354 @@
"""Tests for the AutomaticEDA engine features added in phase 4a.
Covers, with executable evidence, the six render-engine improvements:
1. Bold no longer overlaps the following text in the PDF (real width measured).
2. Zebra striping on data tables (PDF Rectangle fills + PPTX cell fills).
3. Keep-together: a Group moves whole to the next page/slide (heading never gets
stranded from its figure).
4. Every PPTX figure carries a visible caption/title (fallback to the heading).
5. Cover is built last but placed first and reflects an aggregated summary.
6. Glossary is the last chapter; the term "entropía" is a real clickable link in
the PDF (PyMuPDF GOTO annotation) and in the PPTX (native slide-jump run).
Self-contained: synthetic profiles, no DuckDB. Heavy renderer checks (fitz/pptx)
skip cleanly when the optional engine is missing.
"""
import os
import sys
import pytest
_HERE = os.path.dirname(os.path.abspath(__file__))
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
if _FUNCTIONS not in sys.path:
sys.path.insert(0, _FUNCTIONS)
import matplotlib # noqa: E402
matplotlib.use("Agg")
import matplotlib.colors as mcolors # noqa: E402
import matplotlib.pyplot as plt # noqa: E402
from matplotlib.patches import Rectangle # noqa: E402
from datascience.automatic_eda import model # noqa: E402
from datascience.automatic_eda import render_pdf_impl as RP # noqa: E402
from datascience.automatic_eda import render_pptx_impl as RX # noqa: E402
from datascience.automatic_eda import build_document # noqa: E402
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf # noqa: E402
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx # noqa: E402
class _FakePdf:
"""Stand-in for PdfPages so the placers can call _new_page in unit tests."""
def savefig(self, fig): # noqa: D401
pass
def _small_fig():
fig = plt.figure(figsize=(4.0, 1.5))
ax = fig.add_subplot(111)
ax.plot([0, 1, 2], [1, 3, 2])
return fig
def _profile_with_cat_and_num():
"""A tiny profile that triggers cat_distr (→ entropía term) and num_distr."""
return {
"table": "ventas", "n_rows": 120, "n_cols": 2, "quality_score": 91,
"duplicate_pct": 1.5, "null_cell_pct": 0.8,
"columns": [
{"name": "region", "inferred_type": "categorical",
"categorical": {
"top": [{"value": "norte", "count": 50, "pct": 0.42},
{"value": "sur", "count": 40, "pct": 0.33},
{"value": "este", "count": 30, "pct": 0.25}],
"mode": "norte", "n_distinct": 3, "entropy": 1.55,
"imbalance": 0.1}},
{"name": "importe", "inferred_type": "numeric",
"numeric": {"mean": 50.0, "median": 48.0, "std": 10.0,
"min": 10, "max": 99, "iqr": 15,
"histogram": [{"lo": 0, "hi": 50, "count": 40},
{"lo": 50, "hi": 100, "count": 80}]}},
],
}
# --------------------------------------------------------------------------- #
# 1) Bold does not overlap the following text (PDF).
# --------------------------------------------------------------------------- #
def test_pdf_bold_span_does_not_overlap_following_text():
fig = plt.figure(figsize=(RP._W, RP._H))
st = RP._PdfState(_FakePdf(), "t")
st.fig = fig
st.page = 1
# A wide bold token immediately followed by normal text on the SAME line.
rich = [[("PALABRAMUYANCHAENNEGRITA", True, None),
(" texto normal justo después", False, None)]]
RP._place_rich_lines(st, rich, RP._FS_BODY, RP._INK)
renderer = fig.canvas.get_renderer()
boxes = sorted((t.get_window_extent(renderer) for t in fig.texts),
key=lambda b: b.x0)
assert len(boxes) == 2, "se esperaban dos spans dibujados"
# The bold span ends before the normal span starts (no overlap). 1px slack.
assert boxes[0].x1 <= boxes[1].x0 + 1.0, \
"la negrita se solapa con el texto siguiente"
plt.close(fig)
# --------------------------------------------------------------------------- #
# 2) Zebra striping.
# --------------------------------------------------------------------------- #
def _facecolor_eq(artist, hexcolor) -> bool:
want = mcolors.to_rgba(hexcolor)
got = artist.get_facecolor()
return all(abs(a - b) < 0.02 for a, b in zip(got[:3], want[:3]))
def test_pdf_table_has_zebra_striping():
fig = plt.figure(figsize=(RP._W, RP._H))
st = RP._PdfState(_FakePdf(), "t")
st.fig = fig
st.page = 1
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
dt = model.DataTable(header=["A", "B"],
rows=[["1", "x"], ["2", "y"], ["3", "z"], ["4", "w"]])
RP._place_data_table(st, dt)
zebra = [a for a in fig.findobj(Rectangle) if _facecolor_eq(a, RP._ZEBRA)]
# 4 data rows → even rows (1-based 2 and 4) shaded = 2 zebra rectangles.
assert len(zebra) == 2, f"esperadas 2 filas zebra, hay {len(zebra)}"
plt.close(fig)
def test_pptx_table_has_zebra_striping(tmp_path):
pptx = pytest.importorskip("pptx")
from pptx import Presentation
from pptx.dml.color import RGBColor
doc = [model.Chapter(id="c", title="Tabla", version="1.0.0", blocks=[
model.DataTable(header=["A", "B"],
rows=[["1", "x"], ["2", "y"], ["3", "z"], ["4", "w"]])])]
out = str(tmp_path / "zebra.pptx")
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
prs = Presentation(out)
table = None
for slide in prs.slides:
for sh in slide.shapes:
if sh.has_table:
table = sh.table
break
assert table is not None, "no se encontró la tabla en el deck"
zebra = RGBColor(0xF6, 0xF8, 0xFA)
white = RGBColor(0xFF, 0xFF, 0xFF)
# Row 0 = header; data rows follow. Even data rows (table rows 2, 4) shaded.
assert table.cell(1, 0).fill.fore_color.rgb == white
assert table.cell(2, 0).fill.fore_color.rgb == zebra
assert table.cell(4, 0).fill.fore_color.rgb == zebra
# --------------------------------------------------------------------------- #
# 3) Keep-together (Group): heading + figure never split.
# --------------------------------------------------------------------------- #
def test_pdf_group_moves_whole_to_next_page_when_it_does_not_fit():
fig = plt.figure(figsize=(RP._W, RP._H))
st = RP._PdfState(_FakePdf(), "t")
st.fig = fig
st.page = 1
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
grp = model.Group(blocks=[
model.Heading(text="Sección con figura", level=2),
model.Figure(make=_small_fig, caption="cap"),
model.Markdown(text="Descripción breve de la figura."),
])
# Only ~0.4in left: the group does not fit here but fits on a fresh page.
st.y = RP._CONTENT_BOTTOM - 0.4
page_before = st.page
RP._place_group(st, grp)
# Exactly one page break: the whole group (heading+figure+text) stays
# together on the new page — no second break inside it.
assert st.page == page_before + 1
plt.close(st.fig)
def test_pdf_group_does_not_break_when_it_fits():
fig = plt.figure(figsize=(RP._W, RP._H))
st = RP._PdfState(_FakePdf(), "t")
st.fig = fig
st.page = 1
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
grp = model.Group(blocks=[
model.Heading(text="Cabe entera", level=2),
model.Figure(make=_small_fig, caption="cap"),
])
st.y = RP._CONTENT_TOP # empty page → fits, must not break.
page_before = st.page
RP._place_group(st, grp)
assert st.page == page_before
plt.close(st.fig)
def test_pptx_group_moves_whole_to_next_slide(tmp_path):
pytest.importorskip("pptx")
from pptx import Presentation
from pptx.util import Inches
prs = Presentation()
prs.slide_width = Inches(RX._W)
prs.slide_height = Inches(RX._H)
st = RX._PptxState(prs, "t")
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
RX._new_slide(st, cont=False)
grp = model.Group(blocks=[
model.Heading(text="Sección con figura", level=2),
model.Figure(make=_small_fig, caption="cap"),
model.Markdown(text="Descripción breve."),
])
st.y = RX._CONTENT_BOTTOM - 0.4 # does not fit here.
slide_before = st.slide_no
RX._place_group(st, grp)
assert st.slide_no == slide_before + 1 # one jump; group kept together.
# --------------------------------------------------------------------------- #
# 4) Every PPTX figure carries a visible caption/title.
# --------------------------------------------------------------------------- #
def test_pptx_figure_without_caption_gets_heading_title(tmp_path):
pytest.importorskip("pptx")
from pptx import Presentation
from pptx.enum.shapes import MSO_SHAPE_TYPE
doc = [model.Chapter(id="c", title="Cap", version="1.0.0", blocks=[
model.Heading(text="Mi sección gráfica", level=2),
model.Figure(make=_small_fig), # NO caption provided.
])]
out = str(tmp_path / "cap.pptx")
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
prs = Presentation(out)
for slide in prs.slides:
has_pic = any(sh.shape_type == MSO_SHAPE_TYPE.PICTURE
for sh in slide.shapes)
if not has_pic:
continue
italic = [r.text for sh in slide.shapes if sh.has_text_frame
for p in sh.text_frame.paragraphs for r in p.runs
if r.font.italic and r.text.strip()]
assert italic, "la figura no lleva caption visible en su slide"
assert any("Mi sección gráfica" in t for t in italic), \
"el caption no cayó al título de la sección"
return
pytest.fail("no se encontró ningún slide con imagen")
def test_pptx_no_figure_slide_is_ever_untitled(tmp_path):
"""Invariant: across many figures (incl. tall ones), NO slide with an image
lacks a visible caption the caption never spills to the next slide."""
pytest.importorskip("pptx")
from pptx import Presentation
from pptx.enum.shapes import MSO_SHAPE_TYPE
def _tall_fig():
fig = plt.figure(figsize=(5.0, 4.6)) # nearly square → fills the slide.
fig.add_subplot(111).bar([1, 2, 3], [4, 5, 6])
return fig
blocks = []
for i in range(6):
blocks.append(model.Heading(text=f"Gráfico {i}", level=2))
blocks.append(model.Figure(
make=_tall_fig,
caption=("Una descripción de la figura deliberadamente larga para "
"que el caption ocupe más de una línea al envolverse en el "
f"ancho del slide — figura número {i} del bloque.")))
doc = [model.Chapter(id="c", title="Muchas figuras", version="1.0.0",
blocks=blocks)]
out = str(tmp_path / "many.pptx")
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
prs = Presentation(out)
missing = []
pics = 0
for i, slide in enumerate(prs.slides):
if not any(sh.shape_type == MSO_SHAPE_TYPE.PICTURE
for sh in slide.shapes):
continue
pics += 1
italic = [r.text for sh in slide.shapes if sh.has_text_frame
for p in sh.text_frame.paragraphs for r in p.runs
if r.font.italic and r.text.strip()]
if not italic:
missing.append(i)
assert pics >= 6, f"esperadas >=6 figuras, hay {pics}"
assert not missing, f"slides con imagen sin caption: {missing}"
# --------------------------------------------------------------------------- #
# 5) Cover built last, placed first, with an aggregated summary.
# --------------------------------------------------------------------------- #
def test_cover_first_glossary_last_with_summary():
chs = build_document(_profile_with_cat_and_num(), ctx={"dataset_name": "v"})
ids = [c.id for c in chs]
assert ids[0] == "portada", f"la portada no es la primera: {ids}"
assert ids[-1] == "glosario", f"el glosario no es el último: {ids}"
cover = chs[0]
headings = [b.text for b in cover.blocks if b.kind == "heading"]
assert any("Resumen" in h for h in headings), \
"la portada no incluye el resumen agregado"
# The summary reflects the body chapters (e.g. the numeric/categorical ones).
cover_text = " ".join(
b.text for b in cover.blocks if getattr(b, "kind", "") == "markdown")
assert "Distribuciones" in cover_text, \
"el resumen de portada no menciona los capítulos del cuerpo"
# --------------------------------------------------------------------------- #
# 6) Glossary clickable in PDF (PyMuPDF GOTO) and PPTX (native slide jump).
# --------------------------------------------------------------------------- #
def test_pdf_glossary_term_is_clickable(tmp_path):
fitz = pytest.importorskip("fitz")
out = str(tmp_path / "glos.pdf")
res = render_automatic_eda_pdf(_profile_with_cat_and_num(), out,
{"ctx": {"dataset_name": "v"},
"write_manifest": False})
assert res["path"] == out and os.path.exists(out)
doc = fitz.open(out)
goto = [(pno, l) for pno in range(doc.page_count)
for l in doc[pno].get_links() if l.get("kind") == fitz.LINK_GOTO]
doc.close()
assert goto, "no hay ningún enlace interno (entropía → glosario) en el PDF"
# Destination must be a real page in the document (the glossary page).
assert all(0 <= l.get("page", -1) for _p, l in goto)
def test_pptx_glossary_term_is_clickable(tmp_path):
pytest.importorskip("pptx")
from pptx import Presentation
from pptx.oxml.ns import qn
out = str(tmp_path / "glos.pptx")
res = render_automatic_eda_pptx(_profile_with_cat_and_num(), out,
{"ctx": {"dataset_name": "v"},
"write_manifest": False})
assert res["path"] == out and os.path.exists(out)
prs = Presentation(out)
found = False
for slide in prs.slides:
for sh in slide.shapes:
if not sh.has_text_frame:
continue
for p in sh.text_frame.paragraphs:
for r in p.runs:
rpr = r._r.find(qn("a:rPr"))
if rpr is None:
continue
hl = rpr.find(qn("a:hlinkClick"))
if hl is not None and \
hl.get("action") == "ppaction://hlinksldjump":
found = True
assert found, "ningún término tiene hyperlink de salto a slide en el PPTX"
@@ -0,0 +1,748 @@
"""AutomaticEDA Markdown serializer — one self-contained file to paste to an LLM.
Same document model as the PDF/PPTX renderers (an ordered list of
:class:`Chapter`, each a list of format-independent blocks) but emitted as plain
**Markdown** instead of a binary. The goal is different from the other two
renderers: a Markdown EDA is meant to be *pasted into an LLM*, so it prioritises
TEXT and DATA over visuals. Tables become Markdown tables (every row dumped, no
pagination nothing is cut because there are no pages); a ``Figure`` becomes its
caption plus, when possible, the underlying bar/histogram data as a Markdown
table (an LLM cannot see the image); glossary term markers are stripped while
``**bold**`` is kept (it is valid Markdown).
dict-no-throw (the ``eda`` group style): :func:`render_md` never raises. On a
fatal error it returns ``{path: None, ...}`` with a ``note`` explaining why; a
malformed block degrades to a readable note rather than crashing the document.
"""
from __future__ import annotations
import os
import re
from . import model
# Glossary span markers (kept text, dropped markers). We intentionally do NOT use
# ``text_layout.strip_inline_md`` for Markdown blocks because that also removes
# ``**bold**`` — valid Markdown we want to preserve when pasting to an LLM.
_TERM_OPEN_RE = re.compile(r"\[\[term:[A-Za-z0-9_]+\]\]")
_MAX_BAR_ROWS = 100
# --------------------------------------------------------------------------- #
# Small helpers.
# --------------------------------------------------------------------------- #
def _clean_terms(s) -> str:
"""Drop glossary term markers, keeping the visible text (and any **bold**)."""
s = model._safe_str(s)
s = _TERM_OPEN_RE.sub("", s)
return s.replace("[[/term]]", "")
def _cell(v) -> str:
"""Render a value as a safe Markdown table cell.
Escapes pipes (``|`` -> ``\\|``) so they do not break the column layout and
folds newlines to ``<br>`` so a multi-line value stays inside one cell. None
becomes an empty string.
"""
s = model._safe_str(v)
s = s.replace("|", "\\|")
s = s.replace("\r\n", "\n").replace("\r", "\n").replace("\n", "<br>")
return s
def _slug(text: str) -> str:
"""GitHub-style heading anchor: lowercase, spaces->'-', drop other symbols."""
s = model._safe_str(text).strip().lower()
out = []
for ch in s:
if ch.isalnum():
out.append(ch)
elif ch in " -":
out.append("-")
# any other symbol is dropped.
slug = "".join(out)
while "--" in slug:
slug = slug.replace("--", "-")
return slug.strip("-")
def _fmt_num(v) -> str:
"""Compact number for the figure data tables (ints as ints, else 4 sig figs)."""
try:
f = float(v)
except Exception: # noqa: BLE001
return model._safe_str(v)
if f != f: # NaN
return "NaN"
if f == int(f) and abs(f) < 1e15:
return str(int(f))
return f"{f:.4g}"
def _fmt_int(v) -> str:
try:
return str(int(v))
except Exception: # noqa: BLE001
return model._safe_str(v)
def _now_iso() -> str:
from datetime import datetime, timezone
return datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
# --------------------------------------------------------------------------- #
# Document header (title + metadata blockquote + numbered index).
# --------------------------------------------------------------------------- #
def _meta_block(meta: dict) -> list:
"""Build the metadata lines for the header blockquote (omitting absentees)."""
ctx = meta.get("ctx") if isinstance(meta.get("ctx"), dict) else {}
lines: list = []
def add(label, value) -> None:
if value is None:
return
s = model._safe_str(value).strip()
if s and s.lower() != "none":
lines.append(f"**{label}:** {s}")
add("Dataset", ctx.get("dataset_name") or meta.get("dataset_name"))
add("Fuente", ctx.get("source_origin") or meta.get("source_origin"))
add("Almacenamiento", ctx.get("storage") or meta.get("storage"))
n_rows = ctx.get("n_rows", meta.get("n_rows"))
n_cols = ctx.get("n_cols", meta.get("n_cols"))
if n_rows is not None and n_cols is not None:
lines.append(
f"**Dimensiones:** {_fmt_int(n_rows)} filas × {_fmt_int(n_cols)} columnas")
add("Generado", meta.get("generated_at") or _now_iso())
lines.append(f"**Motor:** {model.ENGINE_NAME} v{model.ENGINE_VERSION}")
return lines
# --------------------------------------------------------------------------- #
# Per-block serializers. Each returns a Markdown string (no surrounding blanks;
# the caller separates blocks with a blank line).
# --------------------------------------------------------------------------- #
def _md_heading(block) -> str:
level = int(getattr(block, "level", 1) or 1)
hashes = "#" * min(level + 2, 6) # level1 -> ###; '#'/'##' reserved for doc/chapter.
text = _clean_terms(getattr(block, "text", "")).strip()
return f"{hashes} {text}"
def _md_markdown(block) -> str:
# Keep the text verbatim, dropping only glossary markers (keep **bold**).
return _clean_terms(getattr(block, "text", "")).rstrip("\n")
def _md_kv_table(block) -> str:
lines: list = []
title = getattr(block, "title", None)
if title:
lines.append(f"**{_clean_terms(title).strip()}**")
lines.append("")
lines.append("| Campo | Valor |")
lines.append("| --- | --- |")
for row in (getattr(block, "rows", []) or []):
try:
label, value = row[0], row[1]
except Exception: # noqa: BLE001
label, value = row, ""
lines.append(f"| {_cell(label)} | {_cell(value)} |")
return "\n".join(lines)
def _md_data_table(block) -> str:
lines: list = []
title = getattr(block, "title", None)
if title:
lines.append(f"**{_clean_terms(title).strip()}**")
lines.append("")
header = list(getattr(block, "header", []) or [])
rows = list(getattr(block, "rows", []) or [])
if not header:
ncol = max((len(r) for r in rows), default=1)
header = [f"col{i + 1}" for i in range(ncol)]
ncol = len(header)
lines.append("| " + " | ".join(_cell(h) for h in header) + " |")
lines.append("| " + " | ".join(["---"] * ncol) + " |")
for r in rows: # dump every row — no pagination, nothing cut.
cells = [_cell(r[c]) if c < len(r) else "" for c in range(ncol)]
lines.append("| " + " | ".join(cells) + " |")
note = getattr(block, "note", None)
if note:
lines.append("")
lines.append(f"*{_clean_terms(note).strip()}*")
return "\n".join(lines)
def _bars_table(bars: list, header: tuple = ("Desde", "Hasta", "Frecuencia")) -> str:
"""Render extracted bar/histogram data as a Markdown table.
``header`` is the 3-column header to use. Histogram bars are
``(Desde, Hasta, Frecuencia)``; bar/scree charts (means by group, PCA
explained variance) are *not* bins, so the caller passes a semantically
correct header (e.g. ``(Inicio, Fin, Valor)``) to avoid the misleading
"Frecuencia" label see report 2053, loss #6.
"""
h0, h1, h2 = header
lines = [f"| {h0} | {h1} | {h2} |", "| --- | --- | --- |"]
shown = bars[:_MAX_BAR_ROWS]
for x0, x1, h in shown:
lines.append(f"| {_fmt_num(x0)} | {_fmt_num(x1)} | {_fmt_num(h)} |")
out = "\n".join(lines)
extra = len(bars) - len(shown)
if extra > 0:
out += f"\n\n*… ({extra} filas más)*"
return out
def _is_histogram_caption(caption: str) -> bool:
"""True when a figure caption describes a histogram (genuine numeric bins).
Histograms are the only figures whose bars are real ``[Desde, Hasta)`` bins
with a frequency count. Bar charts (means by group) and the PCA scree plot
carry per-category / per-component values, not bins they must not inherit
the ``Desde/Hasta/Frecuencia`` header.
"""
c = (caption or "").lower()
return "histograma" in c or "distribución" in c or "distribucion" in c
def _extract_bars(fig) -> list:
"""Collect (x_from, x_to, height) of the rectangular bars of a matplotlib fig.
Histogram / bar-chart bars are ``matplotlib.patches.Rectangle`` with positive
width and height; spines, legends and zero-area artists are skipped. Never
raises returns ``[]`` on any problem.
"""
bars: list = []
try:
for ax in fig.get_axes():
# Collect this axes' positive-area rectangles, then keep only the ones
# that look like actual histogram/bar bins. Reference shapes that
# matplotlib also stores in ``ax.patches`` — most notably the ``±1σ``
# band drawn by ``axvspan`` (a single rectangle far wider than a bin)
# and a lone Tukey boxplot box — would otherwise show up as fake
# "bins". A histogram axes has several near-equal-width bars, so we
# drop any rectangle whose width is more than twice the median width
# of that axes' rectangles (the σ-band spans many bins; uniform bins
# all sit at the median width and stay).
ax_bars: list = []
for patch in list(getattr(ax, "patches", []) or []):
try:
w = patch.get_width()
h = patch.get_height()
x = patch.get_x()
except Exception: # noqa: BLE001 — not a Rectangle-like patch.
continue
if w and w > 0 and h and h > 0:
ax_bars.append((x, x + w, h))
if len(ax_bars) >= 3:
widths = sorted(b[1] - b[0] for b in ax_bars)
median_w = widths[len(widths) // 2]
if median_w > 0:
ax_bars = [b for b in ax_bars
if (b[1] - b[0]) <= 2.0 * median_w]
bars.extend(ax_bars)
except Exception: # noqa: BLE001
return []
return bars
def _md_figure(block, meta: dict, out_path: str, counter: list) -> str:
"""Serialize a Figure prioritising TEXT + DATA (an LLM cannot see the image).
Emits the caption, then if the matplotlib figure has bars a Markdown table
of the underlying (Desde, Hasta, Frecuencia) values. Optionally (when
``meta['embed_figures']`` is True) also exports a PNG beside the .md and adds
an image link; off by default so the Markdown stays self-contained.
"""
caption = model._safe_str(getattr(block, "caption", "")).strip()
parts = [f"*Figura: {caption}*" if caption else "*Figura*"]
fig = None
try:
import matplotlib
matplotlib.use("Agg") # defensive: headless rasterization backend.
fig = getattr(block, "fig", None)
make = getattr(block, "make", None)
if fig is None and callable(make):
fig = make()
if fig is not None:
bars = _extract_bars(fig)
if bars:
# A histogram's bars are genuine numeric bins (Desde/Hasta/
# Frecuencia). Bar charts and the PCA scree plot are not bins —
# give them a header that does not lie about "Frecuencia".
header = (("Desde", "Hasta", "Frecuencia")
if _is_histogram_caption(caption)
else ("Inicio", "Fin", "Valor"))
parts.append(_bars_table(bars, header))
if meta.get("embed_figures"):
png = _embed_png(fig, out_path, counter)
if png:
parts.append(f"![{caption}]({png})")
except Exception: # noqa: BLE001 — a bad figure degrades to just its caption.
pass
finally:
if fig is not None:
try:
import matplotlib.pyplot as plt
plt.close(fig)
except Exception: # noqa: BLE001
pass
return "\n\n".join(parts)
def _embed_png(fig, out_path: str, counter: list) -> str:
"""Export the figure to ``<basename>_figN.png`` beside the .md; return its name."""
try:
counter[0] += 1
base = os.path.splitext(os.path.basename(out_path))[0] or "figura"
name = f"{base}_fig{counter[0]}.png"
path = os.path.join(os.path.dirname(os.path.abspath(out_path)), name)
fig.savefig(path, format="png", dpi=120, bbox_inches="tight")
return name
except Exception: # noqa: BLE001
return ""
def _md_image(block) -> str:
path = model._safe_str(getattr(block, "path", ""))
caption = model._safe_str(getattr(block, "caption", "")).strip()
out = f"![{caption}]({path})"
if caption:
out += f"\n\n*{caption}*"
return out
def _md_caption(block) -> str:
return f"*{_clean_terms(getattr(block, 'text', '')).strip()}*"
def _md_note(block) -> str:
text = _clean_terms(getattr(block, "text", "")).strip()
lines = text.split("\n")
return "\n".join((f"> {ln}" if ln.strip() else ">") for ln in lines)
def _md_group(block, meta: dict, out_path: str, counter: list) -> str:
parts: list = []
title = getattr(block, "title", None)
if title:
parts.append(f"### {_clean_terms(title).strip()}")
for b in (getattr(block, "blocks", []) or []):
try:
seg = _serialize_block(b, meta, out_path, counter)
except Exception: # noqa: BLE001
seg = ""
if seg:
parts.append(seg)
return "\n\n".join(parts)
def _md_glossary_entry(block) -> str:
label = (model._safe_str(getattr(block, "label", "")).strip()
or model._safe_str(getattr(block, "key", "")).strip())
definition = _clean_terms(getattr(block, "definition", "")).strip()
out = f"### {label}"
if definition:
out += f"\n\n{definition}"
return out
def _serialize_block(block, meta: dict, out_path: str, counter: list) -> str:
"""Dispatch a single block to its Markdown serializer. Unknown -> note."""
kind = getattr(block, "kind", "")
if kind == "heading":
return _md_heading(block)
if kind == "markdown":
return _md_markdown(block)
if kind == "kv_table":
return _md_kv_table(block)
if kind == "data_table":
return _md_data_table(block)
if kind == "figure":
return _md_figure(block, meta, out_path, counter)
if kind == "image":
return _md_image(block)
if kind == "caption":
return _md_caption(block)
if kind == "note":
return _md_note(block)
if kind == "group":
return _md_group(block, meta, out_path, counter)
if kind == "glossary_entry":
return _md_glossary_entry(block)
# Unknown content -> readable note (mirrors the model's defensive coercion).
return _md_note(model.Note(text=model._safe_str(block)))
# --------------------------------------------------------------------------- #
# Profile appendix — the data the human-facing chapters drop.
#
# The chapter document (shared with the PDF/PPTX renderers) is designed for human
# reading and intentionally omits raw numbers: the correlation matrix shows only
# the top extremes, the numeric blocks skip skew/kurtosis/extended percentiles,
# the model chapter does not list ``scores_by_k`` or the normality test
# statistics. But the Markdown is meant to be *pasted into an LLM*, so it should
# carry EVERYTHING the engine computed. This appendix serializes the full
# ``profile`` (passed via ``meta['profile']``) as Markdown tables, additively:
# the PDF/PPTX are untouched, the .md simply has more than they do. Each section
# is emitted only when its source data is present, so a ``lite`` profile (no
# models) or a profile without correlations degrades cleanly instead of raising.
# See report 2053 for the six losses this closes.
# --------------------------------------------------------------------------- #
def _pair_types(a_type, b_type) -> str:
"""Short ``num↔cat`` label for an association pair's variable types."""
def short(t):
t = model._safe_str(t).lower()
if t.startswith("num"):
return "num"
if t.startswith("cat"):
return "cat"
return t or "?"
return f"{short(a_type)}{short(b_type)}"
def _app_correlations(corr: dict) -> str:
"""Loss #1 — every association pair (not just the top extremes).
Dumps all of ``correlations['pairs']`` as a table (pair · types · method ·
value · p · p-FDR · significant), ordered by |value| desc so the strongest
associations lead while nothing is cut. Includes the ``correlation_ratio``
(numcat) and ``cramers_v`` (catcat) pairs the human chapter never shows.
"""
pairs = list(corr.get("pairs", []) or [])
if not pairs:
return ""
def keyfn(p):
try:
return -abs(float(p.get("value")))
except Exception: # noqa: BLE001
return 0.0
pairs_sorted = sorted(pairs, key=keyfn)
lines = ["### Matriz de asociación — todos los pares",
"",
("| Par | Tipos | Método | Valor | p-value | p-ajustado (FDR) "
"| ¿Sig? |"),
"| --- | --- | --- | --- | --- | --- | --- |"]
for p in pairs_sorted:
par = f"{_cell(p.get('a'))}{_cell(p.get('b'))}"
types = _pair_types(p.get("a_type"), p.get("b_type"))
method = _cell(p.get("method"))
val = _fmt_num(p.get("value"))
pv = _fmt_num(p.get("p_value")) if p.get("p_value") is not None else ""
padj = (_fmt_num(p.get("p_value_adjusted"))
if p.get("p_value_adjusted") is not None else "")
sig = "" if p.get("significant") else "no"
lines.append(
f"| {par} | {types} | {method} | {val} | {pv} | {padj} | {sig} |")
mt = corr.get("multiple_testing") or {}
n_tests = mt.get("n_tests", corr.get("n_tests"))
n_rej = mt.get("n_rejected")
note_bits = [f"{len(pairs)} pares en total"]
if n_tests is not None and n_rej is not None:
note_bits.append(
f"{n_rej} de {n_tests} significativos tras corrección "
f"{model._safe_str(mt.get('method', 'FDR')).upper()}")
lines.append("")
lines.append(f"*{'; '.join(note_bits)}.*")
return "\n".join(lines)
# Numeric statistics, in serialization order: (profile key, column header).
_NUM_STATS = [
("count", "n"), ("mean", "mean"), ("median", "median"), ("mode", "mode"),
("std", "std"), ("variance", "variance"), ("cv", "cv"),
("skew", "skew"), ("kurtosis", "kurtosis"),
("min", "min"), ("p1", "p1"), ("p5", "p5"), ("p25", "p25"), ("p50", "p50"),
("p75", "p75"), ("p95", "p95"), ("p99", "p99"), ("iqr", "iqr"),
("max", "max"), ("n_outliers", "outliers"),
("distribution_type", "distribución"),
]
def _app_numeric_describe(columns: list) -> str:
"""Loss #2 — every numeric statistic for every numeric column.
One row per numeric column with the full describe: mean/median/mode/std/
variance/cv, skew & kurtosis (for ALL columns, not only the skewed ones),
p1/p5/p25/p50/p75/p95/p99, iqr, min/max, outliers and distribution_type.
"""
rows = []
for info in (columns or []):
num = info.get("numeric") if isinstance(info, dict) else None
if not num:
continue
name = _cell(info.get("name"))
cells = [name]
for key, _hdr in _NUM_STATS:
v = num.get("count" if key == "count" else key)
if key == "count":
v = num.get("count", info.get("count"))
if key == "distribution_type":
cells.append(_cell(v))
else:
cells.append(_fmt_num(v) if v is not None else "")
rows.append(cells)
if not rows:
return ""
header = ["Columna"] + [hdr for _k, hdr in _NUM_STATS]
lines = ["### Estadísticos numéricos completos (describe)",
"",
"| " + " | ".join(header) + " |",
"| " + " | ".join(["---"] * len(header)) + " |"]
for cells in rows:
lines.append("| " + " | ".join(cells) + " |")
return "\n".join(lines)
def _app_reexpression(columns: list) -> str:
"""Loss #3 — the concrete recommended re-expression per column.
Names the transform (log1p/sqrt/yeo-johnson/none) instead of a vague
"consider re-expressing", with the ladder power, reason and alternatives.
"""
rows = []
for info in (columns or []):
rx = info.get("reexpression") if isinstance(info, dict) else None
if not rx or not isinstance(rx, dict):
continue
rec = model._safe_str(rx.get("recommended")).strip()
if not rec:
continue
alts = rx.get("alternatives") or []
alt_txt = ", ".join(
model._safe_str(a.get("transform")) for a in alts
if isinstance(a, dict) and a.get("transform")) or ""
rows.append([
_cell(info.get("name")), _cell(rec),
_fmt_num(rx.get("ladder_power")) if rx.get("ladder_power") is not None else "",
_cell(rx.get("reason")), _cell(alt_txt),
])
if not rows:
return ""
lines = ["### Re-expresión recomendada (escalera de Tukey)",
"",
"| Columna | Recomendada | Potencia | Razón | Alternativas |",
"| --- | --- | --- | --- | --- |"]
for r in rows:
lines.append("| " + " | ".join(r) + " |")
return "\n".join(lines)
def _app_kmeans_scores(kmeans: dict) -> str:
"""Loss #4 — KMeans silhouette + inertia per k (justifies the chosen k)."""
scores = list(kmeans.get("scores_by_k", []) or [])
if not scores:
return ""
best_k = kmeans.get("best_k")
lines = ["#### KMeans — selección de k (`scores_by_k`)",
"",
"| k | Silhouette | Inercia | Elegido |",
"| --- | --- | --- | --- |"]
for s in scores:
if not isinstance(s, dict):
continue
k = s.get("k")
chosen = "" if best_k is not None and k == best_k else ""
lines.append(
f"| {_fmt_num(k)} | {_fmt_num(s.get('silhouette'))} "
f"| {_fmt_num(s.get('inertia'))} | {chosen} |")
return "\n".join(lines)
def _app_normality(normality: dict) -> str:
"""Loss #5 — each normality test's statistic next to its p-value."""
if not isinstance(normality, dict) or not normality:
return ""
lines = ["#### Tests de normalidad (estadístico + p-value)",
"",
("| Columna | n | JB stat | JB p | D'Agostino stat | D'Agostino p "
"| Shapiro stat | Shapiro p | ¿Normal? |"),
"| --- | --- | --- | --- | --- | --- | --- | --- | --- |"]
any_row = False
for col, res in normality.items():
if not isinstance(res, dict):
continue
jb = res.get("jarque_bera") or {}
da = res.get("dagostino") or {}
sh = res.get("shapiro") or {}
is_norm = "" if res.get("is_normal") else "no"
lines.append(
f"| {_cell(col)} | {_fmt_num(res.get('n')) if res.get('n') is not None else ''} "
f"| {_fmt_num(jb.get('stat'))} | {_fmt_num(jb.get('p'))} "
f"| {_fmt_num(da.get('stat'))} | {_fmt_num(da.get('p'))} "
f"| {_fmt_num(sh.get('stat'))} | {_fmt_num(sh.get('p'))} | {is_norm} |")
any_row = True
return "\n".join(lines) if any_row else ""
def _profile_appendix(profile: dict) -> str:
"""Build the full-data appendix from a TableProfile dict (additive).
Returns a Markdown ``## Apéndice`` section with one sub-table per loss the
human chapters drop, or ``""`` when the profile carries none of them. Never
raises: a missing/oddly-shaped section is skipped, not fatal.
"""
if not isinstance(profile, dict):
return ""
sections: list = []
try:
corr = profile.get("correlations") or {}
seg = _app_correlations(corr) if isinstance(corr, dict) else ""
if seg:
sections.append(seg)
except Exception: # noqa: BLE001
pass
try:
columns = profile.get("columns") or []
seg = _app_numeric_describe(columns)
if seg:
sections.append(seg)
seg = _app_reexpression(columns)
if seg:
sections.append(seg)
except Exception: # noqa: BLE001
pass
try:
models = profile.get("models") or {}
if isinstance(models, dict):
model_segs = []
seg = _app_kmeans_scores(models.get("kmeans") or {})
if seg:
model_segs.append(seg)
seg = _app_normality(models.get("normality") or {})
if seg:
model_segs.append(seg)
if model_segs:
sections.append(
"### Modelos — detalle\n\n" + "\n\n".join(model_segs))
except Exception: # noqa: BLE001
pass
if not sections:
return ""
intro = ("Volcado completo de los datos que el motor computó y que los "
"capítulos (pensados para lectura humana / PDF) resumen. "
"Pensado para que un LLM reconstruya el análisis entero.")
return ("## Apéndice — Datos completos del perfil\n\n"
f"*{intro}*\n\n" + "\n\n".join(sections))
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def render_md(chapters: list, out_path: str, meta: dict = None) -> dict:
"""Serialize a list of Chapters into a single self-contained Markdown file.
The output leads with ``# <title>``, a metadata blockquote and a numbered
``## Índice`` linking each chapter, then one ``## N. <title>`` section per
chapter with its blocks. Tables become Markdown tables (every row dumped),
figures become caption + underlying data table, glossary markers are stripped
while ``**bold**`` is kept. Designed to be pasted into an LLM.
Args:
chapters: a list of ``Chapter`` (dataclasses or dicts); normalized
defensively with ``model.as_chapters``.
out_path: filesystem path for the ``.md`` (parent dirs are created).
meta: optional dict. Recognised keys: ``title``, ``ctx`` (dict with
``dataset_name``/``source_origin``/``storage``/``n_rows``/``n_cols``),
``generated_at``, ``embed_figures`` (export PNGs beside the .md,
default False).
Returns:
dict (never raises): ``{path: str|None, n_chars: int,
chapters: list[{id, version}], note: str}``. On a fatal error ``path`` is
None and ``note`` explains why.
"""
meta = meta or {}
chapters = model.as_chapters(chapters)
title = model._safe_str(meta.get("title")) or model.ENGINE_NAME
# Edge: nothing to render -> a minimal but valid Markdown document.
if not chapters:
content = (f"# {title}\n\n"
"*(documento vacío — sin capítulos aplicables)*\n")
return _write(out_path, content, [], "documento vacío")
counter = [0] # document-wide figure counter for unique PNG names.
notes: list = []
segments: list = [f"# {title}"]
meta_lines = _meta_block(meta)
if meta_lines:
segments.append("\n".join(f"> {ln}" for ln in meta_lines))
# Numbered index. The anchor matches the chapter heading emitted below
# (``## N. <title>``) in GitHub slug style.
chap_heads = []
idx_lines = ["## Índice"]
for i, ch in enumerate(chapters, 1):
head_text = f"{i}. {model._safe_str(ch.title)}"
anchor = _slug(head_text)
chap_heads.append((head_text, anchor))
idx_lines.append(f"{i}. [{model._safe_str(ch.title)}](#{anchor})")
segments.append("\n".join(idx_lines))
chapters_meta = []
for i, ch in enumerate(chapters, 1):
segments.append("---")
head_text, _anchor = chap_heads[i - 1]
segments.append(f"## {head_text}")
blocks = list(ch.blocks or [])
# Omit a leading level-1 Heading that just repeats the chapter title.
if blocks:
b0 = blocks[0]
if (getattr(b0, "kind", "") == "heading"
and int(getattr(b0, "level", 1) or 1) == 1
and _clean_terms(getattr(b0, "text", "")).strip()
== model._safe_str(ch.title).strip()):
blocks = blocks[1:]
for block in blocks:
try:
seg = _serialize_block(block, meta, out_path, counter)
except Exception as e: # noqa: BLE001
seg = _md_note(model.Note(text=model._safe_str(block)))
notes.append(
f"bloque '{getattr(block, 'kind', '?')}' del capítulo "
f"'{ch.id}' degradado: {e}")
if seg:
segments.append(seg)
chapters_meta.append({"id": ch.id, "version": ch.version})
# Full-data appendix: dump everything the profile holds that the human
# chapters drop (additive — the .md ends up with more than the PDF/PPTX).
# Emitted only when a profile is supplied via meta['profile']; never fatal.
try:
appendix = _profile_appendix(meta.get("profile"))
except Exception as e: # noqa: BLE001
appendix = ""
notes.append(f"apéndice de perfil omitido: {e}")
if appendix:
segments.append("---")
segments.append(appendix)
content = "\n\n".join(segments) + "\n"
note = f"{len(content)} caracteres"
if notes:
note += " · " + "; ".join(notes)
return _write(out_path, content, chapters_meta, note)
def _write(out_path: str, content: str, chapters_meta: list, note: str) -> dict:
"""Write the Markdown to disk (creating parents). dict-no-throw."""
try:
parent = os.path.dirname(os.path.abspath(out_path))
os.makedirs(parent, exist_ok=True)
with open(out_path, "w", encoding="utf-8") as fh:
fh.write(content)
except Exception as e: # noqa: BLE001 — never raise from the writer.
return {"path": None, "n_chars": 0, "chapters": [],
"note": f"no se pudo escribir el Markdown: {e}"}
return {"path": out_path, "n_chars": len(content),
"chapters": chapters_meta, "note": note}
@@ -60,6 +60,8 @@ _FS_BODY, _FS_CELL, _FS_NOTE = 10.5, 9.0, 9.0
_GAP = 0.12 # vertical gap after a block, inches.
_CELL_PAD = 0.06 # horizontal padding inside a table cell, inches.
_ROW_VPAD = 0.05 # vertical padding inside a table row, inches.
_ZEBRA = "#f6f8fa" # very light grey for zebra-striped (even) table rows.
_LINK = "#2a6f97" # accent colour for clickable glossary terms.
class _PdfState:
@@ -73,6 +75,11 @@ class _PdfState:
self.page = 0 # global page counter.
self.chapter = None # current Chapter (for the footer).
self.chapter_pages = 0 # pages produced for the current chapter.
self.last_heading = "" # text of the most recent heading.
# Glossary wiring (mejora 6). Pages are 0-based; rects/points are in PDF
# points (1/72") with a top-left origin — same convention as PyMuPDF.
self.term_sources = [] # [{key, page, rect:[x0,y0,x1,y1]}]
self.term_dests = {} # key -> {page, point:[x,y]}
# --------------------------------------------------------------------------- #
@@ -121,6 +128,35 @@ def _draw_footer(st: _PdfState) -> None:
transform=st.fig.transFigure, color=_RULE, lw=0.6))
def _text_width_in(st: _PdfState, s: str, fs: float, bold: bool) -> float:
"""Real rendered width (inches) of ``s`` at ``fs`` with the given weight.
Measured with the Agg renderer's own font metrics (the same TrueType the PDF
backend embeds), so a **bold** span advances the cursor by its ACTUAL width
fixing the bug where bold text overlapped the following normal text because
the cursor advanced by the normal-weight average-glyph estimate. Falls back to
the deterministic character grid if the renderer is unavailable, so it never
raises.
"""
if not s:
return 0.0
try:
from matplotlib.font_manager import FontProperties
renderer = st.fig.canvas.get_renderer()
prop = FontProperties(family="sans-serif", size=fs,
weight="bold" if bold else "normal")
w_px, _h, _d = renderer.get_text_width_height_descent(s, prop, False)
return w_px / float(st.fig.dpi)
except Exception: # noqa: BLE001 — fall back to the conservative grid metric.
return tl.avg_char_width_in(fs) * len(s)
def _pt_rect(x0_in: float, y_top_in: float, x1_in: float,
y_bottom_in: float) -> list:
"""An inches box (top-left origin) → a PDF-points rect for PyMuPDF links."""
return [x0_in * 72.0, y_top_in * 72.0, x1_in * 72.0, y_bottom_in * 72.0]
def _remaining(st: _PdfState) -> float:
return _CONTENT_BOTTOM - st.y
@@ -138,6 +174,7 @@ def _place_heading(st: _PdfState, block) -> None:
level = max(1, min(3, int(getattr(block, "level", 1) or 1)))
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
text = tl.strip_inline_md(getattr(block, "text", ""))
st.last_heading = text or st.last_heading
max_chars = tl.chars_per_line(_USABLE_W, fs)
lines = tl.wrap(text, max_chars)
lh = tl.line_height_in(fs, leading=1.2)
@@ -169,6 +206,49 @@ def _place_text_lines(st: _PdfState, lines: list, fs: float, color: str,
st.y += lh
def _place_rich_lines(st: _PdfState, rich_lines: list, fs: float, color: str,
indent: float = 0.0, prefixes=None) -> None:
"""Draw pre-wrapped lines of styled segments (bold + clickable term spans).
Each line is a list of ``(text, is_bold)`` or ``(text, is_bold, term_key)``
segments. Segments are placed left-to-right, advancing x by the segment's
REAL rendered width (measured with the renderer's font metrics for the actual
weight) this is what stops a bold span from overlapping the following text:
the cursor no longer advances by the normal-weight estimate. A segment with a
``term_key`` is drawn in the accent colour and its rectangle is recorded in
``st.term_sources`` so it becomes a clickable jump to the glossary entry.
``prefixes`` is an optional ``(first_line, other_lines)`` pair (e.g. a
bullet) drawn before the segments.
"""
lh = tl.line_height_in(fs)
for idx, segs in enumerate(rich_lines):
_ensure_space(st, lh)
x = _ML + indent
if prefixes is not None:
prefix = prefixes[0] if idx == 0 else prefixes[1]
if prefix:
st.fig.text(_xf(x), _yf(st.y), prefix, fontsize=fs, color=color,
ha="left", va="top")
x += _text_width_in(st, prefix, fs, False)
for seg in segs:
if len(seg) == 3:
seg_text, is_bold, term = seg
else:
seg_text, is_bold, term = seg[0], seg[1], None
if seg_text == "":
continue
w = _text_width_in(st, seg_text, fs, bool(is_bold))
st.fig.text(_xf(x), _yf(st.y), seg_text, fontsize=fs,
color=(_LINK if term else color), ha="left", va="top",
fontweight="bold" if is_bold else "normal")
if term:
st.term_sources.append({
"key": term, "page": st.page - 1,
"rect": _pt_rect(x, st.y, x + w, st.y + lh)})
x += w
st.y += lh
def _place_markdown(st: _PdfState, block) -> None:
raw = getattr(block, "text", "") or ""
md_lines = str(raw).split("\n")
@@ -208,29 +288,26 @@ def _place_markdown(st: _PdfState, block) -> None:
i += 1
continue
if stripped.startswith("- ") or stripped.startswith("* "):
content = tl.strip_inline_md(stripped[2:])
content = stripped[2:] # keep inline markers for bold rendering.
bullet_chars = tl.chars_per_line(_USABLE_W - 0.22, _FS_BODY)
wrapped = tl.wrap(content, bullet_chars)
first = True
for w in wrapped:
prefix = "" if first else " "
_place_text_lines(st, [prefix + w], _FS_BODY, _INK,
indent=0.0)
first = False
rich = tl.wrap_rich_terms(content, bullet_chars)
_place_rich_lines(st, rich, _FS_BODY, _INK,
prefixes=("", " "))
i += 1
continue
# Plain paragraph (gather following plain lines into one paragraph).
para = [tl.strip_inline_md(stripped)]
para = [stripped] # keep inline markers; wrap_rich renders **bold**.
j = i + 1
while j < n:
nxt = md_lines[j].strip()
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
break
para.append(tl.strip_inline_md(nxt))
para.append(nxt)
j += 1
text = " ".join(para)
max_chars = tl.chars_per_line(_USABLE_W, _FS_BODY)
_place_text_lines(st, tl.wrap(text, max_chars), _FS_BODY, _INK)
_place_rich_lines(st, tl.wrap_rich_terms(text, max_chars), _FS_BODY,
_INK)
i = j
st.y += _GAP
@@ -297,15 +374,18 @@ def _wrap_row(cells: list, widths: list, fs: float) -> list:
def _draw_table_row(st: _PdfState, cells_lines: list, widths: list, fs: float,
y0: float, header: bool) -> float:
y0: float, header: bool, zebra: bool = False) -> float:
lh = tl.line_height_in(fs)
nlines = max((len(c) for c in cells_lines), default=1)
row_h = lh * nlines + _ROW_VPAD * 2
if header:
# Background: header band, or a faint zebra fill for even data rows. Drawn
# below the text/rule (zorder 0) so striping never hides cell content.
bg = _HEAD_BG if header else (_ZEBRA if zebra else None)
if bg is not None:
st.fig.add_artist(Rectangle(
(_xf(_ML), _yf(y0 + row_h)), _xf(_ML + _USABLE_W) - _xf(_ML),
_yf(y0) - _yf(y0 + row_h), transform=st.fig.transFigure,
color=_HEAD_BG, lw=0, zorder=0))
color=bg, lw=0, zorder=0))
x = _ML
for c, lines in enumerate(cells_lines):
for k, ln in enumerate(lines):
@@ -350,14 +430,18 @@ def _place_data_table(st: _PdfState, block) -> None:
+ _ROW_VPAD * 2
_ensure_space(st, header_h() + max(first_row_h, lh))
draw_header()
for r in rows:
# ``data_idx`` is the LOGICAL row index (not reset across page breaks) so the
# zebra pattern stays coherent when a long table splits and repeats the
# header: even rows (1-based) are shaded → 0-based odd indices.
for data_idx, r in enumerate(rows):
cells_lines = _wrap_row(r, widths, fs)
row_h = lh * max((len(c) for c in cells_lines), default=1) \
+ _ROW_VPAD * 2
if _remaining(st) < row_h:
_new_page(st)
draw_header() # repeat header on the continuation page.
st.y += _draw_table_row(st, cells_lines, widths, fs, st.y, header=False)
st.y += _draw_table_row(st, cells_lines, widths, fs, st.y,
header=False, zebra=(data_idx % 2 == 1))
note = getattr(block, "note", None)
if note:
_place_text_lines(st, tl.wrap(model._safe_str(note),
@@ -386,53 +470,98 @@ def _png_from_figure(fig) -> bytes:
return buf.read()
def _place_image_array(st: _PdfState, arr, caption) -> None:
def _figure_png_cached(block):
"""Rasterize a Figure to PNG bytes ONCE and cache (bytes, aspect).
Measuring (keep-together) and drawing must agree on the REAL aspect ratio:
``bbox_inches='tight'`` changes it vs ``figsize``, so we rasterize once and
reuse the bytes for both. Cached on the block; never raises."""
cached = getattr(block, "_aeda_png", None)
if cached is not None:
return cached
fig, owned = _resolve_figure(block)
data = None
if fig is not None:
try:
data = _png_from_figure(fig)
finally:
if owned:
try:
plt.close(fig)
except Exception: # noqa: BLE001
pass
aspect = 0.66
if data is not None:
try:
arr = mpimg.imread(io.BytesIO(data))
aspect = (arr.shape[0] / arr.shape[1]) if arr.shape[1] else 0.66
except Exception: # noqa: BLE001
aspect = 0.66
try:
block._aeda_png = (data, aspect)
return block._aeda_png
except Exception: # noqa: BLE001 — block may reject attributes; degrade.
return (data, aspect)
def _image_aspect(block) -> float:
"""Real aspect (h/w) of an Image block by path, for measurement."""
path = getattr(block, "path", "")
if path and os.path.exists(path):
try:
arr = mpimg.imread(path)
return (arr.shape[0] / arr.shape[1]) if arr.shape[1] else 0.66
except Exception: # noqa: BLE001
pass
return 0.66
def _place_image_array(st: _PdfState, arr, caption, max_h_in=None) -> None:
h_px, w_px = arr.shape[0], arr.shape[1]
aspect = (h_px / w_px) if w_px else 1.0
# Reserve the caption's REAL (possibly multi-line) height FIRST, then scale
# the image to (max_h - cap_reserve) so figure + caption always fit the same
# page. cap_reserve adds a cushion so the caption never spills to next page.
cap_lines = (tl.wrap(model._safe_str(caption),
tl.chars_per_line(_USABLE_W, _FS_NOTE))
if caption else [])
cap_real = tl.line_height_in(_FS_NOTE) * len(cap_lines) if caption else 0.0
cap_reserve = (cap_real + 0.04 + 0.08) if caption else 0.0
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
# height_in hint (model.Figure/Image): cap the height so a figure in a
# keep-together Group shrinks to leave room for its heading and text.
if isinstance(max_h_in, (int, float)) and max_h_in > 0:
max_h = min(max_h, float(max_h_in))
max_img_h = max(max_h - cap_reserve, 0.6)
target_w = _USABLE_W
target_h = target_w * aspect
if target_h > max_h:
target_h = max_h
if target_h > max_img_h:
target_h = max_img_h
target_w = target_h / aspect if aspect else _USABLE_W
cap_h = tl.line_height_in(_FS_NOTE) + 0.04 if caption else 0.0
# Move whole image to next page if it does not fit in remaining space.
if _remaining(st) < target_h + cap_h:
if (max_h) >= target_h + cap_h:
_new_page(st)
else:
# Taller than a full page even at min — already clamped to max_h.
_new_page(st)
if _remaining(st) < target_h + cap_reserve:
_new_page(st)
left_frac = _xf(_ML + (_USABLE_W - target_w) / 2.0)
bottom_frac = _yf(st.y + target_h)
ax = st.fig.add_axes([left_frac, bottom_frac, target_w / _W, target_h / _H])
ax.imshow(arr)
ax.axis("off")
st.y += target_h + 0.04
if caption:
_place_text_lines(st, tl.wrap(model._safe_str(caption),
tl.chars_per_line(_USABLE_W, _FS_NOTE)),
_FS_NOTE, _MUTED, style="italic")
if cap_lines:
_place_text_lines(st, cap_lines, _FS_NOTE, _MUTED, style="italic")
st.y += _GAP
def _place_figure(st: _PdfState, block) -> None:
fig, owned = _resolve_figure(block)
if fig is None:
png, _aspect = _figure_png_cached(block)
if png is None:
_place_text_lines(st, ["(figura no disponible)"], _FS_NOTE, _MUTED,
style="italic")
st.y += _GAP
return
try:
png = _png_from_figure(fig)
finally:
if owned:
try:
plt.close(fig)
except Exception: # noqa: BLE001
pass
arr = mpimg.imread(io.BytesIO(png))
_place_image_array(st, arr, getattr(block, "caption", None))
_place_image_array(st, arr, getattr(block, "caption", None),
max_h_in=getattr(block, "height_in", None))
def _place_image(st: _PdfState, block) -> None:
@@ -443,7 +572,8 @@ def _place_image(st: _PdfState, block) -> None:
st.y += _GAP
return
arr = mpimg.imread(path)
_place_image_array(st, arr, getattr(block, "caption", None))
_place_image_array(st, arr, getattr(block, "caption", None),
max_h_in=getattr(block, "height_in", None))
def _place_caption(st: _PdfState, block) -> None:
@@ -460,6 +590,244 @@ def _place_note(st: _PdfState, block) -> None:
st.y += _GAP
# --------------------------------------------------------------------------- #
# Block measurement (mejora 3 — keep-together). These estimate a block's height
# WITHOUT drawing it, so a Group can decide to move whole to the next page before
# anything is drawn. Over-estimating is safe: it only triggers an earlier page
# break, never a content cut (the placers keep their own no-cut pagination).
# --------------------------------------------------------------------------- #
def _measure_heading_text(text: str, level: int) -> float:
level = max(1, min(3, int(level or 1)))
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
lines = tl.wrap(tl.strip_inline_md(text), tl.chars_per_line(_USABLE_W, fs))
h = tl.line_height_in(fs, leading=1.2) * len(lines) + 0.06
if level == 1:
h += 0.10
return h + _GAP
def _measure_markdown(block) -> float:
raw = str(getattr(block, "text", "") or "")
md_lines = raw.split("\n")
h = 0.0
i, n = 0, len(md_lines)
while i < n:
stripped = md_lines[i].strip()
if stripped.startswith("|") and stripped.endswith("|"):
j = i
while j < n and md_lines[j].strip().startswith("|") \
and md_lines[j].strip().endswith("|"):
j += 1
h += (tl.line_height_in(_FS_CELL) + _ROW_VPAD * 2) * (j - i) + _GAP
i = j
continue
if stripped == "":
h += tl.line_height_in(_FS_BODY) * 0.5
i += 1
continue
if stripped.startswith("### "):
h += _measure_heading_text(stripped[4:], 3)
i += 1
continue
if stripped.startswith("## "):
h += _measure_heading_text(stripped[3:], 2)
i += 1
continue
if stripped.startswith("# "):
h += _measure_heading_text(stripped[2:], 1)
i += 1
continue
if stripped.startswith("- ") or stripped.startswith("* "):
lines = tl.wrap_rich_terms(
stripped[2:], tl.chars_per_line(_USABLE_W - 0.22, _FS_BODY))
h += tl.line_height_in(_FS_BODY) * len(lines)
i += 1
continue
para = [stripped]
j = i + 1
while j < n:
nxt = md_lines[j].strip()
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
break
para.append(nxt)
j += 1
lines = tl.wrap_rich_terms(" ".join(para),
tl.chars_per_line(_USABLE_W, _FS_BODY))
h += tl.line_height_in(_FS_BODY) * len(lines)
i = j
return h + _GAP
def _measure_figure_like(block) -> float:
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
hint = getattr(block, "height_in", None)
if isinstance(hint, (int, float)) and hint > 0:
target_h = min(float(hint), max_h)
else:
# Real rasterized aspect (cached) so measuring matches drawing.
if getattr(block, "kind", "") == "image":
aspect = _image_aspect(block)
else:
_data, aspect = _figure_png_cached(block)
target_h = min(_USABLE_W * aspect, max_h)
cap = getattr(block, "caption", None)
cap_h = tl.line_height_in(_FS_NOTE) + 0.04 if cap else 0.0
return target_h + 0.04 + cap_h + _GAP
def _measure_kv_table(block) -> float:
"""Faithful height of a KVTable — matches ``_place_kv_table``.
Counts the optional title heading and, per row, the wrapped VALUE column
(the label column never wraps in the placer). The previous estimate assumed
one line per row and ignored the title, so a column's keep-together Group
under-budgeted the figure and the chart spilled to the next page. Keep this in
sync with ``_place_kv_table``."""
h = 0.0
title = getattr(block, "title", None)
if title:
h += _measure_heading_text(title, 2)
rows = getattr(block, "rows", []) or []
key_w = 1.9
val_chars = tl.chars_per_line(_USABLE_W - key_w - 0.1, _FS_BODY)
lh = tl.line_height_in(_FS_BODY)
for row in rows:
try:
value = row[1]
except Exception: # noqa: BLE001
value = ""
v_lines = tl.wrap(model._safe_str(value), val_chars)
h += lh * len(v_lines) + _ROW_VPAD
return h + _GAP
def _measure_data_table(block) -> float:
"""Faithful height of a DataTable — matches ``_place_data_table``.
Counts the optional title heading, the wrapped header row, every wrapped data
row (per-column wrap via the same ``_col_widths``/``_wrap_row`` the placer
uses) and the optional note. Keep this in sync with ``_place_data_table``."""
h = 0.0
title = getattr(block, "title", None)
if title:
h += _measure_heading_text(title, 2)
header = list(getattr(block, "header", []) or [])
rows = list(getattr(block, "rows", []) or [])
fs = _FS_CELL
widths = _col_widths(header, rows, fs)
lh = tl.line_height_in(fs)
if header:
header_lines = _wrap_row(header, widths, fs)
h += lh * max((len(c) for c in header_lines), default=1) + _ROW_VPAD * 2
for r in rows:
cells_lines = _wrap_row(r, widths, fs)
h += lh * max((len(c) for c in cells_lines), default=1) + _ROW_VPAD * 2
note = getattr(block, "note", None)
if note:
nlines = tl.wrap(model._safe_str(note),
tl.chars_per_line(_USABLE_W, _FS_NOTE))
h += tl.line_height_in(_FS_NOTE) * len(nlines)
return h + _GAP
def _measure_block(st: _PdfState, block) -> float:
kind = getattr(block, "kind", "")
try:
if kind == "heading":
return _measure_heading_text(getattr(block, "text", ""),
getattr(block, "level", 1))
if kind == "markdown":
return _measure_markdown(block)
if kind in ("figure", "image"):
return _measure_figure_like(block)
if kind in ("caption", "note"):
lines = tl.wrap(getattr(block, "text", ""),
tl.chars_per_line(_USABLE_W, _FS_NOTE))
return tl.line_height_in(_FS_NOTE) * len(lines) + _GAP
if kind == "kv_table":
return _measure_kv_table(block)
if kind == "data_table":
return _measure_data_table(block)
if kind == "group":
return sum(_measure_block(st, b)
for b in (getattr(block, "blocks", []) or []))
except Exception: # noqa: BLE001 — a measurement never aborts rendering.
pass
return tl.line_height_in(_FS_BODY)
def _shrink_group_figures(st: _PdfState, blocks: list, avail_full: float) -> None:
"""Cap each figure's height (via height_in) so the whole group fits a page.
The figure shrinks just enough to leave room for its heading, text and
caption keep-together puts the chart on the SAME page as its title and
description instead of pushing it to the next page."""
fig_blocks = [b for b in blocks
if getattr(b, "kind", "") in ("figure", "image")]
if not fig_blocks:
return
nonfig_h = sum(_measure_block(st, b) for b in blocks
if getattr(b, "kind", "") not in ("figure", "image"))
fig_overhead = tl.line_height_in(_FS_NOTE) + 0.04 + 0.04 + _GAP
budget = avail_full - nonfig_h - 0.08 * len(fig_blocks)
if budget <= 0.8:
return
per = budget / len(fig_blocks) - fig_overhead
if per <= 0.6:
return
for fb in fig_blocks:
cur = getattr(fb, "height_in", None)
fb.height_in = (min(float(cur), per)
if isinstance(cur, (int, float)) and cur > 0 else per)
def _place_group(st: _PdfState, block) -> None:
"""Render a keep-together Group: move it whole to the next page if needed."""
blocks = getattr(block, "blocks", []) or []
if not blocks:
return
# Opt-in page break: start this group on a fresh page unless the current one
# is still empty (so a chapter can give each unit its own page).
if getattr(block, "page_break_before", False) and st.y > _CONTENT_TOP + 1e-6:
_new_page(st)
avail_full = _CONTENT_BOTTOM - _CONTENT_TOP
_shrink_group_figures(st, blocks, avail_full)
total = sum(_measure_block(st, b) for b in blocks)
if total <= avail_full:
# Fits on one page: keep it together by moving whole when it won't fit.
if total > _remaining(st):
_new_page(st)
elif st.y > _CONTENT_TOP + 1e-6:
# Taller than a full page: at least start it on a fresh page, then flow.
_new_page(st)
for b in blocks:
placer = _PLACERS.get(getattr(b, "kind", ""), _place_note)
try:
placer(st, b)
except Exception: # noqa: BLE001 — a bad block never aborts the group.
pass
def _place_glossary_entry(st: _PdfState, block) -> None:
"""Render one glossary term and register it as a clickable link target."""
key = getattr(block, "key", "")
label = getattr(block, "label", "") or key
definition = getattr(block, "definition", "")
# Reserve the term + its first definition line together, then anchor the
# destination at the resolved page/position before drawing.
_ensure_space(st, tl.line_height_in(_FS_H3, leading=1.2)
+ tl.line_height_in(_FS_BODY) * 2)
if key:
st.term_dests[key] = {"page": st.page - 1,
"point": [_ML * 72.0, st.y * 72.0]}
_place_heading(st, model.Heading(text=str(label), level=3))
if definition:
_place_text_lines(st, tl.wrap(model._safe_str(definition),
tl.chars_per_line(_USABLE_W, _FS_BODY)),
_FS_BODY, _INK)
st.y += _GAP * 0.5
_PLACERS = {
"heading": _place_heading,
"markdown": _place_markdown,
@@ -469,6 +837,8 @@ _PLACERS = {
"image": _place_image,
"caption": _place_caption,
"note": _place_note,
"group": _place_group,
"glossary_entry": _place_glossary_entry,
}
@@ -525,8 +895,42 @@ def render_pdf(chapters: list, out_path: str, meta: dict = None) -> dict:
return {"path": None, "n_pages": 0, "chapters": [],
"note": f"fallo al escribir el PDF: {e}"}
# Mejora 6 — wire clickable glossary links now the PDF is closed on disk.
# PdfPages cannot emit internal hyperlinks, so we post-process with PyMuPDF
# (delegated registry function). Degrades silently if it is unavailable.
n_links = _wire_glossary_links(st, out_path, notes)
note = f"{n_pages} páginas"
if n_links:
note += f" · {n_links} enlaces de glosario"
if notes:
note += " · " + "; ".join(notes)
return {"path": out_path, "n_pages": n_pages, "chapters": chapters_meta,
"note": note}
def _wire_glossary_links(st: _PdfState, out_path: str, notes: list) -> int:
"""Build {source rect → glossary dest} links and apply them via PyMuPDF.
Returns the number of links applied (0 if there is nothing to wire or the
post-processor is unavailable). Never raises."""
try:
links = []
for src in st.term_sources:
dest = st.term_dests.get(src.get("key"))
if not dest:
continue
links.append({
"src_page": src["page"], "src_rect": src["rect"],
"dst_page": dest["page"], "dst_point": dest["point"]})
if not links:
return 0
from datascience.add_pdf_internal_links import add_pdf_internal_links
res = add_pdf_internal_links(out_path, links)
if isinstance(res, dict) and res.get("status") == "ok":
return int(res.get("n_links") or 0)
if isinstance(res, dict) and res.get("error"):
notes.append(f"glosario sin enlaces: {res.get('error')}")
except Exception as e: # noqa: BLE001 — links are best-effort.
notes.append(f"glosario sin enlaces: {e}")
return 0
@@ -43,6 +43,8 @@ _ACCENT = (0x2A, 0x6F, 0x97)
_MUTED = (0x8A, 0x8A, 0x8A)
_HEAD_BG = (0xEE, 0xF3, 0xF6)
_WHITE = (0xFF, 0xFF, 0xFF)
_ZEBRA = (0xF6, 0xF8, 0xFA) # faint grey for even (zebra) data rows.
_LINK = (0x2A, 0x6F, 0x97) # accent colour for clickable glossary terms.
_FS_TITLE = 26
_FS_H1, _FS_H2, _FS_H3 = 20, 16, 13
@@ -59,6 +61,10 @@ class _PptxState:
self.chapter = None
self.slide_no = 0
self.chapter_slides = 0
self.last_heading = "" # text of the most recent heading.
# Glossary wiring (mejora 6): runs to link and per-term target slide.
self.term_runs = [] # [(key, run)]
self.term_anchor_slide = {} # key -> Slide (glossary entry)
def _rgb(c):
@@ -151,10 +157,57 @@ def _add_text(st: _PptxState, lines: list, fs: float, color, bold=False,
st.y += height
def _add_rich_text(st: _PptxState, rich_lines: list, fs: float, color,
indent=0.0, bullet=False) -> None:
"""Add pre-wrapped lines of styled segments as one paragraph per line.
Each line is a list of ``(text, is_bold)`` or ``(text, is_bold, term_key)``
segments; every segment becomes its own run so ``**bold**`` spans render with
native PowerPoint bold (``run.font.bold``) without affecting the measured
height (one paragraph per pre-wrapped line). A segment carrying a
``term_key`` is drawn in the accent colour and its run is recorded in
``st.term_runs`` so it later becomes a native hyperlink jumping to the
glossary slide of that term.
"""
lh = tl.line_height_in(fs)
height = lh * len(rich_lines) + 0.05
_ensure(st, height)
box = st.slide.shapes.add_textbox(
Inches(_ML + indent), Inches(st.y), Inches(_USABLE_W - indent),
Inches(height))
tf = box.text_frame
tf.word_wrap = True
first = True
for segs in rich_lines:
p = tf.paragraphs[0] if first else tf.add_paragraph()
first = False
if bullet:
r0 = p.add_run()
r0.text = ""
r0.font.size = Pt(fs)
r0.font.color.rgb = _rgb(color)
for seg in segs:
if len(seg) == 3:
seg_text, is_bold, term = seg
else:
seg_text, is_bold, term = seg[0], seg[1], None
if seg_text == "":
continue
run = p.add_run()
run.text = seg_text
run.font.size = Pt(fs)
run.font.bold = bool(is_bold)
run.font.color.rgb = _rgb(_LINK if term else color)
if term:
st.term_runs.append((term, run, st.slide))
st.y += height
def _place_heading(st: _PptxState, block) -> None:
level = max(1, min(3, int(getattr(block, "level", 1) or 1)))
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
text = tl.strip_inline_md(getattr(block, "text", ""))
st.last_heading = text or st.last_heading
lines = tl.wrap(text, tl.chars_per_line(_USABLE_W, fs))
_add_text(st, lines, fs, _INK, bold=True)
st.y += 0.04
@@ -196,22 +249,23 @@ def _place_markdown(st: _PptxState, block) -> None:
i += 1
continue
if stripped.startswith("- ") or stripped.startswith("* "):
content = tl.strip_inline_md(stripped[2:])
lines = tl.wrap(content, tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
_add_text(st, lines, _FS_BODY, _INK, bullet=True)
content = stripped[2:] # keep inline markers for bold rendering.
rich = tl.wrap_rich_terms(content,
tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
_add_rich_text(st, rich, _FS_BODY, _INK, bullet=True)
i += 1
continue
para = [tl.strip_inline_md(stripped)]
para = [stripped] # keep inline markers; wrap_rich_terms renders **bold**.
j = i + 1
while j < n:
nxt = md_lines[j].strip()
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
break
para.append(tl.strip_inline_md(nxt))
para.append(nxt)
j += 1
text = " ".join(para)
_add_text(st, tl.wrap(text, tl.chars_per_line(_USABLE_W, _FS_BODY)),
_FS_BODY, _INK)
_add_rich_text(st, tl.wrap_rich_terms(
text, tl.chars_per_line(_USABLE_W, _FS_BODY)), _FS_BODY, _INK)
i = j
st.y += _GAP
@@ -258,7 +312,8 @@ def _row_height_in(cells, widths, fs) -> float:
return lh * maxlines + 0.10
def _emit_table(st: _PptxState, header, chunk, widths, fs) -> None:
def _emit_table(st: _PptxState, header, chunk, widths, fs,
start_index: int = 0) -> None:
nrows = len(chunk) + (1 if header else 0)
ncol = len(widths)
# Pre-measure total height to size the shape (pptx still auto-grows rows).
@@ -282,11 +337,14 @@ def _emit_table(st: _PptxState, header, chunk, widths, fs) -> None:
cell.text = model._safe_str(header[c]) if c < len(header) else ""
_style_cell(cell, fs, _INK, bold=True, fill=_HEAD_BG)
ridx = 1
for r in chunk:
# Zebra striping: shade even data rows (1-based) using the GLOBAL row index
# (start_index offset) so the pattern stays coherent across split chunks.
for k, r in enumerate(chunk):
fill = _ZEBRA if (start_index + k) % 2 == 1 else _WHITE
for c in range(ncol):
cell = gtable.cell(ridx, c)
cell.text = model._safe_str(r[c]) if c < len(r) else ""
_style_cell(cell, fs, _INK, bold=False, fill=_WHITE)
_style_cell(cell, fs, _INK, bold=False, fill=fill)
ridx += 1
st.y += total_h + _GAP
@@ -330,6 +388,7 @@ def _place_data_table(st: _PptxState, block, shaded_header=True,
avail = _remaining(st) - header_h
chunk = []
used = 0.0
chunk_start = idx # global index of the first row in this chunk (zebra).
while idx < n:
rh = _row_height_in(rows[idx], widths, fs)
if used + rh > avail and chunk:
@@ -337,7 +396,7 @@ def _place_data_table(st: _PptxState, block, shaded_header=True,
chunk.append(rows[idx])
used += rh
idx += 1
_emit_table(st, header, chunk, widths, fs)
_emit_table(st, header, chunk, widths, fs, start_index=chunk_start)
note = getattr(block, "note", None)
if note:
_add_text(st, tl.wrap(model._safe_str(note),
@@ -384,54 +443,97 @@ def _resolve_png(block):
pass
def _place_picture_bytes(st: _PptxState, data: bytes, caption) -> None:
def _figure_bytes_cached(block):
"""Rasterize a figure/image to PNG bytes ONCE and cache (bytes, aspect).
Measuring (keep-together) and drawing must agree on the real aspect ratio
``bbox_inches='tight'`` changes it vs ``figsize``, so we rasterize once and
reuse the bytes for both. Cached on the block; never raises."""
cached = getattr(block, "_aeda_png", None)
if cached is not None:
return cached
kind = getattr(block, "kind", "")
data = None
if kind == "image":
path = getattr(block, "path", "")
if path and os.path.exists(path):
try:
with open(path, "rb") as fh:
data = fh.read()
except Exception: # noqa: BLE001
data = None
else:
data = _resolve_png(block)
aspect = 0.66
if data is not None:
w_px, h_px = _img_size_px(data)
aspect = (h_px / w_px) if w_px else 0.66
try:
block._aeda_png = (data, aspect)
return block._aeda_png
except Exception: # noqa: BLE001 — block may reject attributes; degrade.
return (data, aspect)
def _place_picture_bytes(st: _PptxState, data: bytes, caption,
max_h_in=None) -> None:
# Mejora 4 — every figure on a slide carries a visible caption/title. If the
# block has no caption, fall back to the current section heading, then to a
# generic label, so no image is ever shown untitled.
caption = (model._safe_str(caption).strip()
or model._safe_str(st.last_heading).strip() or "Figura")
w_px, h_px = _img_size_px(data)
aspect = (h_px / w_px) if w_px else 0.66
# Reserve the caption's REAL (possibly multi-line) height FIRST, then scale
# the image to (max_h - cap_reserve): a figure never fills the whole slide,
# so its caption always fits on the SAME slide and no image is untitled.
# cap_real = what _add_text consumes; cap_reserve adds the post-image gap and
# a small cushion so the caption never spills to the next slide.
cap_lines = tl.wrap(caption, tl.chars_per_line(_USABLE_W, _FS_NOTE))
cap_real = tl.line_height_in(_FS_NOTE) * len(cap_lines) + 0.05
cap_reserve = cap_real + 0.05 + 0.10
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
# height_in hint (model.Figure/Image): cap the target height so a figure in a
# keep-together Group shrinks to leave room for its heading and text.
if isinstance(max_h_in, (int, float)) and max_h_in > 0:
max_h = min(max_h, float(max_h_in))
max_img_h = max(max_h - cap_reserve, 0.6)
target_w = _USABLE_W
target_h = target_w * aspect
if target_h > max_h:
target_h = max_h
if target_h > max_img_h:
target_h = max_img_h
target_w = target_h / aspect if aspect else _USABLE_W
cap_h = tl.line_height_in(_FS_NOTE) + 0.05 if caption else 0.0
if _remaining(st) < target_h + cap_h:
# Keep the image and its caption together on the same slide.
if _remaining(st) < target_h + cap_reserve:
_new_slide(st, cont=True)
left = _ML + (_USABLE_W - target_w) / 2.0
st.slide.shapes.add_picture(io.BytesIO(data), Inches(left), Inches(st.y),
width=Inches(target_w), height=Inches(target_h))
st.y += target_h + 0.05
if caption:
_add_text(st, tl.wrap(model._safe_str(caption),
tl.chars_per_line(_USABLE_W, _FS_NOTE)), _FS_NOTE, _MUTED,
italic=True)
_add_text(st, cap_lines, _FS_NOTE, _MUTED, italic=True)
st.y += _GAP
def _place_figure(st: _PptxState, block) -> None:
png = _resolve_png(block)
png, _aspect = _figure_bytes_cached(block)
if png is None:
_add_text(st, ["(figura no disponible)"], _FS_NOTE, _MUTED, italic=True)
st.y += _GAP
return
_place_picture_bytes(st, png, getattr(block, "caption", None))
_place_picture_bytes(st, png, getattr(block, "caption", None),
max_h_in=getattr(block, "height_in", None))
def _place_image(st: _PptxState, block) -> None:
path = getattr(block, "path", "")
if not path or not os.path.exists(path):
data, _aspect = _figure_bytes_cached(block)
if data is None:
path = getattr(block, "path", "")
_add_text(st, [f"(imagen no encontrada: {path})"], _FS_NOTE, _MUTED,
italic=True)
st.y += _GAP
return
try:
with open(path, "rb") as fh:
data = fh.read()
except Exception as e: # noqa: BLE001
_add_text(st, [f"(no se pudo leer la imagen: {e})"], _FS_NOTE, _MUTED,
italic=True)
st.y += _GAP
return
_place_picture_bytes(st, data, getattr(block, "caption", None))
_place_picture_bytes(st, data, getattr(block, "caption", None),
max_h_in=getattr(block, "height_in", None))
def _place_caption(st: _PptxState, block) -> None:
@@ -445,6 +547,302 @@ def _place_note(st: _PptxState, block) -> None:
_place_caption(st, block)
# --------------------------------------------------------------------------- #
# Block measurement (mejora 3 — keep-together). Estimate a block's slide height
# WITHOUT drawing it so a Group can move whole to the next slide before drawing.
# Over-estimating only triggers an earlier slide break, never a content cut.
# --------------------------------------------------------------------------- #
def _measure_heading_text(text: str, level: int) -> float:
level = max(1, min(3, int(level or 1)))
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
lines = tl.wrap(tl.strip_inline_md(text), tl.chars_per_line(_USABLE_W, fs))
return tl.line_height_in(fs) * len(lines) + 0.05 + 0.04
def _measure_markdown(block) -> float:
raw = str(getattr(block, "text", "") or "")
md_lines = raw.split("\n")
h = 0.0
i, n = 0, len(md_lines)
while i < n:
stripped = md_lines[i].strip()
if stripped.startswith("|") and stripped.endswith("|"):
j = i
while j < n and md_lines[j].strip().startswith("|") \
and md_lines[j].strip().endswith("|"):
j += 1
h += (tl.line_height_in(_FS_CELL) + 0.10) * (j - i) + _GAP
i = j
continue
if stripped == "":
h += tl.line_height_in(_FS_BODY) * 0.4
i += 1
continue
if stripped.startswith("### "):
h += _measure_heading_text(stripped[4:], 3)
i += 1
continue
if stripped.startswith("## "):
h += _measure_heading_text(stripped[3:], 2)
i += 1
continue
if stripped.startswith("# "):
h += _measure_heading_text(stripped[2:], 1)
i += 1
continue
if stripped.startswith("- ") or stripped.startswith("* "):
lines = tl.wrap_rich_terms(
stripped[2:], tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
h += tl.line_height_in(_FS_BODY) * len(lines) + 0.05
i += 1
continue
para = [stripped]
j = i + 1
while j < n:
nxt = md_lines[j].strip()
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
break
para.append(nxt)
j += 1
lines = tl.wrap_rich_terms(" ".join(para),
tl.chars_per_line(_USABLE_W, _FS_BODY))
h += tl.line_height_in(_FS_BODY) * len(lines) + 0.05
i = j
return h + _GAP
def _measure_figure_like(block) -> float:
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
hint = getattr(block, "height_in", None)
if isinstance(hint, (int, float)) and hint > 0:
max_h = min(max_h, float(hint))
# Use the REAL rasterized aspect (cached) so measuring matches drawing — this
# is what keeps a figure together with its heading instead of splitting.
_data, aspect = _figure_bytes_cached(block)
target_h = min(_USABLE_W * aspect, max_h)
# Caption is always emitted now (mejora 4), so always reserve its line.
cap_h = tl.line_height_in(_FS_NOTE) + 0.05
return target_h + 0.05 + cap_h + _GAP
def _measure_kv_table(block) -> float:
"""Faithful KVTable height — matches ``_place_kv_table`` (rendered as a
Campo/Valor data table with wrapped cells). The previous estimate assumed one
line per row and ignored the title, so a keep-together Group under-budgeted
the figure and the chart spilled to the next slide. Keep in sync."""
h = 0.0
title = getattr(block, "title", None)
if title:
h += _measure_heading_text(title, 2)
rows = getattr(block, "rows", []) or []
data_rows = []
for row in rows:
try:
label, value = row[0], row[1]
except Exception: # noqa: BLE001
label, value = str(row), ""
data_rows.append([model._safe_str(label), model._safe_str(value)])
header = ["Campo", "Valor"]
widths = _col_widths(header, data_rows)
fs = _FS_CELL
h += _row_height_in(header, widths, fs)
for r in data_rows:
h += _row_height_in(r, widths, fs)
return h + _GAP
def _measure_data_table(block) -> float:
"""Faithful DataTable height — matches ``_place_data_table`` (title heading +
wrapped header + every wrapped row + optional note). Keep in sync."""
h = 0.0
title = getattr(block, "title", None)
if title:
h += _measure_heading_text(title, 2)
header = list(getattr(block, "header", []) or [])
rows = list(getattr(block, "rows", []) or [])
fs = _FS_CELL
widths = _col_widths(header, rows)
if header:
h += _row_height_in(header, widths, fs)
for r in rows:
h += _row_height_in(r, widths, fs)
note = getattr(block, "note", None)
if note:
nlines = tl.wrap(model._safe_str(note),
tl.chars_per_line(_USABLE_W, _FS_NOTE))
h += tl.line_height_in(_FS_NOTE) * len(nlines) + 0.05
return h + _GAP
def _measure_block(st: _PptxState, block) -> float:
kind = getattr(block, "kind", "")
try:
if kind == "heading":
return _measure_heading_text(getattr(block, "text", ""),
getattr(block, "level", 1))
if kind == "markdown":
return _measure_markdown(block)
if kind in ("figure", "image"):
return _measure_figure_like(block)
if kind in ("caption", "note"):
lines = tl.wrap(getattr(block, "text", ""),
tl.chars_per_line(_USABLE_W, _FS_NOTE))
return tl.line_height_in(_FS_NOTE) * len(lines) + 0.05 + _GAP
if kind == "kv_table":
return _measure_kv_table(block)
if kind == "data_table":
return _measure_data_table(block)
if kind == "group":
return sum(_measure_block(st, b)
for b in (getattr(block, "blocks", []) or []))
except Exception: # noqa: BLE001 — a measurement never aborts rendering.
pass
return tl.line_height_in(_FS_BODY)
def _shrink_group_figures(st: _PptxState, blocks: list, avail_full: float) -> None:
"""Cap each figure's height (via height_in) so the whole group fits a slide.
The figure shrinks just enough to leave room for its heading, text and
caption that is how keep-together puts a chart on the SAME slide as its
title and description instead of pushing it to the next slide."""
fig_blocks = [b for b in blocks
if getattr(b, "kind", "") in ("figure", "image")]
if not fig_blocks:
return
nonfig_h = sum(_measure_block(st, b) for b in blocks
if getattr(b, "kind", "") not in ("figure", "image"))
fig_overhead = tl.line_height_in(_FS_NOTE) + 0.05 + 0.05 + _GAP
budget = avail_full - nonfig_h - 0.10 * len(fig_blocks)
# Low thresholds: a 16:9 slide is short, so a content-heavy column (cardinality
# table + top-k + chart) only fits if the chart is allowed to shrink small.
# Prefer a small-but-present chart on the SAME slide over splitting the column
# across slides (matches the PDF renderer's keep-together philosophy).
if budget <= 0.6:
return # not enough room to keep together; let it flow (degrade).
per = budget / len(fig_blocks) - fig_overhead
if per <= 0.35:
return
for fb in fig_blocks:
cur = getattr(fb, "height_in", None)
fb.height_in = (min(float(cur), per)
if isinstance(cur, (int, float)) and cur > 0 else per)
# Minimum height (inches) reserved for a figure inside a keep-together group on
# the short 16:9 slide. When a high-cardinality column's table(s) would otherwise
# leave no room, the data table is trimmed (with an honest note) so the chart
# stays on the SAME slide next to its table instead of spilling to the next one.
_GROUP_MIN_FIG_H = 1.3
def _trim_data_table_to_budget(block, budget: float):
"""Return a copy of a DataTable whose rows fit within ``budget`` inches.
Keeps the title, header, as many leading rows as fit (at least one) and an
honest note reporting how many of the original rows are shown. NEVER mutates
the original block the same Chapter blocks are rendered by the PDF renderer,
which keeps the full table (an A5 page fits it)."""
header = list(getattr(block, "header", []) or [])
rows = list(getattr(block, "rows", []) or [])
title = getattr(block, "title", None)
fs = _FS_CELL
widths = _col_widths(header, rows)
fixed = 0.0
if title:
fixed += _measure_heading_text(title, 2)
if header:
fixed += _row_height_in(header, widths, fs)
note_h = tl.line_height_in(_FS_NOTE) + 0.05
avail_rows = budget - fixed - note_h - _GAP
kept = []
used = 0.0
for r in rows:
rh = _row_height_in(r, widths, fs)
if used + rh > avail_rows and kept:
break
kept.append(r)
used += rh
if len(kept) >= len(rows):
return block # already fits; keep the original (with its own note).
note = (f"top {len(kept)} de {len(rows)} categorías mostradas "
"(recortado para caber en el slide; el PDF muestra más)")
return model.DataTable(header=header, rows=kept, title=title, note=note)
def _fit_group_blocks(st: _PptxState, blocks: list, avail_full: float) -> list:
"""Return a slide-fitting copy of a keep-together group's blocks.
On the short 16:9 slide a high-cardinality column's top-k table plus its
chart can overflow. Reserve ``_GROUP_MIN_FIG_H`` for the (later shrunk) figure
and trim the data table(s) to what is left, so every column keeps its chart
next to its table on ONE slide. No-op when the group has no figure+table pair
(e.g. id-like columns already drop the top-k upstream, or it already fits)."""
has_fig = any(getattr(b, "kind", "") in ("figure", "image") for b in blocks)
tbls = [b for b in blocks if getattr(b, "kind", "") == "data_table"]
if not (has_fig and tbls):
return blocks
fixed_h = sum(_measure_block(st, b) for b in blocks
if getattr(b, "kind", "") not in ("figure", "image",
"data_table"))
tables_h = sum(_measure_block(st, b) for b in tbls)
budget_tables = avail_full - fixed_h - _GROUP_MIN_FIG_H
if tables_h <= budget_tables:
return blocks # already fits next to a min-height figure; leave intact.
out = []
for b in blocks:
if getattr(b, "kind", "") != "data_table":
out.append(b)
continue
trimmed = _trim_data_table_to_budget(b, max(budget_tables, 0.8))
out.append(trimmed)
budget_tables -= _measure_data_table(trimmed)
return out
def _place_group(st: _PptxState, block) -> None:
"""Render a keep-together Group: move it whole to the next slide if needed."""
blocks = getattr(block, "blocks", []) or []
if not blocks:
return
# Opt-in slide break: start this group on a fresh slide unless the current one
# is still empty (so a chapter can give each unit its own slide).
if getattr(block, "page_break_before", False) and st.y > _CONTENT_TOP + 1e-6:
_new_slide(st, cont=True)
avail_full = _CONTENT_BOTTOM - _CONTENT_TOP
# Trim oversized tables first (keeps the chart on the same slide), then shrink
# the figure to share the remaining room.
blocks = _fit_group_blocks(st, blocks, avail_full)
_shrink_group_figures(st, blocks, avail_full)
total = sum(_measure_block(st, b) for b in blocks)
if total <= avail_full:
if total > _remaining(st):
_new_slide(st, cont=True)
elif st.y > _CONTENT_TOP + 1e-6:
_new_slide(st, cont=True)
for b in blocks:
placer = _PLACERS.get(getattr(b, "kind", ""), _place_note)
try:
placer(st, b)
except Exception: # noqa: BLE001 — a bad block never aborts the group.
pass
def _place_glossary_entry(st: _PptxState, block) -> None:
"""Render one glossary term and register its slide as the link target."""
key = getattr(block, "key", "")
label = getattr(block, "label", "") or key
definition = getattr(block, "definition", "")
_ensure(st, tl.line_height_in(_FS_H3) + tl.line_height_in(_FS_BODY) * 2)
if key:
st.term_anchor_slide[key] = st.slide
_place_heading(st, model.Heading(text=str(label), level=3))
if definition:
_add_text(st, tl.wrap(model._safe_str(definition),
tl.chars_per_line(_USABLE_W, _FS_BODY)), _FS_BODY, _INK)
st.y += _GAP
_PLACERS = {
"heading": _place_heading,
"markdown": _place_markdown,
@@ -454,6 +852,8 @@ _PLACERS = {
"image": _place_image,
"caption": _place_caption,
"note": _place_note,
"group": _place_group,
"glossary_entry": _place_glossary_entry,
}
@@ -505,6 +905,9 @@ def render_pptx(chapters: list, out_path: str, meta: dict = None) -> dict:
_new_slide(st, cont=False)
_place_note(st, model.Note(
"(documento vacío — sin capítulos aplicables)"))
# Mejora 6 — wire clickable glossary terms to their entry slide (native
# PowerPoint slide-jump). Delegated registry function; degrades silently.
n_links = _wire_glossary_links(st, notes)
prs.save(out_path)
n_slides = st.slide_no
except Exception as e: # noqa: BLE001
@@ -512,7 +915,35 @@ def render_pptx(chapters: list, out_path: str, meta: dict = None) -> dict:
"note": f"fallo al escribir el PPTX: {e}"}
note = f"{n_slides} slides"
if n_links:
note += f" · {n_links} enlaces de glosario"
if notes:
note += " · " + "; ".join(notes)
return {"path": out_path, "n_slides": n_slides, "chapters": chapters_meta,
"note": note}
def _wire_glossary_links(st: _PptxState, notes: list) -> int:
"""Turn each recorded term run into a native jump to its glossary slide.
Returns the number of links applied. A term whose only appearance is inside
its own glossary entry (source slide == target slide) is skipped. Never
raises."""
if not st.term_runs or not st.term_anchor_slide:
return 0
linked = 0
try:
from datascience.pptx_link_run_to_slide import pptx_link_run_to_slide
except Exception as e: # noqa: BLE001
notes.append(f"glosario sin enlaces: {e}")
return 0
for key, run, src_slide in st.term_runs:
tgt = st.term_anchor_slide.get(key)
if tgt is None or tgt is src_slide:
continue
try:
if pptx_link_run_to_slide(run, src_slide, tgt):
linked += 1
except Exception: # noqa: BLE001 — links are best-effort.
pass
return linked
@@ -15,8 +15,22 @@ overflowing — that is wrapping, not loss: every character is still rendered.
from __future__ import annotations
import re
import textwrap
# Inline span markers: ``**bold**`` / ``__bold__`` (rendered bold) and
# `` `code` `` (markers removed, not styled). Matched non-greedily so the
# shortest balanced pair wins. Unbalanced leftovers are stripped afterwards so
# the visible text matches ``strip_inline_md`` exactly.
_INLINE_SPAN_RE = re.compile(r"(\*\*.+?\*\*|__.+?__|`.+?`)")
# Glossary term span: ``[[term:key]]texto visible[[/term]]``. The visible text
# (which may itself contain ``**bold**``) is kept and tagged with ``key`` so the
# renderers can turn each appearance into a clickable jump to the glossary entry.
_TERM_SPAN_RE = re.compile(r"\[\[term:([A-Za-z0-9_]+)\]\](.*?)\[\[/term\]\]",
re.S)
_TERM_OPEN_RE = re.compile(r"\[\[term:[A-Za-z0-9_]+\]\]")
def avg_char_width_in(fontsize_pt: float) -> float:
"""Approximate average glyph width in inches for a sans-serif font.
@@ -79,11 +93,264 @@ def strip_inline_md(text: str) -> str:
if not text:
return ""
s = str(text)
# Drop glossary term markers, keeping the visible inner text.
s = _TERM_SPAN_RE.sub(lambda m: m.group(2), s)
s = _TERM_OPEN_RE.sub("", s) # leftover unbalanced open marker.
s = s.replace("[[/term]]", "") # leftover unbalanced close marker.
for marker in ("**", "__", "`"):
s = s.replace(marker, "")
return s
def _strip_term_markers(s: str) -> str:
"""Remove any (balanced or leftover) glossary term markers, keeping text."""
s = _TERM_OPEN_RE.sub("", s)
return s.replace("[[/term]]", "")
def _strip_leftover_markers(s: str) -> str:
"""Drop any unbalanced inline markers from a plain (non-span) fragment.
Keeps the visible text identical to :func:`strip_inline_md` even when a
``**`` / ``__`` / `` ` `` has no matching closing marker.
"""
for marker in ("**", "__", "`"):
s = s.replace(marker, "")
return s
def parse_inline_bold(text: str):
"""Split ``text`` into ``[(fragment, is_bold), ...]`` preserving order.
``**...**`` and ``__...__`` spans become bold fragments (markers removed);
`` `code` `` keeps its text without the backticks and is not bold; any other
text is emitted verbatim with unbalanced markers stripped. The concatenation
of all fragment texts equals :func:`strip_inline_md` of the input so the
*visible* characters (and therefore line wrapping) are unchanged; only the
bold flag is added. Adjacent fragments of the same weight are merged.
"""
s = "" if text is None else str(text)
if not s:
return []
out = []
def _emit(fragment: str, bold: bool) -> None:
if fragment == "":
return
if out and out[-1][1] == bold:
out[-1] = (out[-1][0] + fragment, bold)
else:
out.append((fragment, bold))
pos = 0
for m in _INLINE_SPAN_RE.finditer(s):
if m.start() > pos:
_emit(_strip_leftover_markers(s[pos:m.start()]), False)
tok = m.group(0)
if tok.startswith("**") and tok.endswith("**"):
_emit(tok[2:-2], True)
elif tok.startswith("__") and tok.endswith("__"):
_emit(tok[2:-2], True)
else: # `code`
_emit(tok[1:-1], False)
pos = m.end()
if pos < len(s):
_emit(_strip_leftover_markers(s[pos:]), False)
return out
def _hard_split(word: str, max_chars: int):
"""Split a single long token into <= max_chars chunks (never loses chars)."""
return [word[i:i + max_chars] for i in range(0, len(word), max_chars)] or [""]
def wrap_rich(text: str, max_chars: int):
"""Word-wrap ``text`` to ``max_chars`` while preserving inline bold spans.
Returns ``list[list[(fragment, is_bold)]]`` one inner list of styled
fragments per output line; concatenating an inner list's fragment texts is
the visible line. Wrapping is word-aware and hard-splits over-long tokens, so
no line exceeds ``max_chars`` (the renderers measure these very lines, so the
no-cut guarantee holds). Bold spans never widen a line: only the bold flag is
carried, the visible width is identical to :func:`wrap`.
"""
if max_chars < 1:
max_chars = 1
spans = parse_inline_bold(text)
if not spans:
return [[("", False)]]
# Flatten to (word, is_bold) tokens, honoring hard newlines as line breaks.
# A token list of None marks a forced line break.
tokens = [] # each: (word, bold) or ("\n", None)
for frag, bold in spans:
parts = frag.split("\n")
for pi, part in enumerate(parts):
if pi > 0:
tokens.append(("\n", None))
for word in part.split(" "):
if word == "":
continue
tokens.append((word, bold))
lines = [] # list[list[(seg, bold)]]
cur = [] # list[(word, bold)]
cur_len = 0
def _flush():
nonlocal cur, cur_len
# Merge adjacent same-weight words (with separating spaces) into segments.
merged = []
for k, (word, bold) in enumerate(cur):
piece = word if k == 0 else " " + word
if merged and merged[-1][1] == bold:
merged[-1] = (merged[-1][0] + piece, bold)
else:
merged.append((piece, bold))
lines.append(merged or [("", False)])
cur = []
cur_len = 0
for word, bold in tokens:
if bold is None: # forced newline
_flush()
continue
if len(word) > max_chars:
if cur:
_flush()
chunks = _hard_split(word, max_chars)
for ci, chunk in enumerate(chunks):
if ci < len(chunks) - 1:
lines.append([(chunk, bold)])
else:
cur = [(chunk, bold)]
cur_len = len(chunk)
continue
add = len(word) if cur_len == 0 else cur_len + 1 + len(word)
if cur_len != 0 and add > max_chars:
_flush()
cur = [(word, bold)]
cur_len = len(word)
else:
cur.append((word, bold))
cur_len = add
if cur:
_flush()
return lines or [[("", False)]]
def parse_inline_rich(text: str):
"""Split ``text`` into ``[(fragment, is_bold, term_key), ...]``.
Extends :func:`parse_inline_bold` with glossary term spans
``[[term:key]]visible[[/term]]``: the inner ``visible`` text is parsed for
``**bold**`` as usual and every resulting fragment carries ``term_key`` so the
renderers can make it clickable. Text outside a term span gets ``term_key =
None``. Unbalanced term markers are stripped (kept identical to
:func:`strip_inline_md`). The concatenation of all fragment texts equals
``strip_inline_md(text)`` visible characters and wrapping are unchanged; only
the bold flag and the term key are added. Adjacent fragments with the same
(bold, term) are merged.
"""
s = "" if text is None else str(text)
if not s:
return []
out = []
def _emit(fragment: str, bold: bool, term) -> None:
if fragment == "":
return
if out and out[-1][1] == bold and out[-1][2] == term:
out[-1] = (out[-1][0] + fragment, bold, term)
else:
out.append((fragment, bold, term))
def _emit_bolded(segment: str, term) -> None:
# Reuse the bold parser on a term-marker-free segment.
for frag, bold in parse_inline_bold(_strip_term_markers(segment)):
_emit(frag, bold, term)
pos = 0
for m in _TERM_SPAN_RE.finditer(s):
if m.start() > pos:
_emit_bolded(s[pos:m.start()], None)
_emit_bolded(m.group(2), m.group(1))
pos = m.end()
if pos < len(s):
_emit_bolded(s[pos:], None)
return out
def wrap_rich_terms(text: str, max_chars: int):
"""Like :func:`wrap_rich` but preserving glossary term keys per fragment.
Returns ``list[list[(fragment, is_bold, term_key)]]`` one inner list per
output line. Wrapping is word-aware and hard-splits over-long tokens so no
line exceeds ``max_chars`` (the renderers measure these very lines). Term and
bold flags never widen a line: the visible width matches :func:`wrap`.
"""
if max_chars < 1:
max_chars = 1
spans = parse_inline_rich(text)
if not spans:
return [[("", False, None)]]
tokens = [] # each: (word, bold, term) or ("\n", None, None)
for frag, bold, term in spans:
parts = frag.split("\n")
for pi, part in enumerate(parts):
if pi > 0:
tokens.append(("\n", None, None))
for word in part.split(" "):
if word == "":
continue
tokens.append((word, bold, term))
lines = []
cur = []
cur_len = 0
def _flush():
nonlocal cur, cur_len
merged = []
for k, (word, bold, term) in enumerate(cur):
piece = word if k == 0 else " " + word
if merged and merged[-1][1] == bold and merged[-1][2] == term:
merged[-1] = (merged[-1][0] + piece, bold, term)
else:
merged.append((piece, bold, term))
lines.append(merged or [("", False, None)])
cur = []
cur_len = 0
for word, bold, term in tokens:
if bold is None: # forced newline
_flush()
continue
if len(word) > max_chars:
if cur:
_flush()
chunks = _hard_split(word, max_chars)
for ci, chunk in enumerate(chunks):
if ci < len(chunks) - 1:
lines.append([(chunk, bold, term)])
else:
cur = [(chunk, bold, term)]
cur_len = len(chunk)
continue
add = len(word) if cur_len == 0 else cur_len + 1 + len(word)
if cur_len != 0 and add > max_chars:
_flush()
cur = [(word, bold, term)]
cur_len = len(word)
else:
cur.append((word, bold, term))
cur_len = add
if cur:
_flush()
return lines or [[("", False, None)]]
def parse_md_table(lines: list):
"""Parse consecutive ``| a | b |`` lines into ``(header, rows)`` or None.
@@ -0,0 +1,114 @@
---
name: build_eda_render_ctx
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def build_eda_render_ctx(db_path: str, table: str, profile: dict, backend: str = 'duckdb', sample: int = 5000, base_ctx: dict = None) -> dict"
description: "Constructor del `ctx` de datos crudos del motor AutomaticEDA. Dado un db_path+table (DuckDB o Postgres) y el TableProfile AGREGADO ya calculado por profile_table, produce el dict ctx que los renderers (render_automatic_eda_pdf/_pptx -> build_document(profile, ctx)) pasan a los capitulos que necesitan DATOS CRUDOS no presentes en el perfil agregado: modelos (project_clusters_2d en vivo), timeseries, geospatial y agregacion (groupby/pivot push-down). NO trae tablas enteras a RAM: muestrea con LIMIT sample y delega el push-down de la serie en extract_timeseries_raw. Construye el lector read-only query_fn(sql)->dict igual que profile_table (closure sobre duckdb_query_readonly / pg_query). Estilo dict-no-throw del grupo eda: NUNCA lanza; si una pieza falla, degrada esa clave a ausente/[] y sigue. Devuelve el ctx dict directamente (NO un wrapper {status,...}); se pasa tal cual como meta={'ctx': <ese dict>}. Claves de datos que produce: raw_numeric (muestra cruda alineada por fila), timeseries_raw (fechas+series), geo_points (lats/lons) y db_path+table para el push-down de agregacion. Respeta base_ctx: parte de una copia y solo AÑADE las claves de datos; las de presentacion (dataset_name, source_origin, ...) no se pisan."
tags: [eda, datascience, automatic-eda, render, ctx, extraction, read-only, duckdb, postgres, python]
uses_functions: [detect_time_column_py_datascience, extract_timeseries_raw_py_datascience, detect_latlon_columns_py_datascience, duckdb_query_readonly_py_infra, pg_query_py_infra]
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
params:
- name: db_path
desc: "ruta al archivo DuckDB, o DSN PostgreSQL si backend='postgres'. Se guarda tal cual en ctx['db_path'] (el capitulo agregacion lo usa para el groupby/pivot push-down via DuckDB) y se inyecta en el closure query_fn. No se valida aqui: si la base no existe, las queries devuelven status error y las claves de datos se omiten."
- name: table
desc: "nombre de la tabla. Se escapa con comillas dobles en las queries (raw_numeric y timeseries) y se guarda en ctx['table']."
- name: profile
desc: "TableProfile AGREGADO producido por profile_table. Solo se lee su clave `columns` (lista de ColumnProfile dict con name / inferred_type / numeric.{min,max} / semantic_type). Lectura defensiva: si no es dict o no tiene columns, se trata como []. NO se traen las filas crudas de aqui — se muestrean de la base."
- name: backend
desc: "'duckdb' (default) o 'postgres'. Selecciona el lector read-only del registry (duckdb_query_readonly / pg_query). Cualquier otro valor devuelve el base_ctx tal cual, SIN añadir claves de datos (ni siquiera db_path/table)."
- name: sample
desc: "maximo de filas a muestrear (clausula LIMIT) tanto para raw_numeric (una sola query SELECT de las numericas) como para timeseries_raw (max_rows de extract_timeseries_raw). Default 5000. Acota memoria y tiempo de render."
- name: base_ctx
desc: "dict opcional con claves de PRESENTACION ya preparadas (dataset_name, source_origin, ...). Se parte de una copia y NO se pisan sus claves; solo se añaden las de datos. Default None -> {}."
output: "El dict `ctx` directamente (NO un wrapper {status,...}); se pasa tal cual como meta={'ctx': <ese dict>} a render_automatic_eda_pdf/pptx. Nunca lanza. Para backends validos contiene SIEMPRE db_path + table, y opcionalmente: raw_numeric {col:[float|None,...]} (muestra cruda alineada por fila; omitida si no hay numericas o falla la query), timeseries_raw {time_col, t:[iso...], series:{col:[float|None,...]}} (solo si hay columna temporal + numericas y trae filas), geo_points {lats:[...], lons:[...]} (solo si se detecta par lat/lon y ambas estan en raw_numeric). Ante fallo global devuelve al menos {**base_ctx, 'db_path': db_path, 'table': table}. Backend desconocido -> base_ctx tal cual sin claves de datos."
tested: true
tests: ["test_db_path_y_table_en_ctx", "test_raw_numeric_con_columnas_numericas", "test_timeseries_raw_con_fecha", "test_geo_points_con_latlon", "test_sin_fecha_no_hay_timeseries", "test_base_ctx_preservado"]
test_file_path: "python/functions/datascience/build_eda_render_ctx_test.py"
file_path: "python/functions/datascience/build_eda_render_ctx.py"
---
## Ejemplo
```python
import sys, os
sys.path.insert(0, os.path.join("python", "functions"))
from datascience import build_eda_render_ctx, render_automatic_eda_pdf
from datascience import profile_table # opcional: para obtener el TableProfile
# 1) Perfil agregado de la tabla (push-down, sin RAM).
prof = profile_table("data/ventas.duckdb", "ventas_geo", write_report=False)["profile"]
# 2) ctx de datos crudos para los capitulos (muestrea con LIMIT, no carga todo).
ctx = build_eda_render_ctx(
"data/ventas.duckdb", "ventas_geo", prof,
backend="duckdb", sample=5000,
base_ctx={"dataset_name": "Ventas con geolocalizacion"},
)
# ctx == {
# "dataset_name": "Ventas con geolocalizacion", # preservado del base_ctx
# "db_path": "data/ventas.duckdb", "table": "ventas_geo",
# "raw_numeric": {"ventas": [1200.5, ...], "lat": [40.41, ...], "lon": [-3.70, ...]},
# "timeseries_raw": {"time_col": "fecha", "t": ["2024-01-01", ...], "series": {...}},
# "geo_points": {"lats": [40.41, ...], "lons": [-3.70, ...]},
# }
# 3) Se entrega tal cual a los renderers via meta={"ctx": ctx}.
render_automatic_eda_pdf(prof, "reports/eda.pdf", meta={"ctx": ctx})
```
## Cuando usarla
Justo antes de renderizar un AutomaticEDA (PDF o PPTX), cuando ya tienes el
TableProfile AGREGADO de `profile_table` pero los capitulos de modelos,
timeseries, geospatial y agregacion necesitan DATOS CRUDOS que el perfil
agregado no lleva (la muestra numerica alineada por fila, la serie cronologica,
el par lat/lon, y el db_path/table para el push-down del groupby/pivot). Es el
puente entre el perfil agregado y `build_document(profile, ctx)`: una sola
llamada produce el `ctx` completo muestreando con `LIMIT` en vez de cargar la
tabla entera en memoria.
## Gotchas
- **Impura**: lee de la base de datos a traves de `query_fn` (closure sobre
`duckdb_query_readonly` / `pg_query`). No abre conexiones fuera de esos
wrappers del registry. Estilo dict-no-throw del grupo `eda`: NUNCA lanza; ante
cualquier fallo (query, deteccion, render de una clave) degrada esa clave a
ausente/`[]` y sigue. Ante un fallo global devuelve al menos
`{**base_ctx, "db_path": db_path, "table": table}`.
- **`error_type` en el frontmatter es `error_go_core` por convencion del
registry** (toda funcion impura debe declararlo y el indexer lo exige), pero el
codigo NO lanza esa excepcion: degrada al ctx parcial. Es metadata, no
comportamiento.
- **Devuelve el ctx dict directamente, NO un wrapper `{status,...}`**: a
diferencia de `extract_timeseries_raw` / `profile_table`, esta funcion es el
ultimo eslabon antes del render y su salida se pasa tal cual como
`meta={"ctx": <ese dict>}`. No envuelvas su retorno.
- **Backend desconocido**: con un `backend` que no sea `duckdb` ni `postgres`
devuelve el `base_ctx` tal cual, SIN claves de datos (ni siquiera
`db_path`/`table`). Comprueba el backend antes si dependes de esas claves.
- **Alineacion por fila de `raw_numeric`**: `raw_numeric[col]` tiene una entrada
por fila muestreada (un valor no convertible a float queda como `None`, no se
descarta la fila) porque `project_clusters_2d` descarta filas listwise: todas
las columnas deben tener la MISMA longitud. `geo_points` se construye desde
`raw_numeric` para heredar esa alineacion.
- **`geo_points` exige lat/lon en `raw_numeric`**: el par lat/lon solo se adjunta
si ambas columnas se detectaron (nombre+rango) Y figuran en `raw_numeric`
(es decir, son numericas en el perfil). Si la tabla guarda lat/lon como texto
no promovido a numeric, no apareceran; el capitulo geospatial sabe degradar.
- **`timeseries_raw` depende del orden del backend**: hereda el `ORDER BY
"time_col"` de `extract_timeseries_raw`. Si la columna temporal esta guardada
como texto no ordenable lexicograficamente (p.ej. `DD/MM/YYYY`), el orden no
sera el cronologico real — normaliza la columna a date/timestamp antes.
- **`LIMIT sample`**: con tablas grandes obtienes el primer tramo (raw_numeric
por orden fisico, timeseries por orden cronologico), no un muestreo uniforme.
Sube `sample` si necesitas mas cobertura.
- **No loguear los datos crudos**: `raw_numeric` / `timeseries_raw` /
`geo_points` pueden contener datos sensibles. En trazas usa solo conteos y
nombres de columna, no el ctx completo.
@@ -0,0 +1,224 @@
"""build_eda_render_ctx — constructor del `ctx` de datos crudos del motor AutomaticEDA.
Funcion impura (lee de la base de datos) del grupo de capacidad `eda`. Dado un
``db_path`` + ``table`` (DuckDB o PostgreSQL) y el ``TableProfile`` AGREGADO ya
calculado por ``profile_table``, produce el dict ``ctx`` que los renderers
(``render_automatic_eda_pdf`` / ``render_automatic_eda_pptx`` ->
``build_document(profile, ctx)``) pasan a los capitulos que necesitan DATOS
CRUDOS no presentes en el perfil agregado: modelos (``project_clusters_2d`` en
vivo), timeseries, geospatial y agregacion (groupby/pivot push-down).
NO trae tablas enteras a RAM: muestrea con ``LIMIT sample`` y, para la serie
temporal, delega el push-down en ``extract_timeseries_raw`` (una sola query
ordenada). El lector read-only ``query_fn(sql) -> dict`` se construye igual que
en ``profile_table`` (un closure sobre ``duckdb_query_readonly`` / ``pg_query``)
y nunca abre conexiones fuera de esos wrappers.
Estilo dict-no-throw del grupo `eda`: la funcion NUNCA lanza. Si una pieza falla
(query, deteccion, render de una clave), esa clave se degrada a ausente / lista
vacia y el resto del ctx se construye igual. Ante un fallo global devuelve al
menos ``{**base_ctx, "db_path": db_path, "table": table}``.
Claves de DATOS que produce (las consumen los capitulos):
- ``head_rows`` : [ {col: valor, ...}, ... ] primeras filas CRUDAS de la
tabla (``SELECT * LIMIT head_n``), una entrada por fila.
La lee el capitulo OVERVIEW para mostrar df.head real en
lugar del placeholder "df.head no disponible".
- ``raw_numeric`` : {col: [float|None, ...]} muestra cruda de las columnas
numericas, ALINEADA POR FILA (una entrada por fila aunque
sea None). La leen modelos (clustering 2D en vivo) y
geospatial (lat/lon salen de aqui).
- ``timeseries_raw`` : {time_col, t: [iso...], series: {col: [float|None, ...]}}.
La lee el capitulo TIMESERIES.
- ``geo_points`` : {lats: [...], lons: [...]} listas alineadas (ya floats).
La lee el capitulo GEOSPATIAL.
- ``db_path``, ``table`` : los usa el capitulo AGREGACION para el groupby/pivot
push-down via DuckDB.
Las claves de PRESENTACION que traiga ``base_ctx`` (dataset_name, source_origin,
...) NO se pisan: esta funcion solo AÑADE las claves de datos sobre una copia.
"""
def _to_float(value):
"""Convierte un valor a float de forma defensiva. None si no es convertible.
Un bool es subclase de int en Python pero nunca es un valor numerico de
serie/coordenada valido, asi que se trata como None (mismo criterio que
extract_timeseries_raw / detect_latlon_columns).
"""
if value is None or isinstance(value, bool):
return None
if isinstance(value, (int, float)):
return float(value)
s = str(value).strip()
if not s:
return None
try:
return float(s)
except (TypeError, ValueError):
return None
def build_eda_render_ctx(db_path, table, profile, backend="duckdb", sample=5000, base_ctx=None, head_n=10):
"""Construye el ctx de datos crudos para los renderers de AutomaticEDA.
Args:
db_path: ruta al archivo DuckDB, o DSN PostgreSQL si backend="postgres".
Se guarda tal cual en ctx["db_path"] (el capitulo agregacion lo usa
para el push-down).
table: nombre de la tabla. Se escapa con comillas dobles en las queries y
se guarda en ctx["table"].
profile: TableProfile agregado producido por profile_table. Solo se lee
su clave ``columns`` (lista de ColumnProfile dict con name /
inferred_type / numeric.{min,max} / semantic_type). Lectura
defensiva: si no es dict o no tiene columns, se trata como [].
backend: "duckdb" (default) o "postgres". Selecciona el lector read-only
(duckdb_query_readonly / pg_query). Cualquier otro valor devuelve el
base_ctx tal cual, sin añadir claves de datos.
sample: maximo de filas a muestrear (clausula LIMIT) tanto para
raw_numeric como para timeseries_raw. Default 5000.
base_ctx: dict opcional con claves de presentacion ya preparadas
(dataset_name, source_origin, ...). Se parte de una copia y NO se
pisan sus claves; solo se añaden las de datos. Default None -> {}.
head_n: numero de filas crudas a muestrear para ``ctx["head_rows"]``
(df.head del capitulo OVERVIEW). Default 10. <=0 omite la clave.
Returns:
El dict ``ctx`` directamente (NO un wrapper {status,...}): se pasa tal
cual como ``meta={"ctx": <ese dict>}`` a render_automatic_eda_pdf/pptx.
Nunca lanza. Claves que puede contener: head_rows, raw_numeric,
timeseries_raw, geo_points (omitidas si no aplican o fallan), y siempre
db_path + table para backends validos.
"""
# Copia de base_ctx: nunca mutamos el dict del caller. Las claves de
# presentacion que ya traiga se conservan; las de datos se añaden encima.
ctx = dict(base_ctx) if isinstance(base_ctx, dict) else {}
try:
# 1) Lector read-only del backend activo, construido EXACTAMENTE como en
# profile_table (closure sobre el wrapper del registry). Imports perezosos
# dentro de la funcion: este modulo vive en el paquete `datascience`, asi
# que importar sus hermanas a nivel de modulo crearia un ciclo al cargar
# el __init__ del paquete. Lazy import rompe el ciclo y respeta el
# contrato (imports explicitos, sin `import *`).
if backend == "duckdb":
from infra import duckdb_query_readonly
def query_fn(sql):
return duckdb_query_readonly(db_path, sql)
elif backend == "postgres":
from infra import pg_query
def query_fn(sql):
return pg_query(db_path, sql)
else:
# Backend desconocido: devolver base_ctx tal cual, sin claves de datos.
return ctx
# 7) db_path + table SIEMPRE (para backends validos): el capitulo
# agregacion los necesita para el groupby/pivot push-down via DuckDB.
ctx["db_path"] = db_path
ctx["table"] = table
# 1.5) head_rows: primeras filas CRUDAS de la tabla (SELECT * LIMIT n)
# para que el capitulo OVERVIEW muestre df.head real en vez del
# placeholder. Una sola query, dict-no-throw: si falla, se omite la
# clave (el capitulo degrada a su nota honesta). No se pisa una clave
# head_rows que ya viniera en base_ctx (presentacion).
if head_n and int(head_n) > 0 and "head_rows" not in ctx:
try:
hq = query_fn(f'SELECT * FROM "{table}" LIMIT {int(head_n)}')
if isinstance(hq, dict) and hq.get("status") == "ok":
hrows = [
dict(r) for r in (hq.get("rows") or [])
if isinstance(r, dict)
]
if hrows:
ctx["head_rows"] = hrows
except Exception: # noqa: BLE001 - dict-no-throw: omitir la clave
pass
# 2) Columnas del perfil agregado (lectura defensiva).
cols = profile.get("columns") if isinstance(profile, dict) else None
cols = cols or []
# 3) Deteccion temporal/numerica con la funcion PURA del registry.
from datascience import detect_time_column
det = detect_time_column(cols)
time_col = det.get("time_col")
numeric_cols = det.get("numeric_cols") or []
# 4) raw_numeric: muestra de las columnas numericas CRUDAS, ALINEADAS POR
# FILA en UNA sola query. Cada columna queda con una entrada por fila
# (None si no parsea) para no desalinear filas: project_clusters_2d
# descarta filas listwise, asi que las listas deben tener igual longitud.
raw_numeric = {}
if numeric_cols:
try:
cols_sql = ", ".join(f'"{c}"' for c in numeric_cols)
sql = f'SELECT {cols_sql} FROM "{table}" LIMIT {int(sample)}'
q = query_fn(sql)
if isinstance(q, dict) and q.get("status") == "ok":
rows = q.get("rows", []) or []
raw_numeric = {c: [] for c in numeric_cols}
for row in rows:
for c in numeric_cols:
raw_numeric[c].append(_to_float(row.get(c)))
except Exception: # noqa: BLE001 - dict-no-throw: degradar la clave
raw_numeric = {}
if raw_numeric:
ctx["raw_numeric"] = raw_numeric
# 5) timeseries_raw: SOLO si hay columna temporal y numericas. Se delega
# el push-down en la funcion impura extract_timeseries_raw (una sola query
# ordenada cronologicamente). Solo se adjunta si trae filas.
if time_col and numeric_cols:
try:
from datascience import extract_timeseries_raw
ts = extract_timeseries_raw(
query_fn, table, time_col, numeric_cols, max_rows=sample
)
if (
isinstance(ts, dict)
and ts.get("status") == "ok"
and (ts.get("n") or 0) > 0
):
ctx["timeseries_raw"] = {
"time_col": ts["time_col"],
"t": ts["t"],
"series": ts["series"],
}
except Exception: # noqa: BLE001 - dict-no-throw: omitir la clave
pass
# 6) geo_points: detecta el par lat/lon con la funcion PURA del registry.
# Solo se adjunta si AMBAS columnas estan en raw_numeric (ya floats,
# alineadas por fila). Si no hay par o no estan, se omite: el capitulo
# geospatial sabe degradar.
try:
from datascience import detect_latlon_columns
geo = detect_latlon_columns(cols)
lat_col = geo.get("lat_col")
lon_col = geo.get("lon_col")
if lat_col and lon_col and lat_col in raw_numeric and lon_col in raw_numeric:
ctx["geo_points"] = {
"lats": raw_numeric[lat_col],
"lons": raw_numeric[lon_col],
}
except Exception: # noqa: BLE001 - dict-no-throw: omitir la clave
pass
return ctx
except Exception: # noqa: BLE001 - dict-no-throw global: nunca reventar.
# Fallback minimo: copia de base_ctx + db_path/table para que el capitulo
# agregacion siga teniendo lo imprescindible.
out = dict(base_ctx) if isinstance(base_ctx, dict) else {}
out["db_path"] = db_path
out["table"] = table
return out
@@ -0,0 +1,153 @@
"""Tests para build_eda_render_ctx.
Self-contained: crea un DuckDB temporal pequeño con una columna fecha, varias
numericas y un par lat/lon, construye un TableProfile minimo a mano (con la forma
de columnas del grupo `eda`: name / inferred_type / numeric.{min,max} /
semantic_type) y verifica que el ctx producido contiene las claves de datos que
consumen los capitulos del AutomaticEDA.
"""
import os
import sys
# El test importa funciones del registry como una app del registry: inserta el
# directorio raiz `python/functions` en sys.path y luego `from datascience import`.
_FUNCTIONS_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
if _FUNCTIONS_ROOT not in sys.path:
sys.path.insert(0, _FUNCTIONS_ROOT)
import duckdb # noqa: E402
from datascience import build_eda_render_ctx # noqa: E402
_TABLE = "ventas_geo"
# Filas: fecha creciente, 2 columnas numericas (ventas, unidades) y un par lat/lon
# (Madrid -> lat ~40, lon ~-3, dentro de [-90,90] y [-180,180]).
_ROWS = [
("2024-01-01", 1200.5, 12, 40.41, -3.70),
("2024-01-02", 980.0, 9, 41.38, 2.17),
("2024-01-03", 1500.25, 15, 37.39, -5.99),
("2024-01-04", 1100.0, 11, 39.47, -0.38),
("2024-01-05", 1750.75, 18, 43.26, -2.93),
]
def _make_db(tmp_path):
"""Crea un DuckDB temporal con la tabla de prueba y devuelve su ruta."""
db_path = os.path.join(str(tmp_path), "eda_ctx.duckdb")
con = duckdb.connect(db_path)
try:
con.execute(
f'CREATE TABLE "{_TABLE}" '
"(fecha DATE, ventas DOUBLE, unidades INTEGER, lat DOUBLE, lon DOUBLE)"
)
con.executemany(
f'INSERT INTO "{_TABLE}" VALUES (?, ?, ?, ?, ?)', _ROWS
)
finally:
con.close()
return db_path
def _profile_with_date():
"""TableProfile minimo con columna fecha + numericas + lat/lon."""
return {
"columns": [
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
{
"name": "ventas",
"inferred_type": "numeric",
"semantic_type": "decimal",
"numeric": {"min": 980.0, "max": 1750.75},
},
{
"name": "unidades",
"inferred_type": "numeric",
"semantic_type": "integer",
"numeric": {"min": 9, "max": 18},
},
{
"name": "lat",
"inferred_type": "numeric",
"semantic_type": "decimal",
"numeric": {"min": 37.39, "max": 43.26},
},
{
"name": "lon",
"inferred_type": "numeric",
"semantic_type": "decimal",
"numeric": {"min": -5.99, "max": 2.17},
},
]
}
def _profile_without_date():
"""Mismo perfil pero SIN columna temporal (solo numericas)."""
prof = _profile_with_date()
prof["columns"] = [c for c in prof["columns"] if c["name"] != "fecha"]
return prof
def test_db_path_y_table_en_ctx(tmp_path):
db_path = _make_db(tmp_path)
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
assert ctx["db_path"] == db_path
assert ctx["table"] == _TABLE
def test_raw_numeric_con_columnas_numericas(tmp_path):
db_path = _make_db(tmp_path)
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
raw = ctx["raw_numeric"]
# Las 4 columnas numericas (ventas, unidades, lat, lon), listas no vacias y
# alineadas por fila (misma longitud == nº de filas).
for col in ("ventas", "unidades", "lat", "lon"):
assert col in raw
assert len(raw[col]) == len(_ROWS)
assert raw["ventas"][0] == 1200.5
assert raw["unidades"][0] == 12.0 # int promovido a float
def test_timeseries_raw_con_fecha(tmp_path):
db_path = _make_db(tmp_path)
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
ts = ctx["timeseries_raw"]
assert ts["time_col"] == "fecha"
assert len(ts["t"]) == len(_ROWS) # fechas ISO no vacias
# Las numericas aparecen como series paralelas a t.
for col in ("ventas", "unidades", "lat", "lon"):
assert col in ts["series"]
assert len(ts["series"][col]) == len(_ROWS)
def test_geo_points_con_latlon(tmp_path):
db_path = _make_db(tmp_path)
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
geo = ctx["geo_points"]
assert len(geo["lats"]) == len(_ROWS)
assert len(geo["lons"]) == len(_ROWS)
# Listas alineadas, ya floats, leidas de raw_numeric.
assert geo["lats"][0] == 40.41
assert geo["lons"][0] == -3.70
def test_sin_fecha_no_hay_timeseries(tmp_path):
db_path = _make_db(tmp_path)
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_without_date())
assert "timeseries_raw" not in ctx
# raw_numeric y geo_points siguen presentes (no dependen de la fecha).
assert "raw_numeric" in ctx
assert "geo_points" in ctx
def test_base_ctx_preservado(tmp_path):
db_path = _make_db(tmp_path)
base = {"dataset_name": "ventas_geo_demo", "source_origin": "test"}
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date(), base_ctx=base)
# Las claves de presentacion del base_ctx no se pisan.
assert ctx["dataset_name"] == "ventas_geo_demo"
assert ctx["source_origin"] == "test"
# Y las de datos se añaden encima.
assert ctx["db_path"] == db_path
assert "raw_numeric" in ctx
@@ -0,0 +1,115 @@
---
id: categorical_cardinality_block_py_datascience
name: categorical_cardinality_block
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def categorical_cardinality_block(cat: dict, n_rows: int) -> dict"
description: "Deriva métricas de cardinalidad listas para renderizar a partir de la salida de summarize_categorical para UNA columna categórica más el número total de filas. Calcula pct_distinct, entropy_max=log2(n_distinct), entropy_norm (recortada a [0,1]), n_singletons (sobre el top visible) y los flags id_like / dominated. NO recalcula la entropía ni reimplementa summarize_categorical: la consume. Estilo dict-no-throw del grupo eda — nunca lanza."
tags: [eda, categorical, cardinality, entropy, profiling, datascience, pure]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [math]
example: |
from categorical_cardinality_block import categorical_cardinality_block
cat = {"top": [{"value": "a", "count": 5, "pct": 0.5}], "mode": "a",
"mode_pct": 0.5, "n_distinct": 4, "entropy": 1.685, "imbalance": 5.0,
"len_min": 1, "len_mean": 1.0, "len_max": 1}
block = categorical_cardinality_block(cat, n_rows=10)
tested: true
tests:
- "test_normal_case"
- "test_empty_cat_does_not_raise"
- "test_none_cat_does_not_raise"
- "test_n_rows_zero_no_zero_division"
- "test_id_like_when_distinct_near_rows"
- "test_dominated_when_mode_pct_high"
- "test_mode_pct_fallback_from_top_fraction"
- "test_n_singletons_partial_when_top_truncated"
- "test_single_distinct_value_entropy_norm_none"
test_file_path: "python/functions/datascience/categorical_cardinality_block_test.py"
file_path: "python/functions/datascience/categorical_cardinality_block.py"
params:
- name: cat
desc: "Dict producido por summarize_categorical para UNA columna categórica. Claves leídas (todas opcionales, lectura defensiva): top (list de {value,count,pct}), mode, mode_pct (puede faltar), n_distinct, entropy (Shannon en bits), imbalance, len_min, len_mean, len_max. None o no-dict se tratan como {}."
- name: n_rows
desc: "Número total de filas del dataset. Usado para pct_distinct. Si es 0 o None, pct_distinct sale None (sin ZeroDivisionError)."
output: "Dict con exactamente 16 claves, todas siempre presentes: n_distinct, n_rows, pct_distinct, entropy, entropy_max, entropy_norm, mode, mode_pct, imbalance, n_singletons, n_singletons_partial, len_min, len_mean, len_max, id_like, dominated. Valores None/False cuando no son derivables; la función nunca lanza. pct_distinct en escala 0-100. entropy_max=log2(n_distinct) (0.0 si n_distinct in {0,1}). entropy_norm=entropy/entropy_max recortada a [0,1]. n_singletons = nº de elementos de top con count==1 (None si top vacío). n_singletons_partial=True si n_distinct>len(top). id_like=pct_distinct>=99. dominated=mode_pct>=90."
---
## Ejemplo
```python
from categorical_cardinality_block import categorical_cardinality_block
# Salida típica de summarize_categorical para una columna, con n_rows del dataset.
cat = {
"top": [
{"value": "a", "count": 5, "pct": 0.5},
{"value": "b", "count": 3, "pct": 0.3},
{"value": "c", "count": 1, "pct": 0.1},
{"value": "d", "count": 1, "pct": 0.1},
],
"mode": "a",
"mode_pct": 0.5,
"n_distinct": 4,
"entropy": 1.685, # Shannon en bits (<= log2(4) = 2.0)
"imbalance": 5.0,
"len_min": 1, "len_mean": 1.0, "len_max": 1,
}
categorical_cardinality_block(cat, n_rows=10)
# {
# "n_distinct": 4, "n_rows": 10,
# "pct_distinct": 40.0, # 4 / 10 * 100
# "entropy": 1.685,
# "entropy_max": 2.0, # log2(4)
# "entropy_norm": 0.8425, # 1.685 / 2.0, recortado a [0,1]
# "mode": "a", "mode_pct": 0.5,
# "imbalance": 5.0,
# "n_singletons": 2, # c y d con count == 1
# "n_singletons_partial": False, # top cubre los 4 distintos
# "len_min": 1, "len_mean": 1.0, "len_max": 1,
# "id_like": False, # pct_distinct 40 < 99
# "dominated": False, # mode_pct 0.5 < 90
# }
```
## Cuando usarla
Úsala justo después de `summarize_categorical`, cuando vayas a renderizar el
bloque de cardinalidad de una columna categórica en un EDA: necesitas el ratio
de valores distintos (`pct_distinct`), la entropía normalizada al rango `[0,1]`
para comparar columnas con cardinalidades distintas, el conteo de singletons, y
las banderas heurísticas `id_like` (la columna parece un identificador) y
`dominated` (una sola categoría domina). Pásale el dict crudo de
`summarize_categorical` para esa columna y el `n_rows` total del dataset. No
reimplementa nada: solo deriva métricas de presentación a partir de lo ya
calculado.
## Gotchas
- **`mode_pct` se pasa tal cual viene en `cat`.** `summarize_categorical`
produce `mode_pct` como **fracción** (01), no como porcentaje. El flag
`dominated` compara `mode_pct >= 90.0`, así que con la salida cruda de
`summarize_categorical` (fracciones) `dominated` no se dispara: aliméntalo con
`mode_pct` en escala 0100 si quieres usar esa bandera. Solo el camino de
*fallback* (cuando `cat` no trae `mode_pct` y se deriva de `top[0]['pct']`)
normaliza una fracción `<= 1` multiplicándola por 100.
- **`n_singletons` solo cubre el `top` visible.** Si `summarize_categorical` se
llamó con `top_k` pequeño, hay valores fuera del top; en ese caso
`n_singletons_partial` es `True` para avisar de que el conteo es parcial.
- **`pct_distinct` es `None` si `n_rows` es 0 o `None`** (no lanza
`ZeroDivisionError`); por tanto `id_like` queda `False` en ese caso.
- **`entropy_norm` es `None` cuando `entropy_max <= 0`** (columna constante,
`n_distinct in {0,1}`): no hay división por cero y no se inventa un 0/1.
- **No recalcula la entropía.** Si `cat['entropy']` es incoherente con
`n_distinct`, `entropy_norm` se recorta a `[0,1]` pero el valor de entrada no
se corrige.
- **`bool` no cuenta como número.** Un `True`/`False` en una clave numérica de
`cat` se trata como ausente (`None`), por la guarda defensiva.
@@ -0,0 +1,132 @@
"""Pure EDA helper: cardinality metrics block from a `summarize_categorical` output.
Part of the `eda` capability group. Consumes the per-column dict produced by
``summarize_categorical`` (for a single categorical/text column) plus the total
row count of the dataset and derives render-ready cardinality metrics: distinct
ratio, normalized entropy, singleton count, and the ``id_like`` / ``dominated``
flags.
It does NOT recompute the entropy nor reimplement ``summarize_categorical`` it
only reads that function's output. Dict-no-throw style of the `eda` group: it
never raises. Missing or malformed inputs yield ``None``/``False``/``0`` for the
affected keys, never an exception. Stdlib only (``math.log2``).
"""
from math import log2
def _num(value):
"""Return ``value`` unchanged if it is a real (non-bool) number, else ``None``.
``bool`` is rejected on purpose: in Python ``True`` is an ``int`` but it is
never a meaningful count/ratio here.
"""
if isinstance(value, bool):
return None
if isinstance(value, (int, float)):
return value
return None
def categorical_cardinality_block(cat: dict, n_rows: int) -> dict:
"""Derive cardinality metrics for one categorical column.
Args:
cat: The per-column dict produced by ``summarize_categorical`` for a
single categorical/text column. Expected (all optional, read
defensively) keys: ``top`` (list of ``{value, count, pct}``),
``mode``, ``mode_pct``, ``n_distinct``, ``entropy`` (Shannon, bits),
``imbalance``, ``len_min``, ``len_mean``, ``len_max``. ``None`` or a
non-dict is treated as ``{}``.
n_rows: Total number of rows in the dataset (used for ``pct_distinct``).
Returns:
Dict with exactly these keys, every one always present:
``n_distinct``, ``n_rows``, ``pct_distinct``, ``entropy``,
``entropy_max``, ``entropy_norm``, ``mode``, ``mode_pct``,
``imbalance``, ``n_singletons``, ``n_singletons_partial``, ``len_min``,
``len_mean``, ``len_max``, ``id_like``, ``dominated``. Values are
``None``/``False`` when not derivable; the function never raises.
"""
cat = cat if isinstance(cat, dict) else {}
# --- passthroughs (numeric-validated, type preserved) ---
n_distinct = _num(cat.get("n_distinct"))
n_rows_out = _num(n_rows)
entropy = _num(cat.get("entropy"))
imbalance = _num(cat.get("imbalance"))
len_min = _num(cat.get("len_min"))
len_mean = _num(cat.get("len_mean"))
len_max = _num(cat.get("len_max"))
mode = cat.get("mode") # any value (or None); passthrough as-is
# --- pct_distinct ---
if n_distinct is None or n_rows_out is None or n_rows_out == 0:
pct_distinct = None
else:
pct_distinct = n_distinct / n_rows_out * 100.0
# --- entropy_max = log2(n_distinct) ---
if n_distinct is None:
entropy_max = None
elif n_distinct > 1:
entropy_max = log2(n_distinct)
else: # n_distinct in {0, 1}
entropy_max = 0.0
# --- entropy_norm = entropy / entropy_max, clipped to [0, 1] ---
if entropy_max is not None and entropy_max > 0 and entropy is not None:
entropy_norm = entropy / entropy_max
entropy_norm = max(0.0, min(1.0, entropy_norm))
else:
entropy_norm = None
# --- mode_pct: prefer cat['mode_pct']; else derive from top[0].pct ---
mode_pct = _num(cat.get("mode_pct"))
top = cat.get("top")
has_top = isinstance(top, (list, tuple)) and len(top) > 0
if mode_pct is None and has_top:
first = top[0]
if isinstance(first, dict):
first_pct = _num(first.get("pct"))
if first_pct is not None:
# Normalize to 0-100: a fraction (<= 1) becomes a percentage.
mode_pct = first_pct * 100.0 if first_pct <= 1 else first_pct
# --- singletons (count == 1) within the visible top ---
if has_top:
n_singletons = sum(
1
for item in top
if isinstance(item, dict) and _num(item.get("count")) == 1
)
else:
n_singletons = None
# The singleton count only covers the visible top; there may be more
# distinct values (and thus more singletons) outside it.
top_len = len(top) if isinstance(top, (list, tuple)) else 0
n_singletons_partial = bool(n_distinct is not None and n_distinct > top_len)
# --- derived flags ---
id_like = pct_distinct is not None and pct_distinct >= 99.0
dominated = mode_pct is not None and mode_pct >= 90.0
return {
"n_distinct": n_distinct,
"n_rows": n_rows_out,
"pct_distinct": pct_distinct,
"entropy": entropy,
"entropy_max": entropy_max,
"entropy_norm": entropy_norm,
"mode": mode,
"mode_pct": mode_pct,
"imbalance": imbalance,
"n_singletons": n_singletons,
"n_singletons_partial": n_singletons_partial,
"len_min": len_min,
"len_mean": len_mean,
"len_max": len_max,
"id_like": id_like,
"dominated": dominated,
}
@@ -0,0 +1,216 @@
"""Tests para categorical_cardinality_block."""
import sys
import os
from math import log2
sys.path.insert(0, os.path.dirname(__file__))
from categorical_cardinality_block import categorical_cardinality_block
# Output contract: every call returns exactly these 16 keys.
EXPECTED_KEYS = {
"n_distinct",
"n_rows",
"pct_distinct",
"entropy",
"entropy_max",
"entropy_norm",
"mode",
"mode_pct",
"imbalance",
"n_singletons",
"n_singletons_partial",
"len_min",
"len_mean",
"len_max",
"id_like",
"dominated",
}
def _sample_cat():
"""A realistic summarize_categorical output for one column."""
return {
"top": [
{"value": "a", "count": 5, "pct": 0.5},
{"value": "b", "count": 3, "pct": 0.3},
{"value": "c", "count": 1, "pct": 0.1},
{"value": "d", "count": 1, "pct": 0.1},
],
"mode": "a",
"mode_pct": 0.5,
"n_distinct": 4,
"entropy": 1.685, # <= log2(4) = 2.0
"imbalance": 5.0,
"len_min": 1,
"len_mean": 1.0,
"len_max": 1,
}
def test_normal_case():
"""Caso normal: pct_distinct, entropy_max=log2(n_distinct), entropy_norm in [0,1], n_singletons."""
cat = _sample_cat()
result = categorical_cardinality_block(cat, n_rows=10)
assert set(result.keys()) == EXPECTED_KEYS
# passthroughs
assert result["n_distinct"] == 4
assert result["n_rows"] == 10
assert result["entropy"] == 1.685
assert result["imbalance"] == 5.0
assert result["mode"] == "a"
assert result["mode_pct"] == 0.5 # passthrough, not normalized
assert result["len_min"] == 1
assert result["len_max"] == 1
# pct_distinct = 4 / 10 * 100
assert abs(result["pct_distinct"] - 40.0) < 1e-12
# entropy_max = log2(4) = 2.0
assert abs(result["entropy_max"] - log2(4)) < 1e-12
assert abs(result["entropy_max"] - 2.0) < 1e-12
# entropy_norm = 1.685 / 2.0 = 0.8425, within [0, 1]
assert abs(result["entropy_norm"] - 1.685 / 2.0) < 1e-12
assert 0.0 <= result["entropy_norm"] <= 1.0
# singletons: c and d have count == 1
assert result["n_singletons"] == 2
# top covers all distinct values (4 == 4)
assert result["n_singletons_partial"] is False
# neither id-like (40%) nor dominated (mode_pct 0.5)
assert result["id_like"] is False
assert result["dominated"] is False
def test_empty_cat_does_not_raise():
"""Caso cat={}: no lanza, claves derivadas None y flags False."""
result = categorical_cardinality_block({}, n_rows=100)
assert set(result.keys()) == EXPECTED_KEYS
for key in (
"n_distinct",
"pct_distinct",
"entropy",
"entropy_max",
"entropy_norm",
"mode",
"mode_pct",
"imbalance",
"n_singletons",
"len_min",
"len_mean",
"len_max",
):
assert result[key] is None
assert result["n_singletons_partial"] is False
assert result["id_like"] is False
assert result["dominated"] is False
# n_rows is a passthrough of the argument, still coherent.
assert result["n_rows"] == 100
def test_none_cat_does_not_raise():
"""Caso cat=None: tratado como {}, mismas garantias que el dict vacio."""
result = categorical_cardinality_block(None, n_rows=None)
assert set(result.keys()) == EXPECTED_KEYS
assert result["n_distinct"] is None
assert result["pct_distinct"] is None
assert result["entropy_max"] is None
assert result["entropy_norm"] is None
assert result["id_like"] is False
assert result["dominated"] is False
def test_n_rows_zero_no_zero_division():
"""Caso n_rows=0: pct_distinct None sin ZeroDivisionError."""
cat = _sample_cat()
result = categorical_cardinality_block(cat, n_rows=0)
assert result["pct_distinct"] is None
# n_distinct still passes through.
assert result["n_distinct"] == 4
assert result["id_like"] is False
def test_id_like_when_distinct_near_rows():
"""id_like True cuando n_distinct ~ n_rows (pct_distinct >= 99)."""
cat = {"n_distinct": 99, "entropy": 6.6, "top": [], "mode": None}
result = categorical_cardinality_block(cat, n_rows=100)
assert abs(result["pct_distinct"] - 99.0) < 1e-12
assert result["id_like"] is True
# exact identity column: 100 / 100 = 100%
cat_full = {"n_distinct": 100, "top": []}
result_full = categorical_cardinality_block(cat_full, n_rows=100)
assert result_full["id_like"] is True
def test_dominated_when_mode_pct_high():
"""dominated True cuando mode_pct alto (>= 90)."""
cat = {
"n_distinct": 3,
"entropy": 0.3,
"mode": "x",
"mode_pct": 95.0,
"top": [
{"value": "x", "count": 95, "pct": 0.95},
{"value": "y", "count": 3, "pct": 0.03},
{"value": "z", "count": 2, "pct": 0.02},
],
"imbalance": 47.5,
}
result = categorical_cardinality_block(cat, n_rows=100)
assert result["mode_pct"] == 95.0
assert result["dominated"] is True
def test_mode_pct_fallback_from_top_fraction():
"""Sin mode_pct: deriva del pct del primer top, fraccion <=1 escala a 0-100."""
cat = {
"n_distinct": 3,
"top": [
{"value": "x", "count": 95, "pct": 0.95},
{"value": "y", "count": 5, "pct": 0.05},
],
}
result = categorical_cardinality_block(cat, n_rows=100)
# 0.95 (fraction) -> 95.0 (percentage)
assert abs(result["mode_pct"] - 95.0) < 1e-12
assert result["dominated"] is True
def test_n_singletons_partial_when_top_truncated():
"""n_distinct > len(top): n_singletons cubre solo el top visible, partial True."""
cat = {
"n_distinct": 10,
"top": [
{"value": "a", "count": 4, "pct": 0.4},
{"value": "b", "count": 1, "pct": 0.1},
{"value": "c", "count": 1, "pct": 0.1},
],
"entropy": 2.5,
}
result = categorical_cardinality_block(cat, n_rows=12)
assert result["n_singletons"] == 2 # only b, c visible
assert result["n_singletons_partial"] is True
def test_single_distinct_value_entropy_norm_none():
"""n_distinct=1: entropy_max=0.0 -> entropy_norm None (no division by zero)."""
cat = {
"n_distinct": 1,
"entropy": 0.0,
"mode": "only",
"mode_pct": 1.0,
"top": [{"value": "only", "count": 7, "pct": 1.0}],
"imbalance": 1.0,
}
result = categorical_cardinality_block(cat, n_rows=7)
assert result["entropy_max"] == 0.0
assert result["entropy_norm"] is None
assert result["n_singletons"] == 0
@@ -0,0 +1,108 @@
---
id: categorical_top_pie_figure_py_datascience
name: categorical_top_pie_figure
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def categorical_top_pie_figure(top: list, n_distinct: int = 0, title: str = \"\", top_k: int = 6, n_rows=None) -> \"matplotlib.figure.Figure\""
description: "Construye una figura matplotlib tipo donut (pie con agujero central) de las top_k categorías más frecuentes de una columna categórica, agregando el resto en un sector gris \"Otros (N categorías)\". Consume el bloque `top` de summarize_categorical y devuelve un matplotlib.figure.Figure listo para rasterizar por el renderer del informe EDA. Backend Agg sin pyplot global; defensivo ante top vacío/None."
tags: [eda, categorical, pie, donut, matplotlib, figure, visualization, datascience, impure]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: [matplotlib]
example: |
from categorical_top_pie_figure import categorical_top_pie_figure
top = [
{"value": "rojo", "count": 40, "pct": 0.4},
{"value": "azul", "count": 30, "pct": 0.3},
{"value": "verde", "count": 20, "pct": 0.2},
]
fig = categorical_top_pie_figure(top, n_distinct=12, title="color", top_k=6, n_rows=100)
tested: true
tests:
- "test_returns_figure"
- "test_ten_items_topk_six_yields_seven_wedges"
- "test_empty_top_does_not_raise_and_returns_figure"
- "test_long_value_truncated_in_legend"
- "test_none_value_and_none_count_are_handled"
- "test_n_rows_adds_exact_others_slice"
test_file_path: "python/functions/datascience/categorical_top_pie_figure_test.py"
file_path: "python/functions/datascience/categorical_top_pie_figure.py"
params:
- name: top
desc: "Lista de dicts {value, count, pct} ordenada de mayor a menor por count (salida del bloque `top` de summarize_categorical). Puede venir vacía o con dicts incompletos: items no-dict, sin count, con count None o count <= 0 se descartan. value None se admite (sin etiqueta)."
- name: n_distinct
desc: "Nº total de categorías distintas de la columna. Etiqueta el sector agregado como \"Otros (n_distinct - top_k)\" (mínimo 0). Si no supera el nº de sectores mostrados, se usa el overflow real de `top` como nº de categorías agregadas. Default 0."
- name: title
desc: "Título de la figura (nombre de la columna). Se trunca a ~48 chars con elipsis si es muy largo. Default \"\" (sin título)."
- name: top_k
desc: "Nº máximo de sectores explícitos. Default 6. El sector \"Otros\" no cuenta contra este límite. Con top_k <= 0 se muestra al menos la categoría mayor."
- name: n_rows
desc: "Opcional. Total de filas del dataset. Si se da y la suma de counts mostrados < n_rows, el sector \"Otros\" usa (n_rows - suma_mostrada) como count para que los ángulos sean exactos respecto al total real. Si se omite, \"Otros\" usa la suma de counts fuera del top_k mostrado (solo cuando top trae más de top_k items). Default None."
output: "Un matplotlib.figure.Figure (figsize 6.4x4.0, dpi 150) con un Axes donut (wedgeprops width 0.42) más una leyenda lateral con value truncado a 20 chars + count; el sector \"Otros\" en gris. Anotación central con el total n. Si no hay counts válidos, devuelve igualmente una Figure con un texto centrado \"sin datos categóricos\" (nunca lanza). El caller rasteriza/cierra la figura; la función no la muestra ni la guarda."
---
## Ejemplo
```python
from categorical_top_pie_figure import categorical_top_pie_figure
# `top` es la salida del bloque "top" de summarize_categorical (ya ordenado desc).
top = [
{"value": "rojo", "count": 40, "pct": 0.40},
{"value": "azul", "count": 30, "pct": 0.30},
{"value": "verde", "count": 20, "pct": 0.20},
{"value": "amarillo", "count": 5, "pct": 0.05},
]
fig = categorical_top_pie_figure(
top,
n_distinct=12, # 12 categorías distintas en total
title="color_producto",
top_k=6, # hasta 6 sectores explícitos
n_rows=100, # "Otros" = 100 - 95 = 5, sobre 8 categorías agregadas
)
# El renderer del informe lo rasteriza; aquí solo persistimos para inspección.
fig.savefig("/tmp/donut_color.png")
```
## Cuando usarla
Úsala dentro de un informe EDA cuando quieras visualizar la composición de una
columna categórica de un vistazo: cuántas filas caen en las categorías
dominantes frente a la cola larga. Pásale directamente el bloque `top` de
`summarize_categorical` (ya ordenado de mayor a menor) más `n_distinct` para que
el sector "Otros" indique cuántas categorías quedan agrupadas. Es la pareja
"composición" del gráfico de barras top-k: el donut comunica proporciones del
total, las barras comunican magnitudes comparables.
## Gotchas
- **Impura por matplotlib.** Toca la maquinaria de render. Usa el backend `Agg`
y la API orientada a objetos `Figure`/`add_subplot` — NUNCA `pyplot.*` aquí,
para no tocar el estado global ni filtrar figuras entre llamadas. `pyplot` NO
es thread-safe; esta función evita ese riesgo construyendo el `Figure`
directamente, así que es segura de llamar en bucle desde el renderer.
- **El caller cierra la figura.** La función devuelve el `Figure` pero no lo
muestra ni lo guarda. Quien la consume debe rasterizarla y luego liberarla
(`fig.clf()` / `matplotlib.pyplot.close(fig)` si se usó pyplot en el caller)
para no acumular memoria en lotes grandes de columnas.
- **Pie engaña con muchos sectores.** Por eso `top_k` por defecto es 6 y el
resto se agrega en "Otros": donuts con 15+ sectores son ilegibles. Para
cardinalidad muy alta el donut solo muestra la cabeza de la distribución; la
cola vive en el sector gris.
- **Ángulos exactos solo con `n_rows`.** Sin `n_rows`, el sector "Otros" se
calcula con el overflow presente en `top`; si `top` ya viene recortado a
`top_k` por el productor, no habrá "Otros" aunque existan más categorías. Pasa
`n_rows` (total de filas del dataset) para ángulos correctos respecto al total
real.
- **Defensiva, nunca lanza.** `top=[]`, `value=None`, `count=None` o counts no
numéricos se manejan sin error: en el peor caso devuelve una `Figure` con
"sin datos categóricos". No envuelvas la llamada en try/except por miedo a un
raise — no lo hay.
@@ -0,0 +1,230 @@
"""Impure EDA helper: donut figure of the most common categories (`eda` group).
Builds a matplotlib donut (pie with a central hole) of the ``top_k`` most
frequent categories of a categorical column, folding everything else into a
single "Otros (N categorías)" slice. Returns a ready-to-rasterize
``matplotlib.figure.Figure``; it never shows nor saves it.
Impure because it touches matplotlib's rendering machinery. It uses the headless
Agg backend and the object-oriented ``Figure`` API (no ``pyplot``) so it leaks no
global state and is safe to call repeatedly from a report renderer.
"""
import matplotlib
matplotlib.use("Agg")
from matplotlib.figure import Figure # noqa: E402
# Gray reserved for the aggregated "Otros" slice.
_OTHER_COLOR = "#9e9e9e"
# Muted gray for secondary text (title fallback, center annotation, no-data).
_MUTED_TEXT = "#5f6b7a"
# Pleasant, colour-blind-friendly qualitative palette for the explicit slices.
_PALETTE = [
"#4C72B0",
"#DD8452",
"#55A868",
"#C44E52",
"#8172B3",
"#937860",
"#DA8BC3",
"#8C8C8C",
"#CCB974",
"#64B5CD",
]
def _truncate(text, width: int = 20) -> str:
"""Truncate ``text`` to ``width`` chars, appending an ellipsis if cut."""
s = "" if text is None else str(text)
if len(s) <= width:
return s
if width <= 1:
return s[:width]
return s[: width - 1] + ""
def categorical_top_pie_figure(
top: list,
n_distinct: int = 0,
title: str = "",
top_k: int = 6,
n_rows=None,
) -> "matplotlib.figure.Figure":
"""Build a donut figure of the most common categories of a column.
Renders the ``top_k`` most frequent categories as explicit donut slices and
aggregates every remaining category into a single gray "Otros (N
categorías)" slice. Category names are not painted on the wedges; they are
listed in a lateral legend (truncated value + count) to avoid overlap on
narrow (mobile) figures.
The function is fully defensive: empty input, missing/``None`` values or
counts never raise. When there is nothing valid to draw it still returns a
``Figure`` carrying a centered "sin datos categóricos" message.
Args:
top: List of ``{value, count, pct}`` dicts, already sorted by ``count``
descending (the ``top`` block of ``summarize_categorical``). May be
empty or carry incomplete/``None`` entries; non-dict items, items
without a positive numeric ``count`` and ``None`` counts are skipped.
n_distinct: Total number of distinct categories in the column. Used to
label the aggregated slice as "Otros (n_distinct - top_k)" (floored
at 0). Ignored when it does not exceed the number of shown slices.
title: Figure title (the column name). Truncated when too long.
top_k: Maximum number of explicit slices. Default 6. The "Otros" slice
does not count against this limit.
n_rows: Optional total row count of the dataset. When given and the sum
of shown counts is below ``n_rows``, the "Otros" slice uses
``n_rows - sum_shown`` as its count so the wedge angles are exact
with respect to the real total. When omitted, "Otros" uses the sum
of the counts that fall outside the shown ``top_k`` (only when
``top`` carries more than ``top_k`` items).
Returns:
A ``matplotlib.figure.Figure`` with a single donut Axes plus a lateral
legend. The caller is responsible for rasterizing/closing it.
"""
fig = Figure(figsize=(6.4, 4.0), dpi=150)
ax = fig.add_subplot(111)
safe_title = _truncate(title, 48)
# --- Defensive parse: keep only well-formed {value, count} with count > 0.
cleaned = []
if isinstance(top, list):
for item in top:
if not isinstance(item, dict):
continue
count = item.get("count")
if count is None:
continue
try:
count = float(count)
except (TypeError, ValueError):
continue
if count <= 0:
continue
cleaned.append((item.get("value"), count))
if not cleaned:
ax.axis("off")
ax.text(
0.5,
0.5,
"sin datos categóricos",
ha="center",
va="center",
fontsize=12,
color=_MUTED_TEXT,
transform=ax.transAxes,
)
if safe_title:
ax.set_title(safe_title, fontsize=12, loc="center", pad=8)
fig.tight_layout()
return fig
# --- Split into shown slices and the aggregated remainder.
shown = cleaned[: max(int(top_k), 0)]
if not shown: # top_k <= 0 — show at least the largest category.
shown = cleaned[:1]
sum_shown = sum(c for _, c in shown)
overflow_count = sum(c for _, c in cleaned[len(shown):])
# How many categories are folded into "Otros".
try:
nd = int(n_distinct)
except (TypeError, ValueError):
nd = 0
others_categories = max(nd - len(shown), 0)
# If n_distinct is unknown/too small, fall back to the overflow we actually
# have in `top` beyond the shown slices.
overflow_items = len(cleaned) - len(shown)
if others_categories == 0 and overflow_items > 0:
others_categories = overflow_items
# Count attributed to the "Otros" slice for exact angles.
others_count = 0.0
if n_rows is not None:
try:
total_rows = float(n_rows)
except (TypeError, ValueError):
total_rows = None
if total_rows is not None and total_rows > sum_shown:
others_count = total_rows - sum_shown
if others_count <= 0:
others_count = overflow_count
labels = [v for v, _ in shown]
values = [c for _, c in shown]
colors = [_PALETTE[i % len(_PALETTE)] for i in range(len(shown))]
has_others = others_count > 0 and others_categories > 0
if has_others:
values.append(others_count)
labels.append("Otros")
colors.append(_OTHER_COLOR)
total = sum(values)
def _autopct(pct: float) -> str:
# Hide tiny labels to avoid crowding the wedges.
return f"{pct:.0f}%" if pct >= 5 else ""
wedges, _texts, autotexts = ax.pie(
values,
colors=colors,
startangle=90,
counterclock=False,
wedgeprops={"width": 0.42, "edgecolor": "white", "linewidth": 1.0},
autopct=_autopct,
pctdistance=0.79,
textprops={"fontsize": 8},
)
for at in autotexts:
at.set_color("white")
at.set_fontweight("bold")
ax.set_aspect("equal")
# --- Lateral legend: truncated value + count (+ "(N categorías)" for Otros).
legend_labels = []
for idx, (lab, val) in enumerate(zip(labels, values)):
if has_others and idx == len(labels) - 1:
legend_labels.append(
f"Otros ({others_categories} categorías) — {int(round(val))}"
)
else:
legend_labels.append(f"{_truncate(lab, 20)}{int(round(val))}")
ax.legend(
wedges,
legend_labels,
title="Categorías",
loc="center left",
bbox_to_anchor=(1.02, 0.5),
fontsize=8,
title_fontsize=9,
frameon=False,
)
if safe_title:
ax.set_title(safe_title, fontsize=13, loc="left", pad=10)
# Center annotation: total count covered by the donut.
ax.text(
0,
0,
f"n={int(round(total))}",
ha="center",
va="center",
fontsize=11,
color=_MUTED_TEXT,
fontweight="bold",
)
# Leave room on the right for the legend (avoid clipping it).
fig.subplots_adjust(left=0.02, right=0.62, top=0.88, bottom=0.06)
return fig
@@ -0,0 +1,104 @@
"""Tests para categorical_top_pie_figure (donut de categorías top, grupo eda).
Usa el backend Agg sin pyplot; no muestra ni guarda figuras. Cada test cierra
explícitamente la Figure construida (matplotlib.pyplot.close) para no acumular
estado entre tests.
"""
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt # noqa: E402
from matplotlib.figure import Figure # noqa: E402
from categorical_top_pie_figure import categorical_top_pie_figure
def _make_top(n):
"""n items {value, count, pct} ordenados desc por count."""
return [
{"value": f"cat_{i}", "count": n - i, "pct": (n - i) / sum(range(1, n + 1))}
for i in range(n)
]
def _wedges(ax):
"""Devuelve los wedges (sectores) de un Axes con un pie."""
from matplotlib.patches import Wedge
return [p for p in ax.patches if isinstance(p, Wedge)]
def test_returns_figure():
fig = categorical_top_pie_figure(_make_top(3), n_distinct=3, title="col")
assert isinstance(fig, Figure)
plt.close(fig)
def test_ten_items_topk_six_yields_seven_wedges():
top = _make_top(10)
fig = categorical_top_pie_figure(top, n_distinct=10, title="muchas", top_k=6)
ax = fig.axes[0]
wedges = _wedges(ax)
# 6 categorías explícitas + 1 sector "Otros".
assert len(wedges) == 7
plt.close(fig)
def test_empty_top_does_not_raise_and_returns_figure():
fig = categorical_top_pie_figure([], n_distinct=0, title="vacía")
assert isinstance(fig, Figure)
# Sin datos: no debe haber sectores de pie.
assert len(_wedges(fig.axes[0])) == 0
plt.close(fig)
def test_long_value_truncated_in_legend():
long_value = "una_categoria_con_un_nombre_larguisimo_que_excede_el_limite"
top = [
{"value": long_value, "count": 10, "pct": 0.5},
{"value": "corta", "count": 10, "pct": 0.5},
]
fig = categorical_top_pie_figure(top, n_distinct=2, title="col", top_k=6)
ax = fig.axes[0]
legend = ax.get_legend()
assert legend is not None
texts = [t.get_text() for t in legend.get_texts()]
# El valor largo aparece truncado con elipsis y NO en su forma completa.
assert any("" in t for t in texts)
assert long_value not in " ".join(texts)
plt.close(fig)
def test_none_value_and_none_count_are_handled():
top = [
{"value": None, "count": 5, "pct": 0.5},
{"value": "b", "count": None, "pct": 0.0}, # count None -> se descarta
{"value": "c", "count": 5, "pct": 0.5},
]
fig = categorical_top_pie_figure(top, n_distinct=2, title="con nones", top_k=6)
assert isinstance(fig, Figure)
# Solo 2 items válidos, sin overflow -> 2 wedges, sin "Otros".
assert len(_wedges(fig.axes[0])) == 2
plt.close(fig)
def test_n_rows_adds_exact_others_slice():
# 3 categorías mostradas suman 30, dataset real 100 -> "Otros" = 70.
top = _make_top(3) # counts 3,2,1 -> reescalamos abajo
top = [
{"value": "a", "count": 15, "pct": 0.15},
{"value": "b", "count": 10, "pct": 0.10},
{"value": "c", "count": 5, "pct": 0.05},
]
fig = categorical_top_pie_figure(
top, n_distinct=20, title="col", top_k=3, n_rows=100
)
ax = fig.axes[0]
# 3 explícitas + Otros.
assert len(_wedges(ax)) == 4
legend_texts = [t.get_text() for t in ax.get_legend().get_texts()]
# El sector Otros refleja n_distinct - top_k = 17 categorías y count 70.
assert any("Otros (17 categorías)" in t and "70" in t for t in legend_texts)
plt.close(fig)
@@ -0,0 +1,68 @@
---
name: classify_relationship_type
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def classify_relationship_type(xs: list, ys: list) -> dict"
description: "Clasifica el TIPO de relacion entre dos variables numericas pareadas por indice para el EDA automatico del grupo eda. Limpia los pares de forma defensiva (descarta None/bool/NaN/inf), reusa pearson y spearman_corr del registry y ajusta polinomios de grado 2 y 3 con numpy.polyfit (R^2 manual), y a partir de esas senales etiqueta la forma: 'lineal', 'polinomica (grado 2/3)', 'monotona no-lineal' o 'debil/sin forma'. Orden de decision: debil -> monotona -> polinomica -> lineal (la primera que matchea gana), con umbrales calibrados para datos reales discretos/ruidosos. Devuelve ademas los coeficientes del mejor modelo en orden de numpy.polyval para pintar la curva de ajuste sobre el scatter. Funcion pura no-throw: ante datos insuficientes (menos de 5 pares validos o varianza ~0) o cualquier fallo devuelve el dict canonico con tipo='debil/sin forma' y el resto a None."
tags: [eda, correlation, relationship, classification, polyfit, datascience, pure]
params:
- name: xs
desc: "Lista (o tupla) de valores numericos de la primera variable, pareada por indice con ys. Cada par xs[i],ys[i] se descarta si cualquiera de los dos es None, bool, NaN o inf. Lectura defensiva."
- name: ys
desc: "Lista (o tupla) de valores numericos de la segunda variable, pareada por indice con xs. Mismas reglas de limpieza que xs."
output: "Dict con SIEMPRE las mismas 8 claves: tipo (str: 'lineal' | 'polinómica (grado 2)' | 'polinómica (grado 3)' | 'monótona no-lineal' | 'débil/sin forma'); pearson (float|None: coeficiente de Pearson r); r2_linear (float|None: r**2 del ajuste lineal); spearman (float|None: rho de Spearman); r2_poly2 (float|None: R^2 del ajuste polinomico de grado 2); r2_poly3 (float|None: R^2 del ajuste de grado 3); best_degree (int|None: grado del modelo elegido — 1 lineal, 2/3 polinomico, None si monotona/debil); coeffs (list|None: coeficientes del mejor modelo en orden de numpy.polyval para pintar la curva, o None). Ante datos insuficientes o error: tipo='débil/sin forma' y el resto de claves a None."
uses_functions: [pearson_py_datascience, spearman_corr_py_datascience]
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [numpy]
tested: true
tests: ["test_lineal", "test_polinomica_cuadratica", "test_monotona_no_lineal", "test_monotona_exponencial", "test_debil_sin_forma", "test_lista_vacia_no_lanza", "test_longitudes_distintas_no_lanza", "test_todos_none_no_lanza", "test_entradas_none_no_lanza", "test_constante_no_lanza", "test_filtra_nan_inf_bool"]
test_file_path: "python/functions/datascience/classify_relationship_type_test.py"
file_path: "python/functions/datascience/classify_relationship_type.py"
---
## Ejemplo
```python
import sys, os
sys.path.insert(0, os.path.join("python", "functions"))
from datascience.classify_relationship_type import classify_relationship_type
import numpy as np
# Relacion claramente cuadratica (forma de parabola) sobre dominio simetrico.
x = list(np.linspace(-10, 10, 60))
y = [v * v for v in x]
res = classify_relationship_type(x, y)
print(res["tipo"]) # 'polinómica (grado 2)'
print(res["best_degree"]) # 2
print(res["r2_linear"]) # 0.0 -> el Pearson lineal no ve la parabola
print(res["r2_poly2"]) # 1.0
print(res["coeffs"]) # [1.0, -0.0, -0.0] -> numpy.polyval(coeffs, x) ~ x**2
# El capitulo pinta la curva de ajuste cuando coeffs no es None:
# if res["coeffs"] is not None:
# xs_fit = np.linspace(min(x), max(x), 200)
# ys_fit = np.polyval(res["coeffs"], xs_fit)
# ax.plot(xs_fit, ys_fit) # curva sobre el ax.scatter(x, y)
```
## Cuando usarla
- Usala en el capitulo de relaciones/correlaciones del EDA automatico, despues de detectar dos columnas numericas con alguna asociacion, para decidir QUE curva de ajuste pintar sobre el scatter (recta, parabola, cubica o ninguna) y poner una etiqueta legible al tipo de relacion.
- Cuando un Pearson bajo no signifique "sin relacion": esta funcion cruza Pearson con Spearman y con ajustes polinomicos para distinguir una relacion lineal debil de una monotona no-lineal (que el rango si capta) o de una curva polinomica.
- Cuando necesites un punto de entrada determinista y no-throw que, con los mismos datos, devuelva siempre el mismo `tipo` y los mismos `coeffs` listos para `numpy.polyval` sin tener que ajustar modelos a mano en el capitulo.
## Gotchas
- Funcion pura, deterministica y no-throw: ante menos de 5 pares validos, varianza ~0 (xs o ys constante) o cualquier excepcion interna devuelve el dict canonico `tipo="débil/sin forma"` con el resto de claves a `None`. El dict SIEMPRE trae las 8 claves: nunca compruebes existencia, comprueba `None`.
- El orden de decision importa: `débil -> monótona -> polinómica -> lineal` (la primera que matchee gana). La monotonia se evalua ANTES que el ajuste polinomico, asi que una curva monotona suave (exp, log, potencias) sale `monótona no-lineal` aunque un cubico tambien la ajuste — la dominancia del rango (Spearman >> Pearson) es la senal mas interpretable. Solo cae en `polinómica` una forma curva NO monotona (p.ej. una parabola, Spearman ~0 pero R^2 polinomico alto).
- Umbrales fijos (calibrados para EDA con datos discretos/ruidosos, no para inferencia formal): `débil/sin forma` si las tres senales son bajas a la vez (`abs(pearson) < 0.3` y `abs(spearman) < 0.3` y `mejor_poly < 0.3`); `monótona no-lineal` si `abs(spearman) - abs(pearson) >= 0.1` y `abs(spearman) >= 0.4`; `polinómica (grado N)` si el mejor polinomico mejora `>= 0.1` sobre el lineal y su R^2 `>= 0.3`; en cualquier otro caso con senal (no debil) `lineal`. El suelo de 0.3 evita llamar "debil" a relaciones reales pero discretas (conteos, escalas ordinales) con R^2 bajo pero direccion clara.
- `coeffs` va en orden de `numpy.polyval` (grado descendente). Para `lineal` es `[pendiente, intercepto]` (grado 1); para `polinómica` los del grado elegido; para `monótona no-lineal` y `débil/sin forma` es `None` (el scatter pintara una curva suavizada o nada — lo decide el capitulo, no esta funcion).
- `best_degree` prefiere el grado 2 sobre el 3 cuando empatan dentro de 0.02 de R^2 (parsimonia): no esperes grado 3 salvo que mejore claramente.
- Los pares con `None`, `bool`, `NaN` o `inf` se descartan por indice en silencio; `bool` cuenta como no-numerico (un `True` no es `1`). El dominio de los datos afecta al resultado: una parabola sobre un dominio simetrico da Pearson ~0 (sale `polinómica`), pero sobre un dominio asimetrico el Pearson sube y puede salir `lineal`.
@@ -0,0 +1,187 @@
"""Clasifica el TIPO de relacion entre dos variables numericas pareadas.
Funcion pura del grupo eda. Dadas dos listas numericas pareadas por indice,
limpia los pares de forma defensiva, calcula correlaciones lineal (Pearson) y de
rangos (Spearman) y ajustes polinomicos de grado 2 y 3, y a partir de esas
senales etiqueta la forma de la relacion para el EDA automatico:
"lineal" | "polinómica (grado 2)" | "polinómica (grado 3)" |
"monótona no-lineal" | "débil/sin forma"
Ademas devuelve los coeficientes del mejor modelo (en orden de numpy.polyval)
para que el capitulo pinte la curva de ajuste sobre el scatter. Reusa las
funciones del registry `pearson` y `spearman_corr` en vez de reimplementarlas.
NUNCA lanza: ante cualquier fallo o dato insuficiente devuelve el dict canonico
con tipo="débil/sin forma" y el resto de claves a None.
"""
import math
import warnings
import numpy as np
from datascience.datascience import pearson
from datascience.spearman_corr import spearman_corr
# Forma canonica de la respuesta cuando no se puede clasificar (datos
# insuficientes, varianza nula o error interno). Siempre las mismas claves.
_WEAK = {
"tipo": "débil/sin forma",
"pearson": None,
"r2_linear": None,
"spearman": None,
"r2_poly2": None,
"r2_poly3": None,
"best_degree": None,
"coeffs": None,
}
def _is_num(v) -> bool:
"""True si v es un numero real finito (int/float, no bool, no NaN, no inf)."""
return (
isinstance(v, (int, float))
and not isinstance(v, bool)
and not (isinstance(v, float) and (math.isnan(v) or math.isinf(v)))
)
def _poly_r2(coeffs, x_arr, y_arr, ss_tot: float) -> float:
"""R^2 de un ajuste polinomico: 1 - SS_res/SS_tot. 0 si SS_tot==0."""
if ss_tot == 0.0:
return 0.0
pred = np.polyval(coeffs, x_arr)
ss_res = float(np.sum((y_arr - pred) ** 2))
return 1.0 - ss_res / ss_tot
def classify_relationship_type(xs: list, ys: list) -> dict:
"""Clasifica el tipo de relacion entre dos variables numericas pareadas.
Empareja xs[i],ys[i] por indice y descarta el par si cualquiera de los dos
es None, bool, NaN o inf. Sobre los pares limpios calcula Pearson r
(r2_linear = r**2), Spearman rho y los R^2 de ajustes polinomicos de grado 2
y 3 (con numpy.polyfit + R^2 manual). Con esas senales decide la etiqueta.
Orden de evaluacion de la etiqueta (la primera que matchee gana). Los
umbrales estan calibrados para datos reales, a menudo discretos y ruidosos
(conteos, escalas ordinales): una relacion con |r| >= 0.3, |rho| >= 0.3 o un
polinomio con R^2 >= 0.3 ya tiene FORMA y no debe etiquetarse como "debil".
1. "débil/sin forma" todas las senales bajas a la vez:
abs(pearson) < 0.3 y abs(spearman) < 0.3 y mejor_poly < 0.3.
2. "monótona no-lineal" el rango (Spearman) capta una monotonia que el
Pearson lineal no: abs(spearman) - abs(pearson) >= 0.1 y
abs(spearman) >= 0.4. No se fuerza un polinomio (coeffs/best_degree =
None); el capitulo dibuja la tendencia ordenada sobre el scatter.
3. "polinómica (grado N)" el mejor polinomico mejora claramente sobre
el lineal (mejor_poly - r2_linear >= 0.1) y mejor_poly >= 0.3. N es el
grado (2 o 3) con mejor R^2, prefiriendo el 2 si empatan dentro de 0.02
(parsimonia).
4. "lineal" el resto: hay senal (no es debil) y la forma que existe es
esencialmente lineal. best_degree=1, coeffs del ajuste de grado 1.
Si hay menos de 5 pares validos, o la varianza de xs o de ys es ~0
(constante), devuelve directamente "débil/sin forma".
Args:
xs: lista (o tupla) de valores numericos de la primera variable,
pareada por indice con ys. Pares con None/bool/NaN/inf se descartan.
ys: lista (o tupla) de valores numericos de la segunda variable,
pareada por indice con xs.
Returns:
dict con SIEMPRE las mismas claves:
tipo (str), pearson (float|None), r2_linear (float|None),
spearman (float|None), r2_poly2 (float|None), r2_poly3 (float|None),
best_degree (int|None: 1, 2, 3 o None),
coeffs (list|None: coeficientes en orden de numpy.polyval, o None).
Nunca lanza: ante fallo o datos insuficientes devuelve el dict debil.
"""
try:
if xs is None or ys is None:
return dict(_WEAK)
pairs = [
(float(x), float(y))
for x, y in zip(xs, ys)
if _is_num(x) and _is_num(y)
]
# Datos insuficientes para hablar de forma de la relacion.
if len(pairs) < 5:
return dict(_WEAK)
clean_x = [p[0] for p in pairs]
clean_y = [p[1] for p in pairs]
# Varianza ~0 en cualquiera de las series => relacion indefinida.
if len(set(clean_x)) < 2 or len(set(clean_y)) < 2:
return dict(_WEAK)
x_arr = np.asarray(clean_x, dtype=float)
y_arr = np.asarray(clean_y, dtype=float)
if float(np.var(x_arr)) < 1e-15 or float(np.var(y_arr)) < 1e-15:
return dict(_WEAK)
# Correlaciones reutilizando las funciones del registry.
r = pearson(clean_x, clean_y)
spearman = spearman_corr(clean_x, clean_y)
r2_linear = r ** 2
# Ajustes polinomicos grado 2 y 3 con R^2 manual.
ss_tot = float(np.sum((y_arr - float(np.mean(y_arr))) ** 2))
with warnings.catch_warnings():
warnings.simplefilter("ignore")
c1 = np.polyfit(x_arr, y_arr, 1)
c2 = np.polyfit(x_arr, y_arr, 2)
c3 = np.polyfit(x_arr, y_arr, 3)
r2_poly2 = _poly_r2(c2, x_arr, y_arr, ss_tot)
r2_poly3 = _poly_r2(c3, x_arr, y_arr, ss_tot)
mejor_poly = max(r2_poly2, r2_poly3)
# Grado del mejor polinomico, con preferencia por la parsimonia: solo se
# elige el grado 3 si supera al grado 2 por mas de 0.02.
best_poly_degree = 3 if (r2_poly3 - r2_poly2) > 0.02 else 2
abs_s = abs(spearman)
abs_p = abs(r)
# Decision en orden: debil-temprano -> monotona -> polinomica -> lineal.
if abs_p < 0.3 and abs_s < 0.3 and mejor_poly < 0.3:
# Ninguna senal supera el suelo de forma: relacion debil/sin forma.
tipo = "débil/sin forma"
best_degree = None
coeffs = None
elif (abs_s - abs_p) >= 0.1 and abs_s >= 0.4:
# Spearman (rango) capta una monotonia que el Pearson lineal no:
# relacion monotona no-lineal. No se fuerza un polinomio que tal vez
# no ajusta bien; el capitulo dibuja la tendencia ordenada.
tipo = "monótona no-lineal"
best_degree = None
coeffs = None
elif (mejor_poly - r2_linear) >= 0.1 and mejor_poly >= 0.3:
tipo = "polinómica (grado {})".format(best_poly_degree)
best_degree = best_poly_degree
best_coeffs = c2 if best_poly_degree == 2 else c3
coeffs = [float(c) for c in best_coeffs]
else:
# Hay senal (no es debil) y no es ni monotona-pura ni polinomica:
# la correlacion que existe es esencialmente lineal.
tipo = "lineal"
best_degree = 1
coeffs = [float(c) for c in c1]
return {
"tipo": tipo,
"pearson": round(float(r), 6),
"r2_linear": round(float(r2_linear), 6),
"spearman": round(float(spearman), 6),
"r2_poly2": round(float(r2_poly2), 6),
"r2_poly3": round(float(r2_poly3), 6),
"best_degree": best_degree,
"coeffs": (
[round(c, 8) for c in coeffs] if coeffs is not None else None
),
}
except Exception:
return dict(_WEAK)
@@ -0,0 +1,174 @@
"""Tests para classify_relationship_type."""
import os
import sys
import numpy as np
sys.path.insert(0, os.path.dirname(__file__))
from classify_relationship_type import classify_relationship_type
# Claves que el dict de salida debe contener SIEMPRE.
_EXPECTED_KEYS = {
"tipo", "pearson", "r2_linear", "spearman",
"r2_poly2", "r2_poly3", "best_degree", "coeffs",
}
def _assert_shape(r):
"""Toda salida tiene exactamente las 8 claves canonicas."""
assert isinstance(r, dict)
assert set(r.keys()) == _EXPECTED_KEYS
def test_lineal():
"""Golden: y = 2x + 1 con ruido pequeno -> 'lineal', best_degree=1."""
rng = np.random.default_rng(42)
x = np.linspace(0.0, 10.0, 50)
y = 2.0 * x + 1.0 + rng.normal(0.0, 0.3, 50)
r = classify_relationship_type(list(x), list(y))
_assert_shape(r)
assert r["tipo"] == "lineal"
assert r["best_degree"] == 1
assert r["r2_linear"] >= 0.5
# coeffs ~ [pendiente, intercepto] del ajuste de grado 1.
assert r["coeffs"] is not None and len(r["coeffs"]) == 2
assert abs(r["coeffs"][0] - 2.0) < 0.1 # pendiente ~2
assert abs(r["coeffs"][1] - 1.0) < 0.3 # intercepto ~1
def test_polinomica_cuadratica():
"""Golden: y = x**2 sobre [-10, 10] -> 'polinómica', best_degree in (2, 3)."""
x = np.linspace(-10.0, 10.0, 60)
y = x ** 2
r = classify_relationship_type(list(x), list(y))
_assert_shape(r)
assert r["tipo"].startswith("polinómica")
assert r["best_degree"] in (2, 3)
# Una parabola perfecta queda capturada por el grado 2 (parsimonia).
assert r["best_degree"] == 2
assert r["r2_poly2"] > 0.99
assert r["coeffs"] is not None and len(r["coeffs"]) == r["best_degree"] + 1
def test_monotona_no_lineal():
"""Golden: monotona convexa de cola pesada -> 'monótona no-lineal'.
y = 1/(N+1-i)**2 es estrictamente creciente (Spearman ~ 1) pero su cola
explosiva hace que ni la recta ni un polinomio de grado 2/3 la ajusten
(R^2 polinomico < 0.5), de modo que el Pearson lineal NO capta la relacion
que el rango (Spearman) si ve. Construccion deterministica (sin azar).
"""
n = 200
i = np.arange(n, dtype=float)
y = 1.0 / (n + 1 - i) ** 2
r = classify_relationship_type(list(i), list(y))
_assert_shape(r)
assert r["tipo"] == "monótona no-lineal"
assert r["best_degree"] is None
assert r["coeffs"] is None
# Spearman fuerte y claramente por encima del Pearson.
assert abs(r["spearman"]) >= 0.5
assert abs(r["spearman"]) - abs(r["pearson"]) >= 0.15
def test_monotona_exponencial():
"""DoD literal: y = exp(x) (monotona no-lineal) -> 'monótona no-lineal'.
exp es estrictamente creciente (Spearman = 1) pero el Pearson lineal queda
claramente por debajo (~0.86), así que la dominancia del rango la marca como
monótona no-lineal en vez de lineal o polinómica.
"""
x = np.linspace(0.0, 5.0, 80)
y = np.exp(x)
r = classify_relationship_type(list(x), list(y))
_assert_shape(r)
assert r["tipo"] == "monótona no-lineal"
assert r["best_degree"] is None and r["coeffs"] is None
assert abs(r["spearman"]) >= 0.9
assert abs(r["spearman"]) - abs(r["pearson"]) >= 0.1
def test_debil_sin_forma():
"""Golden: x e y independientes (semilla fija) -> 'débil/sin forma'."""
rng = np.random.default_rng(0)
x = rng.normal(0.0, 1.0, 200)
y = rng.normal(0.0, 1.0, 200)
r = classify_relationship_type(list(x), list(y))
_assert_shape(r)
assert r["tipo"] == "débil/sin forma"
assert r["best_degree"] is None
assert r["coeffs"] is None
# Todas las senales son bajas.
assert abs(r["pearson"]) < 0.3
assert r["r2_linear"] < 0.1
def test_lista_vacia_no_lanza():
"""Edge: listas vacias -> dict debil canonico, sin lanzar."""
r = classify_relationship_type([], [])
_assert_shape(r)
assert r["tipo"] == "débil/sin forma"
assert r["pearson"] is None
assert r["r2_linear"] is None
assert r["spearman"] is None
assert r["r2_poly2"] is None
assert r["r2_poly3"] is None
assert r["best_degree"] is None
assert r["coeffs"] is None
def test_longitudes_distintas_no_lanza():
"""Edge: listas de distinta longitud -> empareja por indice, no lanza."""
# zip trunca a la longitud minima: solo 3 pares (< 5) -> debil.
r = classify_relationship_type([1, 2, 3, 4, 5, 6, 7, 8], [1.0, 2.0, 3.0])
_assert_shape(r)
assert r["tipo"] == "débil/sin forma"
assert r["best_degree"] is None
def test_todos_none_no_lanza():
"""Edge: todos los valores None -> ningun par valido -> debil, no lanza."""
r = classify_relationship_type([None, None, None, None, None, None],
[None, None, None, None, None, None])
_assert_shape(r)
assert r["tipo"] == "débil/sin forma"
assert r["coeffs"] is None
def test_entradas_none_no_lanza():
"""Edge: xs/ys None directamente -> debil, no lanza."""
assert classify_relationship_type(None, None)["tipo"] == "débil/sin forma"
assert classify_relationship_type([1.0, 2.0], None)["tipo"] == "débil/sin forma"
def test_constante_no_lanza():
"""Edge: ys constante (varianza ~0) -> debil, no lanza."""
r = classify_relationship_type([1, 2, 3, 4, 5, 6, 7], [5, 5, 5, 5, 5, 5, 5])
_assert_shape(r)
assert r["tipo"] == "débil/sin forma"
def test_filtra_nan_inf_bool():
"""Edge: pares con NaN/inf/bool/None se descartan por indice."""
nan = float("nan")
inf = float("inf")
# Solo i=0,1,2,3,4 quedan validos (5 pares) y forman una recta perfecta.
xs = [0.0, 1.0, 2.0, 3.0, 4.0, nan, inf, True, None]
ys = [1.0, 3.0, 5.0, 7.0, 9.0, 1.0, 2.0, 3.0, 4.0]
r = classify_relationship_type(xs, ys)
_assert_shape(r)
# Los 5 pares validos son y = 2x + 1 exacto -> lineal.
assert r["tipo"] == "lineal"
assert r["best_degree"] == 1
@@ -4,10 +4,10 @@ name: column_quality_score
kind: function
lang: py
domain: datascience
version: "1.0.0"
version: "2.0.0"
purity: pure
signature: "def column_quality_score(col: dict) -> dict"
description: "Calcula un score de calidad de datos 0-100 para un ColumnProfile del grupo eda, con desglose completeness/validity/consistency y lista de issues legibles. Funcion pura, no muta el input."
description: "Calcula un score de calidad de datos 0-100 para un ColumnProfile del grupo eda. Combina completeness (0.6) y validity (0.4) con renormalizacion por aplicabilidad; los outliers, columnas constantes e ids NO bajan el score (van a observations). Devuelve desglose por dimension, issues (defectos) y observations (señales analiticas). Funcion pura, no muta el input."
tags: [eda, data-quality, profiling, scoring, datascience]
uses_functions: []
uses_types: []
@@ -17,20 +17,26 @@ error_type: ""
imports: []
example: |
from datascience import column_quality_score
col = {"name": "precio", "inferred_type": "float", "null_pct": 0.2,
"unique_pct": 0.4, "flags": [], "numeric": {"outlier_pct": 0.08}}
col = {"name": "precio", "inferred_type": "numeric", "null_pct": 0.2,
"unique_pct": 0.4, "flags": [], "numeric": {"outlier_pct": 8.0}}
column_quality_score(col)
# {"score": 86.8, "completeness": 0.8, "validity": 0.92,
# "consistency": 1.0, "issues": ["20% nulos", "8% outliers"]}
# {"score": 88.0, "completeness": 0.8, "validity": 1.0,
# "applicable": ["completeness", "validity"], "issues": ["20% nulos"],
# "observations": ["8% de valores atípicos (z-score>3): ..."]}
tested: true
tests:
- "test_clean_column_high_score"
- "test_half_null_lowers_completeness_and_score"
- "test_constant_column_flags_issue"
- "test_weights_60_40_native_type"
- "test_outliers_do_not_penalize_score"
- "test_nulls_lower_score_more_than_outliers"
- "test_validity_from_parse_rate_lowers_score"
- "test_validity_from_match_rate"
- "test_free_text_renormalizes_to_completeness_only"
- "test_all_null_column_scores_zero"
- "test_constant_column_scores_full_and_is_observation"
- "test_high_cardinality_id_scores_full_and_is_observation"
- "test_mostly_null_no_double_counts_validity"
- "test_empty_dict_does_not_crash"
- "test_outliers_penalize_validity"
- "test_mostly_null_flag_halves_validity"
- "test_high_cardinality_text_flagged_as_id"
- "test_none_values_treated_defensively"
- "test_does_not_mutate_input"
test_file_path: "python/functions/datascience/column_quality_score_test.py"
@@ -38,16 +44,22 @@ file_path: "python/functions/datascience/column_quality_score.py"
params:
- name: col
desc: >
ColumnProfile dict del grupo eda (p.ej. salida de summarize_table_duckdb).
Se leen sus claves de forma defensiva con .get(...) y se toleran valores
None. Claves usadas: null_pct (0-1), inferred_type, semantic_type,
unique_pct (0-1), flags (list[str], reconoce "constant"/"mostly_null"),
numeric ({outlier_pct: 0-1, ...}|None) y match_rate (opcional, 0-1).
ColumnProfile dict del grupo eda (p.ej. salida de summarize_table_duckdb /
profile_table). Se leen sus claves de forma defensiva con .get(...) y se
toleran valores None. Claves usadas: null_pct (0-1), n_rows, empty_count
(texto), inferred_type, semantic_type, validity_rate (0-1, lo expone
profile_table al promocionar texto a numero/fecha), match_rate (0-1),
unique_pct (0-1), flags (list[str], reconoce
"constant"/"possible_id"/"high_cardinality") y numeric ({outlier_pct: 0-100,
skew, ...}|None).
output: >
dict con score (float 0-100, redondeado a 1 decimal), completeness (0-1),
validity (0-1), consistency (0-1) e issues (list[str] de descripciones
legibles de los problemas detectados). score = round(100 * (0.5*completeness
+ 0.3*validity + 0.2*consistency), 1).
dict con score (float 0-100, 1 decimal), completeness (0-1), validity (0-1 o
None si no aplicable), dimensions ({completeness, validity}), applicable
(list[str] de dimensiones que entraron en el score), issues (list[str] SOLO de
defectos de calidad: nulos, vacios, valores no conformes) y observations
(list[str] de señales analiticas que NO bajan el score: outliers, columna
constante, posible id, asimetria). score = round(100 * (0.6*completeness +
0.4*validity) / pesos_aplicables, 1), renormalizado cuando validity no aplica.
---
## Ejemplo
@@ -59,51 +71,71 @@ from datascience import column_quality_score
col = {
"name": "precio",
"physical_type": "DOUBLE",
"inferred_type": "float",
"inferred_type": "numeric",
"semantic_type": "",
"count": 800,
"n_rows": 1000,
"null_count": 200,
"null_pct": 0.20,
"distinct_count": 400,
"unique_pct": 0.40,
"flags": [],
"numeric": {"outlier_pct": 0.08},
"numeric": {"outlier_pct": 8.0, "skew": 0.3},
"categorical": None,
"datetime": None,
}
column_quality_score(col)
# {
# "score": 86.8,
# "completeness": 0.8, # 1 - 0.20
# "validity": 0.92, # 1 - min(0.08, 0.3)
# "consistency": 1.0,
# "issues": ["20% nulos", "8% outliers"],
# "score": 88.0, # 100 * (0.6*0.8 + 0.4*1.0)
# "completeness": 0.8, # 1 - 0.20
# "validity": 1.0, # numerica nativa: el tipo es conforme
# "dimensions": {"completeness": 0.8, "validity": 1.0},
# "applicable": ["completeness", "validity"],
# "issues": ["20% nulos"], # SOLO defectos de calidad
# "observations": ["8% de valores atípicos (z-score>3): ..."], # NO bajan score
# }
```
## Cuando usarla
Cuando hayas perfilado una tabla con el grupo `eda` (p.ej.
`summarize_table_duckdb`) y necesites un numero 0-100 por columna para
ordenar/priorizar limpieza de datos, pintar semaforos de calidad en un
dashboard, o decidir que columnas descartar antes de modelar. Es la capa de
scoring sobre el ColumnProfile crudo: lee el perfil, no toca los datos.
`summarize_table_duckdb` / `profile_table`) y necesites un numero 0-100 por
columna para ordenar/priorizar limpieza de datos, pintar semaforos de calidad,
o decidir que columnas descartar antes de modelar. Separa los **defectos de
calidad reales** (`issues`: nulos, vacios, valores que no parsean a su tipo) de
las **observaciones analiticas** (`observations`: outliers, columnas constantes,
ids), que se reportan pero no penalizan. Es la capa de scoring sobre el
ColumnProfile crudo: lee el perfil, no toca los datos.
## Notas
## Gotchas
Funcion pura, sin I/O ni dependencias externas, no muta `col`. Lee todas las
claves con `.get(...)` y tolera que vengan en `None` (un ColumnProfile recien
salido de `summarize_table_duckdb` trae muchas claves a `None`), por lo que
nunca falla por claves ausentes — un `{}` produce un resultado bien definido.
Funcion pura, sin I/O, no muta `col`. Aun asi conviene saber:
Pesos del score: completeness 0.5, validity 0.3, consistency 0.2.
- **Los outliers NO bajan el score.** Un valor extremo puede ser real y correcto
(un cliente que compra mucho); detectar atipicos es analisis de la
distribucion, no un juicio de correccion. Salen en `observations`, no en
`issues`. Mismo trato para columnas constantes e identificadores de alta
cardinalidad: son observaciones, no defectos.
- **`validity` puede ser `None`** (no aplicable): texto libre sin `semantic_type`
ni `validity_rate`, o columna 100% nula. En ese caso el score se renormaliza a
solo `completeness` (la columna no se premia ni castiga por algo no medible).
- **`outlier_pct` se interpreta en escala 0-100** (la que emite
`describe_numeric`, z-score>3). Pasar una fraccion 0-1 produce un texto de
observacion con el % equivocado, pero NUNCA afecta al score.
- **`validity_rate` lo puebla `profile_table`** al promocionar una columna de
texto a numero/fecha (fraccion que parsea). Si no esta presente y el tipo es
nativo numerico/fecha/bool, `validity = 1.0`.
- Sin doble conteo: la falta de datos cuenta solo en `completeness` (el antiguo
castigo de `mostly_null` sobre `validity` se elimino).
- **completeness** = `1 - null_pct` (None -> 0 nulls -> 1.0).
- **validity**: parte de 1.0 y penaliza `min(outlier_pct, 0.3)` en columnas
numericas, `0.5 * (1 - match_rate)` si hay `semantic_type` declarado con
`match_rate` bajo disponible, y multiplica por 0.5 si el flag `mostly_null`
esta presente.
- **consistency**: 1.0 salvo flag `constant` (-> 0.3, columna poco informativa)
o texto con `unique_pct > 0.9` (-> 0.6, posible id de alta cardinalidad).
## Capability growth log
- v2.0.0 (2026-06-30) — nueva formula de calidad (report 2046): pesos 60/40
(completeness/validity) con renormalizacion por aplicabilidad; se elimina la
dimension `consistency`-como-informatividad y el doble castigo de
`mostly_null`; los outliers/constantes/ids salen del score a `observations`;
validity mide conformidad real (parse rate / match rate / tipo nativo). Salida
ampliada con `dimensions`, `applicable` y `observations`.
- v1.0.0 — version inicial: pesos 50/30/20 (completeness/validity/consistency),
los outliers penalizaban validity (con bug de escala) y consistency penalizaba
informatividad.
@@ -1,34 +1,78 @@
"""Score de calidad de datos (0-100) para un ColumnProfile del grupo eda.
Funcion pura: dado el perfil de una columna producido por el grupo de
capacidad `eda` (p.ej. summarize_table_duckdb), calcula un score agregado
de calidad junto a su desglose en completeness / validity / consistency y
una lista de issues legibles. No realiza I/O ni muta el input.
capacidad `eda` (p.ej. summarize_table_duckdb / profile_table), calcula un
score agregado de calidad junto a su desglose por dimension y dos listas
legibles separadas: `issues` (defectos de calidad reales que SI bajan el
score) y `observations` (señales analiticas que NO bajan el score). No
realiza I/O ni muta el input.
Modelo (DAMA-DMBOK / ISO 8000), ver report 2046:
- Solo entran en el score las dimensiones medibles automaticamente desde el
perfil, sin fuente externa de verdad: completeness y validity por columna.
- Renormalizacion por aplicabilidad: si una dimension no es medible en la
columna (texto libre sin semantica -> validity no aplica; columna 100% nula
-> validity no medible), se excluye y los pesos se renormalizan sobre las
aplicables. Una columna ni se premia ni se castiga por algo no medible.
- Sin doble conteo: la falta de datos cuenta solo en completeness (se elimino
el antiguo castigo extra de `mostly_null` sobre validity).
- Los OUTLIERS NO bajan la calidad. Un valor extremo puede ser real y
correcto; detectar atipicos es analisis de la distribucion, no un juicio de
coreccion. Outliers, columnas constantes e identificadores de alta
cardinalidad pasan a `observations`, nunca a `issues`.
"""
# Pesos base de las dimensiones de columna (se renormalizan por aplicabilidad).
_W_COMPLETENESS = 0.6
_W_VALIDITY = 0.4
# Tipos inferidos cuyo almacen garantiza la conformidad de tipo (validity=1.0)
# cuando NO vienen de una promocion de texto (en cuyo caso manda validity_rate).
_NATIVE_TYPED = ("numeric", "integer", "float", "datetime", "date", "boolean", "bool")
def column_quality_score(col: dict) -> dict:
"""Calcula un score de calidad de datos 0-100 para un ColumnProfile.
El score pondera tres dimensiones:
- completeness (0.5): proporcion de valores no nulos.
- validity (0.3): ausencia de outliers / heuristicas de validez.
- consistency (0.2): la columna aporta informacion (no constante, no ruido).
El score combina solo dimensiones de calidad medibles desde el perfil, con
renormalizacion por aplicabilidad:
- completeness (peso base 0.6, siempre aplica): proporcion de valores
presentes = 1 - null_pct. En texto, las celdas vacias (`empty_count`)
tambien cuentan como faltantes.
- validity (peso base 0.4, cuando hay un criterio de validacion real):
fraccion de valores no nulos conformes a su tipo/semantica. Tipo nativo
numerico/fecha/bool = 1.0; texto promovido a numero/fecha = parse rate
(`validity_rate`); texto con `semantic_type` regexable = `match_rate`;
texto libre o columna 100% nula = NO aplicable (renormaliza a solo
completeness).
Los outliers, columnas constantes, identificadores y asimetria fuerte NO
bajan el score: se devuelven en `observations`.
Args:
col: ColumnProfile dict del grupo eda. Se leen las claves de forma
defensiva con .get(...) y se tolera que muchas vengan en None.
Claves relevantes: null_pct, inferred_type, semantic_type,
unique_pct, flags (list[str]), numeric ({outlier_pct, ...}|None),
match_rate (opcional).
Claves relevantes: null_pct (0-1), n_rows, empty_count,
inferred_type, semantic_type, validity_rate (0-1, lo expone
profile_table al promocionar texto a numero/fecha), match_rate
(0-1), unique_pct (0-1), flags (list[str], reconoce
"constant"/"possible_id"/"high_cardinality"), numeric
({outlier_pct: 0-100, skew, ...}|None).
Returns:
dict con:
score (float, 0-100, redondeado a 1 decimal),
completeness (float, 0-1),
validity (float, 0-1),
consistency (float, 0-1),
issues (list[str]) descripciones legibles de los problemas.
score (float 0-100, redondeado a 1 decimal),
completeness (float 0-1),
validity (float 0-1 | None si no aplicable),
dimensions ({completeness, validity}),
applicable (list[str] de dimensiones que entraron en el score),
issues (list[str]) SOLO defectos de calidad (nulos, vacios,
valores no conformes a su tipo/semantica),
observations (list[str]) señales analiticas que NO bajan el score
(outliers, columna constante, posible id, asimetria).
"""
if not isinstance(col, dict):
col = {}
@@ -39,103 +83,153 @@ def column_quality_score(col: dict) -> dict:
flags = set(flags)
issues: list[str] = []
observations: list[str] = []
inferred_type = col.get("inferred_type") or ""
semantic_type = col.get("semantic_type") or ""
# --- completeness -------------------------------------------------
null_pct = col.get("null_pct")
if null_pct is None:
null_pct = 0.0
try:
null_pct = float(null_pct)
except (TypeError, ValueError):
null_pct = 0.0
null_pct = _clamp(null_pct, 0.0, 1.0)
# Falta de datos = nulos + (en texto) celdas vacias. Es el unico sitio
# donde la falta de datos cuenta: nunca se duplica en validity.
null_pct = _clamp(_num(col.get("null_pct"), 0.0), 0.0, 1.0)
completeness = 1.0 - null_pct
if null_pct > 0:
issues.append(f"{round(null_pct * 100)}% nulos")
issues.append(f"{_pct(null_pct)} nulos")
# --- validity -----------------------------------------------------
validity = 1.0
inferred_type = col.get("inferred_type") or ""
empty_frac = 0.0
n_rows = col.get("n_rows")
empty_count = col.get("empty_count")
if (
isinstance(n_rows, (int, float)) and not isinstance(n_rows, bool) and n_rows > 0
and isinstance(empty_count, (int, float)) and not isinstance(empty_count, bool)
and empty_count > 0
):
empty_frac = _clamp(float(empty_count) / float(n_rows), 0.0, 1.0)
completeness = _clamp(completeness - empty_frac, 0.0, 1.0)
issues.append(f"{_pct(empty_frac)} vacíos")
numeric = col.get("numeric")
is_numeric = inferred_type in ("integer", "float", "numeric") or isinstance(numeric, dict)
if isinstance(numeric, dict):
outlier_pct = numeric.get("outlier_pct")
if outlier_pct is not None:
try:
outlier_pct = float(outlier_pct)
except (TypeError, ValueError):
outlier_pct = 0.0
outlier_pct = _clamp(outlier_pct, 0.0, 1.0)
if outlier_pct > 0:
penalty = min(outlier_pct, 0.3)
validity -= penalty
issues.append(f"{round(outlier_pct * 100)}% outliers")
# semantic_type declarado pero con baja tasa de match (si la conocemos).
semantic_type = col.get("semantic_type") or ""
match_rate = col.get("match_rate")
if semantic_type and match_rate is not None:
try:
match_rate = float(match_rate)
except (TypeError, ValueError):
match_rate = None
if match_rate is not None:
match_rate = _clamp(match_rate, 0.0, 1.0)
if match_rate < 1.0:
shortfall = 1.0 - match_rate
validity -= 0.5 * shortfall
issues.append(
f"semantic_type '{semantic_type}' con baja coincidencia "
f"({round(match_rate * 100)}%)"
)
if "mostly_null" in flags:
validity *= 0.5
issues.append("mayoritariamente nula")
validity = _clamp(validity, 0.0, 1.0)
# --- consistency --------------------------------------------------
consistency = 1.0
if "constant" in flags:
consistency = 0.3
issues.append("columna constante")
# --- validity (con renormalizacion por aplicabilidad) -------------
# None = no medible -> se excluye del score (no penaliza ni premia).
validity = None
if completeness <= 0.0:
# Columna 100% faltante: no hay valores no nulos sobre los que medir
# conformidad. validity no aplica -> el score sale solo de completeness
# (= 0). Es el peor defecto de calidad posible.
validity = None
else:
unique_pct = col.get("unique_pct")
if unique_pct is not None:
try:
unique_pct = float(unique_pct)
except (TypeError, ValueError):
unique_pct = None
if (
inferred_type == "text"
validity_rate = col.get("validity_rate")
match_rate = col.get("match_rate")
if validity_rate is not None:
# Texto promovido a numero/fecha: parse rate real de la muestra.
v = _num(validity_rate, None)
if v is not None:
validity = _clamp(v, 0.0, 1.0)
if validity < 1.0:
kind = (
"número" if inferred_type == "numeric"
else "fecha" if inferred_type == "datetime"
else inferred_type or "su tipo"
)
issues.append(
f"{_pct(1.0 - validity)} no parsea al tipo {kind}"
)
elif inferred_type in _NATIVE_TYPED:
# Tipo nativo garantizado por el almacen: no hay valores que no
# parseen. validity = 1.0 (no se confunde con tener outliers).
validity = 1.0
elif semantic_type and match_rate is not None:
v = _num(match_rate, None)
if v is not None:
validity = _clamp(v, 0.0, 1.0)
if validity < 1.0:
issues.append(
f"{_pct(1.0 - validity)} no casa con el "
f"formato «{semantic_type}»"
)
else:
# Texto libre / categorica sin semantica: no hay criterio honesto
# de validez. No aplica.
validity = None
# --- observations (NO bajan el score) -----------------------------
numeric = col.get("numeric")
if isinstance(numeric, dict):
# outlier_pct viene en escala 0-100 desde describe_numeric (z-score>3).
outlier_pct = _num(numeric.get("outlier_pct"), None)
if outlier_pct is not None and outlier_pct >= 0.05:
observations.append(
f"{_pct(outlier_pct / 100.0)} de valores atípicos (z-score>3): "
"revisar si son errores u observaciones legítimas"
)
skew = _num(numeric.get("skew"), None)
if skew is not None and abs(skew) >= 1.0:
observations.append(
f"asimetría fuerte (skew={round(skew, 2)}): considerar "
"re-expresión antes de modelar"
)
if "constant" in flags:
observations.append(
"columna constante: aporta poca información para el análisis"
)
unique_pct = _num(col.get("unique_pct"), None)
is_id = (
"possible_id" in flags
or "high_cardinality" in flags
or (
inferred_type in ("text", "categorical")
and unique_pct is not None
and _clamp(unique_pct, 0.0, 1.0) > 0.9
):
consistency = 0.6
issues.append("posible id de alta cardinalidad")
consistency = _clamp(consistency, 0.0, 1.0)
# --- score agregado ----------------------------------------------
score = round(
100.0 * (0.5 * completeness + 0.3 * validity + 0.2 * consistency),
1,
)
)
if is_id:
observations.append(
"valores casi únicos: posible identificador (no es un defecto de calidad)"
)
# Silencia warnings sobre la variable de tipo no usada.
_ = is_numeric
# --- score agregado con renormalizacion ---------------------------
applicable = ["completeness"]
num = _W_COMPLETENESS * completeness
den = _W_COMPLETENESS
if validity is not None:
applicable.append("validity")
num += _W_VALIDITY * validity
den += _W_VALIDITY
score = round(100.0 * num / den, 1) if den > 0 else 0.0
return {
"score": score,
"completeness": completeness,
"validity": validity,
"consistency": consistency,
"dimensions": {"completeness": completeness, "validity": validity},
"applicable": applicable,
"issues": issues,
"observations": observations,
}
def _pct(frac: float) -> str:
"""Formatea una fraccion 0-1 como porcentaje honesto: «N%» si >=1%, «0.N%»
por debajo (para no mostrar «0%» cuando hay un defecto real pequeño)."""
p = frac * 100.0
if p >= 1.0:
return f"{round(p)}%"
return f"{p:.1f}%"
def _num(x, default):
"""Convierte x a float; devuelve `default` si es None o no parseable."""
if x is None:
return default
if isinstance(x, bool):
return default
try:
return float(x)
except (TypeError, ValueError):
return default
def _clamp(x: float, lo: float, hi: float) -> float:
"""Recorta x al rango [lo, hi]."""
if x < lo:
@@ -1,4 +1,12 @@
"""Tests para column_quality_score."""
"""Tests para column_quality_score (nueva fórmula, report 2046).
Verifica las invariantes de la fórmula de calidad:
- completeness (0.6) + validity (0.4) con renormalización por aplicabilidad.
- Los OUTLIERS no bajan el score (van a observations, no a issues).
- Columnas constantes e ids no bajan el score (observations).
- Sin doble conteo de la falta de datos.
- all-null -> score 0; función pura (no muta el input).
"""
import os
import sys
@@ -9,11 +17,11 @@ from column_quality_score import column_quality_score
def _clean_numeric_col() -> dict:
"""ColumnProfile de una columna numerica sana, sin problemas."""
"""ColumnProfile de una columna numérica nativa sana, sin problemas."""
return {
"name": "edad",
"physical_type": "INTEGER",
"inferred_type": "integer",
"inferred_type": "numeric",
"semantic_type": "",
"count": 1000,
"n_rows": 1000,
@@ -28,85 +36,163 @@ def _clean_numeric_col() -> dict:
}
# --------------------------------------------------------------------------- #
# Golden
# --------------------------------------------------------------------------- #
def test_clean_column_high_score():
out = column_quality_score(_clean_numeric_col())
assert out["score"] > 90
assert out["score"] == 100.0
assert out["completeness"] == 1.0
assert out["validity"] == 1.0
assert out["consistency"] == 1.0
assert out["applicable"] == ["completeness", "validity"]
assert out["issues"] == []
assert out["observations"] == []
def test_half_null_lowers_completeness_and_score():
def test_weights_60_40_native_type():
"""30% nulos en numérica nativa: score = 100*(0.6*0.7 + 0.4*1.0) = 82."""
col = _clean_numeric_col()
col["null_count"] = 500
col["null_pct"] = 0.5
clean_score = column_quality_score(_clean_numeric_col())["score"]
col["null_pct"] = 0.30
col["null_count"] = 300
out = column_quality_score(col)
assert out["completeness"] == 0.5
assert out["score"] < clean_score
assert any("nulos" in issue for issue in out["issues"])
assert out["completeness"] == 0.7
assert out["validity"] == 1.0
assert out["score"] == 82.0
assert any("nulos" in i for i in out["issues"])
def test_constant_column_flags_issue():
# --------------------------------------------------------------------------- #
# Outliers FUERA del score
# --------------------------------------------------------------------------- #
def test_outliers_do_not_penalize_score():
"""Columna con outliers pero sin nulos -> score máximo; outliers en observations."""
col = _clean_numeric_col()
col["numeric"] = {"outlier_pct": 18.0, "skew": 0.2} # 18% atípicos (escala 0-100)
out = column_quality_score(col)
assert out["score"] == 100.0 # los outliers NO bajan la calidad
assert out["validity"] == 1.0
# No aparecen como problema de calidad...
assert not any("atípic" in i or "outlier" in i for i in out["issues"])
# ...sino como observación analítica.
assert any("atípic" in o for o in out["observations"])
def test_nulls_lower_score_more_than_outliers():
"""Vacíos sí penalizan; outliers no: comparar las dos columnas."""
con_nulos = _clean_numeric_col()
con_nulos["null_pct"] = 0.30
con_outliers = _clean_numeric_col()
con_outliers["numeric"] = {"outlier_pct": 30.0}
assert column_quality_score(con_nulos)["score"] < \
column_quality_score(con_outliers)["score"]
# --------------------------------------------------------------------------- #
# Validity: aplicabilidad y renormalización
# --------------------------------------------------------------------------- #
def test_validity_from_parse_rate_lowers_score():
"""Numérica como texto con 20% basura: validity=0.8 -> score=92."""
col = {
"name": "precio_txt", "inferred_type": "numeric", "semantic_type": "decimal",
"null_pct": 0.0, "validity_rate": 0.80, "flags": [], "numeric": None,
}
out = column_quality_score(col)
assert out["validity"] == 0.8
assert out["score"] == 92.0 # 100*(0.6 + 0.4*0.8)
assert any("no parsea" in i for i in out["issues"])
def test_validity_from_match_rate():
"""Texto con semantic_type y 5% no conforme: validity=0.95."""
col = {
"name": "email", "inferred_type": "text", "semantic_type": "email",
"null_pct": 0.0, "match_rate": 0.95, "unique_pct": 0.5, "flags": [],
}
out = column_quality_score(col)
assert out["validity"] == 0.95
assert out["score"] == 98.0 # 100*(0.6 + 0.4*0.95)
assert any("no casa" in i for i in out["issues"])
def test_free_text_renormalizes_to_completeness_only():
"""Texto libre sin semántica: validity no aplica -> score = 100*completeness."""
col = {
"name": "comentario", "inferred_type": "text", "semantic_type": "",
"null_pct": 0.30, "unique_pct": 0.5, "flags": [], "numeric": None,
}
out = column_quality_score(col)
assert out["validity"] is None
assert out["applicable"] == ["completeness"]
assert out["completeness"] == 0.7
assert out["score"] == 70.0 # renormalizado a solo completeness
# --------------------------------------------------------------------------- #
# Casos límite (report §4.6)
# --------------------------------------------------------------------------- #
def test_all_null_column_scores_zero():
col = _clean_numeric_col()
col["null_pct"] = 1.0
col["null_count"] = 1000
out = column_quality_score(col)
assert out["completeness"] == 0.0
assert out["validity"] is None # no medible sin valores no nulos
assert out["score"] == 0.0
def test_constant_column_scores_full_and_is_observation():
"""Columna constante: dato válido y completo -> score 100; baja info = observación."""
col = _clean_numeric_col()
col["flags"] = ["constant"]
col["distinct_count"] = 1
col["unique_pct"] = 0.001
out = column_quality_score(col)
assert out["consistency"] == 0.3
assert any("constante" in issue for issue in out["issues"])
assert out["score"] == 100.0 # NO se castiga la baja informatividad
assert not any("constante" in i for i in out["issues"])
assert any("constante" in o for o in out["observations"])
def test_high_cardinality_id_scores_full_and_is_observation():
"""Id de alta cardinalidad: unicidad perfecta -> score 100; posible id = observación."""
col = {
"name": "uuid", "inferred_type": "text", "semantic_type": "",
"null_pct": 0.0, "unique_pct": 0.99, "flags": ["possible_id"],
"numeric": None,
}
out = column_quality_score(col)
assert out["score"] == 100.0
assert not any("identificador" in i for i in out["issues"])
assert any("identificador" in o for o in out["observations"])
def test_mostly_null_no_double_counts_validity():
"""85% nulos: solo completeness penaliza; validity nativa sigue 1.0 (sin doble castigo)."""
col = _clean_numeric_col()
col["null_pct"] = 0.85
col["flags"] = ["mostly_null"]
out = column_quality_score(col)
assert out["validity"] == 1.0 # ya no se multiplica por 0.5
# score = 100*(0.6*0.15 + 0.4*1.0) = 49
assert out["score"] == 49.0
assert not any("mayoritariamente" in o for o in out["observations"])
# --------------------------------------------------------------------------- #
# Robustez
# --------------------------------------------------------------------------- #
def test_empty_dict_does_not_crash():
out = column_quality_score({})
assert isinstance(out["score"], float)
assert out["completeness"] == 1.0
assert 0.0 <= out["score"] <= 100.0
assert isinstance(out["issues"], list)
def test_outliers_penalize_validity():
col = _clean_numeric_col()
col["numeric"] = {"outlier_pct": 0.2}
out = column_quality_score(col)
assert out["validity"] < 1.0
assert any("outliers" in issue for issue in out["issues"])
def test_mostly_null_flag_halves_validity():
col = _clean_numeric_col()
col["null_pct"] = 0.85
col["flags"] = ["mostly_null"]
out = column_quality_score(col)
assert out["validity"] == 0.5
assert any("mayoritariamente nula" in issue for issue in out["issues"])
def test_high_cardinality_text_flagged_as_id():
col = {
"name": "uuid",
"inferred_type": "text",
"semantic_type": "",
"null_pct": 0.0,
"unique_pct": 0.99,
"flags": [],
"numeric": None,
}
out = column_quality_score(col)
assert out["consistency"] < 1.0
assert any("alta cardinalidad" in issue for issue in out["issues"])
assert isinstance(out["observations"], list)
def test_none_values_treated_defensively():
col = {
"name": "x",
"inferred_type": None,
"semantic_type": None,
"null_pct": None,
"unique_pct": None,
"flags": None,
"numeric": None,
"name": "x", "inferred_type": None, "semantic_type": None,
"null_pct": None, "unique_pct": None, "flags": None, "numeric": None,
}
out = column_quality_score(col)
assert out["completeness"] == 1.0
@@ -0,0 +1,102 @@
---
id: compute_text_duplicates_py_datascience
name: compute_text_duplicates
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def compute_text_duplicates(texts, near_threshold=0.85, sample_max=2000) -> dict"
description: "Detecta documentos duplicados en un corpus de texto. Los duplicados EXACTOS se calculan siempre con la stdlib: cada documento se normaliza (colapsa espacios, strip, lower) y se hashea con SHA-1; n_exact_dup es cuántos docs repiten uno ya visto y exact_dup_pct su porcentaje. Los CASI-duplicados (near-dup) usan la dependencia OPCIONAL datasketch (MinHash + LSH sobre 3-shingles de palabras); si no está instalada, esa parte degrada a available:False sin afectar al resto. Estilo dict-no-throw del grupo eda — nunca lanza."
tags: [eda, datascience, text, nlp, duplicates, minhash, pure, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [hashlib, re]
example: |
from datascience.compute_text_duplicates import compute_text_duplicates
texts = ["El gato come pescado", "El gato come pescado", "Un perro ladra"]
result = compute_text_duplicates(texts)
# {"n_docs": 3, "n_exact_dup": 1, "exact_dup_pct": 33.33, "n_unique": 2,
# "near_dup": {"available": False, "n_near_dup_docs": 0}}
tested: true
tests:
- "test_duplicados_exactos"
- "test_sin_duplicados"
- "test_vacio"
- "test_near_dup_degrada"
test_file_path: "python/functions/datascience/compute_text_duplicates_test.py"
file_path: "python/functions/datascience/compute_text_duplicates.py"
params:
- name: texts
desc: "Lista de documentos de texto. Los elementos None o que no sean str se descartan silenciosamente; n_docs cuenta solo los documentos válidos. None como argumento se trata como lista vacía."
- name: near_threshold
desc: "Umbral de similitud Jaccard (01) para considerar dos documentos casi-duplicados en el cálculo near-dup vía MinHashLSH. Solo aplica si datasketch está instalada. Default 0.85."
- name: sample_max
desc: "Número máximo de documentos muestreados (los primeros) para el cálculo near-dup, que es O(n) en memoria de MinHashes. No afecta al conteo de duplicados exactos, que siempre recorre todo el corpus. Default 2000."
output: "Dict con exactamente 5 claves, siempre presentes: n_docs (int, docs válidos), n_exact_dup (int, docs que repiten un texto normalizado ya visto = n_docs - n_unique), exact_dup_pct (float a 2 decimales = n_exact_dup/n_docs*100, o None si el corpus está vacío), n_unique (int, nº de textos normalizados distintos), y near_dup (sub-dict con available:bool y n_near_dup_docs:int; cuando available es True incluye además threshold con el near_threshold usado). La función nunca lanza: captura toda excepción y degrada."
---
## Ejemplo
```python
from datascience.compute_text_duplicates import compute_text_duplicates
# Tres copias del mismo texto (con espacios/casing distintos) + dos únicos.
texts = [
"El gato come pescado",
"El gato come pescado",
"el GATO come pescado", # mismo tras normalizar
"Un perro ladra",
"La luna brilla",
]
compute_text_duplicates(texts)
# {
# "n_docs": 5,
# "n_exact_dup": 2, # 3 copias del primer texto => 2 repeticiones
# "exact_dup_pct": 40.0, # 2 / 5 * 100
# "n_unique": 3, # 3 textos normalizados distintos
# "near_dup": {"available": False, "n_near_dup_docs": 0}, # datasketch ausente
# }
# Corpus vacío: contrato estable, exact_dup_pct None, sin excepción.
compute_text_duplicates([])
# {"n_docs": 0, "n_exact_dup": 0, "exact_dup_pct": None, "n_unique": 0,
# "near_dup": {"available": False, "n_near_dup_docs": 0}}
```
## Cuando usarla
Úsala en la fase de calidad de un EDA de texto, cuando quieras saber cuánto de
tu corpus es ruido duplicado antes de entrenar, vectorizar o muestrear: te da
el porcentaje de duplicados exactos (`exact_dup_pct`), el número de documentos
únicos (`n_unique`) y, si tienes `datasketch` instalada, una estimación de
casi-duplicados (paráfrasis, copias con pequeñas ediciones) vía MinHash + LSH.
Pásale directamente la columna/lista de textos crudos; la función filtra None y
no-str por ti y nunca lanza, así que es segura para encadenar en pipelines de
perfilado.
## Gotchas
- **Near-dup requiere `datasketch` (opcional).** Si la librería no está
instalada, `near_dup` degrada a `{"available": False, "n_near_dup_docs": 0}`
(sin clave `threshold`) y el resto del resultado se calcula igual. Los
duplicados **exactos** funcionan siempre porque solo usan la stdlib (hash).
- **Normalización de exactos.** Dos textos cuentan como el mismo duplicado
exacto si coinciden tras `" ".join(doc.split()).strip().lower()`: se colapsan
espacios/tabuladores/saltos, se recortan extremos y se ignora el caso. Cambios
de puntuación o acentos SÍ los distinguen (no se eliminan).
- **`n_exact_dup` cuenta repeticiones, no grupos.** Con 3 copias de un mismo
texto, `n_exact_dup` es 2 (las dos copias extra), no 1. Equivale a
`n_docs - n_unique`.
- **`exact_dup_pct` es `None` con corpus vacío** (no `ZeroDivisionError`); en
cualquier otro caso es un float redondeado a 2 decimales.
- **`sample_max` solo limita el near-dup.** El conteo de duplicados exactos
recorre todo el corpus; el near-dup muestrea los primeros `sample_max`
documentos para acotar memoria. Si el corpus está ordenado, considera barajar
antes para que la muestra sea representativa.
- **Elementos no-str se descartan.** `True`/`False` no cuentan como str y se
ignoran igual que `None`; `n_docs` refleja solo los documentos válidos.
@@ -0,0 +1,128 @@
"""Detección de documentos duplicados en un corpus de texto.
Función pura, estilo dict-no-throw del grupo `eda`: nunca lanza, siempre
devuelve el mismo contrato de claves. Los duplicados EXACTOS se calculan
siempre con la stdlib (normalización + hash SHA-1). Los CASI-duplicados
(near-dup) requieren la dependencia opcional `datasketch`; si no está
instalada, esa parte degrada limpiamente a ``available: False`` sin afectar
al resto del cálculo.
"""
import hashlib
import re
def _compute_near_dup(valid, near_threshold, sample_max):
"""Cuenta documentos con al menos otro casi-duplicado vía MinHash + LSH.
Import perezoso de ``datasketch``. Si la librería no está disponible (o
cualquier paso falla), degrada a ``{"available": False, "n_near_dup_docs": 0}``
sin propagar la excepción.
Args:
valid: lista de str ya filtrada (sin None ni no-str).
near_threshold: umbral de similitud Jaccard para LSH.
sample_max: número máximo de documentos a muestrear.
Returns:
dict con ``available`` (bool) y ``n_near_dup_docs`` (int). Cuando
``available`` es True, incluye además ``threshold``.
"""
try:
from datasketch import MinHash, MinHashLSH
except Exception:
return {"available": False, "n_near_dup_docs": 0}
try:
docs = valid[:sample_max]
num_perm = 128
lsh = MinHashLSH(threshold=near_threshold, num_perm=num_perm)
minhashes = {}
for i, doc in enumerate(docs):
tokens = re.findall(r"\w+", doc.lower())
shingles = set()
for j in range(len(tokens) - 2):
shingles.add(" ".join(tokens[j:j + 3]))
# Documentos con menos de 3 tokens no generan 3-shingles: caemos a
# los tokens sueltos para no perderlos del todo.
if not shingles:
shingles = set(tokens)
if not shingles:
# Documento sin tokens (cadena vacía / solo símbolos): se omite.
continue
m = MinHash(num_perm=num_perm)
for sh in shingles:
m.update(sh.encode("utf-8"))
key = "d{}".format(i)
minhashes[key] = m
lsh.insert(key, m)
n_near = 0
for key, m in minhashes.items():
matches = lsh.query(m)
if len(matches) > 1:
n_near += 1
return {
"available": True,
"n_near_dup_docs": int(n_near),
"threshold": near_threshold,
}
except Exception:
return {"available": False, "n_near_dup_docs": 0}
def compute_text_duplicates(texts, near_threshold=0.85, sample_max=2000) -> dict:
"""Detecta duplicados exactos y casi-duplicados en un corpus de texto.
Args:
texts: lista de documentos. Los elementos None o que no sean str se
descartan; ``n_docs`` cuenta solo los válidos.
near_threshold: umbral de similitud Jaccard para considerar dos
documentos casi-duplicados (solo near-dup, requiere datasketch).
sample_max: tope de documentos muestreados para el cálculo near-dup.
Returns:
dict con las claves ``n_docs``, ``n_exact_dup``, ``exact_dup_pct``
(float redondeado a 2 decimales, o None si el corpus está vacío),
``n_unique`` y ``near_dup`` (sub-dict con ``available`` y
``n_near_dup_docs``, más ``threshold`` cuando está disponible).
Nunca lanza: captura toda excepción y degrada.
"""
# Filtrado defensivo de documentos válidos.
try:
valid = [t for t in texts if isinstance(t, str)] if texts is not None else []
except Exception:
valid = []
n_docs = len(valid)
# Duplicados exactos: normalizar + hash SHA-1 (stdlib, siempre disponible).
try:
seen = set()
n_exact_dup = 0
for doc in valid:
norm = " ".join(doc.split()).strip().lower()
digest = hashlib.sha1(norm.encode("utf-8")).hexdigest()
if digest in seen:
n_exact_dup += 1
else:
seen.add(digest)
n_unique = len(seen)
except Exception:
n_exact_dup = 0
n_unique = 0
exact_dup_pct = round(n_exact_dup / n_docs * 100, 2) if n_docs > 0 else None
# Casi-duplicados: opcional vía datasketch, degrada solo.
near_dup = _compute_near_dup(valid, near_threshold, sample_max)
return {
"n_docs": n_docs,
"n_exact_dup": n_exact_dup,
"exact_dup_pct": exact_dup_pct,
"n_unique": n_unique,
"near_dup": near_dup,
}
@@ -0,0 +1,77 @@
"""Tests para compute_text_duplicates.
Importa el modulo hoja directamente (`datascience.compute_text_duplicates`)
para no depender de que el paquete reexporte la funcion en su __init__.
datasketch normalmente NO esta instalada en el venv, asi que near_dup
degrada a available=False; los tests no requieren la libreria.
"""
from datascience.compute_text_duplicates import compute_text_duplicates
EXPECTED_KEYS = {"n_docs", "n_exact_dup", "exact_dup_pct", "n_unique", "near_dup"}
def test_duplicados_exactos():
"""3 copias del mismo texto + 2 únicos: n_exact_dup=2, pct>0."""
texts = [
"El gato come pescado",
"El gato come pescado",
"el GATO come pescado", # mismo tras normalizar (espacios + case)
"Un perro ladra",
"La luna brilla",
]
result = compute_text_duplicates(texts)
assert set(result.keys()) == EXPECTED_KEYS
assert result["n_docs"] == 5
# 3 copias del primer texto (2 son repeticion) + 2 textos unicos.
assert result["n_exact_dup"] == 2
assert result["n_unique"] == 3
assert result["exact_dup_pct"] is not None
assert result["exact_dup_pct"] > 0
# 2 / 5 * 100 = 40.0
assert abs(result["exact_dup_pct"] - 40.0) < 1e-9
def test_sin_duplicados():
"""Corpus sin repeticiones: n_exact_dup=0, n_unique==n_docs."""
texts = [
"primero documento distinto",
"segundo documento distinto",
"tercero documento distinto",
]
result = compute_text_duplicates(texts)
assert result["n_docs"] == 3
assert result["n_exact_dup"] == 0
assert result["n_unique"] == 3
assert abs(result["exact_dup_pct"] - 0.0) < 1e-9
def test_vacio():
"""Corpus vacio: n_docs 0, exact_dup_pct None, no lanza."""
result = compute_text_duplicates([])
assert set(result.keys()) == EXPECTED_KEYS
assert result["n_docs"] == 0
assert result["n_exact_dup"] == 0
assert result["exact_dup_pct"] is None
assert result["n_unique"] == 0
assert result["near_dup"]["n_near_dup_docs"] == 0
def test_near_dup_degrada():
"""near_dup expone 'available' (bool) y no lanza aunque falte datasketch."""
texts = ["uno dos tres cuatro", "uno dos tres cuatro cinco", "algo distinto"]
result = compute_text_duplicates(texts)
near = result["near_dup"]
assert "available" in near
assert isinstance(near["available"], bool)
assert "n_near_dup_docs" in near
assert isinstance(near["n_near_dup_docs"], int)
# Tambien tolera None y entradas no-str sin lanzar.
mixed = compute_text_duplicates(["hola", None, 123, "hola"])
assert mixed["n_docs"] == 2
assert mixed["n_exact_dup"] == 1
@@ -0,0 +1,86 @@
---
id: compute_text_length_stats_py_datascience
name: compute_text_length_stats
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def compute_text_length_stats(texts, n_bins=20) -> dict"
description: "Profiles the length distribution of a corpus of text documents for EDA: per-document characters, words (unicode \\w+ tokens) and sentences (segments split on .!?… with a minimum of 1 per non-empty doc), each summarized with mean/p50/p90/p99/min/max (nearest-rank percentiles), plus an equal-width histogram of per-document word counts. None and non-str items are discarded. Dict-no-throw: never raises. Stdlib only (re)."
tags: [eda, datascience, text, nlp, length, statistics, pure, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [re, math]
example: |
from datascience.compute_text_length_stats import compute_text_length_stats
result = compute_text_length_stats(["Hola mundo.", "Una frase mas larga aqui."], n_bins=5)
tested: true
tests:
- "test_basico"
- "test_vacio"
- "test_descarta_none"
- "test_un_documento"
test_file_path: "python/functions/datascience/compute_text_length_stats_test.py"
file_path: "python/functions/datascience/compute_text_length_stats.py"
params:
- name: texts
desc: "List of text documents (str). None entries and any non-str items (ints, floats, etc.) are discarded before any computation. An empty string \"\" is kept (chars 0, words 0, sentences 0)."
- name: n_bins
desc: "Number of equal-width bins for the per-document word-count histogram. Default 20. When all docs have the same word count, there are <2 docs, or n_bins < 1, a single covering bin is returned instead."
output: "Dict with keys n_docs (int), chars, words, sentences and word_hist. Each of the three axis sub-dicts has the exact keys mean (float, 2 decimals), p50, p90, p99, min, max (ints). When there are no valid documents, n_docs is 0, every axis statistic is None and word_hist is []. word_hist is a list of {lo: float, hi: float, count: int} bins; the sum of all bin counts equals n_docs."
---
## Ejemplo
```python
from datascience.compute_text_length_stats import compute_text_length_stats
compute_text_length_stats(
[
"Hola mundo.",
"Una frase mas larga con varias palabras aqui.",
"Esto. Tiene. Tres frases distintas!",
],
n_bins=5,
)
# {
# "n_docs": 3,
# "chars": {"mean": 30.33, "p50": 35, "p90": 45, "p99": 45, "min": 11, "max": 45},
# "words": {"mean": 5.0, "p50": 5, "p90": 8, "p99": 8, "min": 2, "max": 8},
# "sentences": {"mean": 1.67, "p50": 1, "p90": 3, "p99": 3, "min": 1, "max": 3},
# "word_hist": [
# {"lo": 2.0, "hi": 3.2, "count": 1},
# {"lo": 3.2, "hi": 4.4, "count": 0},
# {"lo": 4.4, "hi": 5.6, "count": 1},
# {"lo": 5.6, "hi": 6.8, "count": 0},
# {"lo": 6.8, "hi": 8.0, "count": 1},
# ],
# }
```
## Cuando usarla
Úsala al perfilar una columna o corpus de texto libre en un EDA: cuando
necesites saber lo largos que son los documentos (en caracteres, palabras y
frases) y cómo se reparte esa longitud antes de tokenizar, vectorizar o decidir
truncados/ventanas para un modelo. Pásale la lista de strings crudos de la
columna; `None` y valores no-texto se descartan solos. Encaja en el grupo `eda`
como bloque de longitud junto a `summarize_categorical`.
## Gotchas
- Función pura, solo stdlib (`re`). No usa numpy, pandas ni sklearn.
- Percentiles por método **nearest-rank** (devuelven un valor real de la lista,
no interpolan); por eso p50/p90/p99/min/max son enteros y `mean` es el único
float (redondeado a 2 decimales).
- El conteo de frases es una **aproximación** por puntuación (`.!?…`): un texto
sin esa puntuación cuenta como 1 frase si no está vacío; abreviaturas o
ellipsis pueden inflar o reducir el conteo.
- `word_hist` es equal-width entre min y max de palabras: con todos los docs
del mismo tamaño, menos de 2 docs, o `n_bins < 1`, devuelve un único bin.
- Dict-no-throw: ante input inesperado devuelve la forma vacía
(`n_docs` 0, ejes `None`, `word_hist` []) en vez de lanzar.
@@ -0,0 +1,168 @@
"""Pure EDA helper: document length distribution for the `eda` group.
Given a list of text documents, computes the length distribution along three
axes (characters, words and sentences) plus an equal-width histogram of the
per-document word counts. Stdlib only (``re`` + ``statistics`` semantics via a
hand-rolled nearest-rank percentile). No numpy, no sklearn.
The function is dict-no-throw: it never raises. On any unexpected input it
degrades to the empty-shape result.
"""
import math
import re
_WORD_RE = re.compile(r"\w+", re.UNICODE)
_SENT_RE = re.compile(r"[.!?…]+")
def _empty_axis() -> dict:
"""Return an axis sub-dict with every statistic set to ``None``."""
return {"mean": None, "p50": None, "p90": None, "p99": None, "min": None, "max": None}
def _pct(sorted_vals, q):
"""Nearest-rank percentile of an already-sorted list.
Args:
sorted_vals: List of numbers sorted ascending.
q: Percentile in the 0..100 range.
Returns:
The value at the nearest rank, or ``None`` for an empty list.
"""
n = len(sorted_vals)
if n == 0:
return None
if q <= 0:
return sorted_vals[0]
rank = math.ceil(q / 100.0 * n)
if rank < 1:
rank = 1
if rank > n:
rank = n
return sorted_vals[rank - 1]
def _axis_stats(values) -> dict:
"""Compute mean/p50/p90/p99/min/max over a list of integer counts.
``mean`` is rounded to 2 decimals; every other statistic is an integer
(they are counts). Returns an all-``None`` axis for an empty list.
"""
if not values:
return _empty_axis()
sv = sorted(values)
return {
"mean": round(sum(sv) / len(sv), 2),
"p50": int(_pct(sv, 50)),
"p90": int(_pct(sv, 90)),
"p99": int(_pct(sv, 99)),
"min": int(sv[0]),
"max": int(sv[-1]),
}
def _word_hist(word_counts, n_bins) -> list:
"""Equal-width histogram of per-document word counts.
Builds ``n_bins`` bins between ``min`` and ``max`` of the word counts. When
every document has the same number of words, there are fewer than 2
documents, or ``n_bins`` is not at least 1, a single covering bin is
returned. With no documents the result is ``[]``. The sum of bin ``count``
always equals ``len(word_counts)``.
"""
if not word_counts:
return []
wmin = min(word_counts)
wmax = max(word_counts)
if wmax == wmin or len(word_counts) < 2 or n_bins < 1:
return [{"lo": float(wmin), "hi": float(wmax), "count": len(word_counts)}]
width = (wmax - wmin) / n_bins
bins = []
for i in range(n_bins):
lo = wmin + i * width
hi = wmin + (i + 1) * width
bins.append({"lo": float(lo), "hi": float(hi), "count": 0})
# Pin the last upper edge to the real maximum to avoid float drift.
bins[-1]["hi"] = float(wmax)
for wc in word_counts:
if wc >= wmax:
idx = n_bins - 1
else:
idx = int((wc - wmin) / width)
if idx < 0:
idx = 0
elif idx >= n_bins:
idx = n_bins - 1
bins[idx]["count"] += 1
return bins
def compute_text_length_stats(texts, n_bins=20) -> dict:
"""Summarize the length distribution of a corpus of text documents.
For each document three lengths are measured: characters (``len(doc)``),
words (count of ``\\w+`` unicode tokens) and sentences (non-empty segments
after splitting on ``.!?``, with a minimum of 1 for any non-empty
document). For each axis the mean, p50, p90, p99, min and max are reported,
plus an equal-width histogram of the per-document word counts.
``None`` entries and any non-``str`` items in ``texts`` are discarded.
The function never raises: on empty/``None`` input or any internal error it
returns the empty-shape result (``n_docs`` 0, all-``None`` axes, ``[]``
histogram).
Args:
texts: List of text documents (``str``). ``None`` and non-``str``
items are dropped.
n_bins: Number of equal-width bins for the word-count histogram.
Default 20.
Returns:
Dict with keys ``n_docs``, ``chars``, ``words``, ``sentences`` and
``word_hist``. Each of the three axes is a sub-dict with ``mean``
(float, 2 decimals), ``p50``, ``p90``, ``p99``, ``min`` and ``max``
(ints), all ``None`` when there are no documents. ``word_hist`` is a
list of ``{lo, hi, count}`` bins whose ``count`` sums to ``n_docs``.
"""
empty_axis = _empty_axis()
fallback = {
"n_docs": 0,
"chars": dict(empty_axis),
"words": dict(empty_axis),
"sentences": dict(empty_axis),
"word_hist": [],
}
try:
if not texts:
return fallback
docs = [t for t in texts if isinstance(t, str)]
n_docs = len(docs)
if n_docs == 0:
return fallback
char_counts = [len(d) for d in docs]
word_counts = [len(_WORD_RE.findall(d)) for d in docs]
sent_counts = []
for d in docs:
segments = [s for s in _SENT_RE.split(d) if s.strip()]
n = len(segments)
if d and n == 0:
# Non-empty document with no detectable sentence: count as 1.
n = 1
sent_counts.append(n)
return {
"n_docs": n_docs,
"chars": _axis_stats(char_counts),
"words": _axis_stats(word_counts),
"sentences": _axis_stats(sent_counts),
"word_hist": _word_hist(word_counts, n_bins),
}
except Exception:
return fallback
@@ -0,0 +1,70 @@
"""Tests para compute_text_length_stats.
Inserta `python/functions` en sys.path (relativo a este archivo) para importar
el modulo hoja por su paquete `datascience`, sin depender de que el paquete lo
reexporte en su __init__.
"""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from datascience.compute_text_length_stats import compute_text_length_stats
def test_basico():
"""Varios textos de longitudes distintas: stats y histograma coherentes."""
texts = [
"Hola mundo.", # 2 words, 1 sentence
"Una frase mas larga con varias palabras aqui.", # 8 words, 1 sentence
"Corto.", # 1 word, 1 sentence
"Esto. Tiene. Tres frases distintas!", # 5 words, 3 sentences
]
result = compute_text_length_stats(texts)
assert result["n_docs"] == 4
# Diferentes longitudes en palabras -> max estrictamente mayor que min.
assert result["words"]["max"] > result["words"]["min"]
# El histograma de palabras no esta vacio.
assert result["word_hist"] != []
# La suma de counts del histograma cubre todos los documentos.
assert sum(b["count"] for b in result["word_hist"]) == result["n_docs"]
# mean es float redondeado; min/max son enteros.
assert isinstance(result["words"]["mean"], float)
assert isinstance(result["words"]["min"], int)
assert isinstance(result["words"]["max"], int)
# El documento con 3 frases empuja el max de sentences a >= 3.
assert result["sentences"]["max"] >= 3
def test_vacio():
"""Lista vacia: n_docs 0, subdicts None, word_hist []."""
result = compute_text_length_stats([])
assert result["n_docs"] == 0
for axis in ("chars", "words", "sentences"):
for key in ("mean", "p50", "p90", "p99", "min", "max"):
assert result[axis][key] is None
assert result["word_hist"] == []
def test_descarta_none():
"""None y valores no-str se descartan del computo."""
result = compute_text_length_stats(["hello world", None, 123, 4.5, "foo bar baz"])
# Solo dos strings validos.
assert result["n_docs"] == 2
assert result["words"]["min"] == 2 # "hello world"
assert result["words"]["max"] == 3 # "foo bar baz"
assert sum(b["count"] for b in result["word_hist"]) == 2
def test_un_documento():
"""Un solo documento: word_hist tiene exactamente un bin con count 1."""
result = compute_text_length_stats(["solo un documento aqui"])
assert result["n_docs"] == 1
assert len(result["word_hist"]) == 1
assert result["word_hist"][0]["count"] == 1
# Con un unico documento, p50 == min == max == su numero de palabras (4).
assert result["words"]["min"] == 4
assert result["words"]["max"] == 4
assert result["words"]["p50"] == 4
@@ -0,0 +1,88 @@
---
id: compute_text_readability_py_datascience
name: compute_text_readability
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def compute_text_readability(texts, sample_max=500) -> dict"
description: "Calcula la legibilidad Flesch Reading Ease de un corpus de texto usando textstat con import perezoso y degradación. Filtra None/no-str/vacíos, muestrea hasta sample_max documentos (los primeros) y agrega los scores Flesch en {mean, p50, min, max}. Si textstat no está instalada devuelve available=False sin lanzar. Estilo dict-no-throw del grupo eda — nunca lanza."
tags: [eda, datascience, text, nlp, readability, flesch, textstat, pure, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [math, textstat]
example: |
from datascience.compute_text_readability import compute_text_readability
out = compute_text_readability(["The cat sat on the mat. It was warm and sunny."])
# {"available": True, "n_scored": 1, "flesch": {"mean": 109.0, "p50": 109.0, "min": 108.96..., "max": 108.96...}}
tested: true
tests:
- "test_prosa_ingles"
- "test_vacio"
- "test_degradacion"
test_file_path: "python/functions/datascience/compute_text_readability_test.py"
file_path: "python/functions/datascience/compute_text_readability.py"
params:
- name: texts
desc: "Lista de str (documentos del corpus). Los elementos None, no-str o vacíos tras strip() se descartan silenciosamente. El orden se respeta: el muestreo toma los primeros documentos válidos."
- name: sample_max
desc: "Número máximo de documentos válidos a puntuar (los primeros). Default 500. Acota el coste en corpus grandes. Valores no convertibles a int caen a 500; negativos se tratan como 0."
output: "Dict con exactamente 3 claves siempre presentes: available (bool: True si textstat se pudo importar), n_scored (int: nº de documentos efectivamente puntuados), flesch (dict con mean, p50, min, max). mean y p50 redondeados a 1 decimal; p50 por nearest-rank sobre los scores ordenados; min/max son los scores extremos sin redondear. Todos los valores de flesch son None cuando n_scored es 0. La función nunca lanza: cualquier excepción global (incluida ImportError de textstat) degrada a available=False, n_scored=0 y flesch todo None."
---
## Ejemplo
```python
from datascience.compute_text_readability import compute_text_readability
textos = [
"The cat sat on the mat. It was a warm and sunny day in the park.",
"Reading is a wonderful habit. Books open doors to new worlds and ideas.",
"He ran quickly to the store to buy some fresh bread and a bottle of milk.",
]
compute_text_readability(textos)
# {
# "available": True,
# "n_scored": 3,
# "flesch": {"mean": 91.4, "p50": 95.4, "min": 70.08..., "max": 108.83...}
# }
# Corpus vacío (textstat presente): available True pero nada que puntuar.
compute_text_readability([])
# {"available": True, "n_scored": 0,
# "flesch": {"mean": None, "p50": None, "min": None, "max": None}}
```
## Cuando usarla
Úsala en un EDA de texto cuando necesites una métrica única y comparable de
**lo fácil que es de leer** un corpus de documentos (descripciones, reviews,
artículos, tickets). Devuelve el resumen Flesch Reading Ease agregado
(`mean`/`p50`/`min`/`max`) listo para un report o un bloque del notebook, sin
tener que iterar `textstat` a mano. Pásale la lista de textos crudos y, si el
corpus es grande, limita el coste con `sample_max`. El estilo dict-no-throw
permite incrustarla en pipelines del grupo `eda` sin envolver en try/except.
## Gotchas
- **`textstat` es una dependencia opcional.** Si no está instalada (o falla al
importar) la función NO lanza: devuelve `available=False`, `n_scored=0` y
`flesch` todo `None`. Comprueba `available` antes de interpretar los números.
- **Flesch Reading Ease está pensado para prosa en inglés.** Aplicado a otros
idiomas o a texto no-prosa (código, listas, tablas, cadenas muy cortas) los
scores no son interpretables, aunque se calculen sin error.
- **Escala Flesch:** valores **altos** = más fácil de leer (≈90100 muy fácil),
valores **bajos** = más difícil (puede ser negativo en texto muy denso). No
se recortan a ningún rango: se reportan tal cual los devuelve `textstat`.
- **`available=True` con `n_scored=0`** significa que `textstat` está presente
pero el corpus no aportó documentos puntuables (vacío, solo None/no-str, o
todos los docs fallaron al puntuar). Es distinto de `available=False`.
- **Muestreo = los primeros `sample_max`**, no aleatorio. Si el orden del corpus
está sesgado, el resumen reflejará ese sesgo.
- **`mean` y `p50` redondean a 1 decimal**; `min`/`max` se devuelven sin
redondear (los scores extremos reales).
@@ -0,0 +1,121 @@
"""Legibilidad Flesch Reading Ease de un corpus de texto.
Función pura del grupo `eda`, estilo dict-no-throw: nunca lanza. Usa la
librería `textstat` con import perezoso y degradación: si `textstat` no está
instalada (o falla al importar), devuelve un resultado con `available=False`
en lugar de propagar el error.
"""
def _percentile_nearest_rank(sorted_values, pct):
"""Percentil por nearest-rank sobre una lista ya ordenada ascendente.
rank = ceil(pct/100 * n); índice 1-based recortado a [1, n].
Devuelve None si la lista está vacía.
"""
n = len(sorted_values)
if n == 0:
return None
import math
rank = math.ceil((pct / 100.0) * n)
if rank < 1:
rank = 1
if rank > n:
rank = n
return sorted_values[rank - 1]
def compute_text_readability(texts, sample_max=500) -> dict:
"""Calcula la legibilidad Flesch Reading Ease de un corpus.
Args:
texts: lista de str. Los elementos None, no-str o vacíos (tras strip)
se descartan. Se muestrean los primeros `sample_max` documentos
válidos.
sample_max: número máximo de documentos a puntuar (los primeros).
Returns:
Dict con la forma exacta::
{"available": bool, "n_scored": int,
"flesch": {"mean": float|None, "p50": float|None,
"min": float|None, "max": float|None}}
`available` es True si `textstat` se pudo importar. La función nunca
lanza: cualquier excepción global degrada a `available=False`.
"""
empty = {
"available": False,
"n_scored": 0,
"flesch": {"mean": None, "p50": None, "min": None, "max": None},
}
try:
# Import perezoso con degradación: textstat es una dependencia opcional.
try:
import textstat
except Exception:
return {
"available": False,
"n_scored": 0,
"flesch": {"mean": None, "p50": None, "min": None, "max": None},
}
# Filtrar y muestrear documentos válidos (los primeros sample_max).
docs = []
if texts is not None:
try:
limit = int(sample_max)
except Exception:
limit = 500
if limit < 0:
limit = 0
for item in texts:
if not isinstance(item, str):
continue
if item.strip() == "":
continue
docs.append(item)
if len(docs) >= limit:
break
scores = []
for doc in docs:
try:
score = textstat.flesch_reading_ease(doc)
except Exception:
continue
try:
score = float(score)
except Exception:
continue
scores.append(score)
n_scored = len(scores)
if n_scored == 0:
# textstat presente pero corpus vacío / sin puntuar.
return {
"available": True,
"n_scored": 0,
"flesch": {"mean": None, "p50": None, "min": None, "max": None},
}
mean_val = round(sum(scores) / n_scored, 1)
sorted_scores = sorted(scores)
p50_raw = _percentile_nearest_rank(sorted_scores, 50)
p50_val = round(p50_raw, 1) if p50_raw is not None else None
min_val = sorted_scores[0]
max_val = sorted_scores[-1]
return {
"available": True,
"n_scored": n_scored,
"flesch": {
"mean": mean_val,
"p50": p50_val,
"min": min_val,
"max": max_val,
},
}
except Exception:
return empty
@@ -0,0 +1,74 @@
"""Tests para compute_text_readability."""
import sys
import os
import builtins
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
from datascience.compute_text_readability import compute_text_readability
EXPECTED_KEYS = {"available", "n_scored", "flesch"}
FLESCH_KEYS = {"mean", "p50", "min", "max"}
def test_prosa_ingles():
"""Varios textos en prosa inglesa: available True, n_scored>0, mean no None."""
texts = [
"The cat sat on the mat. It was a warm and sunny day in the park.",
"She sells sea shells by the sea shore. The shells she sells are surely sea shells.",
"Reading is a wonderful habit. Books open doors to new worlds and ideas.",
"He ran quickly to the store to buy some fresh bread and a bottle of milk.",
]
out = compute_text_readability(texts)
assert set(out.keys()) == EXPECTED_KEYS
assert out["available"] is True
assert out["n_scored"] > 0
assert set(out["flesch"].keys()) == FLESCH_KEYS
assert out["flesch"]["mean"] is not None
assert out["flesch"]["p50"] is not None
assert out["flesch"]["min"] is not None
assert out["flesch"]["max"] is not None
# min <= mean/p50 <= max coherente.
assert out["flesch"]["min"] <= out["flesch"]["max"]
def test_vacio():
"""Corpus vacío con textstat presente: available True, n_scored 0, flesch None."""
out = compute_text_readability([])
assert set(out.keys()) == EXPECTED_KEYS
assert out["available"] is True
assert out["n_scored"] == 0
assert out["flesch"]["mean"] is None
assert out["flesch"]["p50"] is None
assert out["flesch"]["min"] is None
assert out["flesch"]["max"] is None
# Elementos no-str / vacíos también se descartan -> n_scored 0.
out2 = compute_text_readability([None, "", " ", 123])
assert out2["available"] is True
assert out2["n_scored"] == 0
def test_degradacion(monkeypatch):
"""Sin textstat (ImportError forzado): degrada a available False sin lanzar."""
import datascience.compute_text_readability as m
real = builtins.__import__
def fake(name, *a, **k):
if name == "textstat" or name.startswith("textstat."):
raise ImportError("simulado")
return real(name, *a, **k)
monkeypatch.setattr(builtins, "__import__", fake)
out = m.compute_text_readability(["The cat sat on the mat. It was happy and warm."])
assert out["available"] is False
assert out["n_scored"] == 0
assert out["flesch"]["mean"] is None
assert out["flesch"]["p50"] is None
assert out["flesch"]["min"] is None
assert out["flesch"]["max"] is None
@@ -0,0 +1,103 @@
---
id: compute_top_ngrams_py_datascience
name: compute_top_ngrams
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def compute_top_ngrams(texts, n=2, top_k=15, remove_stopwords=True) -> dict"
description: "Calcula los n-gramas de palabras más frecuentes de un corpus de texto (n=1 unigramas, 2 bigramas, 3 trigramas...). Tokeniza a minúsculas con re.findall(r'\\w+', ...), descarta tokens numéricos y, si remove_stopwords=True, elimina stopwords ES+EN ANTES de formar los n-gramas (n-gramas contiguos sobre la secuencia de tokens de contenido, sin cruzar documentos). Pura y autocontenida con collections.Counter, sin sklearn. Estilo dict-no-throw del grupo eda: nunca lanza."
tags: [eda, datascience, text, nlp, ngrams, bigrams, trigrams, pure, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [re, collections]
example: |
from datascience.compute_top_ngrams import compute_top_ngrams
texts = ["machine learning rocks", "we love machine learning"]
compute_top_ngrams(texts, n=2, top_k=5)
# {"n": 2, "top": [{"ngram": "machine learning", "count": 2}, ...]}
tested: true
tests:
- "test_bigramas"
- "test_trigramas"
- "test_vacio"
- "test_stopwords"
test_file_path: "python/functions/datascience/compute_top_ngrams_test.py"
file_path: "python/functions/datascience/compute_top_ngrams.py"
params:
- name: texts
desc: "Lista (o tupla) de cadenas. Los elementos None o que no sean str se descartan silenciosamente. Cada documento se tokeniza por separado; los n-gramas no cruzan la frontera entre documentos."
- name: n
desc: "Tamaño del n-grama: 1 unigramas, 2 bigramas, 3 trigramas, etc. Valores < 1 o no enteros producen top vacío (se conserva tal cual en la clave 'n' del retorno)."
- name: top_k
desc: "Número máximo de n-gramas a devolver, ordenados por frecuencia descendente con desempate alfabético determinista. Default 15. Valores negativos se tratan como 0."
- name: remove_stopwords
desc: "Si True (default) elimina las stopwords ES+EN de una lista inline (~130 términos de altísima frecuencia) ANTES de formar los n-gramas, de modo que los n-gramas se construyen sobre la secuencia de tokens de contenido."
output: "Dict con exactamente 2 claves: n (el n recibido, sin normalizar) y top (lista de dicts {'ngram': str, 'count': int} ordenada por count descendente, longitud <= top_k). ngram es la unión de los tokens del n-grama por un espacio. Corpus vacío, tokens insuficientes para formar n-gramas o cualquier excepción interna degradan a {'n': n, 'top': []}. La función nunca lanza."
---
## Ejemplo
```python
from datascience.compute_top_ngrams import compute_top_ngrams
texts = [
"machine learning rocks",
"machine learning is fun",
"we love machine learning",
]
# Bigramas (n=2): "machine learning" aparece en los 3 documentos.
compute_top_ngrams(texts, n=2, top_k=5)
# {
# "n": 2,
# "top": [
# {"ngram": "machine learning", "count": 3},
# {"ngram": "learning fun", "count": 1},
# {"ngram": "learning rocks", "count": 1},
# {"ngram": "love machine", "count": 1},
# ],
# }
# Unigramas con stopwords fuera (default): solo palabras de contenido.
compute_top_ngrams(["the cat sat on the mat"], n=1, top_k=3)
# {"n": 1, "top": [{"ngram": "cat", "count": 1},
# {"ngram": "mat", "count": 1},
# {"ngram": "sat", "count": 1}]}
```
## Cuando usarla
Úsala en la fase de EDA de texto cuando, además del vocabulario suelto, necesites
ver qué **combinaciones de palabras contiguas** dominan un corpus: colocaciones,
frases técnicas recurrentes ("machine learning", "data analyst"), o patrones de
trigramas en titulares/descripciones. Es el complemento natural de un perfil de
vocabulario: pasa de "qué palabras aparecen" a "qué secuencias aparecen". Llámala
con `n=1` para unigramas, `n=2` para bigramas y `n=3` para trigramas, y ajusta
`top_k` al tamaño de la tabla que vas a renderizar. Deja `remove_stopwords=True`
para que los n-gramas reflejen contenido y no conectores gramaticales.
## Gotchas
- **Las stopwords se eliminan ANTES de formar los n-gramas.** Con
`remove_stopwords=True` la frase "data of analysis" produce el bigrama
"data analysis" (el "of" intermedio desaparece y los tokens de contenido se
vuelven contiguos), no "data of" ni "of analysis". Si quieres preservar la
adyacencia literal del texto original, pasa `remove_stopwords=False`.
- **Los n-gramas NO cruzan documentos.** Cada elemento de `texts` se tokeniza y
recorre por separado; el último token de un documento nunca se combina con el
primero del siguiente.
- **Tokens puramente numéricos se descartan** (`tok.isdigit()`), pero los
alfanuméricos mixtos no: "3d" o "covid19" sí cuentan como tokens. Un decimal
como "3.5" se parte en "3" y "5" por `\w+` y ambos se descartan por numéricos.
- **La lista de stopwords es inline ES+EN**, pensada para textos generales en
esos dos idiomas. Para otros idiomas o jerga específica de dominio puede dejar
pasar conectores; en ese caso filtra el corpus aguas arriba o usa
`remove_stopwords=False` y posfiltra.
- **`top` puede tener menos de `top_k` elementos** si el corpus no tiene tantos
n-gramas distintos. El desempate por frecuencia es alfabético (determinista),
no por orden de aparición.
@@ -0,0 +1,94 @@
"""Top n-gramas de palabras más frecuentes de un corpus de texto.
Función pura, autocontenida (solo stdlib: re + collections.Counter). No depende
de scikit-learn ni de ninguna otra librería externa. Estilo dict-no-throw del
grupo `eda`: ante cualquier entrada degenerada o excepción interna devuelve
``{"n": n, "top": []}`` en vez de lanzar.
"""
import re
from collections import Counter
# Lista inline de stopwords ES + EN (~80 términos de altísima frecuencia).
# Se eliminan ANTES de formar los n-gramas: los n-gramas se construyen sobre la
# secuencia de tokens de contenido, no sobre el texto original.
_STOPWORDS = frozenset({
# Español
"de", "la", "que", "el", "en", "y", "a", "los", "del", "se", "las", "por",
"un", "para", "con", "no", "una", "su", "al", "lo", "como", "más", "mas",
"pero", "sus", "le", "ya", "o", "este", "", "si", "porque", "esta",
"entre", "cuando", "muy", "sin", "sobre", "también", "tambien", "me",
"hasta", "hay", "donde", "quien", "desde", "todo", "nos", "durante",
"todos", "uno", "les", "ni", "contra", "otros", "ese", "eso", "ante",
"ellos", "e", "esto", "", "antes", "algunos", "qué", "unos", "yo",
"otro", "otras", "otra", "él", "tanto", "esa", "estos", "mucho", "quienes",
"nada", "muchos", "cual", "poco", "ella", "estar", "estas", "algunas",
"algo", "nosotros",
# Inglés
"the", "of", "and", "to", "in", "is", "it", "for", "on", "with", "as",
"are", "was", "be", "this", "that", "by", "an", "or", "at", "from", "but",
"not", "have", "has", "had", "they", "you", "we", "he", "she", "his",
"her", "their", "its", "i", "my", "me", "our", "us", "do", "does", "did",
"will", "would", "can", "could", "should", "there", "which", "who", "what",
"when", "where", "how", "all", "if", "so", "than", "then", "out", "up",
})
def compute_top_ngrams(texts, n=2, top_k=15, remove_stopwords=True) -> dict:
"""Calcula los n-gramas de palabras más frecuentes de un corpus.
Args:
texts: lista de cadenas. Los elementos ``None`` o que no sean ``str`` se
descartan silenciosamente.
n: tamaño del n-grama (1 = unigramas, 2 = bigramas, 3 = trigramas...).
Valores < 1 o no enteros producen ``top`` vacío.
top_k: número máximo de n-gramas a devolver, ordenados por frecuencia
descendente (con desempate alfabético determinista).
remove_stopwords: si ``True`` elimina las stopwords ES+EN ANTES de
formar los n-gramas, de modo que los n-gramas se construyen sobre la
secuencia de tokens de contenido (no cruzando documentos).
Returns:
``{"n": n, "top": [{"ngram": "w1 w2", "count": int}, ...]}``. Corpus
vacío, sin tokens suficientes o cualquier excepción interna degrada a
``{"n": n, "top": []}``. Nunca lanza.
"""
try:
if not isinstance(n, int) or n < 1:
return {"n": n, "top": []}
try:
limit = int(top_k)
except (TypeError, ValueError):
limit = 0
if limit < 0:
limit = 0
if not isinstance(texts, (list, tuple)):
return {"n": n, "top": []}
counter = Counter()
for doc in texts:
if not isinstance(doc, str):
continue
tokens = [
tok
for tok in re.findall(r"\w+", doc.lower(), re.UNICODE)
if not tok.isdigit()
]
if remove_stopwords:
tokens = [tok for tok in tokens if tok not in _STOPWORDS]
if len(tokens) < n:
continue
for i in range(len(tokens) - n + 1):
ngram = " ".join(tokens[i:i + n])
counter[ngram] += 1
if not counter:
return {"n": n, "top": []}
ordered = sorted(counter.items(), key=lambda kv: (-kv[1], kv[0]))
top = [{"ngram": ngram, "count": count} for ngram, count in ordered[:limit]]
return {"n": n, "top": top}
except Exception:
return {"n": n, "top": []}
@@ -0,0 +1,65 @@
"""Tests para compute_top_ngrams."""
import sys
import os
# sys.path estándar: añade `python/functions/` para importar por paquete raíz.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
from datascience.compute_top_ngrams import compute_top_ngrams
def test_bigramas():
# "machine learning" se repite en cada documento -> bigrama más frecuente.
texts = [
"machine learning rocks",
"machine learning is fun",
"we love machine learning",
]
result = compute_top_ngrams(texts, n=2, top_k=5)
assert result["n"] == 2
assert result["top"], "esperaba al menos un bigrama"
assert result["top"][0]["ngram"] == "machine learning"
assert result["top"][0]["count"] == 3
# Cada entrada respeta el contrato {"ngram": str, "count": int}.
for item in result["top"]:
assert isinstance(item["ngram"], str)
assert isinstance(item["count"], int)
def test_trigramas():
texts = [
"alpha beta gamma delta",
"alpha beta gamma omega",
]
# Con stopwords desactivadas para no descartar tokens de contenido.
result = compute_top_ngrams(texts, n=3, top_k=5, remove_stopwords=False)
assert result["n"] == 3
ngrams = {item["ngram"]: item["count"] for item in result["top"]}
# "alpha beta gamma" aparece en ambos documentos.
assert ngrams.get("alpha beta gamma") == 2
# Trigramas únicos de cada documento.
assert ngrams.get("beta gamma delta") == 1
assert ngrams.get("beta gamma omega") == 1
def test_vacio():
assert compute_top_ngrams([], n=2) == {"n": 2, "top": []}
# Documentos no-str / None se descartan -> corpus efectivamente vacío.
assert compute_top_ngrams([None, 123, {"a": 1}], n=2) == {"n": 2, "top": []}
def test_stopwords():
# "the cat" debería desaparecer al quitar stopwords ("the" es stopword EN).
texts = ["the cat the cat the cat"]
con = compute_top_ngrams(texts, n=2, top_k=10, remove_stopwords=True)
sin = compute_top_ngrams(texts, n=2, top_k=10, remove_stopwords=False)
con_ngrams = {item["ngram"] for item in con["top"]}
sin_ngrams = {item["ngram"] for item in sin["top"]}
# Sin filtrar, el bigrama dominante es "the cat".
assert "the cat" in sin_ngrams
# Al filtrar stopwords, ya no aparece "the cat" (queda solo "cat cat").
assert "the cat" not in con_ngrams
assert con_ngrams != sin_ngrams
@@ -0,0 +1,91 @@
---
id: compute_vocabulary_stats_py_datascience
name: compute_vocabulary_stats
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def compute_vocabulary_stats(texts: list, top_k: int = 20, remove_stopwords: bool = True) -> dict"
description: "Profiles the vocabulary of a text corpus for EDA: tokenises a list of documents, counts term frequencies and derives lexical-richness measures — total tokens, unique types, type-token ratio (TTR), hapax legomena and the top-k most frequent terms. Pure, stdlib only (re + collections.Counter); no nltk, no sklearn. Inline ES+EN stopword list, opt-out via remove_stopwords. Never raises: empty/degenerate input returns the zeroed result."
tags: [eda, datascience, text, nlp, vocabulary, ttr, hapax, pure, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [re, collections]
example: |
from datascience.compute_vocabulary_stats import compute_vocabulary_stats
result = compute_vocabulary_stats(["el gato y el perro", "gato veloz"], top_k=5)
tested: true
tests:
- "test_basico"
- "test_vacio"
- "test_stopwords_quitadas"
- "test_stopwords_conservadas"
test_file_path: "python/functions/datascience/compute_vocabulary_stats_test.py"
file_path: "python/functions/datascience/compute_vocabulary_stats.py"
params:
- name: texts
desc: "List of documents (strings) forming the corpus. Entries that are None or not a str are silently discarded. Tokens are extracted per document with re.findall(r'\\w+', doc.lower(), re.UNICODE); purely numeric tokens (tok.isdigit()) are dropped."
- name: top_k
desc: "Maximum number of most-frequent terms to return in top_terms. Default 20. Does not affect n_tokens/n_types/ttr/hapax — only the length of the top_terms list."
- name: remove_stopwords
desc: "When True (default) common Spanish+English stopwords from the inline _STOPWORDS set (~120 entries) are removed from the token stream before any counting. Set False to keep every word (raw lexical profile)."
output: "Dict with the exact keys n_tokens (int), n_types (int), ttr (float|None, n_types/n_tokens rounded to 4 dp), n_hapax (int, terms occurring exactly once), hapax_pct (float|None, n_hapax/n_types*100 rounded to 2 dp) and top_terms (list of {term, count, pct} sorted by count descending, pct = count/n_tokens*100 rounded to 2 dp). For an empty corpus (no tokens after filtering): n_tokens=0, n_types=0, ttr=None, n_hapax=0, hapax_pct=None, top_terms=[]. Any exception degrades to that same empty result — the function never throws."
---
## Ejemplo
```python
from datascience.compute_vocabulary_stats import compute_vocabulary_stats
compute_vocabulary_stats(
["el gato y el perro", "gato veloz corre", "perro perro perro"],
top_k=5,
)
# {
# "n_tokens": 6, # stopwords (el, y) eliminadas por defecto
# "n_types": 3, # gato, perro, veloz, corre -> tras quitar stopwords
# "ttr": 0.5, # n_types / n_tokens
# "n_hapax": 2, # veloz, corre (1 aparicion cada uno)
# "hapax_pct": 50.0, # n_hapax / n_types * 100
# "top_terms": [
# {"term": "perro", "count": 4, "pct": 44.44},
# {"term": "gato", "count": 2, "pct": 22.22},
# ...
# ],
# }
# Perfil lexico crudo (sin filtrar stopwords):
compute_vocabulary_stats(["the cat and the dog"], remove_stopwords=False)
```
## Cuando usarla
Úsala al perfilar una columna o corpus de texto libre en un EDA del grupo `eda`:
cuando necesites medir la riqueza léxica (cuántos tokens y cuántas palabras
distintas, type-token ratio, porcentaje de palabras que solo aparecen una vez) y
ver qué términos dominan el vocabulario (top-k frecuencias). Pásale la lista de
documentos crudos (filas de la columna); `None` y valores no-string se ignoran
solos. Es el equivalente para texto largo de `summarize_categorical`, que perfila
categorías cortas.
## Gotchas
- Función pura y stdlib-only, pero el resultado depende del **idioma**: la lista
`_STOPWORDS` cubre español e inglés. Para otros idiomas pon
`remove_stopwords=False` o filtra fuera, o el perfil mezclará stopwords no
reconocidas en `top_terms`.
- La tokenización es `\w+` con `re.UNICODE`: separa por puntuación y conserva
acentos/ñ, pero NO hace stemming ni lematización — "gato" y "gatos" cuentan
como tipos distintos. Tampoco hace stripping de acentos, así que "más" (con
tilde) y "mas" son tokens diferentes (ambos están en la stoplist).
- Los tokens **puramente numéricos** (`"123"`) se descartan siempre; un token
alfanumérico mixto (`"covid19"`) se conserva.
- `ttr` baja artificialmente en corpus grandes (más texto, más repetición): no
compares TTR entre corpus de tamaños muy distintos sin normalizar.
- Nunca lanza: entrada vacía, `None`, o cualquier excepción interna devuelven el
resultado con ceros/`None`/`[]`. Comprueba `n_tokens == 0` para detectar el
caso degenerado.
@@ -0,0 +1,99 @@
"""Profile the vocabulary of a text corpus for EDA (pure, stdlib only).
Tokenises a list of documents, counts term frequencies and derives lexical
richness measures (type-token ratio, hapax legomena) plus the top-k terms.
No external NLP dependencies (no nltk, no sklearn) only ``re`` and
``collections`` from the standard library.
"""
import re
from collections import Counter
# Common Spanish + English stopwords. Inline, lowercase, no accents stripped
# beyond what already appears here. Filtering is opt-in via remove_stopwords.
_STOPWORDS = {
# Spanish
"de", "la", "que", "el", "en", "y", "a", "los", "del", "se", "las", "por",
"un", "para", "con", "no", "una", "su", "al", "es", "lo", "como", "mas",
"más", "pero", "sus", "le", "ya", "o", "este", "si", "", "porque",
"esta", "entre", "cuando", "muy", "sin", "sobre", "tambien", "también",
"me", "hasta", "hay", "donde", "quien", "desde", "todo", "nos", "durante",
"todos", "uno", "les", "ni", "contra", "otros", "ese", "eso", "ante",
"ellos", "e", "esto", "antes", "algunos", "que", "unos", "yo", "otro",
"otras", "otra", "el", "tanto", "esa", "estos", "mucho", "nada", "muchos",
# English
"the", "of", "and", "to", "in", "is", "it", "for", "on", "with", "as",
"was", "but", "are", "this", "that", "an", "be", "by", "or", "not", "at",
"from", "my", "i", "you", "he", "she", "we", "they", "his", "her", "its",
"our", "their", "what", "which", "who", "whom", "has", "have", "had", "do",
"does", "did", "will", "would", "can", "could", "should", "may", "might",
"must", "if", "then", "than", "so", "too", "very", "just", "also", "were",
"been", "being", "there", "here", "all", "any", "some", "more", "most",
"out", "up", "down", "into", "over", "such", "only", "own", "same",
}
def compute_vocabulary_stats(texts, top_k=20, remove_stopwords=True) -> dict:
"""Profile the vocabulary of a corpus of documents.
Args:
texts: List of strings (the corpus). Entries that are None or not a
string are discarded silently.
top_k: Maximum number of most-frequent terms to include in
``top_terms``. Default 20. Does not affect the other measures.
remove_stopwords: When True (default) common ES+EN stopwords are
dropped from the token stream before any counting.
Returns:
A dict with the exact keys ``n_tokens``, ``n_types``, ``ttr``,
``n_hapax``, ``hapax_pct`` and ``top_terms``. For an empty corpus (no
tokens after filtering): n_tokens=0, n_types=0, ttr=None, n_hapax=0,
hapax_pct=None, top_terms=[]. Never raises any exception degrades to
the empty-corpus result.
"""
empty = {
"n_tokens": 0,
"n_types": 0,
"ttr": None,
"n_hapax": 0,
"hapax_pct": None,
"top_terms": [],
}
try:
tokens = []
for doc in texts or []:
if not isinstance(doc, str):
continue
for tok in re.findall(r"\w+", doc.lower(), re.UNICODE):
if tok.isdigit():
continue
if remove_stopwords and tok in _STOPWORDS:
continue
tokens.append(tok)
n_tokens = len(tokens)
if n_tokens == 0:
return dict(empty)
counts = Counter(tokens)
n_types = len(counts)
ttr = round(n_types / n_tokens, 4)
n_hapax = sum(1 for c in counts.values() if c == 1)
hapax_pct = round(n_hapax / n_types * 100, 2)
top_terms = [
{"term": term, "count": count, "pct": round(count / n_tokens * 100, 2)}
for term, count in counts.most_common(top_k)
]
return {
"n_tokens": n_tokens,
"n_types": n_types,
"ttr": ttr,
"n_hapax": n_hapax,
"hapax_pct": hapax_pct,
"top_terms": top_terms,
}
except Exception:
return dict(empty)
@@ -0,0 +1,74 @@
"""Tests para compute_vocabulary_stats."""
import os
import sys
sys.path.insert(
0, os.path.join(os.path.dirname(__file__), "..", "..", "functions")
)
from datascience.compute_vocabulary_stats import compute_vocabulary_stats
def test_basico():
# Corpus con repeticiones y hapax. Stopwords desactivadas para controlar
# exactamente que tokens entran.
texts = ["gato gato perro", "perro perro raton", "elefante"]
r = compute_vocabulary_stats(texts, top_k=10, remove_stopwords=False)
# n_types < n_tokens cuando hay repeticiones.
assert r["n_types"] < r["n_tokens"]
assert r["n_tokens"] == 7
assert r["n_types"] == 4 # gato, perro, raton, elefante
# ttr en (0, 1].
assert 0 < r["ttr"] <= 1
assert r["ttr"] == round(4 / 7, 4)
# top_terms ordenado por count descendente.
counts = [t["count"] for t in r["top_terms"]]
assert counts == sorted(counts, reverse=True)
assert r["top_terms"][0]["term"] == "perro"
assert r["top_terms"][0]["count"] == 3
# hapax: raton y elefante aparecen exactamente una vez.
assert r["n_hapax"] == 2
assert r["hapax_pct"] == round(2 / 4 * 100, 2)
# pct coherente con count/n_tokens.
assert r["top_terms"][0]["pct"] == round(3 / 7 * 100, 2)
def test_vacio():
# Sin documentos validos -> ceros / None / [].
for arg in ([], None, [None, 123, ""], ["123 456"]):
r = compute_vocabulary_stats(arg)
assert r["n_tokens"] == 0
assert r["n_types"] == 0
assert r["ttr"] is None
assert r["n_hapax"] == 0
assert r["hapax_pct"] is None
assert r["top_terms"] == []
def test_stopwords_quitadas():
texts = ["the gato the perro", "de la casa azul"]
r = compute_vocabulary_stats(texts, remove_stopwords=True)
terms = {t["term"] for t in r["top_terms"]}
# Stopwords ES+EN no deben aparecer.
assert "the" not in terms
assert "de" not in terms
assert "la" not in terms
# Palabras de contenido si.
assert "gato" in terms
assert "casa" in terms
def test_stopwords_conservadas():
texts = ["the gato the perro", "de la casa azul"]
r = compute_vocabulary_stats(texts, remove_stopwords=False)
terms = {t["term"] for t in r["top_terms"]}
# Con el filtro desactivado, las stopwords se conservan.
assert "the" in terms
assert "de" in terms
assert "la" in terms
@@ -0,0 +1,87 @@
---
name: confidence_interval_mean
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def confidence_interval_mean(data: list, other: list = None, confidence: float = 0.95) -> dict"
description: "Intervalo de confianza (IC) de la media de una muestra con la t de Student, o de la DIFERENCIA de medias de dos muestras independientes con el metodo de Welch (sin asumir varianzas iguales). Una muestra: df=n-1, se=sd_muestral/sqrt(n) (sd con ddof=1), tcrit=t.ppf((1+confidence)/2, df), ci=mean+/-tcrit*se. Dos muestras: IC de mean(data)-mean(other) con se=sqrt(se1^2+se2^2) y grados de libertad de Welch-Satterthwaite. Pura y robusta: nunca lanza; ante casos degenerados (muestra vacia, n<2) devuelve nan + clave note, y con varianza cero el IC colapsa al punto (no es error). Usa scipy.stats y numpy."
tags: [papers, statistics, confidence-interval, welch, t-test, python]
params:
- name: data
desc: "muestra de observaciones numericas (lista de numeros). Si other es None, el IC es el de la media de data."
- name: other
desc: "segunda muestra independiente (lista de numeros) o None (default). Si se da, el IC es el de la diferencia de medias mean(data)-mean(other) calculada con Welch (no asume varianzas iguales)."
- name: confidence
desc: "nivel de confianza en (0, 1); 0.95 = IC del 95% (default). El cuantil critico es t.ppf((1+confidence)/2, df)."
output: "dict {mean, ci_low, ci_high, se, df, confidence, n}. mean = media de data (una muestra) o la diferencia mean(data)-mean(other) (dos muestras). En el caso de dos muestras se anaden ademas n1 y n2 (y n = n1+n2). df son los grados de libertad de la t (Welch-Satterthwaite si dos muestras). Casos degenerados (muestra vacia, n<2) anaden la clave note y dejan ci_low/ci_high/se (y a veces df) en nan; con varianza cero y n>=2 el IC colapsa a [mean, mean] con se=0 (con note, sin nan). Nunca None ni excepcion."
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [scipy, numpy]
tested: true
tests: ["test_one_sample_golden_contra_scipy", "test_one_sample_distinto_nivel_confianza", "test_welch_diferencia_golden_contra_scipy", "test_edge_un_solo_elemento_no_lanza_nan_note", "test_edge_lista_vacia_no_lanza_note", "test_edge_varianza_cero_colapsa_al_punto", "test_edge_welch_muestra_vacia_no_lanza_note", "test_edge_welch_n1_uno_no_lanza_note"]
test_file_path: "python/functions/datascience/confidence_interval_mean_test.py"
file_path: "python/functions/datascience/confidence_interval_mean.py"
---
## Ejemplo
```python
from datascience import confidence_interval_mean
# IC del 95% de la media de una muestra (t de Student).
data = [2, 4, 4, 4, 5, 5, 7, 9]
ci = confidence_interval_mean(data, confidence=0.95)
print(ci["mean"]) # -> 5.0
print(ci["df"]) # -> 7.0 (n - 1)
print(round(ci["ci_low"], 5), round(ci["ci_high"], 5))
# -> 3.21251 6.78749 (se con sd muestral ddof=1 ~ 2.13809)
# IC del 95% de la DIFERENCIA de medias (Welch, no asume varianzas iguales).
control = [23.0, 21.0, 25.0, 22.0, 24.0, 26.0]
tratado = [18.0, 20.0, 17.0, 19.0, 21.0]
diff = confidence_interval_mean(control, tratado, confidence=0.95)
print(diff["mean"]) # -> 4.5 (mean(control) - mean(tratado))
print(round(diff["ci_low"], 4), round(diff["ci_high"], 4))
# Si el intervalo no incluye 0, la diferencia es significativa al 5%.
# Degenerados: nunca lanza.
print(confidence_interval_mean([5])["note"]) # n < 2: ... indefinidos
print(confidence_interval_mean([3, 3, 3])["se"]) # -> 0.0 (IC colapsa a [3, 3])
```
## Cuando usarla
Cuando quieras cuantificar la **incertidumbre de una media estimada** a partir de
una muestra: reporta `[ci_low, ci_high]` en vez de un punto suelto para mostrar
el rango plausible del valor real al nivel de confianza pedido. Usala tambien
para **comparar dos grupos** (A/B test, control vs tratamiento, antes vs
despues con grupos independientes): pasa las dos muestras y, si el IC de la
diferencia **no incluye el 0**, la diferencia es significativa al nivel
`1 - confidence`. Es el complemento del p-valor: ademas de "hay efecto", te dice
"de que tamano y con que margen". Para dos muestras usa Welch por defecto, asi
que no necesitas comprobar antes si las varianzas son iguales.
## Gotchas
- Pura y determinista (no hace I/O, no muta las entradas), pero **no** es
stdlib-only: depende de `scipy.stats` y `numpy` (ambos en el venv del proyecto).
- Con `other` usa **Welch** (df de Welch-Satterthwaite): NO asume varianzas
iguales ni tamanos de muestra iguales. Si necesitas el t-test clasico de
varianzas agrupadas (pooled), esta funcion no lo hace.
- `sd` se calcula con **ddof=1** (sd muestral), que es lo correcto para el IC de
una media con la t. Atajos como `sd_poblacional/sqrt(n)` (ddof=0) dan un
intervalo demasiado estrecho.
- En el caso de dos muestras, `mean` es la **diferencia** `mean(data) - mean(other)`
(no la media de data). El orden importa: el signo del IC depende de cual va
primero.
- Nunca lanza. Casos degenerados devuelven `nan` en `ci_low`/`ci_high`/`se`
(y a veces `df`) mas una clave `note`: muestra vacia o `n < 2` en cualquiera de
las muestras. **Excepcion**: con varianza cero y `n >= 2` el IC colapsa al
punto `[mean, mean]` con `se = 0` (no es un error, no hay `nan`).
- Comprueba `"note" in out` antes de usar `ci_low`/`ci_high` si la muestra puede
ser degenerada.
@@ -0,0 +1,176 @@
"""Intervalo de confianza de la media (una muestra) o de la diferencia de medias (Welch).
Funcion pura del grupo papers. Calcula el intervalo de confianza (IC) de la media
de una muestra usando la t de Student, o el IC de la diferencia de medias de dos
muestras independientes con el metodo de Welch (sin asumir varianzas iguales).
- Una muestra: ``df = n - 1``, ``se = sd / sqrt(n)`` (sd con ddof=1),
``tcrit = t.ppf((1 + confidence) / 2, df)``, ``ci = mean +/- tcrit * se``.
- Dos muestras (Welch): IC de ``mean(data) - mean(other)``, con
``se = sqrt(se1^2 + se2^2)`` y grados de libertad de Welch-Satterthwaite.
No lanza excepciones: ante casos degenerados (muestras vacias, ``n < 2``,
varianza cero) devuelve un dict coherente con ``ci_low``/``ci_high``/``se`` en
``nan`` (salvo el sub-caso de varianza cero, donde el IC colapsa al punto) y una
clave ``note`` explicando el caso. Usa ``scipy.stats`` y ``numpy``.
"""
from __future__ import annotations
import math
import numpy as np
from scipy import stats
def confidence_interval_mean(
data: list, other: list = None, confidence: float = 0.95
) -> dict:
"""Intervalo de confianza de la media o de la diferencia de medias (Welch).
Si ``other`` es ``None``, calcula el IC de la media de ``data`` con la t de
Student. Si se proporciona ``other``, calcula el IC de la diferencia
``mean(data) - mean(other)`` con el metodo de Welch (no asume varianzas
iguales) y grados de libertad de Welch-Satterthwaite.
Es una funcion pura y determinista: no hace I/O ni muta las entradas. No
lanza excepcion ante datos degenerados; en su lugar devuelve un dict con la
clave ``note`` y los campos numericos indefinidos a ``nan``.
Args:
data: muestra de observaciones numericas (lista de numeros).
other: segunda muestra independiente. Si se da, el IC es el de la
diferencia de medias ``mean(data) - mean(other)`` con Welch. Si es
``None`` (default), el IC es el de la media de ``data``.
confidence: nivel de confianza en (0, 1), p.ej. 0.95 para el 95%.
Returns:
dict con las claves:
mean: media de ``data`` (una muestra) o la diferencia
``mean(data) - mean(other)`` (dos muestras).
ci_low: extremo inferior del intervalo de confianza.
ci_high: extremo superior del intervalo de confianza.
se: error estandar de la media (o de la diferencia).
df: grados de libertad de la t (Welch-Satterthwaite si dos muestras).
confidence: nivel de confianza aplicado (float).
n: tamano de la muestra (una muestra) o tamano total ``n1 + n2``
(dos muestras; ademas se incluyen ``n1`` y ``n2``).
En el caso de dos muestras se incluyen ademas ``n1`` y ``n2``. Casos
degenerados (muestra vacia, ``n < 2``, etc.) anaden la clave ``note`` y
dejan ``ci_low``/``ci_high``/``se`` (y a veces ``df``) en ``nan``.
"""
conf = float(confidence)
if other is None:
return _ci_one_sample(data, conf)
return _ci_welch(data, other, conf)
def _ci_one_sample(data: list, conf: float) -> dict:
"""IC de la media de una sola muestra con la t de Student."""
arr = np.asarray(list(data), dtype=float)
n = int(arr.size)
base = {
"mean": float("nan"),
"ci_low": float("nan"),
"ci_high": float("nan"),
"se": float("nan"),
"df": float("nan"),
"confidence": conf,
"n": n,
}
if n == 0:
base["note"] = "muestra vacia: media e intervalo indefinidos"
return base
mean = float(arr.mean())
base["mean"] = mean
if n < 2:
base["note"] = "n < 2: error estandar y grados de libertad indefinidos"
return base
df = n - 1
base["df"] = float(df)
sd = float(arr.std(ddof=1))
se = sd / math.sqrt(n)
base["se"] = se
# Varianza cero: el IC colapsa al punto (no es un error).
if se == 0.0:
base["ci_low"] = mean
base["ci_high"] = mean
base["note"] = "varianza cero: el intervalo colapsa a la media"
return base
tcrit = float(stats.t.ppf((1.0 + conf) / 2.0, df))
margin = tcrit * se
base["ci_low"] = mean - margin
base["ci_high"] = mean + margin
return base
def _ci_welch(data: list, other: list, conf: float) -> dict:
"""IC de la diferencia de medias de dos muestras con el metodo de Welch."""
a = np.asarray(list(data), dtype=float)
b = np.asarray(list(other), dtype=float)
n1 = int(a.size)
n2 = int(b.size)
base = {
"mean": float("nan"),
"ci_low": float("nan"),
"ci_high": float("nan"),
"se": float("nan"),
"df": float("nan"),
"confidence": conf,
"n": n1 + n2,
"n1": n1,
"n2": n2,
}
if n1 == 0 or n2 == 0:
base["note"] = "alguna muestra esta vacia: diferencia e intervalo indefinidos"
return base
mean1 = float(a.mean())
mean2 = float(b.mean())
diff = mean1 - mean2
base["mean"] = diff
if n1 < 2 or n2 < 2:
base["note"] = (
"n < 2 en alguna muestra: error estandar y grados de libertad indefinidos"
)
return base
sd1 = float(a.std(ddof=1))
sd2 = float(b.std(ddof=1))
se1 = sd1 / math.sqrt(n1)
se2 = sd2 / math.sqrt(n2)
se = math.sqrt(se1 * se1 + se2 * se2)
base["se"] = se
# Ambas varianzas cero: el IC de la diferencia colapsa al punto.
if se == 0.0:
base["ci_low"] = diff
base["ci_high"] = diff
base["df"] = float("nan")
base["note"] = "varianza cero en ambas muestras: el intervalo colapsa a la diferencia"
return base
# Grados de libertad de Welch-Satterthwaite.
df = (se1 * se1 + se2 * se2) ** 2 / (
(se1**4) / (n1 - 1) + (se2**4) / (n2 - 1)
)
base["df"] = float(df)
tcrit = float(stats.t.ppf((1.0 + conf) / 2.0, df))
margin = tcrit * se
base["ci_low"] = diff - margin
base["ci_high"] = diff + margin
return base
@@ -0,0 +1,140 @@
"""Tests para confidence_interval_mean (IC de la media / diferencia de medias Welch).
Importa el modulo hoja directamente (`confidence_interval_mean`) para no depender
de que el paquete reexporte la funcion en su __init__ (lo integra el orquestador
al cerrar el grupo).
Los golden se calculan con scipy dentro del propio test para que sean robustos:
la funcion bajo prueba debe coincidir con la referencia de scipy a ~1e-9.
"""
import math
import numpy as np
from scipy import stats
from confidence_interval_mean import confidence_interval_mean
def test_one_sample_golden_contra_scipy():
# mean=5.0, n=8. Este dataset tiene sd POBLACIONAL (ddof=0) exactamente 2.0,
# pero la sd MUESTRAL (ddof=1, la que exige la spec y la que es correcta para
# el IC de una media con la t) es sqrt(32/7) ~ 2.13809. El golden robusto se
# calcula con scipy usando se con ddof=1, no con el atajo 2.0/sqrt(8).
data = [2, 4, 4, 4, 5, 5, 7, 9]
out = confidence_interval_mean(data, confidence=0.95)
n = len(data)
mean = float(np.mean(data))
sd = float(np.std(data, ddof=1)) # sample sd ~ 2.13809
se = sd / math.sqrt(n)
lo, hi = stats.t.interval(0.95, df=n - 1, loc=mean, scale=se)
assert abs(out["mean"] - 5.0) < 1e-9
assert abs(out["se"] - se) < 1e-12
assert out["df"] == 7.0
assert out["n"] == 8
assert out["confidence"] == 0.95
assert abs(out["ci_low"] - lo) < 1e-9
assert abs(out["ci_high"] - hi) < 1e-9
# Valores tabulados correctos para ddof=1 (no los 3.32793/6.67207 del
# enunciado, que asumian erroneamente sd=2.0 / ddof=0).
assert abs(out["ci_low"] - 3.21251) < 1e-3
assert abs(out["ci_high"] - 6.78749) < 1e-3
assert "note" not in out
def test_one_sample_distinto_nivel_confianza():
data = [10.0, 12.0, 11.0, 13.0, 9.0, 14.0]
out = confidence_interval_mean(data, confidence=0.99)
n = len(data)
mean = float(np.mean(data))
se = float(np.std(data, ddof=1)) / math.sqrt(n)
lo, hi = stats.t.interval(0.99, df=n - 1, loc=mean, scale=se)
assert abs(out["mean"] - mean) < 1e-12
assert abs(out["ci_low"] - lo) < 1e-9
assert abs(out["ci_high"] - hi) < 1e-9
assert out["df"] == float(n - 1)
def test_welch_diferencia_golden_contra_scipy():
data = [23.0, 21.0, 25.0, 22.0, 24.0, 26.0]
other = [18.0, 20.0, 17.0, 19.0, 21.0]
conf = 0.95
out = confidence_interval_mean(data, other, confidence=conf)
a = np.asarray(data, dtype=float)
b = np.asarray(other, dtype=float)
n1, n2 = a.size, b.size
mean1, mean2 = float(a.mean()), float(b.mean())
diff = mean1 - mean2
se1 = float(a.std(ddof=1)) / math.sqrt(n1)
se2 = float(b.std(ddof=1)) / math.sqrt(n2)
se = math.sqrt(se1**2 + se2**2)
df = (se1**2 + se2**2) ** 2 / (se1**4 / (n1 - 1) + se2**4 / (n2 - 1))
lo, hi = stats.t.interval(conf, df=df, loc=diff, scale=se)
assert abs(out["mean"] - diff) < 1e-9
assert abs(out["mean"] - (mean1 - mean2)) < 1e-9
assert abs(out["se"] - se) < 1e-12
assert abs(out["df"] - df) < 1e-9
assert abs(out["ci_low"] - lo) < 1e-9
assert abs(out["ci_high"] - hi) < 1e-9
assert out["n1"] == n1
assert out["n2"] == n2
assert out["n"] == n1 + n2
assert "note" not in out
def test_edge_un_solo_elemento_no_lanza_nan_note():
out = confidence_interval_mean([5], confidence=0.95)
assert out["mean"] == 5.0 # la media si esta definida con n=1
assert math.isnan(out["se"])
assert math.isnan(out["ci_low"])
assert math.isnan(out["ci_high"])
assert math.isnan(out["df"])
assert out["n"] == 1
assert "note" in out
def test_edge_lista_vacia_no_lanza_note():
out = confidence_interval_mean([], confidence=0.95)
assert math.isnan(out["mean"])
assert math.isnan(out["ci_low"])
assert math.isnan(out["ci_high"])
assert math.isnan(out["se"])
assert out["n"] == 0
assert "note" in out
def test_edge_varianza_cero_colapsa_al_punto():
out = confidence_interval_mean([3, 3, 3], confidence=0.95)
assert out["mean"] == 3.0
assert out["se"] == 0.0
assert out["ci_low"] == 3.0
assert out["ci_high"] == 3.0
assert not math.isnan(out["ci_low"])
assert out["n"] == 3
assert "note" in out
def test_edge_welch_muestra_vacia_no_lanza_note():
out = confidence_interval_mean([1.0, 2.0, 3.0], [], confidence=0.95)
assert math.isnan(out["mean"])
assert math.isnan(out["ci_low"])
assert math.isnan(out["se"])
assert out["n1"] == 3
assert out["n2"] == 0
assert "note" in out
def test_edge_welch_n1_uno_no_lanza_note():
out = confidence_interval_mean([5.0], [1.0, 2.0, 3.0], confidence=0.95)
# La diferencia de medias si esta definida.
assert abs(out["mean"] - (5.0 - 2.0)) < 1e-9
assert math.isnan(out["se"])
assert math.isnan(out["ci_low"])
assert math.isnan(out["df"])
assert "note" in out
@@ -0,0 +1,80 @@
---
name: detect_corpus_language
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def detect_corpus_language(texts, top_k=10, sample_max=1000) -> dict"
description: "Estima la distribucion de idiomas de un corpus de textos con la libreria langdetect (import perezoso). Funcion pura y defensiva del grupo eda: filtra documentos None/no-str/vacios, muestrea hasta sample_max docs, clasifica cada uno con detect() ignorando los que langdetect no puede resolver (LangDetectException), y devuelve la distribucion top_k por frecuencia mas el idioma dominante. Si langdetect no esta instalada o algo falla, degrada a {available: False, ...} y NUNCA lanza (dict-no-throw). Seed fija (DetectorFactory.seed=0) para deteccion determinista."
tags: [eda, datascience, text, nlp, language-detection, langdetect, pure, python]
params:
- name: texts
desc: "Lista de strings (documentos). Los elementos None, no-str o vacios tras strip se descartan antes de clasificar."
- name: top_k
desc: "Numero maximo de idiomas a devolver en distribution, ordenados por count descendente (desempate por codigo ISO ascendente). Default 10."
- name: sample_max
desc: "Numero maximo de documentos a clasificar (se toman los primeros del corpus) para acotar el coste. Default 1000."
output: >
Dict con forma fija (dict-no-throw, nunca lanza):
{"available": bool, "n_detected": int,
"distribution": [{"lang": str, "count": int, "pct": float}, ...],
"dominant": str|None}.
available=True si langdetect es importable; lang son codigos ISO 639-1 ("es","en","fr",...);
pct = count/n_detected*100 redondeado a 2 decimales; n_detected = docs clasificados con exito;
dominant = idioma mas frecuente (None si no hubo detecciones). Corpus vacio con langdetect
presente -> available True, n_detected 0, distribution [], dominant None. Sin langdetect (o
fallo global) -> available False y el resto de campos a su valor vacio.
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: [langdetect]
tested: true
tests: ["test_mixto_es_en", "test_vacio", "test_degradacion"]
test_file_path: "python/functions/datascience/detect_corpus_language_test.py"
file_path: "python/functions/datascience/detect_corpus_language.py"
---
## Ejemplo
```python
import sys, os
sys.path.insert(0, os.path.join("python", "functions"))
from datascience.detect_corpus_language import detect_corpus_language
corpus = [
"este es un texto bastante largo en español para detectar el idioma correctamente",
"la inteligencia artificial transforma la manera en que trabajamos cada dia",
"this is a fairly long english text to detect the language correctly without issues",
]
out = detect_corpus_language(corpus)
# {"available": True, "n_detected": 3,
# "distribution": [{"lang": "es", "count": 2, "pct": 66.67},
# {"lang": "en", "count": 1, "pct": 33.33}],
# "dominant": "es"}
```
## Cuando usarla
Cuando perfiles una columna o corpus de texto en un EDA y necesites saber en
que idioma(s) esta escrito antes de elegir tokenizadores, stopwords, modelos
NLP o stemmers. Util tambien como check de calidad: detectar corpus mezclados
o un idioma inesperado. Llamala con la lista de textos crudos; la funcion
limpia, muestrea y resume sola.
## Gotchas
- `langdetect` es **opcional**: si no esta instalada, la funcion no lanza —
devuelve `{"available": False, "n_detected": 0, "distribution": [], "dominant": None}`.
Comprueba `out["available"]` antes de usar la distribucion.
- **Textos cortos** (pocas palabras o sin features lingüisticas) pueden no
detectarse: langdetect lanza `LangDetectException`, que se ignora y el doc no
cuenta en `n_detected`. Pasa frases razonablemente largas para resultados fiables.
- **Determinismo**: se fija `DetectorFactory.seed = 0` en cada llamada para que la
deteccion sea reproducible; sin esa semilla langdetect puede dar resultados
ligeramente distintos entre ejecuciones.
- `distribution` esta truncada a `top_k`; si el corpus tiene mas idiomas que
`top_k`, la suma de los `count` mostrados puede ser menor que `n_detected`
(pero `dominant` siempre refleja el idioma mas frecuente del corpus completo).
@@ -0,0 +1,91 @@
"""Detecta la distribucion de idiomas de un corpus de textos.
Funcion pura y defensiva: el computo es determinista y local (sin I/O de red).
La libreria opcional `langdetect` se importa de forma perezosa dentro de la
funcion; si no esta instalada (o cualquier paso falla), la funcion degrada
limpiamente a `available=False` y NUNCA lanza excepciones.
"""
def detect_corpus_language(texts, top_k=10, sample_max=1000) -> dict:
"""Estima la distribucion de idiomas de un corpus con `langdetect`.
Args:
texts: lista de strings (documentos). Los elementos None, no-str o
vacios tras strip se descartan.
top_k: numero maximo de idiomas a devolver en `distribution`,
ordenados por frecuencia descendente.
sample_max: numero maximo de documentos a clasificar (se toman los
primeros) para acotar el coste.
Returns:
dict con la forma fija (dict-no-throw):
{
"available": bool, # True si langdetect es importable
"n_detected": int, # documentos clasificados con exito
"distribution": [{"lang": str, "count": int, "pct": float}, ...],
"dominant": str | None,
}
"""
degraded = {
"available": False,
"n_detected": 0,
"distribution": [],
"dominant": None,
}
try:
# Import perezoso con degradacion: si langdetect no esta disponible,
# devolvemos el dict degradado sin lanzar.
try:
from langdetect import detect, DetectorFactory
# Semilla fija -> deteccion determinista entre ejecuciones.
DetectorFactory.seed = 0
except Exception:
return dict(degraded)
# Normaliza y filtra el corpus.
docs = []
if texts:
for t in texts:
if isinstance(t, str):
s = t.strip()
if s:
docs.append(s)
# Muestreo de los primeros `sample_max` documentos.
if sample_max is not None and sample_max >= 0:
docs = docs[:sample_max]
# Conteo por idioma; langdetect lanza LangDetectException en textos
# sin features detectables -> se ignora y se sigue.
counts: dict = {}
for doc in docs:
try:
lang = detect(doc)
except Exception:
continue
counts[lang] = counts.get(lang, 0) + 1
n_detected = sum(counts.values())
# Orden estable: por count descendente, desempate por codigo de idioma.
ordered = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))
k = top_k if (top_k is not None and top_k >= 0) else len(ordered)
distribution = []
for lang, count in ordered[:k]:
pct = round(count / n_detected * 100, 2) if n_detected else 0.0
distribution.append({"lang": lang, "count": count, "pct": pct})
dominant = ordered[0][0] if ordered else None
return {
"available": True,
"n_detected": n_detected,
"distribution": distribution,
"dominant": dominant,
}
except Exception:
# Cualquier fallo global degrada a available False sin lanzar.
return dict(degraded)
@@ -0,0 +1,58 @@
"""Tests para detect_corpus_language."""
import builtins
import os
import sys
# Anade python/functions a sys.path para importar el paquete `datascience`.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from datascience.detect_corpus_language import detect_corpus_language
_ES = [
"este es un texto bastante largo en español para detectar el idioma correctamente sin problemas",
"la inteligencia artificial transforma la manera en que trabajamos cada dia en muchos sectores",
]
_EN = [
"this is a fairly long english text to detect the language correctly without any length issues",
"machine learning models can classify documents into many different categories quite reliably",
]
def test_mixto_es_en():
"""Golden: corpus mixto ES+EN claro -> available True, >=2 idiomas, counts coherentes."""
out = detect_corpus_language(_ES + _EN)
assert out["available"] is True
assert out["dominant"] in {"es", "en"}
assert len(out["distribution"]) >= 2
total = sum(item["count"] for item in out["distribution"])
assert total == out["n_detected"]
assert out["n_detected"] == 4
def test_vacio():
"""Edge: lista vacia con langdetect presente -> available True, sin detecciones."""
out = detect_corpus_language([])
assert out["available"] is True
assert out["n_detected"] == 0
assert out["distribution"] == []
assert out["dominant"] is None
def test_degradacion(monkeypatch):
"""Error path: si langdetect no es importable -> degrada a available False sin lanzar."""
import datascience.detect_corpus_language as m
real_import = builtins.__import__
def fake_import(name, *a, **k):
if name == "langdetect" or name.startswith("langdetect."):
raise ImportError("simulado")
return real_import(name, *a, **k)
monkeypatch.setattr(builtins, "__import__", fake_import)
out = m.detect_corpus_language(["hola mundo", "hello world"])
assert out["available"] is False
assert out["n_detected"] == 0
assert out["distribution"] == []
assert out["dominant"] is None
@@ -0,0 +1,107 @@
---
name: detect_declared_keys_duckdb
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def detect_declared_keys_duckdb(db_path: str, table: str = None) -> dict"
description: "Detecta las claves DECLARADAS (constraints reales) de un schema DuckDB leyendo la table function duckdb_constraints(): extrae PRIMARY KEY, FOREIGN KEY y UNIQUE (ignora NOT NULL y CHECK) y las devuelve normalizadas con sus columnas, y para las FK con su tabla y columnas referenciadas. Con table=None procesa todas las tablas; con table='X' filtra a PK/UNIQUE de X y a FK cuyo origen es X (case-sensitive). A diferencia de infer_fk_containment_duckdb (que INFIERE FKs candidatas por containment de valores cuando el schema no las declara), esta funcion devuelve las relaciones de clave REALES del schema. Estilo dict-no-throw: nunca lanza. Parte del grupo eda (relaciones de clave)."
tags: [eda, duckdb, datascience, relations, primary-key, foreign-key, schema, exploratory-data-analysis]
params:
- name: db_path
desc: "Ruta al archivo DuckDB. Debe existir (lectura read-only via duckdb_query_readonly; no se crea). Un path inexistente devuelve {status:'error', ...}."
- name: table
desc: "Si se pasa, filtra los resultados a esa tabla: incluye PRIMARY KEY y UNIQUE cuya tabla sea `table`, y FOREIGN KEY cuya tabla ORIGEN sea `table` (no la referenciada). None (default) devuelve los constraints de todas las tablas. La comparacion es case-sensitive (nombres tal cual los devuelve DuckDB)."
output: "dict dict-no-throw. En exito {status:'ok', primary_keys:[{table:str, columns:[str,...]}, ...], foreign_keys:[{table:str, columns:[str,...], referenced_table:str, referenced_columns:[str,...]}, ...], unique:[{table:str, columns:[str,...]}, ...], tables:[str,...]} donde tables es la lista ordenada de tablas (origen) que poseen al menos un constraint PK/FK/UNIQUE emitido. Solo se emiten constraints de clave: NOT NULL y CHECK se ignoran. En error {status:'error', error:str}."
uses_functions: [duckdb_query_readonly_py_infra]
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
tested: true
tests: ["test_golden_detecta_pks_y_fk", "test_golden_ignora_not_null_y_check", "test_edge_filtra_por_tabla_orders", "test_edge_filtra_por_tabla_customers", "test_edge_unique_declarado", "test_edge_sin_constraints_listas_vacias", "test_error_db_inexistente_no_lanza", "test_shape_resultado"]
test_file_path: "python/functions/datascience/detect_declared_keys_duckdb_test.py"
file_path: "python/functions/datascience/detect_declared_keys_duckdb.py"
---
## Ejemplo
```python
import sys, os, duckdb
sys.path.insert(0, os.path.join("python", "functions"))
from datascience import detect_declared_keys_duckdb
# Base de ejemplo en /tmp: orders.customer_id -> customers.id (FK declarada)
path = "/tmp/declared_keys_demo.duckdb"
if os.path.exists(path):
os.remove(path)
con = duckdb.connect(path)
con.execute("CREATE TABLE customers(id INTEGER PRIMARY KEY, name TEXT)")
con.execute(
"CREATE TABLE orders("
" id INTEGER PRIMARY KEY,"
" customer_id INTEGER REFERENCES customers(id),"
" amt DOUBLE)"
)
con.close()
res = detect_declared_keys_duckdb(path)
if res["status"] == "ok":
for pk in res["primary_keys"]:
print(f"PK {pk['table']}({', '.join(pk['columns'])})")
for fk in res["foreign_keys"]:
print(f"FK {fk['table']}({', '.join(fk['columns'])}) -> "
f"{fk['referenced_table']}({', '.join(fk['referenced_columns'])})")
# PK customers(id)
# PK orders(id)
# FK orders(customer_id) -> customers(id)
else:
print("error:", res["error"])
# Filtrar a una tabla concreta (PK/UNIQUE de orders + FK con origen orders):
solo_orders = detect_declared_keys_duckdb(path, table="orders")
print(solo_orders["tables"]) # ['orders']
```
## Cuando usarla
- Cuando exploras un esquema DuckDB y quieres mostrar las relaciones de clave REALES (PK/FK/UNIQUE) que el schema ha declarado, sin inferir nada.
- Como paso del capitulo RELACIONES del grupo `eda`: primero mira las claves declaradas con esta funcion; si el schema no declara FKs, complementa con `infer_fk_containment_duckdb` (inferencia por containment).
- Antes de documentar o migrar un esquema, para listar el contrato de integridad referencial que el motor ya conoce.
- Para validar que las constraints que esperas (esa FK que creaste con `REFERENCES`) realmente estan declaradas en la base materializada.
## Gotchas
- **Impura**: lee de disco via la primitiva read-only `duckdb_query_readonly` (no crea ni modifica la base). El `db_path` debe existir; un path inexistente devuelve `{status:'error'}` (read_only NO crea la base).
- **Requiere `duckdb_constraints()`**: usa la table function `duckdb_constraints()`, disponible en DuckDB modernos (verificado en 1.5.2). En versiones antiguas sin esa funcion, la query falla y se devuelve `{status:'error'}`.
- **Solo claves DECLARADAS**: devuelve lo que el schema declaro con `PRIMARY KEY` / `FOREIGN KEY (... REFERENCES ...)` / `UNIQUE`. Una tabla materializada con `CREATE TABLE AS SELECT` NO lleva constraints — para esos casos no habra claves que mostrar y hay que INFERIRLAS (`infer_fk_containment_duckdb`).
- **NOT NULL y CHECK se ignoran**: `duckdb_constraints()` tambien emite filas `NOT NULL` (DuckDB genera una por cada columna PK) y `CHECK`; esta funcion las descarta y solo conserva PK/FK/UNIQUE.
- **Nombres case-sensitive**: el filtro `table='Orders'` no casa con una tabla `orders`. Se comparan los nombres tal cual los devuelve DuckDB.
- **FK atribuida al origen**: una FOREIGN KEY se atribuye a su tabla ORIGEN (el `table` de la entrada), no a la referenciada. El filtro `table='X'` trae las FK cuyo origen es X, no las que apuntan a X.
- **`tables` = tablas dueñas de constraints emitidos**: la lista `tables` contiene solo las tablas que poseen al menos un PK/FK/UNIQUE en el resultado (su campo `table`), ordenadas. No incluye tablas referenciadas que no tengan constraint propio en la salida.
- **Columnas como listas**: `constraint_column_names` y `referenced_column_names` son columnas LIST de DuckDB; en 1.5.2 llegan como listas Python. La funcion las normaliza a listas de strings con una red de seguridad por si llegaran como string.
## Notas
`duckdb_constraints()` devuelve una fila por constraint con los campos
`table_name`, `constraint_type`, `constraint_column_names`, `referenced_table`,
`referenced_column_names`. Mapeo a la salida:
```text
PRIMARY KEY -> primary_keys[]: {table, columns}
UNIQUE -> unique[]: {table, columns}
FOREIGN KEY -> foreign_keys[]: {table, columns, referenced_table, referenced_columns}
NOT NULL -> ignorado
CHECK -> ignorado
```
Para una FK, `referenced_table` y `referenced_column_names` vienen poblados; para
PK/UNIQUE, `referenced_table` es NULL y `referenced_column_names` una lista vacia.
Complementa a `infer_fk_containment_duckdb`: esta funcion devuelve las relaciones
de clave REALES del schema (declaradas); la otra INFIERE FKs candidatas por
containment de valores cuando el schema no las declaro. En el capitulo RELACIONES
de AutomaticEDA se usan en orden: primero las declaradas, luego la inferencia como
respaldo.
@@ -0,0 +1,127 @@
"""detect_declared_keys_duckdb — lee las claves DECLARADAS de un schema DuckDB.
Funcion impura: lee de disco a traves de la primitiva read-only del grupo
`duckdb` (duckdb_query_readonly). Pertenece al grupo de capacidad `eda`
(relaciones de clave): a diferencia de infer_fk_containment_duckdb, que INFIERE
FOREIGN KEYs candidatas por containment de valores, esta funcion devuelve las
constraints REALES que el schema ha declarado (PRIMARY KEY / FOREIGN KEY /
UNIQUE) leyendo la table function `duckdb_constraints()`.
Es la pieza del capitulo RELACIONES de AutomaticEDA que muestra las relaciones de
clave reales cuando existen frente a la inferencia, que se usa cuando el schema
no las declaro.
Estilo dict-no-throw del grupo duckdb: nunca lanza; captura cualquier error y
devuelve {status:'error', error:str}.
"""
from infra import duckdb_query_readonly
def _as_list(value) -> list:
"""Normaliza el valor de una columna LIST de DuckDB a una lista de strings.
En DuckDB 1.5.2, `constraint_column_names` y `referenced_column_names` llegan
ya como listas Python a traves de duckdb_query_readonly. Este helper es solo
una red de seguridad: si por cualquier motivo llegara como string (p.ej. la
representacion `[id, customer_id]`), la parsea de forma defensiva.
"""
if value is None:
return []
if isinstance(value, (list, tuple)):
return [str(v) for v in value]
if isinstance(value, str):
s = value.strip()
if s.startswith("[") and s.endswith("]"):
s = s[1:-1]
if not s.strip():
return []
return [
part.strip().strip("'\"")
for part in s.split(",")
if part.strip().strip("'\"")
]
return [str(value)]
def detect_declared_keys_duckdb(db_path: str, table: str = None) -> dict:
"""Detecta las claves PRIMARY KEY / FOREIGN KEY / UNIQUE declaradas en DuckDB.
Lee la table function `duckdb_constraints()` y extrae solo las constraints de
clave (PRIMARY KEY, FOREIGN KEY, UNIQUE), ignorando NOT NULL y CHECK.
Args:
db_path: ruta al archivo DuckDB. Debe existir (lectura read-only; no se
crea). Un path inexistente devuelve {status:'error', ...} sin lanzar.
table: si se pasa, filtra los resultados a esa tabla: incluye PRIMARY KEY
y UNIQUE cuya tabla sea `table`, y FOREIGN KEY cuya tabla ORIGEN sea
`table`. None (default) devuelve los constraints de todas las tablas.
La comparacion de nombres es case-sensitive (tal cual los devuelve
DuckDB).
Returns:
dict dict-no-throw. En exito:
{status:'ok',
primary_keys:[{table:str, columns:[str, ...]}, ...],
foreign_keys:[{table:str, columns:[str, ...],
referenced_table:str,
referenced_columns:[str, ...]}, ...],
unique:[{table:str, columns:[str, ...]}, ...],
tables:[str, ...]} # tablas (origen) con algun PK/FK/UNIQUE emitido
En error (sin lanzar): {status:'error', error:str}.
"""
try:
sql = (
"SELECT table_name, constraint_type, constraint_column_names, "
"referenced_table, referenced_column_names FROM duckdb_constraints()"
)
res = duckdb_query_readonly(db_path, sql)
if res["status"] != "ok":
return {"status": "error", "error": res["error"]}
primary_keys = []
foreign_keys = []
unique = []
tables = set()
for row in res["rows"]:
ctype = row["constraint_type"]
tname = row["table_name"]
# Filtro por tabla origen: para PK/FK/UNIQUE el dueño del constraint es
# `table_name`. Una FK se atribuye a su tabla origen (no a la
# referenciada), igual que el filtro pide.
if table is not None and tname != table:
continue
cols = _as_list(row["constraint_column_names"])
if ctype == "PRIMARY KEY":
primary_keys.append({"table": tname, "columns": cols})
tables.add(tname)
elif ctype == "UNIQUE":
unique.append({"table": tname, "columns": cols})
tables.add(tname)
elif ctype == "FOREIGN KEY":
foreign_keys.append(
{
"table": tname,
"columns": cols,
"referenced_table": row["referenced_table"],
"referenced_columns": _as_list(
row["referenced_column_names"]
),
}
)
tables.add(tname)
# NOT NULL y CHECK se ignoran: no son relaciones de clave.
return {
"status": "ok",
"primary_keys": primary_keys,
"foreign_keys": foreign_keys,
"unique": unique,
"tables": sorted(tables),
}
except Exception as e: # noqa: BLE001
return {"status": "error", "error": str(e)}
@@ -0,0 +1,167 @@
"""Tests para detect_declared_keys_duckdb."""
import duckdb
import pytest
from .detect_declared_keys_duckdb import detect_declared_keys_duckdb
@pytest.fixture
def db(tmp_path):
"""DuckDB temporal con claves declaradas.
- customers(id PRIMARY KEY, name)
- orders(id PRIMARY KEY, customer_id REFERENCES customers(id), amt)
Esto declara dos PRIMARY KEY (customers.id, orders.id) y una FOREIGN KEY
(orders.customer_id -> customers.id). DuckDB ademas genera constraints
NOT NULL para las columnas PK, que la funcion debe ignorar.
"""
path = str(tmp_path / "keys_test.duckdb")
con = duckdb.connect(path)
con.execute("CREATE TABLE customers(id INTEGER PRIMARY KEY, name TEXT)")
con.execute(
"CREATE TABLE orders("
" id INTEGER PRIMARY KEY,"
" customer_id INTEGER REFERENCES customers(id),"
" amt DOUBLE"
")"
)
con.close()
return path
def _pk_for(res, table):
"""Devuelve la entrada primary_keys cuya tabla es `table`, o None."""
for pk in res["primary_keys"]:
if pk["table"] == table:
return pk
return None
def test_golden_detecta_pks_y_fk(db):
"""Golden: detecta las dos PK y la FK declaradas, con valores concretos."""
res = detect_declared_keys_duckdb(db)
assert res["status"] == "ok"
# PRIMARY KEY de customers y de orders.
pk_customers = _pk_for(res, "customers")
pk_orders = _pk_for(res, "orders")
assert pk_customers is not None
assert pk_customers["columns"] == ["id"]
assert pk_orders is not None
assert pk_orders["columns"] == ["id"]
# FOREIGN KEY orders.customer_id -> customers.id.
assert len(res["foreign_keys"]) == 1
fk = res["foreign_keys"][0]
assert fk["table"] == "orders"
assert fk["columns"] == ["customer_id"]
assert fk["referenced_table"] == "customers"
assert fk["referenced_columns"] == ["id"]
# tables incluye ambas (origen de algun constraint).
assert res["tables"] == ["customers", "orders"]
def test_golden_ignora_not_null_y_check(db):
"""NOT NULL (auto-generado por las PK) no aparece como clave."""
res = detect_declared_keys_duckdb(db)
assert res["status"] == "ok"
# Solo 2 PK reales (no las NOT NULL que DuckDB genera por cada columna PK).
assert len(res["primary_keys"]) == 2
# No hay UNIQUE declarado en este schema.
assert res["unique"] == []
def test_edge_filtra_por_tabla_orders(db):
"""Edge table='orders': PK de orders + su FK; NO la PK de customers."""
res = detect_declared_keys_duckdb(db, table="orders")
assert res["status"] == "ok"
# Solo la PK de orders.
assert len(res["primary_keys"]) == 1
assert res["primary_keys"][0]["table"] == "orders"
assert res["primary_keys"][0]["columns"] == ["id"]
# La PK de customers NO esta.
assert _pk_for(res, "customers") is None
# La FK de orders si esta (origen = orders).
assert len(res["foreign_keys"]) == 1
assert res["foreign_keys"][0]["table"] == "orders"
assert res["foreign_keys"][0]["referenced_table"] == "customers"
# tables solo contiene orders (la dueña de los constraints emitidos).
assert res["tables"] == ["orders"]
def test_edge_filtra_por_tabla_customers(db):
"""Edge table='customers': solo su PK; ninguna FK (orders queda fuera)."""
res = detect_declared_keys_duckdb(db, table="customers")
assert res["status"] == "ok"
assert len(res["primary_keys"]) == 1
assert res["primary_keys"][0]["table"] == "customers"
assert res["foreign_keys"] == []
assert res["tables"] == ["customers"]
def test_edge_unique_declarado(tmp_path):
"""Edge: una constraint UNIQUE declarada aparece en `unique`."""
path = str(tmp_path / "unique_test.duckdb")
con = duckdb.connect(path)
con.execute("CREATE TABLE products(sku INTEGER UNIQUE, name TEXT)")
con.close()
res = detect_declared_keys_duckdb(path)
assert res["status"] == "ok"
assert len(res["unique"]) == 1
assert res["unique"][0]["table"] == "products"
assert res["unique"][0]["columns"] == ["sku"]
assert res["primary_keys"] == []
assert res["foreign_keys"] == []
assert res["tables"] == ["products"]
def test_edge_sin_constraints_listas_vacias(tmp_path):
"""Edge: tabla sin PK/FK/UNIQUE -> todas las listas vacias, status ok."""
path = str(tmp_path / "no_keys.duckdb")
con = duckdb.connect(path)
con.execute("CREATE TABLE log(a INTEGER, b INTEGER)")
con.close()
res = detect_declared_keys_duckdb(path)
assert res["status"] == "ok"
assert res["primary_keys"] == []
assert res["foreign_keys"] == []
assert res["unique"] == []
assert res["tables"] == []
def test_error_db_inexistente_no_lanza(tmp_path):
"""Error: db_path inexistente -> status error, sin lanzar excepcion."""
path = str(tmp_path / "does_not_exist.duckdb")
res = detect_declared_keys_duckdb(path)
assert res["status"] == "error"
assert isinstance(res["error"], str)
assert res["error"] != ""
def test_shape_resultado(db):
"""El retorno tiene exactamente las claves esperadas."""
res = detect_declared_keys_duckdb(db)
assert set(res.keys()) == {
"status",
"primary_keys",
"foreign_keys",
"unique",
"tables",
}
for pk in res["primary_keys"]:
assert set(pk.keys()) == {"table", "columns"}
for fk in res["foreign_keys"]:
assert set(fk.keys()) == {
"table",
"columns",
"referenced_table",
"referenced_columns",
}
@@ -0,0 +1,68 @@
---
name: detect_time_column
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: pure
signature: "def detect_time_column(columns: list) -> dict"
description: "Detecta, a partir de la lista de ColumnProfile de un TableProfile del grupo eda, cual es la columna de orden temporal y que columnas numericas hay para graficar una serie en el tiempo. Una columna es temporal si inferred_type=='datetime' o semantic_type in {datetime_iso, date_eu}; time_col es la primera temporal en orden. Es la pieza que usa el capitulo TIMESERIES del AutomaticEDA para decidir si aplica. Lectura defensiva dict-no-throw: nunca lanza, siempre devuelve las mismas claves."
tags: [eda, timeseries, datetime, profiling, column-detection, automatic-eda, datascience, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: ""
imports: []
params:
- name: columns
desc: "lista de ColumnProfile dict de un TableProfile del grupo eda. Cada elemento suele tener name, inferred_type, semantic_type y numeric. Elementos que no sean dict se ignoran; None/no-lista/vacia -> dict 'no aplica'."
output: "dict SIEMPRE con: time_col (str|None, columna temporal elegida = primera temporal), time_semantic (str, semantic_type de la temporal o ''), numeric_cols (list[str], columnas con inferred_type=='numeric' en orden), n_datetime_cols (int), datetime_cols (list[str], todas las temporales en orden de aparicion), reason (str en espanol explicando la eleccion). Nunca lanza excepcion."
tested: true
tests: ["test_golden_datetime_y_numericas", "test_deteccion_por_semantic_type_date_eu", "test_sin_columna_temporal", "test_columns_none_no_revienta", "test_columns_vacia_no_revienta", "test_columns_no_lista_no_revienta", "test_elementos_basura_se_ignoran", "test_varias_datetime_elige_la_primera"]
test_file_path: "python/functions/datascience/detect_time_column_test.py"
file_path: "python/functions/datascience/detect_time_column.py"
---
## Ejemplo
```python
from datascience import detect_time_column
columns = [
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
{"name": "ventas", "inferred_type": "numeric"},
{"name": "unidades", "inferred_type": "numeric"},
{"name": "region", "inferred_type": "text"},
]
res = detect_time_column(columns)
res["time_col"] # -> "fecha"
res["numeric_cols"] # -> ["ventas", "unidades"]
res["n_datetime_cols"] # -> 1
# Sin columna temporal: el capitulo TIMESERIES no aplica.
detect_time_column([{"name": "id", "inferred_type": "numeric"}])["time_col"] # -> None
```
## Cuando usarla
Cuando el capitulo TIMESERIES del AutomaticEDA recibe un TableProfile y necesita
decidir si la tabla admite analisis de serie temporal: si `time_col` es None no
hay eje de tiempo y el capitulo se salta; si hay `time_col` y `numeric_cols`,
úsalas como eje X (orden cronologico) y series Y. Tambien sirve para enrutar el
resto del pipeline (acf_pacf / stl_decompose / adf_kpss_stationarity) sobre las
columnas numericas detectadas.
## Gotchas
- Es pura y stdlib-only (sin numpy ni DuckDB): segura de llamar en cualquier paso.
- `time_col` se elige por ORDEN de aparicion en la lista, no por "mejor candidata".
Si hay varias columnas datetime y quieres otra, filtra `datetime_cols` tu mismo.
- Solo mira metadatos del perfil (`inferred_type`/`semantic_type`); no parsea ni
valida los valores reales de la columna. La calidad de la deteccion depende de
que el profiler (summarize_table_duckdb / infer_semantic_type) haya inferido bien.
- Las claves del semantic_type son exactamente las del profiler: `datetime_iso`
(ISO 8601) y `date_eu` (DD/MM/AAAA). Otros formatos de fecha no se detectan por
semantic_type salvo que `inferred_type` ya sea `"datetime"`.
- `numeric_cols` se basa en `inferred_type == "numeric"` (no en "integer"/"float");
si tu profiler usa otra etiqueta, normalizala antes.
@@ -0,0 +1,112 @@
"""Detecta la columna temporal y las columnas numericas de un TableProfile (grupo eda).
Funcion pura y determinista: a partir de la lista de columnas de un TableProfile
producido por el grupo de capacidad `eda` (cada elemento es un ColumnProfile dict),
decide cual es la columna de orden temporal y que columnas numericas hay disponibles
para graficar una serie en el tiempo. Es la pieza que usa el capitulo TIMESERIES del
AutomaticEDA para decidir si la tabla admite analisis de serie temporal.
Lectura 100% defensiva al estilo "dict-no-throw" del grupo eda: nunca lanza
excepcion, siempre devuelve el mismo conjunto de claves.
"""
# semantic_type que el profiler (infer_semantic_type) emite para fechas/datetimes.
_DATETIME_SEMANTICS = ("datetime_iso", "date_eu")
def detect_time_column(columns: list) -> dict:
"""Detecta la columna temporal y las numericas de una lista de ColumnProfile.
Recorre los ColumnProfile de un TableProfile y clasifica cada columna como
temporal o numerica leyendo de forma defensiva sus claves. Una columna es
temporal si su ``inferred_type == "datetime"`` o si su ``semantic_type`` esta
en {``"datetime_iso"``, ``"date_eu"``}. La columna temporal elegida
(``time_col``) es la PRIMERA temporal en el orden de la lista. Las numericas
(``numeric_cols``) son las de ``inferred_type == "numeric"``, en orden.
Funcion pura: no hace I/O, no muta el input, es determinista.
Args:
columns: lista de ColumnProfile dict del grupo eda. Cada elemento suele
tener claves como ``name``, ``inferred_type``, ``semantic_type`` y
``numeric``. Los elementos que no sean dict se ignoran. Si ``columns``
es None, no es lista o esta vacia, se devuelve el dict "no aplica".
Returns:
Siempre un dict con las mismas claves::
{
"time_col": str | None, # columna temporal elegida (None si no hay)
"time_semantic": str, # semantic_type de la temporal ("" si no aplica)
"numeric_cols": [str, ...], # columnas con inferred_type == "numeric"
"n_datetime_cols": int, # nº de columnas temporales detectadas
"datetime_cols": [str, ...],# todas las temporales, en orden de aparicion
"reason": str, # frase corta (en espanol) que explica la eleccion
}
"""
# Caso "no aplica": entrada invalida o vacia.
if not isinstance(columns, list) or not columns:
return {
"time_col": None,
"time_semantic": "",
"numeric_cols": [],
"n_datetime_cols": 0,
"datetime_cols": [],
"reason": "no se detecto columna de fecha/datetime",
}
datetime_cols: list[str] = []
datetime_semantics: list[str] = []
numeric_cols: list[str] = []
for col in columns:
# Ignora elementos que no sean dict sin fallar.
if not isinstance(col, dict):
continue
name = col.get("name")
if name is None:
name = ""
else:
name = str(name)
inferred_type = col.get("inferred_type") or ""
semantic_type = col.get("semantic_type") or ""
is_datetime = inferred_type == "datetime" or semantic_type in _DATETIME_SEMANTICS
if is_datetime:
datetime_cols.append(name)
datetime_semantics.append(semantic_type)
if inferred_type == "numeric":
numeric_cols.append(name)
if not datetime_cols:
return {
"time_col": None,
"time_semantic": "",
"numeric_cols": numeric_cols,
"n_datetime_cols": 0,
"datetime_cols": [],
"reason": "no se detecto columna de fecha/datetime",
}
time_col = datetime_cols[0]
time_semantic = datetime_semantics[0]
if len(datetime_cols) == 1:
reason = f"columna temporal '{time_col}' detectada"
else:
reason = (
f"{len(datetime_cols)} columnas temporales; se elige la primera "
f"'{time_col}'"
)
return {
"time_col": time_col,
"time_semantic": time_semantic,
"numeric_cols": numeric_cols,
"n_datetime_cols": len(datetime_cols),
"datetime_cols": datetime_cols,
"reason": reason,
}
@@ -0,0 +1,102 @@
"""Tests para detect_time_column (grupo eda). Self-contained, sin DuckDB."""
from detect_time_column import detect_time_column
def test_golden_datetime_y_numericas():
columns = [
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
{"name": "ventas", "inferred_type": "numeric"},
{"name": "unidades", "inferred_type": "numeric"},
{"name": "region", "inferred_type": "text"},
]
res = detect_time_column(columns)
assert res["time_col"] == "fecha"
assert res["time_semantic"] == "datetime_iso"
assert res["numeric_cols"] == ["ventas", "unidades"]
assert res["n_datetime_cols"] == 1
assert res["datetime_cols"] == ["fecha"]
assert isinstance(res["reason"], str) and res["reason"]
def test_deteccion_por_semantic_type_date_eu():
# inferred_type no es datetime, pero semantic_type date_eu => temporal.
columns = [
{"name": "id", "inferred_type": "numeric"},
{"name": "dia", "inferred_type": "text", "semantic_type": "date_eu"},
{"name": "importe", "inferred_type": "numeric"},
]
res = detect_time_column(columns)
assert res["time_col"] == "dia"
assert res["time_semantic"] == "date_eu"
assert res["numeric_cols"] == ["id", "importe"]
assert res["n_datetime_cols"] == 1
assert res["datetime_cols"] == ["dia"]
def test_sin_columna_temporal():
columns = [
{"name": "id", "inferred_type": "numeric"},
{"name": "nombre", "inferred_type": "text"},
{"name": "activo", "inferred_type": "boolean"},
]
res = detect_time_column(columns)
assert res["time_col"] is None
assert res["time_semantic"] == ""
assert res["numeric_cols"] == ["id"]
assert res["n_datetime_cols"] == 0
assert res["datetime_cols"] == []
assert res["reason"] == "no se detecto columna de fecha/datetime"
def test_columns_none_no_revienta():
res = detect_time_column(None)
assert res["time_col"] is None
assert res["time_semantic"] == ""
assert res["numeric_cols"] == []
assert res["n_datetime_cols"] == 0
assert res["datetime_cols"] == []
assert res["reason"] == "no se detecto columna de fecha/datetime"
def test_columns_vacia_no_revienta():
res = detect_time_column([])
assert res["time_col"] is None
assert res["numeric_cols"] == []
assert res["n_datetime_cols"] == 0
def test_columns_no_lista_no_revienta():
# Un dict (no lista) tambien debe caer en el caso "no aplica".
res = detect_time_column({"name": "fecha", "inferred_type": "datetime"})
assert res["time_col"] is None
assert res["numeric_cols"] == []
def test_elementos_basura_se_ignoran():
columns = [
None,
"no soy un dict",
42,
{"name": "ts", "inferred_type": "datetime"},
{"name": "valor", "inferred_type": "numeric"},
]
res = detect_time_column(columns)
assert res["time_col"] == "ts"
assert res["numeric_cols"] == ["valor"]
assert res["n_datetime_cols"] == 1
def test_varias_datetime_elige_la_primera():
columns = [
{"name": "created_at", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
{"name": "metric", "inferred_type": "numeric"},
{"name": "updated_at", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
{"name": "fecha_baja", "inferred_type": "text", "semantic_type": "date_eu"},
]
res = detect_time_column(columns)
assert res["time_col"] == "created_at"
assert res["time_semantic"] == "datetime_iso"
assert res["n_datetime_cols"] == 3
assert res["datetime_cols"] == ["created_at", "updated_at", "fecha_baja"]
assert res["numeric_cols"] == ["metric"]
@@ -0,0 +1,103 @@
---
id: draw_join_graph_figure_py_datascience
name: draw_join_graph_figure
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def draw_join_graph_figure(join_graph: dict, title: str = None) -> \"matplotlib.figure.Figure\""
description: "Rasteriza el join graph de una base (relaciones FK inter-tabla, salida de build_join_graph) a un matplotlib.figure.Figure: nodos circulares con el nombre de cada tabla (hubs en color de acento cálido, el resto neutro) y aristas dirigidas etiquetadas from_col→to_col (más la cardinalidad si viene). Es la contrapartida dibujada del string Mermaid para que el capítulo de relaciones del informe AutomaticEDA muestre un diagrama real. Layout networkx spring_layout determinista (seed=42), backend Agg sin abrir ventanas; defensivo: nunca lanza y nunca hace I/O."
tags: [eda, plot, relations, graph, matplotlib, figure, networkx, datascience, impure]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: [matplotlib, networkx]
example: |
from draw_join_graph_figure import draw_join_graph_figure
join_graph = {
"nodes": [
{"table": "customers", "out_degree": 0, "in_degree": 1, "role": "dimension"},
{"table": "orders", "out_degree": 1, "in_degree": 0, "role": "fact"},
],
"edges": [
{"from_table": "orders", "from_col": "customer_id",
"to_table": "customers", "to_col": "id", "cardinality": "N:1"},
],
"hubs": ["orders"],
}
fig = draw_join_graph_figure(join_graph, title="Relaciones FK")
fig.savefig("/tmp/join_graph.png")
tested: true
tests:
- "test_returns_figure_with_axis"
- "test_savefig_produces_nonempty_png"
- "test_empty_dict_does_not_raise_and_savefig_png"
- "test_none_does_not_raise_and_savefig_png"
test_file_path: "python/functions/datascience/draw_join_graph_figure_test.py"
file_path: "python/functions/datascience/draw_join_graph_figure.py"
params:
- name: join_graph
desc: "Dict producido por build_join_graph. Claves: `nodes` (list[dict] con table, out_degree, in_degree, role), `edges` (list[dict] con from_table, from_col, to_table, to_col y opcional cardinality/inclusion) y `hubs` (list[str] de tablas hub a destacar en color cálido). Claves ausentes, items no-dict, None o {} se toleran (devuelve Figure con texto, sin lanzar). Los nombres de nodo se derivan también de las aristas, así que un grafo con edges pero sin nodes explícitos igual se dibuja."
- name: title
desc: "Título dibujado sobre el diagrama. Si se omite (None) se usa \"Join graph\". Default None."
output: "Un matplotlib.figure.Figure (figsize 7x5) con un único Axes que contiene el diagrama node-link dirigido: tablas como nodos circulares etiquetados (hubs en acento cálido #DD8452, resto en azul neutro #4C72B0) y FKs como flechas dirigidas con etiqueta from_col→to_col (+ cardinalidad). Si join_graph no tiene nodos ni aristas (o es None/{}), devuelve igualmente una Figure con el texto centrado \"Sin relaciones FK detectadas.\"; ante cualquier fallo interno devuelve una Figure con un mensaje genérico (nunca lanza). El caller rasteriza/cierra la figura; la función no la muestra ni la guarda."
---
## Ejemplo
```python
from draw_join_graph_figure import draw_join_graph_figure
# `join_graph` es la salida de build_join_graph (nodes + edges + hubs).
join_graph = {
"nodes": [
{"table": "customers", "out_degree": 0, "in_degree": 1, "role": "dimension"},
{"table": "orders", "out_degree": 2, "in_degree": 0, "role": "fact"},
{"table": "products", "out_degree": 0, "in_degree": 1, "role": "dimension"},
],
"edges": [
{"from_table": "orders", "from_col": "customer_id",
"to_table": "customers", "to_col": "id", "cardinality": "N:1"},
{"from_table": "orders", "from_col": "product_id",
"to_table": "products", "to_col": "id", "cardinality": "N:1"},
],
"hubs": ["orders"], # `orders` se pinta en color de acento (tabla de hechos)
}
fig = draw_join_graph_figure(join_graph, title="Relaciones FK")
# El renderer del informe lo rasteriza; aquí solo persistimos para inspección.
fig.savefig("/tmp/join_graph.png")
```
## Cuando usarla
Úsala en el capítulo de relaciones de un informe AutomaticEDA cuando quieras un
diagrama **dibujado** del esquema relacional, no solo el bloque Mermaid pegable.
Pásale directamente la salida de `build_join_graph` (`nodes` + `edges` + `hubs`)
y obtienes una `matplotlib.figure.Figure` lista para que el renderer perezoso la
rasterice. Es la pareja visual del string Mermaid: Mermaid sirve para pegar en
Markdown/docs que lo soporten; esta función produce la imagen real (PNG/PDF) que
va embebida en informes que no renderizan Mermaid.
## Gotchas
- **Impura por matplotlib.** Fija el backend `Agg` al importar — no abre
ventanas ni depende de un display. Segura de llamar en lotes desde el
renderer.
- **Layout determinista (`seed=42`).** Usa `nx.spring_layout(G, seed=42)`, así
que la misma entrada produce el mismo diagrama (test reproducible). Para
grafos de 0/1 nodos usa una posición fija centrada en vez del spring layout.
- **No hace I/O.** No llama `plt.show()` ni guarda a disco — solo devuelve la
`Figure`. Quien la consume la rasteriza y la libera (`plt.close(fig)`) para no
acumular memoria en informes con muchas tablas.
- **Devuelve una Figure, NO un dict.** A diferencia de `build_join_graph` (que
devuelve el dict del grafo), esta función devuelve el objeto de figura ya
dibujado.
- **Defensiva, nunca lanza.** `None`, `{}`, claves ausentes o items malformados
se manejan sin error: en el peor caso devuelve una `Figure` con
"Sin relaciones FK detectadas." (vacío) o un mensaje genérico (fallo interno).
No la envuelvas en try/except por miedo a un raise — no lo hay.

Some files were not shown because too many files have changed in this diff Show More