Compare commits

...

6 Commits

Author SHA1 Message Date
egutierrez 7ec2bb1b45 feat(eda): el Markdown del AutomaticEDA vuelca TODOS los datos del profile
El .md del grupo `eda` es la salida pensada para pegar a un LLM, así que debe
contener todo lo que el motor computó, aunque el PDF/PPTX (vista humana) resuman.
La evaluación 2053 detectó 6 datos que el .md perdía respecto al profile. Se
cierran de forma aditiva (el .md tiene MÁS que el PDF/PPTX, sin tocar esos
renderers ni los capítulos).

render_automatic_eda.py pasa el profile al serializador Markdown vía
meta['profile'] (un meta propio del MD; el de PDF/PPTX queda intacto).
render_md_impl.py añade un "Apéndice — Datos completos del perfil" al final del
documento, emitido solo cuando hay profile y degradando limpio cuando falta una
sección (lite sin modelos, profile sin correlaciones). El apéndice no se acopla
a los ids de capítulo (que editan otros agentes en paralelo).

Pérdidas cerradas:
1. Matriz de asociación COMPLETA: los N pares de correlations.pairs (no solo el
   top-17), incluidos correlation_ratio (num↔cat) y cramers_v (cat↔cat).
2. Numéricas: describe completo por columna — mean/median/mode/std/variance/cv,
   skew y kurtosis para TODAS (no solo las asimétricas), p1/p5/p25/p50/p75/p95/
   p99, iqr, min/max, outliers, distribution_type.
3. Re-expresión: nombra la transformación concreta (log1p/sqrt/yeo-johnson) con
   potencia, razón y alternativas, no un vago "considerar re-expresión".
4. KMeans: tabla scores_by_k (silhouette + inercia por k) marcando el k elegido.
5. Normalidad: el estadístico (stat) de cada test junto al p-value.
6. Encabezados de figuras de barras/scree dejan de heredar
   "Desde/Hasta/Frecuencia" del histograma; usan "Inicio/Fin/Valor" cuando el
   caption no es un histograma.

Test nuevo md_completeness_test.py: profile sintético, asserta los N pares de
correlación, skew/kurtosis de cada numérica, percentiles extendidos, log1p,
scores_by_k, stat de normalidad, headers de barras y los edges (sin modelos /
sin correlaciones / sin profile, defensivo).

Verificado con titanic (profile_level=full): 28 pares en la tabla (incl.
Sex↔Embarked cramers_v), 7 numéricas con skew+kurtosis, p5/p95/p99, scores_by_k
y JB/D'Agostino/Shapiro stat presentes. PDF/PPTX/manifest siguen saliendo.
Suite automatic_eda + render_automatic_eda_test: 134 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:27:30 +02:00
egutierrez a1e2e3567c merge: 4c cat_distr una hoja por columna (PDF+PPTX 1:1) + sin descripcion entropia redundante + page_break motor (verificado met) 2026-06-30 19:53:57 +02:00
egutierrez 9be84a48ea merge: 4c quitar definiciones redundantes con glosario en calidad/correlacion/modelos/agregacion/relaciones (links intactos, verificado met) 2026-06-30 19:24:22 +02:00
egutierrez fd63261444 refactor(eda): quitar definiciones inline redundantes con el glosario en 5 capítulos
Ahora que el AutomaticEDA tiene un capítulo GLOSARIO con las definiciones de los
términos técnicos (enganchados como links clicables desde el cuerpo), los
capítulos calidad/correlacion/modelos/agregacion/relaciones ya no repiten inline
esas explicaciones largas: se deja el TÉRMINO marcado (clicable, sigue saltando
al glosario) y se elimina el párrafo/oración de definición redundante. Los
HALLAZGOS y datos concretos del análisis se mantienen intactos; solo se quitan
las definiciones generales que el glosario ya cubre.

- calidad: _criteria_intro pasa de un bullet-list con las definiciones de
  completitud/validez/unicidad/calidad + fórmula renormalizada + párrafo de
  outliers a una frase que nombra las dimensiones, sus pesos (60/40) y el
  principio de outliers; los 4 términos siguen marcados.
- modelos: la nota de normalización deja de explicar la fórmula del z-score; la
  intro de PCA ya no define "componentes ortogonales ordenados por varianza"; la
  de KMeans quita "rango −1 a 1: cuanto más alto..." (silhouette); la sección de
  Isolation Forest quita la descripción de árboles/cortes/umbral. Términos
  marcados intactos.
- correlacion: la intro deja de describir cada método y consolida la duplicación
  signo/dirección; los 4 métodos + FDR siguen marcados.
- agregacion: la intro quita la definición de pivot ("cruzan dos categóricas
  sobre una medida") y abrevia la selección de claves; groupby y pivot marcados.
- relaciones: la intro y la sección de candidatas/inter-tabla quitan las
  definiciones de PK ("identifica cada fila"), FK ("referencian a otra tabla") y
  containment ("valores contenidos en la clave de otra"); pk/fk/cardinalidad/
  containment siguen marcados.

Verificado sobre el EDA de titanic (run_models + run_llm, 48 págs): los 23 link
annotations término→glosario se conservan (PyMuPDF), el glosario mantiene las 20
definiciones, y el texto visible de los 5 capítulos baja un 34.7% en conjunto
(calidad −67%, modelos −33%, relaciones −19%, agregacion −15%, correlacion −8%).
Tests actualizados (calidad_test asertaba el texto viejo). Suite EDA + pipeline
verde (118 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 19:15:24 +02:00
egutierrez 4099d88eaf merge: 4b salida markdown del AutomaticEDA (render_md, render_automatic_eda emite aeda_md_path, verificado met) 2026-06-30 18:59:33 +02:00
egutierrez 48de3ce3da feat(eda): salida Markdown del AutomaticEDA para pegar a un LLM
Añade un tercer formato de salida al AutomaticEDA, junto al PDF y el PPTX:
un Markdown autocontenido del MISMO documento por capítulos
(chapters_registry.build_document), optimizado para incorporar a un LLM
(texto plano + tablas markdown reales, sin binarios incrustados).

- render_md_impl.render_md(chapters, out_path, meta): serializa los bloques
  del modelo (Heading/Markdown/KVTable/DataTable/Figure/Image/Caption/Note/
  Group/GlossaryEntry) a Markdown. Cabecera con metadatos + índice navegable
  con anclas GitHub; tablas volcadas enteras (el MD no pagina); marcadores de
  glosario eliminados conservando la negrita; glosario al final.
- Figuras: un LLM no ve la imagen, así que se prioriza texto + datos. Se emite
  el caption y, cuando la figura tiene barras (histograma), se extrae la tabla
  de bins (Desde/Hasta/Frecuencia) de los artistas matplotlib. La banda ±1σ
  (axvspan) se descarta por ancho para que no aparezca como un falso bin.
  PNG opcional vía meta['embed_figures'] (off por defecto → sin binarios).
- render_automatic_eda_markdown: función pública del registry (tag eda),
  espejo de render_automatic_eda_pdf/pptx, acepta lista de capítulos o un
  TableProfile (build_document). dict-no-throw.
- render_automatic_eda (pipeline): emite también el .md (emit_md=True por
  defecto, clave de retorno aeda_md_path). Cambio aditivo: PDF/PPTX/manifest
  siguen saliendo igual.

Tests: golden de todos los kinds + regresión del filtro de la banda ±1σ +
edge documento vacío + profile path. Suite del paquete y del pipeline verde
(122 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 18:52:08 +02:00
14 changed files with 1425 additions and 94 deletions
+2
View File
@@ -64,6 +64,7 @@ from .exploratory_caveats import exploratory_caveats
from .render_eda_pdf import render_eda_pdf, render_eda_pdf_relational
from .render_automatic_eda_pdf import render_automatic_eda_pdf
from .render_automatic_eda_pptx import render_automatic_eda_pptx
from .render_automatic_eda_markdown import render_automatic_eda_markdown
from .detect_time_column import detect_time_column
from .extract_timeseries_raw import extract_timeseries_raw
from .build_eda_render_ctx import build_eda_render_ctx
@@ -82,6 +83,7 @@ __all__ = [
"resample_timeseries",
"render_automatic_eda_pdf",
"render_automatic_eda_pptx",
"render_automatic_eda_markdown",
"decode_qr_image",
"adf_kpss_stationarity",
"acf_pacf",
@@ -36,6 +36,7 @@ from .model import ( # noqa: F401
from .chapters_registry import CHAPTER_ORDER, build_chapter, build_document # noqa: F401
from .render_pdf_impl import render_pdf # noqa: F401
from .render_pptx_impl import render_pptx # noqa: F401
from .render_md_impl import render_md # noqa: F401
__all__ = [
"ENGINE_NAME",
@@ -60,4 +61,5 @@ __all__ = [
"build_document",
"render_pdf",
"render_pptx",
"render_md",
]
@@ -561,13 +561,11 @@ def _intro_blocks(gloss=None, mark_term: bool = False) -> list:
t_groupby = _term(mark_term, "groupby", "**por grupos** (split-apply-combine)")
t_pivot = _term(mark_term, "pivot_table", "**tablas dinámicas** (pivot)")
text = (
f"Este capítulo analiza la tabla {t_groupby}: "
"elige las columnas categóricas más informativas por su cardinalidad "
"y relevancia, no todas contra todas, para no inflar comparaciones "
"espurias — y resume las variables numéricas dentro de cada grupo "
f"(conteo, media, mediana, desviación). Las {t_pivot} "
"cruzan dos categóricas sobre una medida, y los **gráficos de barras** "
"(siempre desde cero) comparan los grupos de un vistazo."
f"Este capítulo analiza la tabla {t_groupby}: elige las columnas "
"categóricas más informativas (por cardinalidad y relevancia, no todas "
"contra todas) y resume las variables numéricas dentro de cada grupo "
f"(conteo, media, mediana, desviación). Se añaden {t_pivot} y "
"**gráficos de barras** (siempre desde cero) para comparar los grupos."
)
return [model.Heading(text=CHAPTER_TITLE, level=1),
model.Markdown(text=text)]
@@ -3,12 +3,13 @@
Builds the quality chapter from a ``TableProfile`` of the ``eda`` group. The
chapter implements the quality model of report 2046:
1. **En qué se basa la calidad** — an intro paragraph explaining the two scored
1. **En qué se basa la calidad** — a concise intro naming the two scored
dimensions and their weights (completitud 60%, validez 40%) plus the
table-level row uniqueness, BEFORE any number, and stating explicitly that
outliers are reported as observations and do **not** lower the score. The
criteria terms (calidad de datos, completitud, validez, unicidad de registro)
are hooked into the shared glossary as clickable jumps.
table-level row uniqueness, BEFORE any number, and stating that outliers are
reported as observations and do **not** lower the score. The criteria terms
(calidad de datos, completitud, validez, unicidad de registro) are hooked
into the shared glossary as clickable jumps; their full definitions live in
the GLOSARIO chapter, not inline here.
2. **Scores por columna** — a table with, per column, the total quality score and
its breakdown into completeness / validity (no consistency dimension).
3. **Problemas de calidad** — a table listing ONLY real quality defects
@@ -309,30 +310,22 @@ def _term(key: str, label: str, mark: bool) -> str:
def _criteria_intro(mark: bool) -> str:
"""Intro paragraph explaining the two scored dimensions and the principle."""
"""Intro: how the score is composed, with every term marked clickable.
Concise on purpose: the definitions of each term (calidad de datos,
completitud, validez, unicidad de registro) now live in the GLOSARIO
chapter, so the body no longer repeats them — it only states how the score
is composed and keeps each term marked so it stays a clickable jump.
"""
calidad = _term("calidad_datos", "calidad de datos", mark)
completitud = _term("completitud", "Completitud (peso 60%)", mark)
validez = _term("validez", "Validez (peso 40%, cuando es medible)", mark)
completitud = _term("completitud", "completitud", mark)
validez = _term("validez", "validez", mark)
unicidad = _term("unicidad_registro", "unicidad de registro", mark)
return (
f"La {calidad} de cada columna es un score de 0 a 100 que combina solo "
"dimensiones medibles desde el perfil de la tabla, sin fuente externa "
"de verdad:\n\n"
f"- {completitud}: proporción de valores presentes (1 % de nulos; en "
"texto, las celdas vacías cuentan como faltantes). Los nulos y vacíos "
"bajan el score.\n"
f"- {validez}: proporción de valores que encajan con su tipo o formato "
"(un número que parsea, una fecha legible, un email con forma de email). "
"Si una columna es texto libre sin formato esperado, la validez no se "
"mide y el score se basa solo en la completitud.\n\n"
f"Score de columna = 100 × (0,6·completitud + 0,4·validez), "
"renormalizado cuando la validez no aplica. A nivel de tabla se añade "
f"la {unicidad} (1 % de filas duplicadas).\n\n"
"**Los valores atípicos (outliers) NO bajan la calidad.** Un valor "
"extremo puede ser real y correcto; detectar atípicos es parte del "
"análisis de la distribución, no un juicio de corrección. Por eso, junto "
"con las columnas constantes y los identificadores, se listan aparte "
"como **observaciones analíticas** que no afectan al score."
f"La {calidad} de cada columna es un score de 0 a 100 que combina "
f"{completitud} (peso 60%) y {validez} (peso 40%, cuando es medible); "
f"a nivel de tabla se añade la {unicidad}. Los valores atípicos no "
"bajan el score: se listan aparte como **observaciones analíticas**."
)
@@ -72,14 +72,16 @@ def test_golden_chapter_estructura_y_version():
assert "markdown" in kinds and "kv_table" in kinds and "data_table" in kinds
def test_golden_intro_explica_dos_dimensiones_y_pesos():
def test_golden_intro_nombra_dos_dimensiones_y_pesos():
# La intro nombra las dos dimensiones, sus pesos y la unicidad, pero ya NO
# repite sus definiciones largas: estas viven ahora en el capítulo GLOSARIO.
ch = build_calidad(_profile(), {})
intro = [b for b in ch.blocks if b.kind == "markdown"][0].text
for needle in ("Completitud", "Validez", "60%", "40%",
for needle in ("completitud", "validez", "60%", "40%",
"unicidad de registro"):
assert needle in intro, f"falta {needle!r} en la intro de criterios"
# El principio: los outliers NO bajan la calidad.
assert "atípicos" in intro and "NO bajan" in intro
assert "atípicos" in intro and "no bajan" in intro
# Ya no se menciona la dimensión consistencia eliminada.
assert "20%" not in intro
@@ -356,12 +356,11 @@ def build_correlacion(profile: dict, ctx: dict):
t_cramers = _term(mark_term, "cramers_v", "Cramér's V")
t_corr_ratio = _term(mark_term, "correlation_ratio", "razón de correlación")
blocks.append(model.Markdown(text=(
"Asociación entre columnas. Cada par se evalúa con la métrica adecuada a "
f"sus tipos ({t_pearson}/{t_spearman} entre numéricas — con **signo**; "
f"{t_cramers} entre categóricas; {t_corr_ratio} num-categórica; "
"información mutua como medida común no lineal). Sólo las correlaciones "
"**num-num** tienen dirección: por eso los pares **negativos** son siempre "
"num-num.")))
"Asociación entre columnas. Cada par se evalúa con la métrica adecuada "
f"a sus tipos: {t_pearson}/{t_spearman} (numéricas), {t_cramers} "
f"(categóricas), {t_corr_ratio} (num-categórica) e información mutua. "
"Sólo las correlaciones **num-num** llevan **signo** (dirección): por "
"eso los pares **negativos** son siempre num-num.")))
# 1) Association matrix (heatmap).
labels, trimmed = _ordered_labels(pairs)
@@ -6,15 +6,16 @@ normality}``). It renders, as structured markdown/tables/figures that the core
paginator never cuts:
1. **Normalization note** — every multivariate model below standardizes the
columns with z-score first; the chapter explains why (different scales would
otherwise dominate distance/variance).
columns with z-score first (the term is marked clickable; its definition
lives in the GLOSARIO chapter, not inline).
2. **PCA** — a scree plot (explained + cumulative variance, single Y axis) plus
variance and top-loadings tables.
3. **KMeans segments** — a PCA scatter **coloured by cluster** (its own
page/slide), the cluster-size table, and a per-cluster LLM micro-analysis
with a title for each segment.
4. **Isolation Forest outliers** — a short explanation of how anomalous rows are
isolated multivariately and how the threshold is chosen, plus the counts.
4. **Isolation Forest outliers** — the multivariate anomaly counts and decision
threshold (the method is marked clickable; its definition lives in the
GLOSARIO chapter, not inline).
5. **Normality** — per-column Jarque-Bera / D'Agostino / Shapiro verdicts.
The raw numeric data needed to colour the cluster scatter is **not** in the
@@ -314,12 +315,8 @@ def _normalization_intro(gloss=None, mark_term: bool = False) -> list:
text = (
"Estos modelos son **no supervisados**: buscan estructura latente sin "
"una variable objetivo. Antes de aplicarlos, todas las columnas "
f"numéricas se {zscore} (cada valor menos la media, dividido por la "
"desviación típica). Sin esta normalización, una variable con escala "
"grande (p.ej. ingresos en euros) dominaría las distancias y la varianza "
"frente a otra de escala pequeña (p.ej. un ratio entre 0 y 1), sesgando "
"tanto el PCA como el KMeans. Tras la estandarización todas las variables "
"pesan por igual."
f"numéricas se {zscore}, para que todas pesen por igual con "
"independencia de su escala."
)
return [model.Heading(text="Modelos no supervisados", level=1),
model.Markdown(text=text)]
@@ -334,11 +331,11 @@ def _pca_section(pca: dict, gloss=None, mark_term: bool = False) -> list:
n_used = pca.get("n_rows_used")
n_feat = pca.get("n_features")
intro = (
f"El {_term(mark_term, 'pca', 'PCA')} resume {_fmt_num(n_feat)} variables "
"numéricas en componentes ortogonales ordenados por la varianza que "
f"capturan ({_fmt_num(n_used)} filas usadas tras eliminar nulos). El "
"gráfico de sedimentación (scree) muestra cuánta varianza aporta cada "
"componente y su acumulado: un codo marca cuántos componentes bastan."
f"El {_term(mark_term, 'pca', 'PCA')} se aplica sobre "
f"{_fmt_num(n_feat)} variables numéricas ({_fmt_num(n_used)} filas "
"usadas tras eliminar nulos). El gráfico de sedimentación (scree) "
"muestra cuánta varianza aporta cada componente y su acumulado: un "
"codo marca cuántos componentes bastan."
)
blocks.append(model.Markdown(text=intro))
@@ -403,9 +400,8 @@ def _kmeans_section(kmeans: dict, projection: dict, titles,
t_sil = _term(mark_term, "silhouette", "*silhouette*")
intro = (
f"{t_kmeans} agrupa las filas en **{_fmt_num(best_k)} segmentos** "
f"elegidos automáticamente maximizando el coeficiente de {t_sil} "
f"(**{_fmt_num(sil)}**, rango 1 a 1: cuanto más alto, segmentos más "
"compactos y separados). Los segmentos se proyectan sobre el plano de "
f"elegidos automáticamente por el coeficiente de {t_sil} "
f"(**{_fmt_num(sil)}**). Los segmentos se proyectan sobre el plano de "
"los dos primeros componentes principales para visualizarlos."
)
blocks.append(model.Markdown(text=intro))
@@ -469,14 +465,10 @@ def _outliers_section(outliers: dict, gloss=None, mark_term: bool = False) -> li
level=2)]
isof = _term(mark_term, "isolation_forest", "**Isolation Forest**")
explain = (
f"{isof} detecta filas anómalas de forma *multivariante*: "
"construye árboles que parten el espacio con cortes aleatorios y mide "
"cuántos cortes hacen falta para aislar cada fila. Las filas raras "
"(combinaciones de valores poco frecuentes considerando **todas las "
"columnas a la vez**, no una sola) se aíslan con muy pocos cortes y "
"obtienen un score bajo. El **umbral** de decisión separa las filas "
"normales de las anómalas según la contaminación esperada del modelo: "
"una fila es outlier cuando su score queda por debajo de ese umbral."
f"{isof} marca filas anómalas de forma *multivariante*: combinaciones "
"de valores poco frecuentes considerando **todas las columnas a la "
"vez**, no una sola. La tabla resume cuántas se detectaron y el umbral "
"de decisión empleado."
)
blocks.append(model.Markdown(text=explain))
blocks.append(model.KVTable(rows=[
@@ -256,14 +256,14 @@ def _pk_candidates_section(profile: dict, mark: bool) -> list:
pk = ("[[term:pk]]**clave primaria**[[/term]]" if mark
else "**clave primaria**")
intro = (
f"Estas columnas son **candidatas a {pk}**: su "
"[[term:cardinalidad]]cardinalidad[[/term]] iguala al número de filas y no "
"tienen nulos, así que cada valor identifica una fila distinta. Son "
"candidatas, no una clave declarada: la base no las marca como tal."
f"Columnas **candidatas a {pk}**: su "
"[[term:cardinalidad]]cardinalidad[[/term]] iguala al número de filas y "
"no tienen nulos. Son candidatas, no una clave declarada: la base no "
"las marca como tal."
if mark else
"Estas columnas son **candidatas a clave primaria**: su cardinalidad "
"iguala al número de filas y no tienen nulos, así que cada valor "
"identifica una fila distinta.")
"Columnas **candidatas a clave primaria**: su cardinalidad iguala al "
"número de filas y no tienen nulos. Son candidatas, no una clave "
"declarada.")
rows = []
for name in keys:
@@ -320,10 +320,10 @@ def _inter_table_section(db_path: str, tables: list, mark: bool) -> list:
blocks = [
model.Heading(text="Claves foráneas candidatas (inter-tabla)", level=2),
model.Markdown(text=(
f"La fuente tiene varias tablas. Estas {fk_term} candidatas se infieren "
f"por señal de nombre y por {containment}: una columna de una tabla cuyos "
"valores están contenidos en la clave de otra. No están declaradas por "
"la base; son la relación más probable según los datos.")),
f"La fuente tiene varias tablas. Estas {fk_term} candidatas se "
f"infieren por señal de nombre y por {containment}. No están "
"declaradas por la base; son la relación más probable según los "
"datos.")),
]
shown = candidates[:MAX_FK_ROWS]
@@ -441,13 +441,12 @@ def _intro_blocks(mark: bool) -> list:
pk = "[[term:pk]]clave primaria[[/term]]" if mark else "clave primaria"
fk = "[[term:fk]]clave foránea[[/term]]" if mark else "clave foránea"
text = (
f"Este capítulo analiza las **relaciones de clave** de la tabla: qué columna "
f"identifica cada fila (la {pk}) y qué columnas referencian a otra tabla (las "
f"{fk}). Cuando la base las **declara** como restricciones del esquema, se "
"muestran tal cual; cuando no, se proponen las más probables a partir de los "
"datos —por inclusión de valores entre tablas (containment) o, en una sola "
"tabla, por una heurística de nombre y cardinalidad— siempre marcadas como "
"candidatas, nunca como hechos.")
f"Este capítulo analiza las **relaciones de clave** de la tabla: cuál es "
f"la {pk} y cuáles son las {fk}. Cuando la base las **declara** como "
"restricciones del esquema, se muestran tal cual; cuando no, se proponen "
"las más probables a partir de los datos —por containment entre tablas o, "
"en una sola tabla, por una heurística de nombre y cardinalidad— siempre "
"marcadas como candidatas, nunca como hechos.")
return [model.Heading(text=CHAPTER_TITLE, level=1), model.Markdown(text=text)]
@@ -0,0 +1,253 @@
"""Tests for the Markdown completeness appendix (report 2053).
The AutomaticEDA Markdown is the output meant to be *pasted into an LLM*, so it
must carry EVERYTHING the engine computed — even the numbers the human-facing
chapters (shared with the PDF/PPTX) drop for readability. ``render_md`` appends a
full-data appendix built from ``meta['profile']`` that closes the six losses the
evaluation found:
1. the complete association matrix (every pair, incl. correlation_ratio /
cramers_v) — not just the top extremes;
2. every numeric statistic for every numeric column (skew/kurtosis/percentiles);
3. the concrete recommended re-expression;
4. KMeans ``scores_by_k``;
5. the normality test statistics;
6. correct headers for bar/scree figure tables (not ``Desde/Hasta/Frecuencia``).
Self-contained: a synthetic profile, no DuckDB, no heavy renderer.
"""
import os
import sys
import pytest # noqa: F401
_HERE = os.path.dirname(os.path.abspath(__file__))
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
if _FUNCTIONS not in sys.path:
sys.path.insert(0, _FUNCTIONS)
from datascience.automatic_eda import model # noqa: E402
from datascience.automatic_eda.render_md_impl import ( # noqa: E402
_bars_table,
_is_histogram_caption,
_profile_appendix,
render_md,
)
# --------------------------------------------------------------------------- #
# Synthetic profile fixtures.
# --------------------------------------------------------------------------- #
def _numeric(skew, kurtosis):
"""A numeric stat block with every key the appendix serializes."""
return {
"count": 100, "min": 0.0, "max": 10.0, "mean": 5.0, "median": 5.0,
"mode": 4.0, "std": 2.0, "variance": 4.0, "cv": 0.4,
"p1": 0.1, "p5": 0.5, "p25": 2.5, "p50": 5.0, "p75": 7.5,
"p95": 9.5, "p99": 9.9, "iqr": 5.0, "skew": skew, "kurtosis": kurtosis,
"n_outliers": 1, "distribution_type": "normal",
}
def _profile():
"""A small but structurally faithful TableProfile (3 numeric, 2 categorical)."""
pairs = [
{"a": "A", "b": "B", "a_type": "numeric", "b_type": "numeric",
"method": "pearson/spearman", "value": 0.8,
"p_value": 1e-9, "p_value_adjusted": 2e-9, "significant": True},
{"a": "A", "b": "C", "a_type": "numeric", "b_type": "numeric",
"method": "pearson/spearman", "value": -0.3,
"p_value": 0.01, "p_value_adjusted": 0.02, "significant": True},
{"a": "A", "b": "Cat1", "a_type": "numeric", "b_type": "categorical",
"method": "correlation_ratio", "value": 0.45,
"p_value": 0.001, "p_value_adjusted": 0.002, "significant": True},
# The single cat-cat pair the human chapter never shows.
{"a": "Cat1", "b": "Cat2", "a_type": "categorical",
"b_type": "categorical", "method": "cramers_v", "value": 0.11,
"p_value": 0.04, "p_value_adjusted": 0.05, "significant": False},
]
return {
"correlations": {
"pairs": pairs,
"multiple_testing": {"method": "bh", "n_tests": 4, "n_rejected": 3},
},
"columns": [
{"name": "A", "count": 100, "numeric": _numeric(0.0, -1.2),
"reexpression": {"recommended": "none", "ladder_power": 1.0,
"reason": "symmetric", "alternatives": []}},
{"name": "B", "count": 100, "numeric": _numeric(4.77, 33.1),
"reexpression": {"recommended": "log1p", "ladder_power": 0.0,
"reason": "skew 4.77 with zeros",
"alternatives": [{"transform": "yeo-johnson"},
{"transform": "sqrt"}]}},
{"name": "C", "count": 100, "numeric": _numeric(-0.6, 0.2)},
{"name": "Cat1", "categorical": {"top": [], "mode": "x"}},
{"name": "Cat2", "categorical": {"top": [], "mode": "y"}},
],
"models": {
"kmeans": {
"best_k": 3,
"scores_by_k": [
{"k": 2, "silhouette": 0.46, "inertia": 900.0},
{"k": 3, "silhouette": 0.50, "inertia": 550.0},
{"k": 4, "silhouette": 0.38, "inertia": 430.0},
],
"cluster_sizes": [40, 35, 25],
},
"normality": {
"A": {"n": 100,
"jarque_bera": {"stat": 18.7, "p": 8e-5, "normal": False},
"dagostino": {"stat": 18.1, "p": 1e-4, "normal": False},
"shapiro": {"stat": 0.98, "p": 7e-8, "normal": False},
"is_normal": False},
"C": {"n": 100,
"jarque_bera": {"stat": 2.1, "p": 0.35, "normal": True},
"dagostino": {"stat": 1.9, "p": 0.38, "normal": True},
"shapiro": {"stat": 0.99, "p": 0.12, "normal": True},
"is_normal": True},
},
},
}
def _dummy_chapters():
"""A minimal one-chapter document so render_md does not early-return empty."""
return model.as_chapters([
{"id": "intro", "title": "Intro",
"blocks": [{"kind": "markdown", "text": "cuerpo del informe"}]},
])
def _render(tmp_path, profile):
out = os.path.join(str(tmp_path), "out.md")
res = render_md(_dummy_chapters(), out, {"title": "EDA — t", "profile": profile})
assert res["path"] == out
return open(out, encoding="utf-8").read()
def _table_rows(md, section_title):
"""Count data rows of the first Markdown table under ``section_title``."""
seg = md.split(section_title, 1)[1]
rows, in_t, seen_sep = 0, False, False
for ln in seg.splitlines():
if ln.startswith("|"):
in_t = True
stripped = ln.replace("|", "").replace(" ", "")
if stripped and set(stripped) == {"-"}:
seen_sep = True
continue
if seen_sep:
rows += 1
elif in_t and not ln.strip():
break
return rows
# --------------------------------------------------------------------------- #
# Golden: every datum the profile holds reaches the .md.
# --------------------------------------------------------------------------- #
def test_appendix_lists_all_correlation_pairs(tmp_path):
md = _render(tmp_path, _profile())
assert "## Apéndice — Datos completos del perfil" in md
# All 4 pairs (the real titanic profile has 28; here 4 synthetic).
assert _table_rows(md, "### Matriz de asociación") == 4
# The cat-cat Cramér's V pair the human chapter drops is present.
assert "Cat1 ↔ Cat2" in md
assert "cramers_v" in md
assert "correlation_ratio" in md
def test_appendix_has_skew_kurtosis_for_every_numeric(tmp_path):
md = _render(tmp_path, _profile())
seg = md.split("### Estadísticos numéricos completos", 1)[1].split("###", 1)[0]
lines = [l for l in seg.splitlines() if l.startswith("|")]
header = [h.strip() for h in lines[0].strip("|").split("|")]
assert "skew" in header and "kurtosis" in header
ski, kui = header.index("skew"), header.index("kurtosis")
data = lines[2:] # skip header + separator
assert len(data) == 3 # exactly the 3 numeric columns
for row in data:
cells = [c.strip() for c in row.strip("|").split("|")]
assert cells[ski] != "", f"missing skew in {cells[0]}"
assert cells[kui] != "", f"missing kurtosis in {cells[0]}"
def test_appendix_has_extended_percentiles(tmp_path):
md = _render(tmp_path, _profile())
seg = md.split("### Estadísticos numéricos completos", 1)[1]
header = [h.strip() for h in seg.splitlines()[2].strip("|").split("|")]
for p in ("p1", "p5", "p25", "p75", "p95", "p99"):
assert p in header, f"percentile {p} missing from describe header"
def test_appendix_names_concrete_reexpression(tmp_path):
md = _render(tmp_path, _profile())
assert "### Re-expresión recomendada" in md
assert "log1p" in md # the concrete transform, not just "consider re-expressing"
assert "yeo-johnson" in md # alternatives listed too
def test_appendix_has_kmeans_scores_by_k(tmp_path):
md = _render(tmp_path, _profile())
assert "scores_by_k" in md
assert _table_rows(md, "#### KMeans — selección de k") == 3 # k=2,3,4
def test_appendix_has_normality_statistics(tmp_path):
md = _render(tmp_path, _profile())
assert "JB stat" in md # the statistic, not only the p-value
assert "Shapiro stat" in md
assert _table_rows(md, "#### Tests de normalidad") == 2 # cols A and C
# --------------------------------------------------------------------------- #
# Edge: a profile missing models / correlations degrades, never raises.
# --------------------------------------------------------------------------- #
def test_lite_profile_without_models(tmp_path):
prof = _profile()
prof.pop("models") # lite: no KMeans/normality
md = _render(tmp_path, prof)
assert "scores_by_k" not in md # section skipped
assert "Matriz de asociación" in md # correlations still dumped
assert "## Apéndice" in md
def test_profile_without_correlations(tmp_path):
prof = _profile()
prof.pop("correlations")
md = _render(tmp_path, prof) # must not raise
assert "Matriz de asociación" not in md
assert "Estadísticos numéricos completos" in md # numeric section still there
def test_no_profile_means_no_appendix(tmp_path):
out = os.path.join(str(tmp_path), "noprof.md")
res = render_md(_dummy_chapters(), out, {"title": "x"})
assert res["path"] == out
assert "## Apéndice" not in open(out, encoding="utf-8").read()
def test_appendix_helper_is_defensive():
assert _profile_appendix(None) == ""
assert _profile_appendix({}) == ""
assert _profile_appendix({"columns": []}) == ""
# --------------------------------------------------------------------------- #
# Loss #6: bar/scree figure tables get a non-misleading header.
# --------------------------------------------------------------------------- #
def test_histogram_caption_detection():
assert _is_histogram_caption("Histograma de Age")
assert _is_histogram_caption("Distribución de Fare")
assert not _is_histogram_caption("Media de Survived por Sex")
assert not _is_histogram_caption("Varianza explicada (scree PCA)")
def test_bars_table_custom_header():
bars = [(0.0, 1.0, 5.0), (1.0, 2.0, 3.0)]
hist = _bars_table(bars) # default histogram header
assert "| Desde | Hasta | Frecuencia |" in hist
bar = _bars_table(bars, ("Inicio", "Fin", "Valor"))
assert "| Inicio | Fin | Valor |" in bar
assert "Frecuencia" not in bar
@@ -0,0 +1,748 @@
"""AutomaticEDA Markdown serializer — one self-contained file to paste to an LLM.
Same document model as the PDF/PPTX renderers (an ordered list of
:class:`Chapter`, each a list of format-independent blocks) but emitted as plain
**Markdown** instead of a binary. The goal is different from the other two
renderers: a Markdown EDA is meant to be *pasted into an LLM*, so it prioritises
TEXT and DATA over visuals. Tables become Markdown tables (every row dumped, no
pagination — nothing is cut because there are no pages); a ``Figure`` becomes its
caption plus, when possible, the underlying bar/histogram data as a Markdown
table (an LLM cannot see the image); glossary term markers are stripped while
``**bold**`` is kept (it is valid Markdown).
dict-no-throw (the ``eda`` group style): :func:`render_md` never raises. On a
fatal error it returns ``{path: None, ...}`` with a ``note`` explaining why; a
malformed block degrades to a readable note rather than crashing the document.
"""
from __future__ import annotations
import os
import re
from . import model
# Glossary span markers (kept text, dropped markers). We intentionally do NOT use
# ``text_layout.strip_inline_md`` for Markdown blocks because that also removes
# ``**bold**`` — valid Markdown we want to preserve when pasting to an LLM.
_TERM_OPEN_RE = re.compile(r"\[\[term:[A-Za-z0-9_]+\]\]")
_MAX_BAR_ROWS = 100
# --------------------------------------------------------------------------- #
# Small helpers.
# --------------------------------------------------------------------------- #
def _clean_terms(s) -> str:
"""Drop glossary term markers, keeping the visible text (and any **bold**)."""
s = model._safe_str(s)
s = _TERM_OPEN_RE.sub("", s)
return s.replace("[[/term]]", "")
def _cell(v) -> str:
"""Render a value as a safe Markdown table cell.
Escapes pipes (``|`` -> ``\\|``) so they do not break the column layout and
folds newlines to ``<br>`` so a multi-line value stays inside one cell. None
becomes an empty string.
"""
s = model._safe_str(v)
s = s.replace("|", "\\|")
s = s.replace("\r\n", "\n").replace("\r", "\n").replace("\n", "<br>")
return s
def _slug(text: str) -> str:
"""GitHub-style heading anchor: lowercase, spaces->'-', drop other symbols."""
s = model._safe_str(text).strip().lower()
out = []
for ch in s:
if ch.isalnum():
out.append(ch)
elif ch in " -":
out.append("-")
# any other symbol is dropped.
slug = "".join(out)
while "--" in slug:
slug = slug.replace("--", "-")
return slug.strip("-")
def _fmt_num(v) -> str:
"""Compact number for the figure data tables (ints as ints, else 4 sig figs)."""
try:
f = float(v)
except Exception: # noqa: BLE001
return model._safe_str(v)
if f != f: # NaN
return "NaN"
if f == int(f) and abs(f) < 1e15:
return str(int(f))
return f"{f:.4g}"
def _fmt_int(v) -> str:
try:
return str(int(v))
except Exception: # noqa: BLE001
return model._safe_str(v)
def _now_iso() -> str:
from datetime import datetime, timezone
return datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
# --------------------------------------------------------------------------- #
# Document header (title + metadata blockquote + numbered index).
# --------------------------------------------------------------------------- #
def _meta_block(meta: dict) -> list:
"""Build the metadata lines for the header blockquote (omitting absentees)."""
ctx = meta.get("ctx") if isinstance(meta.get("ctx"), dict) else {}
lines: list = []
def add(label, value) -> None:
if value is None:
return
s = model._safe_str(value).strip()
if s and s.lower() != "none":
lines.append(f"**{label}:** {s}")
add("Dataset", ctx.get("dataset_name") or meta.get("dataset_name"))
add("Fuente", ctx.get("source_origin") or meta.get("source_origin"))
add("Almacenamiento", ctx.get("storage") or meta.get("storage"))
n_rows = ctx.get("n_rows", meta.get("n_rows"))
n_cols = ctx.get("n_cols", meta.get("n_cols"))
if n_rows is not None and n_cols is not None:
lines.append(
f"**Dimensiones:** {_fmt_int(n_rows)} filas × {_fmt_int(n_cols)} columnas")
add("Generado", meta.get("generated_at") or _now_iso())
lines.append(f"**Motor:** {model.ENGINE_NAME} v{model.ENGINE_VERSION}")
return lines
# --------------------------------------------------------------------------- #
# Per-block serializers. Each returns a Markdown string (no surrounding blanks;
# the caller separates blocks with a blank line).
# --------------------------------------------------------------------------- #
def _md_heading(block) -> str:
level = int(getattr(block, "level", 1) or 1)
hashes = "#" * min(level + 2, 6) # level1 -> ###; '#'/'##' reserved for doc/chapter.
text = _clean_terms(getattr(block, "text", "")).strip()
return f"{hashes} {text}"
def _md_markdown(block) -> str:
# Keep the text verbatim, dropping only glossary markers (keep **bold**).
return _clean_terms(getattr(block, "text", "")).rstrip("\n")
def _md_kv_table(block) -> str:
lines: list = []
title = getattr(block, "title", None)
if title:
lines.append(f"**{_clean_terms(title).strip()}**")
lines.append("")
lines.append("| Campo | Valor |")
lines.append("| --- | --- |")
for row in (getattr(block, "rows", []) or []):
try:
label, value = row[0], row[1]
except Exception: # noqa: BLE001
label, value = row, ""
lines.append(f"| {_cell(label)} | {_cell(value)} |")
return "\n".join(lines)
def _md_data_table(block) -> str:
lines: list = []
title = getattr(block, "title", None)
if title:
lines.append(f"**{_clean_terms(title).strip()}**")
lines.append("")
header = list(getattr(block, "header", []) or [])
rows = list(getattr(block, "rows", []) or [])
if not header:
ncol = max((len(r) for r in rows), default=1)
header = [f"col{i + 1}" for i in range(ncol)]
ncol = len(header)
lines.append("| " + " | ".join(_cell(h) for h in header) + " |")
lines.append("| " + " | ".join(["---"] * ncol) + " |")
for r in rows: # dump every row — no pagination, nothing cut.
cells = [_cell(r[c]) if c < len(r) else "" for c in range(ncol)]
lines.append("| " + " | ".join(cells) + " |")
note = getattr(block, "note", None)
if note:
lines.append("")
lines.append(f"*{_clean_terms(note).strip()}*")
return "\n".join(lines)
def _bars_table(bars: list, header: tuple = ("Desde", "Hasta", "Frecuencia")) -> str:
"""Render extracted bar/histogram data as a Markdown table.
``header`` is the 3-column header to use. Histogram bars are
``(Desde, Hasta, Frecuencia)``; bar/scree charts (means by group, PCA
explained variance) are *not* bins, so the caller passes a semantically
correct header (e.g. ``(Inicio, Fin, Valor)``) to avoid the misleading
"Frecuencia" label — see report 2053, loss #6.
"""
h0, h1, h2 = header
lines = [f"| {h0} | {h1} | {h2} |", "| --- | --- | --- |"]
shown = bars[:_MAX_BAR_ROWS]
for x0, x1, h in shown:
lines.append(f"| {_fmt_num(x0)} | {_fmt_num(x1)} | {_fmt_num(h)} |")
out = "\n".join(lines)
extra = len(bars) - len(shown)
if extra > 0:
out += f"\n\n*… ({extra} filas más)*"
return out
def _is_histogram_caption(caption: str) -> bool:
"""True when a figure caption describes a histogram (genuine numeric bins).
Histograms are the only figures whose bars are real ``[Desde, Hasta)`` bins
with a frequency count. Bar charts (means by group) and the PCA scree plot
carry per-category / per-component values, not bins — they must not inherit
the ``Desde/Hasta/Frecuencia`` header.
"""
c = (caption or "").lower()
return "histograma" in c or "distribución" in c or "distribucion" in c
def _extract_bars(fig) -> list:
"""Collect (x_from, x_to, height) of the rectangular bars of a matplotlib fig.
Histogram / bar-chart bars are ``matplotlib.patches.Rectangle`` with positive
width and height; spines, legends and zero-area artists are skipped. Never
raises — returns ``[]`` on any problem.
"""
bars: list = []
try:
for ax in fig.get_axes():
# Collect this axes' positive-area rectangles, then keep only the ones
# that look like actual histogram/bar bins. Reference shapes that
# matplotlib also stores in ``ax.patches`` — most notably the ``±1σ``
# band drawn by ``axvspan`` (a single rectangle far wider than a bin)
# and a lone Tukey boxplot box — would otherwise show up as fake
# "bins". A histogram axes has several near-equal-width bars, so we
# drop any rectangle whose width is more than twice the median width
# of that axes' rectangles (the σ-band spans many bins; uniform bins
# all sit at the median width and stay).
ax_bars: list = []
for patch in list(getattr(ax, "patches", []) or []):
try:
w = patch.get_width()
h = patch.get_height()
x = patch.get_x()
except Exception: # noqa: BLE001 — not a Rectangle-like patch.
continue
if w and w > 0 and h and h > 0:
ax_bars.append((x, x + w, h))
if len(ax_bars) >= 3:
widths = sorted(b[1] - b[0] for b in ax_bars)
median_w = widths[len(widths) // 2]
if median_w > 0:
ax_bars = [b for b in ax_bars
if (b[1] - b[0]) <= 2.0 * median_w]
bars.extend(ax_bars)
except Exception: # noqa: BLE001
return []
return bars
def _md_figure(block, meta: dict, out_path: str, counter: list) -> str:
"""Serialize a Figure prioritising TEXT + DATA (an LLM cannot see the image).
Emits the caption, then — if the matplotlib figure has bars — a Markdown table
of the underlying (Desde, Hasta, Frecuencia) values. Optionally (when
``meta['embed_figures']`` is True) also exports a PNG beside the .md and adds
an image link; off by default so the Markdown stays self-contained.
"""
caption = model._safe_str(getattr(block, "caption", "")).strip()
parts = [f"*Figura: {caption}*" if caption else "*Figura*"]
fig = None
try:
import matplotlib
matplotlib.use("Agg") # defensive: headless rasterization backend.
fig = getattr(block, "fig", None)
make = getattr(block, "make", None)
if fig is None and callable(make):
fig = make()
if fig is not None:
bars = _extract_bars(fig)
if bars:
# A histogram's bars are genuine numeric bins (Desde/Hasta/
# Frecuencia). Bar charts and the PCA scree plot are not bins —
# give them a header that does not lie about "Frecuencia".
header = (("Desde", "Hasta", "Frecuencia")
if _is_histogram_caption(caption)
else ("Inicio", "Fin", "Valor"))
parts.append(_bars_table(bars, header))
if meta.get("embed_figures"):
png = _embed_png(fig, out_path, counter)
if png:
parts.append(f"![{caption}]({png})")
except Exception: # noqa: BLE001 — a bad figure degrades to just its caption.
pass
finally:
if fig is not None:
try:
import matplotlib.pyplot as plt
plt.close(fig)
except Exception: # noqa: BLE001
pass
return "\n\n".join(parts)
def _embed_png(fig, out_path: str, counter: list) -> str:
"""Export the figure to ``<basename>_figN.png`` beside the .md; return its name."""
try:
counter[0] += 1
base = os.path.splitext(os.path.basename(out_path))[0] or "figura"
name = f"{base}_fig{counter[0]}.png"
path = os.path.join(os.path.dirname(os.path.abspath(out_path)), name)
fig.savefig(path, format="png", dpi=120, bbox_inches="tight")
return name
except Exception: # noqa: BLE001
return ""
def _md_image(block) -> str:
path = model._safe_str(getattr(block, "path", ""))
caption = model._safe_str(getattr(block, "caption", "")).strip()
out = f"![{caption}]({path})"
if caption:
out += f"\n\n*{caption}*"
return out
def _md_caption(block) -> str:
return f"*{_clean_terms(getattr(block, 'text', '')).strip()}*"
def _md_note(block) -> str:
text = _clean_terms(getattr(block, "text", "")).strip()
lines = text.split("\n")
return "\n".join((f"> {ln}" if ln.strip() else ">") for ln in lines)
def _md_group(block, meta: dict, out_path: str, counter: list) -> str:
parts: list = []
title = getattr(block, "title", None)
if title:
parts.append(f"### {_clean_terms(title).strip()}")
for b in (getattr(block, "blocks", []) or []):
try:
seg = _serialize_block(b, meta, out_path, counter)
except Exception: # noqa: BLE001
seg = ""
if seg:
parts.append(seg)
return "\n\n".join(parts)
def _md_glossary_entry(block) -> str:
label = (model._safe_str(getattr(block, "label", "")).strip()
or model._safe_str(getattr(block, "key", "")).strip())
definition = _clean_terms(getattr(block, "definition", "")).strip()
out = f"### {label}"
if definition:
out += f"\n\n{definition}"
return out
def _serialize_block(block, meta: dict, out_path: str, counter: list) -> str:
"""Dispatch a single block to its Markdown serializer. Unknown -> note."""
kind = getattr(block, "kind", "")
if kind == "heading":
return _md_heading(block)
if kind == "markdown":
return _md_markdown(block)
if kind == "kv_table":
return _md_kv_table(block)
if kind == "data_table":
return _md_data_table(block)
if kind == "figure":
return _md_figure(block, meta, out_path, counter)
if kind == "image":
return _md_image(block)
if kind == "caption":
return _md_caption(block)
if kind == "note":
return _md_note(block)
if kind == "group":
return _md_group(block, meta, out_path, counter)
if kind == "glossary_entry":
return _md_glossary_entry(block)
# Unknown content -> readable note (mirrors the model's defensive coercion).
return _md_note(model.Note(text=model._safe_str(block)))
# --------------------------------------------------------------------------- #
# Profile appendix — the data the human-facing chapters drop.
#
# The chapter document (shared with the PDF/PPTX renderers) is designed for human
# reading and intentionally omits raw numbers: the correlation matrix shows only
# the top extremes, the numeric blocks skip skew/kurtosis/extended percentiles,
# the model chapter does not list ``scores_by_k`` or the normality test
# statistics. But the Markdown is meant to be *pasted into an LLM*, so it should
# carry EVERYTHING the engine computed. This appendix serializes the full
# ``profile`` (passed via ``meta['profile']``) as Markdown tables, additively:
# the PDF/PPTX are untouched, the .md simply has more than they do. Each section
# is emitted only when its source data is present, so a ``lite`` profile (no
# models) or a profile without correlations degrades cleanly instead of raising.
# See report 2053 for the six losses this closes.
# --------------------------------------------------------------------------- #
def _pair_types(a_type, b_type) -> str:
"""Short ``num↔cat`` label for an association pair's variable types."""
def short(t):
t = model._safe_str(t).lower()
if t.startswith("num"):
return "num"
if t.startswith("cat"):
return "cat"
return t or "?"
return f"{short(a_type)}{short(b_type)}"
def _app_correlations(corr: dict) -> str:
"""Loss #1 — every association pair (not just the top extremes).
Dumps all of ``correlations['pairs']`` as a table (pair · types · method ·
value · p · p-FDR · significant), ordered by |value| desc so the strongest
associations lead while nothing is cut. Includes the ``correlation_ratio``
(num↔cat) and ``cramers_v`` (cat↔cat) pairs the human chapter never shows.
"""
pairs = list(corr.get("pairs", []) or [])
if not pairs:
return ""
def keyfn(p):
try:
return -abs(float(p.get("value")))
except Exception: # noqa: BLE001
return 0.0
pairs_sorted = sorted(pairs, key=keyfn)
lines = ["### Matriz de asociación — todos los pares",
"",
("| Par | Tipos | Método | Valor | p-value | p-ajustado (FDR) "
"| ¿Sig? |"),
"| --- | --- | --- | --- | --- | --- | --- |"]
for p in pairs_sorted:
par = f"{_cell(p.get('a'))}{_cell(p.get('b'))}"
types = _pair_types(p.get("a_type"), p.get("b_type"))
method = _cell(p.get("method"))
val = _fmt_num(p.get("value"))
pv = _fmt_num(p.get("p_value")) if p.get("p_value") is not None else ""
padj = (_fmt_num(p.get("p_value_adjusted"))
if p.get("p_value_adjusted") is not None else "")
sig = "" if p.get("significant") else "no"
lines.append(
f"| {par} | {types} | {method} | {val} | {pv} | {padj} | {sig} |")
mt = corr.get("multiple_testing") or {}
n_tests = mt.get("n_tests", corr.get("n_tests"))
n_rej = mt.get("n_rejected")
note_bits = [f"{len(pairs)} pares en total"]
if n_tests is not None and n_rej is not None:
note_bits.append(
f"{n_rej} de {n_tests} significativos tras corrección "
f"{model._safe_str(mt.get('method', 'FDR')).upper()}")
lines.append("")
lines.append(f"*{'; '.join(note_bits)}.*")
return "\n".join(lines)
# Numeric statistics, in serialization order: (profile key, column header).
_NUM_STATS = [
("count", "n"), ("mean", "mean"), ("median", "median"), ("mode", "mode"),
("std", "std"), ("variance", "variance"), ("cv", "cv"),
("skew", "skew"), ("kurtosis", "kurtosis"),
("min", "min"), ("p1", "p1"), ("p5", "p5"), ("p25", "p25"), ("p50", "p50"),
("p75", "p75"), ("p95", "p95"), ("p99", "p99"), ("iqr", "iqr"),
("max", "max"), ("n_outliers", "outliers"),
("distribution_type", "distribución"),
]
def _app_numeric_describe(columns: list) -> str:
"""Loss #2 — every numeric statistic for every numeric column.
One row per numeric column with the full describe: mean/median/mode/std/
variance/cv, skew & kurtosis (for ALL columns, not only the skewed ones),
p1/p5/p25/p50/p75/p95/p99, iqr, min/max, outliers and distribution_type.
"""
rows = []
for info in (columns or []):
num = info.get("numeric") if isinstance(info, dict) else None
if not num:
continue
name = _cell(info.get("name"))
cells = [name]
for key, _hdr in _NUM_STATS:
v = num.get("count" if key == "count" else key)
if key == "count":
v = num.get("count", info.get("count"))
if key == "distribution_type":
cells.append(_cell(v))
else:
cells.append(_fmt_num(v) if v is not None else "")
rows.append(cells)
if not rows:
return ""
header = ["Columna"] + [hdr for _k, hdr in _NUM_STATS]
lines = ["### Estadísticos numéricos completos (describe)",
"",
"| " + " | ".join(header) + " |",
"| " + " | ".join(["---"] * len(header)) + " |"]
for cells in rows:
lines.append("| " + " | ".join(cells) + " |")
return "\n".join(lines)
def _app_reexpression(columns: list) -> str:
"""Loss #3 — the concrete recommended re-expression per column.
Names the transform (log1p/sqrt/yeo-johnson/none) instead of a vague
"consider re-expressing", with the ladder power, reason and alternatives.
"""
rows = []
for info in (columns or []):
rx = info.get("reexpression") if isinstance(info, dict) else None
if not rx or not isinstance(rx, dict):
continue
rec = model._safe_str(rx.get("recommended")).strip()
if not rec:
continue
alts = rx.get("alternatives") or []
alt_txt = ", ".join(
model._safe_str(a.get("transform")) for a in alts
if isinstance(a, dict) and a.get("transform")) or ""
rows.append([
_cell(info.get("name")), _cell(rec),
_fmt_num(rx.get("ladder_power")) if rx.get("ladder_power") is not None else "",
_cell(rx.get("reason")), _cell(alt_txt),
])
if not rows:
return ""
lines = ["### Re-expresión recomendada (escalera de Tukey)",
"",
"| Columna | Recomendada | Potencia | Razón | Alternativas |",
"| --- | --- | --- | --- | --- |"]
for r in rows:
lines.append("| " + " | ".join(r) + " |")
return "\n".join(lines)
def _app_kmeans_scores(kmeans: dict) -> str:
"""Loss #4 — KMeans silhouette + inertia per k (justifies the chosen k)."""
scores = list(kmeans.get("scores_by_k", []) or [])
if not scores:
return ""
best_k = kmeans.get("best_k")
lines = ["#### KMeans — selección de k (`scores_by_k`)",
"",
"| k | Silhouette | Inercia | Elegido |",
"| --- | --- | --- | --- |"]
for s in scores:
if not isinstance(s, dict):
continue
k = s.get("k")
chosen = "" if best_k is not None and k == best_k else ""
lines.append(
f"| {_fmt_num(k)} | {_fmt_num(s.get('silhouette'))} "
f"| {_fmt_num(s.get('inertia'))} | {chosen} |")
return "\n".join(lines)
def _app_normality(normality: dict) -> str:
"""Loss #5 — each normality test's statistic next to its p-value."""
if not isinstance(normality, dict) or not normality:
return ""
lines = ["#### Tests de normalidad (estadístico + p-value)",
"",
("| Columna | n | JB stat | JB p | D'Agostino stat | D'Agostino p "
"| Shapiro stat | Shapiro p | ¿Normal? |"),
"| --- | --- | --- | --- | --- | --- | --- | --- | --- |"]
any_row = False
for col, res in normality.items():
if not isinstance(res, dict):
continue
jb = res.get("jarque_bera") or {}
da = res.get("dagostino") or {}
sh = res.get("shapiro") or {}
is_norm = "" if res.get("is_normal") else "no"
lines.append(
f"| {_cell(col)} | {_fmt_num(res.get('n')) if res.get('n') is not None else ''} "
f"| {_fmt_num(jb.get('stat'))} | {_fmt_num(jb.get('p'))} "
f"| {_fmt_num(da.get('stat'))} | {_fmt_num(da.get('p'))} "
f"| {_fmt_num(sh.get('stat'))} | {_fmt_num(sh.get('p'))} | {is_norm} |")
any_row = True
return "\n".join(lines) if any_row else ""
def _profile_appendix(profile: dict) -> str:
"""Build the full-data appendix from a TableProfile dict (additive).
Returns a Markdown ``## Apéndice`` section with one sub-table per loss the
human chapters drop, or ``""`` when the profile carries none of them. Never
raises: a missing/oddly-shaped section is skipped, not fatal.
"""
if not isinstance(profile, dict):
return ""
sections: list = []
try:
corr = profile.get("correlations") or {}
seg = _app_correlations(corr) if isinstance(corr, dict) else ""
if seg:
sections.append(seg)
except Exception: # noqa: BLE001
pass
try:
columns = profile.get("columns") or []
seg = _app_numeric_describe(columns)
if seg:
sections.append(seg)
seg = _app_reexpression(columns)
if seg:
sections.append(seg)
except Exception: # noqa: BLE001
pass
try:
models = profile.get("models") or {}
if isinstance(models, dict):
model_segs = []
seg = _app_kmeans_scores(models.get("kmeans") or {})
if seg:
model_segs.append(seg)
seg = _app_normality(models.get("normality") or {})
if seg:
model_segs.append(seg)
if model_segs:
sections.append(
"### Modelos — detalle\n\n" + "\n\n".join(model_segs))
except Exception: # noqa: BLE001
pass
if not sections:
return ""
intro = ("Volcado completo de los datos que el motor computó y que los "
"capítulos (pensados para lectura humana / PDF) resumen. "
"Pensado para que un LLM reconstruya el análisis entero.")
return ("## Apéndice — Datos completos del perfil\n\n"
f"*{intro}*\n\n" + "\n\n".join(sections))
# --------------------------------------------------------------------------- #
# Entry point.
# --------------------------------------------------------------------------- #
def render_md(chapters: list, out_path: str, meta: dict = None) -> dict:
"""Serialize a list of Chapters into a single self-contained Markdown file.
The output leads with ``# <title>``, a metadata blockquote and a numbered
``## Índice`` linking each chapter, then one ``## N. <title>`` section per
chapter with its blocks. Tables become Markdown tables (every row dumped),
figures become caption + underlying data table, glossary markers are stripped
while ``**bold**`` is kept. Designed to be pasted into an LLM.
Args:
chapters: a list of ``Chapter`` (dataclasses or dicts); normalized
defensively with ``model.as_chapters``.
out_path: filesystem path for the ``.md`` (parent dirs are created).
meta: optional dict. Recognised keys: ``title``, ``ctx`` (dict with
``dataset_name``/``source_origin``/``storage``/``n_rows``/``n_cols``),
``generated_at``, ``embed_figures`` (export PNGs beside the .md,
default False).
Returns:
dict (never raises): ``{path: str|None, n_chars: int,
chapters: list[{id, version}], note: str}``. On a fatal error ``path`` is
None and ``note`` explains why.
"""
meta = meta or {}
chapters = model.as_chapters(chapters)
title = model._safe_str(meta.get("title")) or model.ENGINE_NAME
# Edge: nothing to render -> a minimal but valid Markdown document.
if not chapters:
content = (f"# {title}\n\n"
"*(documento vacío — sin capítulos aplicables)*\n")
return _write(out_path, content, [], "documento vacío")
counter = [0] # document-wide figure counter for unique PNG names.
notes: list = []
segments: list = [f"# {title}"]
meta_lines = _meta_block(meta)
if meta_lines:
segments.append("\n".join(f"> {ln}" for ln in meta_lines))
# Numbered index. The anchor matches the chapter heading emitted below
# (``## N. <title>``) in GitHub slug style.
chap_heads = []
idx_lines = ["## Índice"]
for i, ch in enumerate(chapters, 1):
head_text = f"{i}. {model._safe_str(ch.title)}"
anchor = _slug(head_text)
chap_heads.append((head_text, anchor))
idx_lines.append(f"{i}. [{model._safe_str(ch.title)}](#{anchor})")
segments.append("\n".join(idx_lines))
chapters_meta = []
for i, ch in enumerate(chapters, 1):
segments.append("---")
head_text, _anchor = chap_heads[i - 1]
segments.append(f"## {head_text}")
blocks = list(ch.blocks or [])
# Omit a leading level-1 Heading that just repeats the chapter title.
if blocks:
b0 = blocks[0]
if (getattr(b0, "kind", "") == "heading"
and int(getattr(b0, "level", 1) or 1) == 1
and _clean_terms(getattr(b0, "text", "")).strip()
== model._safe_str(ch.title).strip()):
blocks = blocks[1:]
for block in blocks:
try:
seg = _serialize_block(block, meta, out_path, counter)
except Exception as e: # noqa: BLE001
seg = _md_note(model.Note(text=model._safe_str(block)))
notes.append(
f"bloque '{getattr(block, 'kind', '?')}' del capítulo "
f"'{ch.id}' degradado: {e}")
if seg:
segments.append(seg)
chapters_meta.append({"id": ch.id, "version": ch.version})
# Full-data appendix: dump everything the profile holds that the human
# chapters drop (additive — the .md ends up with more than the PDF/PPTX).
# Emitted only when a profile is supplied via meta['profile']; never fatal.
try:
appendix = _profile_appendix(meta.get("profile"))
except Exception as e: # noqa: BLE001
appendix = ""
notes.append(f"apéndice de perfil omitido: {e}")
if appendix:
segments.append("---")
segments.append(appendix)
content = "\n\n".join(segments) + "\n"
note = f"{len(content)} caracteres"
if notes:
note += " · " + "; ".join(notes)
return _write(out_path, content, chapters_meta, note)
def _write(out_path: str, content: str, chapters_meta: list, note: str) -> dict:
"""Write the Markdown to disk (creating parents). dict-no-throw."""
try:
parent = os.path.dirname(os.path.abspath(out_path))
os.makedirs(parent, exist_ok=True)
with open(out_path, "w", encoding="utf-8") as fh:
fh.write(content)
except Exception as e: # noqa: BLE001 — never raise from the writer.
return {"path": None, "n_chars": 0, "chapters": [],
"note": f"no se pudo escribir el Markdown: {e}"}
return {"path": out_path, "n_chars": len(content),
"chapters": chapters_meta, "note": note}
@@ -0,0 +1,89 @@
---
name: render_automatic_eda_markdown
kind: function
lang: py
domain: datascience
version: "1.0.0"
purity: impure
signature: "def render_automatic_eda_markdown(chapters_or_profile, out_path: str, meta: dict = None) -> dict"
description: "Renderiza un documento AutomaticEDA por CAPÍTULOS (modelo de bloques independiente del formato) en un único MARKDOWN autocontenido pensado para PEGAR A UN LLM. Acepta una lista de capítulos del modelo o directamente un TableProfile del grupo eda (construye los capítulos canónicos con build_document). Prioriza TEXTO + DATOS sobre lo visual: las tablas se vuelcan como tablas markdown con TODAS las filas (sin paginar — no hay páginas que cortar), una figura matplotlib se reduce a su caption más la tabla de datos subyacente (Desde/Hasta/Frecuencia de las barras del histograma) porque un LLM no ve la imagen, y los marcadores de glosario se eliminan conservando el **negrita**. Lleva cabecera (# título), bloque de metadatos en blockquote e índice numerado con anclas GitHub. Espejo de render_automatic_eda_pdf/render_automatic_eda_pptx pero SIN manifest (KISS, el markdown es un único artefacto de texto). dict-no-throw: nunca lanza, devuelve {path, n_chars, chapters, note}; en error fatal path es None y note explica la causa. Flag opcional meta['embed_figures'] exporta PNGs junto al .md (off por defecto)."
tags: [eda, markdown, render, report, llm, automatic-eda, chapters, versioned, no-cut, text, datascience, python]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: [os, re, matplotlib, "datascience.automatic_eda"]
params:
- name: chapters_or_profile
desc: "una lista de capítulos del modelo AutomaticEDA (dataclasses Chapter o dicts {id,title,version,blocks}) O un TableProfile dict del grupo eda. Si es un TableProfile, los capítulos canónicos se construyen con build_document(profile, meta['ctx']). Bloques soportados: heading, markdown, kv_table, data_table, figure, image, caption, note, group, glossary_entry. Lectura defensiva: lo no reconocido se degrada a Note, nunca lanza."
- name: out_path
desc: "ruta del archivo .md de salida. Los directorios padre se crean si faltan. Directorio no escribible → {path:None, note:<causa>} sin lanzar."
- name: meta
desc: "dict opcional. Claves: title (título del documento), ctx (dict con dataset_name→Dataset, source_origin→Fuente, storage→Almacenamiento, n_rows/n_cols→Dimensiones; también lo consumen los builders de capítulo cuando se da un profile), generated_at (timestamp; si falta se genera ISO UTC), embed_figures (True para exportar PNGs <basename>_figN.png junto al .md; por defecto False y el markdown queda autocontenido)."
output: "dict (nunca lanza): {path: str|None, n_chars: int, chapters: list[{id,version}], note: str}. En error fatal (p.ej. directorio no escribible) path es None y note explica la causa. Un documento sin capítulos aplicables produce un markdown mínimo válido con 'documento vacío' y chapters=[]."
tested: true
tests: ["test_golden_bloques_sinteticos_serializa_todo_a_markdown", "test_edge_documento_vacio_no_revienta", "test_profile_path_construye_capitulos_y_escribe"]
test_file_path: "python/functions/datascience/render_automatic_eda_markdown_test.py"
file_path: "python/functions/datascience/render_automatic_eda_markdown.py"
---
## Ejemplo
```python
from datascience import render_automatic_eda_markdown
# Desde un TableProfile del grupo eda (mismo modelo que los renderers PDF/PPTX).
profile = {
"table": "ventas", "source": "/data/ventas.csv",
"n_rows": 1000, "n_cols": 2, "quality_score": 92.5,
"columns": [
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.01,
"numeric": {"mean": 42.5, "median": 40.0, "min": 1.0, "max": 100.0,
"std": 12.3}},
{"name": "categoria", "inferred_type": "categorical", "null_pct": 0.0,
"categorical": {"top": [{"value": "neumaticos", "count": 500}]}},
],
}
res = render_automatic_eda_markdown(
profile, "reports/ventas_aeda.md",
{"title": "EDA — ventas",
"ctx": {"dataset_name": "Ventas", "source_origin": "ERP export",
"n_rows": 1000, "n_cols": 2}})
print(res["path"], res["n_chars"], res["chapters"])
# -> reports/ventas_aeda.md 4123 [{'id':'portada','version':'1.0.0'}, ...]
```
## Cuando usarla
Cuando quieras **pegar el EDA a un LLM** (ChatGPT, Claude, ...) o tenerlo en texto
plano versionable: mismo documento por capítulos que el PDF/PPTX, pero serializado a
Markdown sin binarios. Úsala como tercera salida junto a `render_automatic_eda_pdf`
(móvil) y `render_automatic_eda_pptx` (compartir) desde el MISMO modelo de capítulos.
A diferencia de esas dos, no hay páginas ni slides: todas las filas de cada tabla se
vuelcan (nada se corta) y cada figura se reduce a su caption + la tabla de datos
subyacente, que es lo que un LLM puede leer. Para añadir capítulos al documento, ver
`docs/capabilities/automatic_eda.md`.
## Gotchas
- **Impura**: escribe el `.md` en `out_path` (crea los directorios padre). Con
`meta['embed_figures']=True` además exporta un PNG `<basename>_figN.png` por figura
junto al `.md`; por defecto NO exporta nada y el markdown queda autocontenido.
- **Nunca lanza** (dict-no-throw): un bloque que falle se degrada a una nota y se anota
en `note`; el documento se escribe igual. Un profile/lista vacíos producen un markdown
mínimo válido con `*(documento vacío …)*` y `chapters=[]`.
- **Figuras = datos, no imagen**: un bloque `figure` se serializa como `*Figura: caption*`
más, si la figura matplotlib trae barras (histograma / barras), una tabla
`| Desde | Hasta | Frecuencia |` extraída de los `Rectangle` patches (máx 100 filas;
el resto se trunca con `*… (N filas más)*`). Si no hay barras o algo falla, solo sale
el caption. La figura se cierra (`plt.close`) tras leerla.
- **Glosario vs negrita**: se eliminan SOLO los marcadores de glosario
`[[term:key]]visible[[/term]]` (queda `visible`); el `**negrita**` markdown SE
CONSERVA (es válido). No se usa `strip_inline_md` aquí porque ese también quita el bold.
- **Anclas del índice**: el `## Índice` enlaza cada capítulo con un ancla estilo GitHub
del encabezado `## N. Título` (minúsculas, espacios→`-`, sin signos). Si dos capítulos
comparten título exacto sus anclas colisionan (caso raro; los capítulos canónicos tienen
títulos únicos).
- **Tablas**: las celdas escapan `|` (→ `\|`) y pliegan saltos de línea a `<br>` para no
romper la columna. No hay reparto por ancho — un LLM no lo necesita.
@@ -0,0 +1,55 @@
"""render_automatic_eda_markdown — chapter-based EDA report as one Markdown file.
Public ``eda``-group entry point that serializes an AutomaticEDA document (a list
of chapters, or an ``eda`` TableProfile from which the canonical chapters are
built) into a single self-contained Markdown file optimised to be **pasted into
an LLM**: plain text, Markdown tables (every row dumped — there are no pages to
cut), figures reduced to caption + underlying data, no binaries. It mirrors
``render_automatic_eda_pdf`` / ``render_automatic_eda_pptx`` but for text output;
unlike those it writes no manifest (KISS — Markdown is a single text artefact).
dict-no-throw: never raises. Returns ``{path, n_chars, chapters, note}``; on a
fatal error ``path`` is None and ``note`` explains why.
"""
from __future__ import annotations
from datascience.automatic_eda import build_document, render_md
from datascience.automatic_eda.model import as_chapter, as_chapters
def _coerce_chapters(chapters_or_profile, meta: dict) -> list:
"""Accept chapters OR an eda profile and return a list of Chapter."""
arg = chapters_or_profile
if isinstance(arg, (list, tuple)):
return as_chapters(list(arg))
if isinstance(arg, dict):
if "blocks" in arg and "columns" not in arg:
ch = as_chapter(arg)
return [ch] if ch is not None else []
return build_document(arg, (meta or {}).get("ctx"))
return []
def render_automatic_eda_markdown(chapters_or_profile, out_path: str,
meta: dict = None) -> dict:
"""Render an AutomaticEDA document into a single self-contained Markdown file.
Args:
chapters_or_profile: a list of chapters (``Chapter`` dataclasses or
dicts) or an ``eda`` TableProfile dict (chapters built via
``build_document(profile, meta['ctx'])``).
out_path: filesystem path for the ``.md`` (parent dirs are created).
meta: optional dict. Recognised keys: ``title``, ``ctx`` (dict with
``dataset_name``/``source_origin``/``storage``/``n_rows``/``n_cols``),
``generated_at``, ``embed_figures`` (export PNGs beside the .md,
default False — off keeps the Markdown self-contained).
Returns:
dict (never raises): ``{path: str|None, n_chars: int,
chapters: list[{id, version}], note: str}``. On a fatal error ``path`` is
None and ``note`` explains the cause.
"""
meta = dict(meta or {})
chapters = _coerce_chapters(chapters_or_profile, meta)
return render_md(chapters, out_path, meta)
@@ -0,0 +1,168 @@
"""Tests for render_automatic_eda_markdown — DoD: golden + edge + profile path.
Self-contained synthetic blocks (no DuckDB). Verifies every block kind serializes
to Markdown (heading, markdown with glossary+bold, kv/data tables, a figure whose
histogram bars become a data table, caption, note, group, glossary entry), that a
leading level-1 heading equal to the chapter title is omitted, that an empty
document degrades to a valid minimal Markdown without raising, and that passing a
minimal TableProfile builds chapters and writes the file.
"""
import os
import tempfile
from datascience.render_automatic_eda_markdown import render_automatic_eda_markdown
from datascience.automatic_eda.model import (
Caption, Chapter, DataTable, Figure, GlossaryEntry, Group, Heading, KVTable,
Markdown, Note,
)
def _hist_fig():
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.hist([1, 1, 2, 2, 2, 3, 4, 4, 5, 5, 5, 5], bins=5)
return fig
def _chapters() -> list:
blocks = [
Heading("Demo", 1), # == chapter title -> omitted.
Heading("Seccion dos", 2), # -> ####
Markdown("Texto con [[term:ent]]entropia[[/term]] y **bold** aqui."),
KVTable(rows=[("Filas", 1000), ("Columnas", 5)], title="Resumen"),
DataTable(header=["col", "valor"],
rows=[["alpha", "111"], ["beta", "222"], ["gamma", "333"]],
title="Datos", note="nota inferior"),
Figure(make=_hist_fig, caption="Histograma demo"),
Caption("pie de figura"),
Note("una nota aparte"),
Group(title="Grupo X", blocks=[Markdown("dentro del grupo")]),
GlossaryEntry(key="ent", label="Entropia",
definition="Medida de incertidumbre."),
]
return [Chapter(id="demo", title="Demo", version="1.0.0", blocks=blocks)]
def _read(path: str) -> str:
with open(path, "r", encoding="utf-8") as fh:
return fh.read()
def test_golden_bloques_sinteticos_serializa_todo_a_markdown():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "demo.md")
res = render_automatic_eda_markdown(
_chapters(), out,
{"title": "EDA Demo",
"ctx": {"dataset_name": "Demo", "n_rows": 12, "n_cols": 2}})
assert res["path"] == out
assert os.path.exists(out)
assert res["n_chars"] > 0
assert res["chapters"] == [{"id": "demo", "version": "1.0.0"}]
content = _read(out)
# Document structure.
assert content.startswith("# ")
assert "## Índice" in content
# A Markdown table is present (header + separator row).
assert "| " in content and "| --- " in content
# DataTable values are all dumped.
for v in ("alpha", "111", "beta", "222", "gamma", "333"):
assert v in content
# Glossary markers stripped, bold kept.
assert "[[term" not in content
assert "[[/term]]" not in content
assert "**bold**" in content
assert "entropia" in content # visible glossary text preserved.
# Figure histogram bars became a data table.
assert "| Desde | Hasta | Frecuencia |" in content
# Glossary entry rendered as a level-3 heading.
assert "### Entropia" in content
# Level-2 heading -> ####.
assert "#### Seccion dos" in content
# Leading level-1 heading equal to the title was omitted.
assert "### Demo" not in content
# Group title rendered.
assert "### Grupo X" in content
def _hist_fig_with_span():
"""Histogram with a wide ``axvspan`` (±1σ band) over it.
Reproduces the num_distr figure shape: matplotlib keeps the span as a lone
Rectangle in ``ax.patches`` alongside the bin bars; it must NOT leak into the
extracted bins table as a fake bin (it is ~5x wider than a bin)."""
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
data = [1, 1, 2, 2, 2, 3, 4, 4, 5, 5, 5, 5]
ax.hist(data, bins=5)
ax.axvspan(2.0, 4.0, alpha=0.2) # mean±σ band — a wide stray rectangle.
return fig
def test_figura_descarta_axvspan_de_la_tabla_de_bins():
"""The ±1σ band rectangle must not appear as a row in the bins table."""
blocks = [Figure(make=_hist_fig_with_span, caption="Hist con banda")]
chapters = [Chapter(id="f", title="Fig", version="1.0.0", blocks=blocks)]
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "fig.md")
render_automatic_eda_markdown(chapters, out, {"title": "T"})
content = _read(out)
assert "| Desde | Hasta | Frecuencia |" in content
# Extract the rows of the bins table: lines between the header/separator
# and the next blank line.
lines = content.splitlines()
hi = next(i for i, ln in enumerate(lines)
if ln.startswith("| Desde | Hasta | Frecuencia |"))
rows = []
for ln in lines[hi + 2:]: # skip header + separator
if not ln.startswith("|"):
break
rows.append(ln)
# 5 histogram bins, no extra wide span row.
assert len(rows) == 5, rows
# No row spans a width of ~2.0 (the axvspan from x=2 to x=4).
for ln in rows:
cells = [c.strip() for c in ln.strip("|").split("|")]
lo, hi_v = float(cells[0]), float(cells[1])
assert (hi_v - lo) < 1.5, f"wide span leaked: {ln}"
def test_edge_documento_vacio_no_revienta():
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "empty.md")
res = render_automatic_eda_markdown([], out, {})
assert res["path"] == out
assert os.path.exists(out)
assert res["chapters"] == []
content = _read(out)
assert "documento vacío" in content
assert content.startswith("# ")
def test_profile_path_construye_capitulos_y_escribe():
profile = {
"table": "mini",
"source": "/data/mini.csv",
"n_rows": 10,
"n_cols": 1,
"quality_score": 88.0,
"columns": [
{"name": "x", "inferred_type": "numeric", "null_pct": 0.0,
"null_count": 0,
"numeric": {"mean": 1.0, "median": 1.0, "min": 0.0, "max": 2.0,
"std": 0.5}},
],
}
with tempfile.TemporaryDirectory() as d:
out = os.path.join(d, "mini.md")
res = render_automatic_eda_markdown(
profile, out, {"title": "Mini", "ctx": {"dataset_name": "Mini"}})
assert res["path"] == out # not None — no exception, file written.
assert os.path.exists(out)
assert res["n_chars"] > 0
@@ -1,9 +1,10 @@
"""render_automatic_eda — EDA completo one-shot: perfil → ctx → PDF + PPTX.
"""render_automatic_eda — EDA completo one-shot: perfil → ctx → PDF + PPTX + MD.
Pipeline impuro del grupo de capacidad `eda`. Dada UNA tabla DuckDB (o
PostgreSQL), produce el informe AutomaticEDA COMPLETO en sus dos formatos a la
vez (PDF móvil A5 + PPTX 16:9) con los 11 capítulos POBLADOS, en una sola
llamada. Compone, sin reimplementar su lógica, cuatro funciones del registry:
PostgreSQL), produce el informe AutomaticEDA COMPLETO en sus tres formatos a la
vez (PDF móvil A5 + PPTX 16:9 + Markdown autocontenido para pegar a un LLM) con
los capítulos POBLADOS, en una sola llamada. Compone, sin reimplementar su
lógica, varias funciones del registry:
- profile_table : perfila la tabla end-to-end (TableProfile agregado),
opcionalmente con modelos baratos y análisis de serie.
@@ -12,8 +13,11 @@ llamada. Compone, sin reimplementar su lógica, cuatro funciones del registry:
modelos/geo, timeseries_raw para series, geo_points
para el mapa, db_path/table para la agregación
push-down). Sin él, esos capítulos degradan.
- render_automatic_eda_pdf : renderiza el documento por capítulos a PDF.
- render_automatic_eda_pptx : renderiza el mismo documento a PPTX.
- render_automatic_eda_pdf : renderiza el documento por capítulos a PDF.
- render_automatic_eda_pptx : renderiza el mismo documento a PPTX.
- render_automatic_eda_markdown : serializa el mismo documento a Markdown
autocontenido (texto + tablas markdown, sin
binarios) para incorporar a un LLM.
El TableProfile agregado basta para portada/overview/distribuciones/calidad/
correlación, pero los capítulos `modelos`, `timeseries`, `geospatial` y
@@ -32,6 +36,7 @@ from datetime import datetime, timezone
from datascience import (
build_eda_render_ctx,
render_automatic_eda_markdown,
render_automatic_eda_pdf,
render_automatic_eda_pptx,
run_eda_models,
@@ -93,6 +98,7 @@ def render_automatic_eda(
out_dir: str = "reports",
basename: str = None,
ctx_extra: dict = None,
emit_md: bool = True,
) -> dict:
"""Perfila una tabla y emite el informe AutomaticEDA completo (PDF + PPTX).
@@ -140,13 +146,19 @@ def render_automatic_eda(
ctx_extra: dict opcional con claves de presentación/contexto extra que se
mezclan en el ctx (p.ej. dataset_name, description, source_origin).
No pisan las claves de datos calculadas por build_eda_render_ctx.
emit_md: además del PDF y el PPTX, emite un Markdown autocontenido del
MISMO documento por capítulos (texto plano + tablas markdown, sin
binarios), pensado para pegar a un LLM. Default True. La ruta sale en
la clave de retorno ``aeda_md_path``. No altera las demás salidas.
Returns:
dict (nunca lanza). En éxito::
{"status": "ok", "pdf_path": str, "pptx_path": str,
"manifest_path": str|None, "n_pages": int, "n_slides": int,
"pdf_note": str, "pptx_note": str, "profile": <TableProfile>}
"aeda_md_path": str|None, "manifest_path": str|None,
"n_pages": int, "n_slides": int, "md_chars": int|None,
"pdf_note": str, "pptx_note": str, "md_note": str|None,
"profile": <TableProfile>}
En error: {"status": "error", "error": str}.
"""
@@ -243,15 +255,34 @@ def render_automatic_eda(
rpdf = render_automatic_eda_pdf(prof, pdf_path, meta) or {}
rpptx = render_automatic_eda_pptx(prof, pptx_path, meta) or {}
# Salida Markdown autocontenida (mismo documento por capítulos) para
# pegar a un LLM. Aditiva: no afecta a PDF/PPTX/manifest. dict-no-throw.
rmd = {}
md_path = None
if emit_md:
md_path = os.path.join(out_dir, base + ".md")
# El Markdown es la salida MÁS completa: además del documento por
# capítulos (compartido con PDF/PPTX) volca un apéndice con TODOS los
# datos numéricos del perfil (matriz de asociación completa, describe
# con skew/kurtosis/percentiles, re-expresiones, scores_by_k de
# KMeans, estadísticos de normalidad). Se le pasa el `prof` vía
# meta['profile']; un meta propio evita alterar el de PDF/PPTX.
md_meta = dict(meta)
md_meta["profile"] = prof
rmd = render_automatic_eda_markdown(prof, md_path, md_meta) or {}
return {
"status": "ok",
"pdf_path": rpdf.get("path"),
"pptx_path": rpptx.get("path"),
"aeda_md_path": rmd.get("path"),
"manifest_path": rpdf.get("manifest_path"),
"n_pages": rpdf.get("n_pages"),
"n_slides": rpptx.get("n_slides"),
"md_chars": rmd.get("n_chars"),
"pdf_note": rpdf.get("note"),
"pptx_note": rpptx.get("note"),
"md_note": rmd.get("note"),
"profile": prof,
}
except Exception as e: # noqa: BLE001 — dict-no-throw: degradar, nunca lanzar.