Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d1a3d58a6b | |||
| b5334a2e97 | |||
| 437409641c | |||
| f3d427d9e4 | |||
| f5b30b23dc | |||
| 5eaf3f662e | |||
| 05fe76bce0 | |||
| 864430e988 | |||
| a69d14d38e | |||
| 00cd5274bc | |||
| cd658cc703 | |||
| 81b57f9acd | |||
| 02ee222dde | |||
| ba162ab301 | |||
| 649de07d6b | |||
| af1dd9bcc2 | |||
| fc5bc334c8 | |||
| 03f3dca823 |
+20
-10
@@ -25,9 +25,10 @@ Página madre del grupo: `docs/capabilities/eda.md` (léela primero para cargar
|
||||
- `--models` → `run_models=True` (PCA/KMeans/IsolationForest/normalidad).
|
||||
- `--llm` → `run_llm=True` (1 call LLM sobre el perfil agregado).
|
||||
- `--series` → `run_series=True` (estacionariedad ADF+KPSS, ACF/PACF, STL, retornos por columna numérica).
|
||||
- `--pdf` → `emit_pdf=True` (PDF A5 vertical legible en móvil).
|
||||
- `--pdf` → `emit_pdf=True` (PDF A5 legacy de `render_eda_pdf`, legible en móvil).
|
||||
- `--legacy-only` → emite SOLO el PDF legacy (sin AutomaticEDA), para casos en que solo se quiera el PDF rápido.
|
||||
|
||||
Por defecto, para un EDA "completo" cuando el usuario no especifica, activa `run_models`, `run_series` y `emit_pdf`; deja `run_llm` para cuando lo pida o cuando interese la interpretación semántica (es la única parte que gasta tokens del modelo).
|
||||
Por defecto, **un EDA completo emite SIEMPRE el informe AutomaticEDA en sus dos formatos: PDF (A5 móvil) Y PPTX (16:9 para compartir)** con los 11 capítulos poblados (portada, overview, distribuciones, calidad, correlaciones, modelos, series, geoespacial, agregación, interpretación LLM). Usa el pipeline `render_automatic_eda` (o `profile_table(emit_automatic=True)`), que activa `run_models` y `run_series` para que los capítulos de modelos/series/geoespacial/agregación salgan poblados. Deja `run_llm` para cuando el usuario lo pida o interese la interpretación semántica + narrativa por capítulo (es la única parte que gasta tokens del modelo).
|
||||
|
||||
## Reglas duras
|
||||
|
||||
@@ -35,7 +36,7 @@ Por defecto, para un EDA "completo" cuando el usuario no especifica, activa `run
|
||||
2. **CSV/Parquet/Excel** entran cargándolos antes a DuckDB (`read_csv_auto`/`read_parquet`/`read_xlsx`) — DuckDB es el motor por defecto. No traigas la tabla entera a RAM.
|
||||
3. **Secretos**: si la fuente es un DSN PostgreSQL con credenciales, NO las imprimas en los reports ni en el notebook; resuélvelas vía `resolve_pg_dsn`/`pass` cuando aplique.
|
||||
4. **El report es un artefacto local**: vive en `reports/` (gitignored), no se sube a Gitea ni se versiona. Compartir = pasar la ruta (regla `reports.md`).
|
||||
5. **Entrega las 4 salidas**: JSON sidecar + Markdown + **PDF móvil** + **notebook Jupyter colaborativo ejecutado en vivo**.
|
||||
5. **Entrega las salidas**: el informe **AutomaticEDA PDF + PPTX** (siempre, con `render_automatic_eda` / `emit_automatic=True`) + (opcional) JSON sidecar + Markdown + PDF legacy + **notebook Jupyter colaborativo ejecutado en vivo**. Comparte las rutas de PDF y PPTX.
|
||||
|
||||
## Paso 1 — Perfilar y escribir los reports
|
||||
|
||||
@@ -43,18 +44,26 @@ Una tabla (caso normal):
|
||||
|
||||
```bash
|
||||
PYTHONPATH=python/functions python/.venv/bin/python3 - <<'PYEOF'
|
||||
from pipelines.profile_table import profile_table
|
||||
r = profile_table(
|
||||
from pipelines.render_automatic_eda import render_automatic_eda
|
||||
# Informe AutomaticEDA COMPLETO one-shot: perfil + ctx (datos crudos) + PDF + PPTX
|
||||
# con los 11 capítulos poblados (clusters pintados, evolución temporal, mapa,
|
||||
# tablas de agregación). run_llm=True añade la narrativa LLM por capítulo.
|
||||
r = render_automatic_eda(
|
||||
"/ruta/datos.duckdb", "ventas",
|
||||
run_models=True, run_series=True, emit_pdf=True, run_llm=False,
|
||||
run_models=True, run_series=True, run_llm=False, out_dir="reports",
|
||||
)
|
||||
print("status:", r["status"])
|
||||
print("md: ", r["report_md_path"])
|
||||
print("json: ", r["report_json_path"])
|
||||
print("pdf: ", r["pdf_path"])
|
||||
print("pdf: ", r["pdf_path"], "(", r["n_pages"], "págs )")
|
||||
print("pptx: ", r["pptx_path"], "(", r["n_slides"], "slides )")
|
||||
print("manifest:", r["manifest_path"])
|
||||
PYEOF
|
||||
```
|
||||
|
||||
Si además quieres el report Markdown + JSON sidecar y/o el PDF legacy junto al
|
||||
AutomaticEDA, usa `profile_table(emit_automatic=True, emit_pdf=True, write_report=True)`:
|
||||
emite todo a la vez (`report_md_path`, `report_json_path`, `pdf_path` legacy,
|
||||
`aeda_pdf_path`, `aeda_pptx_path`, `aeda_manifest_path`).
|
||||
|
||||
Una base entera (todas las tablas + relaciones FK):
|
||||
|
||||
```bash
|
||||
@@ -90,6 +99,7 @@ Sigue la memoria `eda-workflow-registry` y la regla `notebook_collaboration.md`:
|
||||
## Notas
|
||||
|
||||
- El `TableProfile` lleva ahora, además del perfilado base y las correlaciones con FDR: `series` (por columna numérica, con `run_series`), `reexpression` por columna numérica (escalera de Tukey) y `caveats` (siempre, avisos exploratorios). El Markdown y el PDF renderizan estas secciones automáticamente cuando están presentes.
|
||||
- El PDF (`emit_pdf`) está pensado para leerse en el móvil (A5 vertical, tipografía grande, gráficos Tufte). Se escribe junto al Markdown en `reports/`.
|
||||
- El informe **AutomaticEDA** (`render_automatic_eda` / `emit_automatic=True`) emite el MISMO documento por capítulos a **PDF (A5 móvil)** y **PPTX (16:9)** con garantía de no-corte (texto envuelto, tablas partidas repitiendo cabecera, figuras escaladas) y negrita real (`**texto**`). Escribe `automatic_eda_manifest.json` con la versión de cada capítulo. Los capítulos modelos/series/geoespacial/agregación se pueblan con los datos crudos que `build_eda_render_ctx` muestrea de la base (no se traen tablas enteras a RAM).
|
||||
- El PDF legacy (`emit_pdf`, `render_eda_pdf`) sigue disponible y es independiente del AutomaticEDA (A5 vertical, gráficos Tufte). Se escribe junto al Markdown en `reports/`.
|
||||
- `run_series` ordena por la primera columna datetime si existe; si no, por el orden físico de filas. Necesita ≥8 puntos válidos por columna.
|
||||
- Fuentes: DuckDB (CSV/Parquet/Excel cargados antes) y PostgreSQL (`backend="postgres"`). `profile_database` (multi-tabla + FK) es solo DuckDB por ahora.
|
||||
|
||||
@@ -25,7 +25,8 @@ cabecera, y figuras/imágenes se escalan para caber enteras.
|
||||
```
|
||||
Document = list[Chapter]
|
||||
Chapter = { id: str, title: str, version: str, blocks: list[Block] }
|
||||
Block = Heading | Markdown | KVTable | DataTable | Figure | Image | Caption | Note
|
||||
Block = Heading | Markdown | KVTable | DataTable | Figure | Image | Caption
|
||||
| Note | Group | GlossaryEntry
|
||||
```
|
||||
|
||||
Importa el modelo desde `datascience.automatic_eda.model` (o
|
||||
@@ -44,6 +45,10 @@ reconocido se degrada a `Note`, nunca lanza).
|
||||
| `Figure(fig=None, make=None, caption=None, height_in=None)` | una `matplotlib.figure.Figure` ya construida (`fig`) o un callable `make()->Figure` (perezoso) | se rasteriza y escala para caber entera (nunca recortada) |
|
||||
| `Image(path, caption=None, height_in=None)` | ruta a PNG/JPG | se escala para caber entera |
|
||||
| `Caption(text)` / `Note(text)` | texto auxiliar pequeño | pie/nota en gris; `Note` es además el fallback de lo desconocido |
|
||||
| `Group(blocks, title=None)` | unidad **keep-together**: sus bloques se mantienen juntos | el renderer mide el grupo entero y lo mueve completo a la página/slide siguiente si no cabe; encoge la figura para dejar sitio al título+texto. Ver §11 |
|
||||
| `GlossaryEntry(key, label, definition)` | una entrada del glosario (destino clicable) | la genera el capítulo `glosario`; registra su posición como destino de los términos marcados. Ver §11 |
|
||||
|
||||
`Figure`/`Image` aceptan `height_in` (hint): el renderer **clampa** la figura a esa altura máxima (lo usa `Group` para encoger la figura). Toda figura escala dejando sitio a su caption en la misma página/slide; en PPTX el caption es **siempre** visible (si no se da `caption`, cae al último heading o a "Figura").
|
||||
|
||||
### Subset de markdown soportado (`Markdown`)
|
||||
|
||||
@@ -84,8 +89,9 @@ El orden canónico está **pre-declarado** en
|
||||
|
||||
```python
|
||||
CHAPTER_ORDER = [
|
||||
"portada", "overview", "num_distr", "cat_distr", "calidad", "correlacion",
|
||||
"modelos", "analisis_llm", "timeseries", "geospatial", "agregacion",
|
||||
"portada", "overview", "analisis_llm", "num_distr", "cat_distr", "calidad",
|
||||
"correlacion", "modelos", "timeseries", "geospatial", "agregacion",
|
||||
"glosario",
|
||||
]
|
||||
```
|
||||
|
||||
@@ -95,6 +101,15 @@ CHAPTER_ORDER = [
|
||||
`CHAPTER_ORDER`) y aparecerá automáticamente en su posición. Esto permite que muchos
|
||||
agentes trabajen **en paralelo** sin contención: cada uno toca solo su archivo.
|
||||
|
||||
**Dos capítulos tienen posición especial** (los gestiona `build_document`, no toques esto):
|
||||
|
||||
- `portada`: se **construye el último** (después del cuerpo) para poder resumir el
|
||||
análisis, pero se **coloca el primero**. Recibe `ctx['document_summary']` (ver §5) con
|
||||
un resumen agregado del resto. Decisión del usuario: la portada refleja hallazgos.
|
||||
- `glosario`: se construye y se **coloca el último**. Lee los términos que los demás
|
||||
capítulos registraron en `ctx['glossary']` (ver §11). Si no se registró ninguno, el
|
||||
capítulo devuelve `None` y desaparece.
|
||||
|
||||
Si tu capítulo usa un `<id>` que aún no está en `CHAPTER_ORDER`, añádelo en la posición
|
||||
correcta (única edición compartida; coordínala con el orquestador).
|
||||
|
||||
@@ -143,6 +158,8 @@ defensivo). Esto habilita el **seguimiento y la mejora continua por capítulo**.
|
||||
| `granularity` | "Cada fila es…" (portada). Default: derivado de `key_candidates` |
|
||||
| `quality_criteria` | criterios del score de calidad (portada) |
|
||||
| `head_rows` | `list[dict]` con `df.head` (overview). Ver §7 |
|
||||
| `glossary` | `GlossaryCollector` compartido — los capítulos registran términos en él. Lo crea `build_document`; ver §11 |
|
||||
| `document_summary` | dict con el resumen agregado del cuerpo (n_rows, n_cols, quality_score, n_numeric, n_categorical, chapter_titles, …). Lo calcula `build_document` y lo consume la portada |
|
||||
|
||||
Un capítulo puede definir y consumir sus propias claves `ctx` — documenta cuáles en su
|
||||
docstring.
|
||||
@@ -279,6 +296,109 @@ sus bloques presentes y el no-corte (texto largo intacto en la salida). Patrón:
|
||||
|
||||
---
|
||||
|
||||
## 11. Glosario, keep-together y zebra (motor, fase 4a)
|
||||
|
||||
Tres capacidades transversales del motor que **todos** los capítulos pueden usar. La 6.1
|
||||
(glosario) requiere que el capítulo coopere (registrar + marcar términos); la 6.2
|
||||
(keep-together) es opt-in por capítulo (envolver bloques en `Group`); la 6.3 (zebra) es
|
||||
automática (no hay nada que hacer).
|
||||
|
||||
### 11.1 Glosario con términos clicables
|
||||
|
||||
El glosario es un capítulo nuevo (`chapters/glosario.py`) que se renderiza **siempre el
|
||||
último** y lista cada término técnico que algún capítulo haya registrado. Cada aparición
|
||||
del término en el texto se vuelve un **clic real** que salta a su entrada: en PDF como
|
||||
*link annotation* interno (post-proceso con PyMuPDF, porque `PdfPages` no soporta
|
||||
hyperlinks internos), en PPTX como *slide-jump* nativo (`ppaction://hlinksldjump`).
|
||||
|
||||
**API exacta para un capítulo (dos pasos):**
|
||||
|
||||
1. **Registrar el término** en el colector compartido `ctx['glossary']` (un
|
||||
`model.GlossaryCollector`, creado por `build_document` y pasado a todos los capítulos):
|
||||
|
||||
```python
|
||||
glossary = ctx.get("glossary")
|
||||
if isinstance(glossary, model.GlossaryCollector):
|
||||
glossary.add("entropia", "Entropía (de Shannon)", "Medida, en bits, de …")
|
||||
```
|
||||
|
||||
`add(key, label, definition)` es idempotente (la primera definición de cada `key` gana).
|
||||
`key` debe ser `[A-Za-z0-9_]+`. Si no hay colector en `ctx` (renderizado suelto), el
|
||||
capítulo simplemente no marca términos — degrada sin romper.
|
||||
|
||||
2. **Marcar cada aparición** en el texto de un bloque `Markdown` con el span inline
|
||||
`[[term:KEY]]texto visible[[/term]]`. El texto visible puede llevar `**negrita**`. El
|
||||
marcador no altera el texto visible (se elimina como cualquier marcador inline); solo
|
||||
añade el destino clicable.
|
||||
|
||||
```python
|
||||
# En cat_distr (ejemplo real ya implementado):
|
||||
"La [[term:entropia]]**entropía de Shannon**[[/term]] mide cómo de repartidos…"
|
||||
```
|
||||
|
||||
Eso es todo: el capítulo `glosario` recoge los términos (orden alfabético por `label`),
|
||||
emite un `GlossaryEntry` por término, y los renderers cablean los enlaces automáticamente.
|
||||
Si ningún capítulo registró términos, el glosario no aparece.
|
||||
|
||||
**Helpers de `text_layout` (no reimplementar):** `parse_inline_rich(text)` →
|
||||
`[(texto, is_bold, term_key), …]`; `wrap_rich_terms(text, max_chars)` → líneas de esos
|
||||
spans sin corte. `strip_inline_md` ya elimina los marcadores `[[term:…]]`/`[[/term]]`.
|
||||
(Las funciones previas `parse_inline_bold` / `wrap_rich` siguen existiendo, sin términos.)
|
||||
|
||||
**Funciones del registry que cablean los enlaces** (grupo `eda`, ya invocadas por los
|
||||
renderers; degradan en silencio si faltan): `add_pdf_internal_links_py_datascience`
|
||||
(PyMuPDF, link GOTO) y `pptx_link_run_to_slide_py_datascience` (salto a slide nativo).
|
||||
Dependencia: `pymupdf` (declarada en `python/pyproject.toml`).
|
||||
|
||||
**Trabajo de la siguiente fase — enganchar más términos.** El mecanismo está hecho y
|
||||
probado de extremo a extremo con `entropia` (en `cat_distr`). Cada capítulo debe registrar
|
||||
y marcar SUS términos con el mismo patrón de dos pasos. Candidatos por capítulo:
|
||||
|
||||
| Capítulo | Términos a enganchar (key sugerida) |
|
||||
|---|---|
|
||||
| `cat_distr` | `entropia` ✅ (hecho) |
|
||||
| `calidad` | `completitud`, `validez`, `consistencia` |
|
||||
| `correlacion` | `cramers_v`, `fdr` (comparaciones múltiples), método de correlación usado |
|
||||
| `modelos` | `pca`, `silhouette`, `isolation_forest` |
|
||||
| `timeseries` | `estacionariedad`, `acf_pacf`, `stl` |
|
||||
| `num_distr` | `iqr`, `curtosis`, `outlier` (vallas de Tukey) |
|
||||
|
||||
Define la definición de cada término en su capítulo (constante local, como
|
||||
`_TERM_ENTROPIA_DEF` en `cat_distr`) y márcalo en su primera aparición.
|
||||
|
||||
### 11.2 Keep-together: gráfico junto a su título y texto (`Group`)
|
||||
|
||||
Para que un encabezado no quede en una página/slide y su figura en la siguiente, envuelve
|
||||
los bloques de una misma idea en un `model.Group`:
|
||||
|
||||
```python
|
||||
blocks.append(model.Group(blocks=[
|
||||
model.Heading(text=str(name), level=2),
|
||||
model.Figure(make=_figura_perezosa(...), caption="…"),
|
||||
model.Markdown(text="explicación…"),
|
||||
]))
|
||||
```
|
||||
|
||||
El renderer **mide el grupo entero** antes de dibujar nada: si no cabe en lo que queda de
|
||||
página/slide pero cabe en una entera, lo mueve **completo** a la siguiente; y **encoge la
|
||||
figura** (vía `height_in`) lo justo para que el título + texto + figura quepan juntos. Si
|
||||
el grupo es más alto que una página entera, empieza en una nueva y fluye (degradación
|
||||
honesta, nunca corta). Ejemplo real implementado: `num_distr` envuelve cada columna
|
||||
(heading + figura histograma/boxplot + nota) en un `Group`.
|
||||
|
||||
Recomendado para `agregacion` y cualquier capítulo donde una figura deba ir pegada a su
|
||||
título/explicación. Coste: si un capítulo inspecciona `chapter.blocks` en sus tests, ahora
|
||||
encontrará `Group`s — aplana con un helper recursivo (ver `num_distr_test.py::_flatten`).
|
||||
|
||||
### 11.3 Zebra striping en tablas (automático)
|
||||
|
||||
Todo `DataTable` se renderiza con **filas pares sombreadas** (gris muy suave `#f6f8fa`) y
|
||||
cabecera con su fondo propio. Es automático en PDF y PPTX; el patrón se mantiene coherente
|
||||
cuando una tabla larga se parte y repite cabecera (el índice de fila es lógico, no por
|
||||
página). No hay nada que hacer en los capítulos.
|
||||
|
||||
---
|
||||
|
||||
## 10. Integración futura con `profile_table` (siguiente fase)
|
||||
|
||||
`profile_table(emit_pdf=True)` usa hoy `render_eda_pdf` (intacto). En la siguiente fase
|
||||
|
||||
@@ -47,6 +47,9 @@ from .trend_slope import trend_slope
|
||||
from .run_eda_models import run_eda_models
|
||||
from .project_clusters_2d import project_clusters_2d
|
||||
from .describe_clusters_llm import describe_clusters_llm
|
||||
from .detect_latlon_columns import detect_latlon_columns
|
||||
from .analyze_geo_extent import analyze_geo_extent
|
||||
from .build_geo_scatter import build_geo_scatter
|
||||
from .eda_llm_insights import eda_llm_insights
|
||||
from .build_eda_notebook import build_eda_notebook
|
||||
from .decode_qr_image import decode_qr_image
|
||||
@@ -60,8 +63,20 @@ from .exploratory_caveats import exploratory_caveats
|
||||
from .render_eda_pdf import render_eda_pdf, render_eda_pdf_relational
|
||||
from .render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from .render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
from .detect_time_column import detect_time_column
|
||||
from .extract_timeseries_raw import extract_timeseries_raw
|
||||
from .build_eda_render_ctx import build_eda_render_ctx
|
||||
from .profile_datetime import profile_datetime
|
||||
from .resample_timeseries import resample_timeseries
|
||||
from .add_pdf_internal_links import add_pdf_internal_links
|
||||
|
||||
__all__ = [
|
||||
"detect_time_column",
|
||||
"extract_timeseries_raw",
|
||||
"build_eda_render_ctx",
|
||||
"add_pdf_internal_links",
|
||||
"profile_datetime",
|
||||
"resample_timeseries",
|
||||
"render_automatic_eda_pdf",
|
||||
"render_automatic_eda_pptx",
|
||||
"decode_qr_image",
|
||||
@@ -95,6 +110,9 @@ __all__ = [
|
||||
"run_eda_models",
|
||||
"project_clusters_2d",
|
||||
"describe_clusters_llm",
|
||||
"detect_latlon_columns",
|
||||
"analyze_geo_extent",
|
||||
"build_geo_scatter",
|
||||
"eda_llm_insights",
|
||||
"build_eda_notebook",
|
||||
"describe_numeric",
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: add_pdf_internal_links
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: impure
|
||||
signature: "def add_pdf_internal_links(pdf_path: str, links: list) -> dict"
|
||||
description: "Postprocesa un PDF YA escrito insertando link annotations internos de tipo GOTO ('ir a') con PyMuPDF (import fitz). Pensado para PDFs generados por matplotlib PdfPages, que NO soporta hyperlinks internos: tras escribir el PDF se reabre y, por cada entrada de `links`, se añade una anotacion clicable desde un rectangulo de una pagina origen (src_page + src_rect en puntos top-left) hasta un punto de una pagina destino (dst_page + dst_point). Caso de uso tipico del grupo eda: hacer clicables los terminos de un AutomaticEDA que apuntan a su entrada en el glosario al final del documento. Estilo dict-no-throw: NUNCA lanza; valida cada link y SALTA (n_skipped++) los malformados o fuera de rango en vez de fallar. Guarda de forma segura escribiendo a un temporal en el mismo directorio y haciendo os.replace atomico (evita corromper el original). Devuelve {status:ok,n_links,n_skipped} o {status:error,error}; si pymupdf no esta disponible o el archivo no existe devuelve status error."
|
||||
tags: [eda, datascience, pdf, links, glossary, pymupdf, fitz, postprocess, python]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: "error_go_core"
|
||||
imports: []
|
||||
params:
|
||||
- name: pdf_path
|
||||
desc: "ruta al PDF existente (str no vacio). Se reescribe IN SITU (in-place) tras añadir los links: se guarda a un temporal `.<base>.tmp_links` en el mismo directorio y se reemplaza atomicamente con os.replace. Si no es str o no existe el archivo -> {status:error}."
|
||||
- name: links
|
||||
desc: "lista de dicts, uno por link a insertar. Cada dict: src_page (int 0-based de la pagina origen), src_rect ([x0,y0,x1,y1] del rectangulo clicable en PUNTOS PDF 1/72\" con origen ARRIBA-IZQUIERDA), dst_page (int 0-based de la pagina destino), dst_point ([x,y] punto destino, mismos puntos top-left). Las entradas que no son dict, con page fuera de rango [0,page_count), src_rect que no tenga 4 numeros o dst_point que no tenga 2 numeros se SALTAN (n_skipped++), no lanzan. None se trata como lista vacia."
|
||||
output: "dict (NUNCA lanza): en exito {\"status\":\"ok\",\"n_links\":int,\"n_skipped\":int} con n_links = anotaciones GOTO insertadas y n_skipped = entradas invalidas saltadas. En fallo {\"status\":\"error\",\"error\":str}: pymupdf no disponible, pdf_path no es str / no existe, links no es lista, o cualquier excepcion global (el PDF original queda intacto porque el replace solo ocurre tras un save correcto)."
|
||||
tested: true
|
||||
tests: ["test_add_goto_link_basico", "test_links_invalidos_se_saltan", "test_archivo_inexistente_devuelve_error"]
|
||||
test_file_path: "python/functions/datascience/add_pdf_internal_links_test.py"
|
||||
file_path: "python/functions/datascience/add_pdf_internal_links.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience import add_pdf_internal_links
|
||||
|
||||
# Tienes un PDF ya escrito por matplotlib PdfPages (sin hyperlinks internos).
|
||||
# Quieres que el texto "Margen bruto" de la pagina 0 (rectangulo en puntos
|
||||
# top-left) salte a su entrada del glosario en la ultima pagina (indice 7).
|
||||
res = add_pdf_internal_links(
|
||||
"reports/eda.pdf",
|
||||
[
|
||||
{"src_page": 0, "src_rect": [72, 120, 180, 134], "dst_page": 7, "dst_point": [72, 200]},
|
||||
{"src_page": 0, "src_rect": [72, 140, 180, 154], "dst_page": 7, "dst_point": [72, 260]},
|
||||
],
|
||||
)
|
||||
# res == {"status": "ok", "n_links": 2, "n_skipped": 0}
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Justo DESPUES de escribir un PDF con matplotlib `PdfPages` (o cualquier motor
|
||||
que no genere hyperlinks internos) cuando necesitas que ciertos terminos o
|
||||
referencias sean clicables y salten a otra pagina del mismo documento — el caso
|
||||
canonico es enlazar los terminos de un AutomaticEDA con su entrada de glosario
|
||||
al final. Es un paso de postproceso: primero generas el PDF y calculas en que
|
||||
rectangulo quedo cada termino (en puntos PDF), luego pasas esa lista a esta
|
||||
funcion para inyectar las anotaciones GOTO.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Impura — reescribe el archivo IN SITU.** El PDF en `pdf_path` se reemplaza
|
||||
por la version con los links. El guardado es seguro: escribe a un temporal
|
||||
`.<base>.tmp_links` en el MISMO directorio y hace `os.replace` atomico tras
|
||||
cerrar el documento, asi un fallo a mitad no corrompe el original. Aun asi,
|
||||
conserva una copia si el PDF es valioso.
|
||||
- **Sistema de coordenadas: puntos top-left, igual que matplotlib.** PyMuPDF y
|
||||
matplotlib (PdfPages) usan ambos PUNTOS PDF (1/72") con el origen ARRIBA-
|
||||
IZQUIERDA, asi que los rectangulos/puntos COINCIDEN: el `src_rect` que calcules
|
||||
con la geometria de la figura matplotlib se pasa tal cual, sin invertir el eje
|
||||
Y. (Ojo: el espacio de datos de matplotlib SI tiene el origen abajo; lo que
|
||||
coincide es el espacio de la PAGINA en puntos.)
|
||||
- **Indices de pagina 0-based.** `src_page` / `dst_page` son indices base 0
|
||||
(la primera pagina es 0). Fuera del rango `[0, page_count)` el link se SALTA
|
||||
(cuenta en `n_skipped`), no lanza.
|
||||
- **dict-no-throw, validacion por-link.** Las entradas malformadas (no dict,
|
||||
page fuera de rango, `src_rect` sin 4 numeros, `dst_point` sin 2 numeros) se
|
||||
saltan individualmente e incrementan `n_skipped`; el resto de links validos se
|
||||
insertan igual. La funcion solo devuelve `{status:error}` ante fallos globales
|
||||
(pymupdf ausente, archivo inexistente, `links` no es lista).
|
||||
- **`error_type: error_go_core` es metadata del registry, no comportamiento.**
|
||||
Toda funcion impura debe declararlo y el indexer lo exige, pero el codigo NUNCA
|
||||
lanza esa excepcion: degrada al dict de estado.
|
||||
- **Requiere PyMuPDF (`import fitz`).** Si no esta instalado devuelve
|
||||
`{"status":"error","error":"pymupdf no disponible: ..."}`. En el registry el
|
||||
venv `python/.venv` ya lo trae.
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Postprocesa un PDF existente insertando link annotations internos (GOTO).
|
||||
|
||||
Motor: PyMuPDF (``import fitz``). Pensado para PDFs generados por matplotlib
|
||||
``PdfPages``, que no soporta hyperlinks internos: tras escribir el PDF, esta
|
||||
funcion lo reabre y le añade anotaciones "ir a" (GOTO) desde un rectangulo de
|
||||
una pagina origen hasta un punto de una pagina destino. Util para hacer
|
||||
clicables terminos que apuntan a su entrada en un glosario al final del
|
||||
documento.
|
||||
|
||||
Estilo dict-no-throw del grupo `eda`: NUNCA lanza; devuelve un dict de estado.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
|
||||
def add_pdf_internal_links(pdf_path: str, links: list) -> dict:
|
||||
"""Añade link annotations internos (GOTO) a un PDF ya escrito.
|
||||
|
||||
Postprocesa un PDF (p.ej. generado por matplotlib PdfPages, que NO soporta
|
||||
hyperlinks internos) insertando, por cada entrada de ``links``, una
|
||||
anotacion de tipo "ir a" desde un rectangulo de una pagina origen hasta un
|
||||
punto de una pagina destino. Sirve para hacer clicables terminos que apuntan
|
||||
a su entrada en un glosario al final del documento.
|
||||
|
||||
Args:
|
||||
pdf_path: ruta al PDF existente (se reescribe in situ).
|
||||
links: lista de dicts, cada uno:
|
||||
{
|
||||
"src_page": int, # indice 0-based de la pagina origen
|
||||
"src_rect": [x0,y0,x1,y1], # rectangulo clicable, en PUNTOS PDF
|
||||
# (1/72") con origen ARRIBA-IZQUIERDA
|
||||
"dst_page": int, # indice 0-based de la pagina destino
|
||||
"dst_point": [x, y], # punto destino, mismos puntos top-left
|
||||
}
|
||||
|
||||
Returns:
|
||||
dict (NUNCA lanza): {"status":"ok","n_links":int,"n_skipped":int}
|
||||
o {"status":"error","error":str}. Si pymupdf no esta disponible o el
|
||||
archivo no existe -> {"status":"error", ...}.
|
||||
"""
|
||||
try:
|
||||
try:
|
||||
import fitz # PyMuPDF
|
||||
except Exception as exc: # ImportError u otro fallo de carga
|
||||
return {"status": "error", "error": f"pymupdf no disponible: {exc}"}
|
||||
|
||||
if not isinstance(pdf_path, str) or not pdf_path:
|
||||
return {"status": "error", "error": "pdf_path debe ser una ruta no vacia"}
|
||||
if not os.path.isfile(pdf_path):
|
||||
return {"status": "error", "error": f"el archivo no existe: {pdf_path}"}
|
||||
|
||||
if links is None:
|
||||
links = []
|
||||
if not isinstance(links, (list, tuple)):
|
||||
return {"status": "error", "error": "links debe ser una lista de dicts"}
|
||||
|
||||
doc = fitz.open(pdf_path)
|
||||
try:
|
||||
n_pages = doc.page_count
|
||||
n_ok = 0
|
||||
n_skipped = 0
|
||||
|
||||
for link in links:
|
||||
if not isinstance(link, dict):
|
||||
n_skipped += 1
|
||||
continue
|
||||
|
||||
src_page = link.get("src_page")
|
||||
dst_page = link.get("dst_page")
|
||||
src_rect = link.get("src_rect")
|
||||
dst_point = link.get("dst_point")
|
||||
|
||||
# src_page / dst_page: enteros 0-based en rango.
|
||||
if not _is_int(src_page) or not _is_int(dst_page):
|
||||
n_skipped += 1
|
||||
continue
|
||||
if not (0 <= src_page < n_pages) or not (0 <= dst_page < n_pages):
|
||||
n_skipped += 1
|
||||
continue
|
||||
|
||||
# src_rect: 4 numeros.
|
||||
if not _is_num_seq(src_rect, 4):
|
||||
n_skipped += 1
|
||||
continue
|
||||
# dst_point: 2 numeros.
|
||||
if not _is_num_seq(dst_point, 2):
|
||||
n_skipped += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
doc[int(src_page)].insert_link(
|
||||
{
|
||||
"kind": fitz.LINK_GOTO,
|
||||
"from": fitz.Rect(*[float(v) for v in src_rect]),
|
||||
"page": int(dst_page),
|
||||
"to": fitz.Point(*[float(v) for v in dst_point]),
|
||||
}
|
||||
)
|
||||
n_ok += 1
|
||||
except Exception:
|
||||
n_skipped += 1
|
||||
continue
|
||||
|
||||
# Guardado seguro: escribir a temporal en el mismo directorio y
|
||||
# reemplazar atomicamente (evita corromper el PDF original).
|
||||
directory = os.path.dirname(os.path.abspath(pdf_path)) or "."
|
||||
base = os.path.basename(pdf_path)
|
||||
tmp_path = os.path.join(directory, f".{base}.tmp_links")
|
||||
doc.save(tmp_path)
|
||||
finally:
|
||||
doc.close()
|
||||
|
||||
os.replace(tmp_path, pdf_path)
|
||||
|
||||
return {"status": "ok", "n_links": n_ok, "n_skipped": n_skipped}
|
||||
except Exception as exc: # degrada cualquier fallo a dict de error
|
||||
return {"status": "error", "error": str(exc)}
|
||||
|
||||
|
||||
def _is_int(value) -> bool:
|
||||
"""True si value es un entero (no bool)."""
|
||||
return isinstance(value, int) and not isinstance(value, bool)
|
||||
|
||||
|
||||
def _is_num_seq(value, length: int) -> bool:
|
||||
"""True si value es una secuencia de `length` numeros (int/float, no bool)."""
|
||||
if not isinstance(value, (list, tuple)) or len(value) != length:
|
||||
return False
|
||||
for v in value:
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
return False
|
||||
return True
|
||||
@@ -0,0 +1,77 @@
|
||||
"""Tests para add_pdf_internal_links."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from add_pdf_internal_links import add_pdf_internal_links
|
||||
|
||||
|
||||
def test_add_goto_link_basico(tmp_path):
|
||||
"""Golden: un PDF de 2 paginas recibe un link GOTO de la pag 0 a la pag 1."""
|
||||
fitz = pytest.importorskip("fitz")
|
||||
|
||||
# 1) PDF temporal de 2 paginas A5 (~419x595 puntos).
|
||||
pdf = str(tmp_path / "doc.pdf")
|
||||
doc = fitz.open()
|
||||
doc.new_page(width=419, height=595)
|
||||
doc.new_page(width=419, height=595)
|
||||
doc.save(pdf)
|
||||
doc.close()
|
||||
|
||||
# 2) Insertar un link interno desde la pag 0 hacia la pag 1.
|
||||
res = add_pdf_internal_links(
|
||||
pdf,
|
||||
[{"src_page": 0, "src_rect": [50, 50, 200, 70], "dst_page": 1, "dst_point": [40, 40]}],
|
||||
)
|
||||
assert res["status"] == "ok"
|
||||
assert res["n_links"] == 1
|
||||
assert res["n_skipped"] == 0
|
||||
|
||||
# 3) Reabrir y verificar que la pag 0 tiene un link GOTO a la pag 1.
|
||||
doc = fitz.open(pdf)
|
||||
try:
|
||||
links = doc[0].get_links()
|
||||
goto = [l for l in links if l.get("kind") == fitz.LINK_GOTO and l.get("page") == 1]
|
||||
assert len(goto) >= 1
|
||||
finally:
|
||||
doc.close()
|
||||
|
||||
|
||||
def test_links_invalidos_se_saltan(tmp_path):
|
||||
"""Edge: entradas malformadas o fuera de rango incrementan n_skipped, no lanzan."""
|
||||
fitz = pytest.importorskip("fitz")
|
||||
|
||||
pdf = str(tmp_path / "doc.pdf")
|
||||
doc = fitz.open()
|
||||
doc.new_page(width=419, height=595)
|
||||
doc.new_page(width=419, height=595)
|
||||
doc.save(pdf)
|
||||
doc.close()
|
||||
|
||||
res = add_pdf_internal_links(
|
||||
pdf,
|
||||
[
|
||||
# valido
|
||||
{"src_page": 0, "src_rect": [10, 10, 90, 30], "dst_page": 1, "dst_point": [20, 20]},
|
||||
# dst_page fuera de rango
|
||||
{"src_page": 0, "src_rect": [10, 40, 90, 60], "dst_page": 9, "dst_point": [20, 20]},
|
||||
# src_rect con 3 numeros
|
||||
{"src_page": 0, "src_rect": [10, 70, 90], "dst_page": 1, "dst_point": [20, 20]},
|
||||
# no es dict
|
||||
"no-soy-un-dict",
|
||||
],
|
||||
)
|
||||
assert res["status"] == "ok"
|
||||
assert res["n_links"] == 1
|
||||
assert res["n_skipped"] == 3
|
||||
|
||||
|
||||
def test_archivo_inexistente_devuelve_error():
|
||||
"""Error path: pdf_path inexistente -> status error sin lanzar."""
|
||||
res = add_pdf_internal_links("/ruta/que/no/existe_xyz.pdf", [])
|
||||
assert res["status"] == "error"
|
||||
assert "error" in res
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
name: analyze_geo_extent
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def analyze_geo_extent(lats: list, lons: list) -> dict"
|
||||
description: "Calcula la extension geografica de una nube de coordenadas (lat/lon) y asigna cada punto a un pais/region mediante un lookup OFFLINE contra una tabla de bounding boxes embebida como constante. Devuelve bounding box, centroide, span de la diagonal (haversine), conteo por region (top-8 + Otros), reparto por hemisferios y una frase resumen en ES. Lectura defensiva: descarta pares None/NaN/fuera de rango y NUNCA lanza. Solo stdlib (math); sin geopandas/shapely. Las cajas de paises son rectangulos aproximados, no reverse-geocoding exacto."
|
||||
tags: [eda, geospatial, geo, coordinates, bounding-box, haversine, datascience]
|
||||
params:
|
||||
- name: lats
|
||||
desc: "Lista de latitudes en grados, rango valido [-90, 90]. Se empareja por indice con lons (gana la longitud minima comun si difieren). Cada valor puede ser None/NaN/no-numerico/fuera de rango: se lee defensivo y se descarta el par."
|
||||
- name: lons
|
||||
desc: "Lista de longitudes en grados, rango valido [-180, 180]. Paralela a lats, emparejada por indice. Valores None/NaN/no-numericos/fuera de rango se descartan junto con su par."
|
||||
output: "Dict con el resumen geografico: {n_points=pares validos usados, bbox={lat_min,lat_max,lon_min,lon_max} o None, centroid={lat,lon}=media de lat/lon validos o None, span_km=distancia haversine (radio 6371 km) de la diagonal SO->NE del bbox, by_region=[{region,count}] descendente por count limitado a top-8 con el resto agregado en 'Otros', hemisphere={north,south,east,west} (ecuador->norte, meridiano 0->este), note=frase ES resumen}. Si no hay pares validos devuelve la forma cero: n_points 0, bbox None, centroid None, span_km 0.0, by_region [], hemisphere a ceros y note 'sin coordenadas validas'. Puntos que no caen en ninguna caja -> region 'Oceano/Otros'."
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: [math]
|
||||
tested: true
|
||||
tests: ["test_nube_en_espana", "test_dos_paises_distintos", "test_listas_vacias", "test_pares_invalidos_filtrados", "test_longitudes_desbalanceadas", "test_span_km_haversine_par_conocido", "test_no_lanza_con_entradas_raras"]
|
||||
test_file_path: "python/functions/datascience/analyze_geo_extent_test.py"
|
||||
file_path: "python/functions/datascience/analyze_geo_extent.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience.analyze_geo_extent import analyze_geo_extent
|
||||
|
||||
# Nube de puntos alrededor de Madrid + un punto en Paris.
|
||||
lats = [40.4, 40.0, 41.0, 48.8]
|
||||
lons = [-3.7, -3.5, -4.0, 2.3]
|
||||
res = analyze_geo_extent(lats, lons)
|
||||
|
||||
print(res["n_points"]) # 4
|
||||
print(res["by_region"]) # [{'region': 'España', 'count': 3}, {'region': 'Francia', 'count': 1}]
|
||||
print(round(res["span_km"], 1)) # diagonal SO->NE del bbox en km
|
||||
print(res["hemisphere"]) # {'north': 4, 'south': 0, 'east': 1, 'west': 3}
|
||||
print(res["note"]) # los puntos se concentran en España (3 de 4)
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
- Usala en el perfilado EDA (grupo `eda`) cuando una tabla tenga columnas de latitud y longitud y quieras un resumen geografico rapido: donde se concentran los puntos, cuanto territorio cubren y a que paises/regiones caen, sin montar geopandas ni un reverse-geocoder.
|
||||
- Cuando necesites un capitulo `geospatial` del `AutomaticEDA`: alimenta el bbox + centroide para centrar un mapa, el `span_km` para elegir el zoom, y `by_region` para una tabla de conteos por pais.
|
||||
- Cuando quieras detectar datos sucios de coordenadas (mezcla de hemisferios inesperada, puntos en `Oceano/Otros`, span enorme) antes de seguir el analisis.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Funcion pura, sin I/O ni red y determinista: mismas entradas -> misma salida. Lectura defensiva, NUNCA lanza; pares con None/NaN o fuera de rango ([-90,90] lat, [-180,180] lon) se descartan en silencio.
|
||||
- El lookup de region es una **aproximacion rectangular**: cada pais/region es un bounding box, NO su frontera real. Un punto en el mar cerca de una costa, o en una esquina del rectangulo, puede asignarse a un pais vecino. No es reverse-geocoding exacto — para precision real hace falta un shapefile (fuera de scope por KISS).
|
||||
- Cajas solapadas se resuelven por orden: gana la PRIMERA que contiene el punto. Los paises se listan antes que los continentes (fallback), y entre vecinos el mas estrecho/occidental va primero (Portugal antes que España, Chile antes que Argentina, EEUU contiguo antes que Canada). Un punto que no cae en ninguna caja -> `Oceano/Otros`.
|
||||
- La tabla cubre ~24 paises grandes + 6 regiones continentales; paises pequeños o no listados caen a su continente o a `Oceano/Otros`. No incluye territorios insulares lejanos (Canarias, Hawaii, etc.).
|
||||
- `span_km` es la diagonal del bounding box (esquina SO a NE), no la dispersion real de la nube ni el area; con un solo punto valido el bbox es degenerado y `span_km` es 0.0.
|
||||
- El ecuador (`lat == 0`) cuenta como hemisferio norte y el meridiano 0 (`lon == 0`) como este, por convencion `>= 0`.
|
||||
@@ -0,0 +1,209 @@
|
||||
"""analyze_geo_extent — geographic extent of a cloud of coordinates (EDA `geospatial`).
|
||||
|
||||
Pure function: no I/O, no network, deterministic. Given two parallel lists of
|
||||
latitudes and longitudes it derives the bounding box, centroid, diagonal span
|
||||
(haversine), per-region counts and hemisphere split of the points, and assigns
|
||||
each point to a country/region via an OFFLINE lookup against a table of
|
||||
rectangular bounding boxes embedded as a constant (`_REGION_BBOXES`).
|
||||
|
||||
It never reads files, never hits the network and depends only on `math`. The
|
||||
country boxes are deliberately coarse rectangles (a KISS approximation, NOT a
|
||||
reverse-geocoder). Reading is defensive throughout and the function NEVER
|
||||
raises: invalid pairs (None / NaN / out of range) are silently discarded and an
|
||||
empty cloud yields a zeroed result the caller can skip.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
# Earth mean radius in km used by the haversine formula.
|
||||
_EARTH_RADIUS_KM = 6371.0
|
||||
|
||||
# How many distinct regions to surface in `by_region` before collapsing the
|
||||
# remainder into a single "Otros" bucket.
|
||||
_TOP_REGIONS = 8
|
||||
|
||||
# Offline region lookup: (name, lat_min, lat_max, lon_min, lon_max).
|
||||
#
|
||||
# Specific countries are listed FIRST and continental fallbacks LAST: each point
|
||||
# is assigned to the FIRST box that contains it, so the more specific country box
|
||||
# wins over the broad continent box. Boxes are coarse rectangles approximating
|
||||
# the mainland extent of each region; overlapping neighbours are ordered so the
|
||||
# narrower/more-western country claims its coastal points (e.g. Portugal before
|
||||
# Spain, Chile before Argentina, the contiguous US before Canada).
|
||||
_REGION_BBOXES = (
|
||||
# --- countries (specific) ---
|
||||
("Portugal", 36.9, 42.2, -9.6, -6.2),
|
||||
("España", 36.0, 43.8, -9.4, 3.4),
|
||||
("Francia", 41.3, 51.1, -5.2, 9.6),
|
||||
("Reino Unido", 49.9, 58.7, -8.6, 1.8),
|
||||
("Irlanda", 51.4, 55.4, -10.6, -5.9),
|
||||
("Países Bajos", 50.7, 53.6, 3.3, 7.2),
|
||||
("Bélgica", 49.5, 51.5, 2.5, 6.4),
|
||||
("Suiza", 45.8, 47.8, 5.9, 10.5),
|
||||
("Alemania", 47.3, 55.1, 5.9, 15.0),
|
||||
("Italia", 36.6, 47.1, 6.6, 18.5),
|
||||
("Marruecos", 27.7, 35.9, -13.2, -1.0),
|
||||
("Egipto", 22.0, 31.7, 25.0, 35.0),
|
||||
("Sudáfrica", -34.8, -22.1, 16.5, 32.9),
|
||||
("China", 18.0, 53.6, 73.5, 135.1),
|
||||
("Japón", 24.0, 45.6, 122.9, 145.9),
|
||||
("India", 6.7, 35.5, 68.1, 97.4),
|
||||
("Australia", -43.7, -10.0, 112.9, 153.7),
|
||||
("México", 14.5, 32.7, -118.4, -86.7),
|
||||
("Estados Unidos", 24.4, 49.4, -125.0, -66.9),
|
||||
("Canadá", 41.7, 83.1, -141.0, -52.6),
|
||||
("Chile", -55.9, -17.5, -75.6, -66.4),
|
||||
("Argentina", -55.1, -21.8, -73.6, -53.6),
|
||||
("Brasil", -33.8, 5.3, -74.0, -34.8),
|
||||
("Rusia", 41.2, 77.0, 19.6, 180.0),
|
||||
# --- continental fallbacks (broad) ---
|
||||
("Europa", 34.0, 72.0, -25.0, 45.0),
|
||||
("África", -35.0, 37.5, -18.0, 52.0),
|
||||
("Asia", 5.0, 78.0, 26.0, 180.0),
|
||||
("América del Norte", 7.0, 84.0, -168.0, -52.0),
|
||||
("América del Sur", -56.0, 13.0, -82.0, -34.0),
|
||||
("Oceanía", -50.0, 0.0, 110.0, 180.0),
|
||||
)
|
||||
|
||||
|
||||
def _coord(value, limit):
|
||||
"""Coerce a coordinate to a valid float in [-limit, limit] or None.
|
||||
|
||||
bool is a subclass of int but never a real coordinate, so True/False are
|
||||
treated as missing. NaN and out-of-range values are rejected.
|
||||
"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
f = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
# NaN is the only value that is not equal to itself.
|
||||
if f != f or f < -limit or f > limit:
|
||||
return None
|
||||
return f
|
||||
|
||||
|
||||
def _haversine_km(lat1, lon1, lat2, lon2):
|
||||
"""Great-circle distance in km between two (lat, lon) points in degrees."""
|
||||
rlat1, rlat2 = math.radians(lat1), math.radians(lat2)
|
||||
dlat = math.radians(lat2 - lat1)
|
||||
dlon = math.radians(lon2 - lon1)
|
||||
a = math.sin(dlat / 2.0) ** 2 + math.cos(rlat1) * math.cos(rlat2) * math.sin(dlon / 2.0) ** 2
|
||||
return 2.0 * _EARTH_RADIUS_KM * math.asin(min(1.0, math.sqrt(a)))
|
||||
|
||||
|
||||
def _region_of(lat, lon):
|
||||
"""Return the name of the first embedded box containing (lat, lon)."""
|
||||
for name, lat_min, lat_max, lon_min, lon_max in _REGION_BBOXES:
|
||||
if lat_min <= lat <= lat_max and lon_min <= lon <= lon_max:
|
||||
return name
|
||||
return "Océano/Otros"
|
||||
|
||||
|
||||
def _empty_result():
|
||||
"""Result shape when there are no valid coordinate pairs."""
|
||||
return {
|
||||
"n_points": 0,
|
||||
"bbox": None,
|
||||
"centroid": None,
|
||||
"span_km": 0.0,
|
||||
"by_region": [],
|
||||
"hemisphere": {"north": 0, "south": 0, "east": 0, "west": 0},
|
||||
"note": "sin coordenadas validas",
|
||||
}
|
||||
|
||||
|
||||
def analyze_geo_extent(lats: list, lons: list) -> dict:
|
||||
"""Summarise the geographic extent of a cloud of lat/lon coordinates.
|
||||
|
||||
Pairs `lats[i]` with `lons[i]` by index (over the common length when the two
|
||||
lists differ in size), discards any pair where either value is None / NaN or
|
||||
outside [-90, 90] (lat) / [-180, 180] (lon), and derives the bounding box,
|
||||
centroid, diagonal span, per-region counts and hemisphere split. Each valid
|
||||
point is matched to a country/region by an offline lookup against coarse
|
||||
rectangular bounding boxes (`_REGION_BBOXES`).
|
||||
|
||||
Args:
|
||||
lats: List of latitudes in degrees ([-90, 90]); read defensively.
|
||||
lons: List of longitudes in degrees ([-180, 180]); read defensively.
|
||||
Paired with `lats` by index; the shorter length wins when they differ.
|
||||
|
||||
Returns:
|
||||
Dict with the geographic summary:
|
||||
{n_points, bbox={lat_min,lat_max,lon_min,lon_max}, centroid={lat,lon},
|
||||
span_km (haversine of the SW->NE bbox diagonal), by_region=[{region,count}]
|
||||
(descending, top-8 with the rest folded into "Otros"),
|
||||
hemisphere={north,south,east,west}, note (Spanish summary phrase)}.
|
||||
With no valid pairs returns the zeroed shape: n_points 0, bbox None,
|
||||
centroid None, span_km 0.0, empty by_region, zeroed hemisphere and the
|
||||
note "sin coordenadas validas". Never raises.
|
||||
"""
|
||||
if not isinstance(lats, (list, tuple)) or not isinstance(lons, (list, tuple)):
|
||||
return _empty_result()
|
||||
|
||||
valid = []
|
||||
# zip already stops at the shorter list -> unbalanced lengths are handled.
|
||||
for raw_lat, raw_lon in zip(lats, lons):
|
||||
lat = _coord(raw_lat, 90.0)
|
||||
lon = _coord(raw_lon, 180.0)
|
||||
if lat is None or lon is None:
|
||||
continue
|
||||
valid.append((lat, lon))
|
||||
|
||||
if not valid:
|
||||
return _empty_result()
|
||||
|
||||
n = len(valid)
|
||||
lat_vals = [p[0] for p in valid]
|
||||
lon_vals = [p[1] for p in valid]
|
||||
|
||||
lat_min, lat_max = min(lat_vals), max(lat_vals)
|
||||
lon_min, lon_max = min(lon_vals), max(lon_vals)
|
||||
|
||||
centroid_lat = sum(lat_vals) / n
|
||||
centroid_lon = sum(lon_vals) / n
|
||||
|
||||
# Diagonal span: SW corner (lat_min, lon_min) to NE corner (lat_max, lon_max).
|
||||
span_km = _haversine_km(lat_min, lon_min, lat_max, lon_max)
|
||||
|
||||
# Hemisphere split: the equator/prime-meridian go to north/east respectively.
|
||||
north = sum(1 for lat in lat_vals if lat >= 0.0)
|
||||
south = n - north
|
||||
east = sum(1 for lon in lon_vals if lon >= 0.0)
|
||||
west = n - east
|
||||
|
||||
# Count points per region (offline bbox lookup).
|
||||
counts = {}
|
||||
for lat, lon in valid:
|
||||
region = _region_of(lat, lon)
|
||||
counts[region] = counts.get(region, 0) + 1
|
||||
|
||||
# Descending by count, then by name for a deterministic tie-break.
|
||||
ranked = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))
|
||||
by_region = [{"region": name, "count": count} for name, count in ranked[:_TOP_REGIONS]]
|
||||
rest = sum(count for _, count in ranked[_TOP_REGIONS:])
|
||||
if rest > 0:
|
||||
by_region.append({"region": "Otros", "count": rest})
|
||||
|
||||
top_region, top_count = ranked[0]
|
||||
note = (
|
||||
"los puntos se concentran en {region} ({count} de {n})".format(
|
||||
region=top_region, count=top_count, n=n
|
||||
)
|
||||
)
|
||||
|
||||
return {
|
||||
"n_points": n,
|
||||
"bbox": {
|
||||
"lat_min": lat_min,
|
||||
"lat_max": lat_max,
|
||||
"lon_min": lon_min,
|
||||
"lon_max": lon_max,
|
||||
},
|
||||
"centroid": {"lat": centroid_lat, "lon": centroid_lon},
|
||||
"span_km": span_km,
|
||||
"by_region": by_region,
|
||||
"hemisphere": {"north": north, "south": south, "east": east, "west": west},
|
||||
"note": note,
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Tests para analyze_geo_extent."""
|
||||
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from analyze_geo_extent import analyze_geo_extent, _haversine_km
|
||||
|
||||
# Keys that a non-empty result dict must always contain.
|
||||
_EXPECTED_KEYS = {
|
||||
"n_points", "bbox", "centroid", "span_km",
|
||||
"by_region", "hemisphere", "note",
|
||||
}
|
||||
|
||||
|
||||
def test_nube_en_espana():
|
||||
"""Golden: nube de puntos alrededor de Madrid -> region top = España."""
|
||||
# Cuatro puntos en torno a Madrid (lat ~40, lon ~-3.7), con algo de spread.
|
||||
lats = [40.4, 40.0, 41.0, 39.5]
|
||||
lons = [-3.7, -3.5, -4.0, -3.2]
|
||||
res = analyze_geo_extent(lats, lons)
|
||||
|
||||
assert set(res.keys()) == _EXPECTED_KEYS
|
||||
assert res["n_points"] == 4
|
||||
|
||||
# Todos caen en España -> by_region una sola entrada.
|
||||
assert res["by_region"][0]["region"] == "España"
|
||||
assert res["by_region"][0]["count"] == 4
|
||||
|
||||
# Centroide coherente: media de lat y lon.
|
||||
assert math.isclose(res["centroid"]["lat"], sum(lats) / 4, rel_tol=1e-9)
|
||||
assert math.isclose(res["centroid"]["lon"], sum(lons) / 4, rel_tol=1e-9)
|
||||
|
||||
# bbox correcto.
|
||||
assert res["bbox"]["lat_min"] == 39.5
|
||||
assert res["bbox"]["lat_max"] == 41.0
|
||||
assert res["bbox"]["lon_min"] == -4.0
|
||||
assert res["bbox"]["lon_max"] == -3.2
|
||||
|
||||
# Hay spread -> diagonal > 0.
|
||||
assert res["span_km"] > 0.0
|
||||
|
||||
# Hemisferio norte (lat>0) y oeste (lon<0).
|
||||
assert res["hemisphere"]["north"] == 4
|
||||
assert res["hemisphere"]["south"] == 0
|
||||
assert res["hemisphere"]["east"] == 0
|
||||
assert res["hemisphere"]["west"] == 4
|
||||
|
||||
assert "España" in res["note"]
|
||||
|
||||
|
||||
def test_dos_paises_distintos():
|
||||
"""Golden: puntos en España y Francia -> by_region con 2 entradas."""
|
||||
# Madrid (España) x2 y Paris (Francia) x1.
|
||||
lats = [40.4, 40.0, 48.8]
|
||||
lons = [-3.7, -3.5, 2.3]
|
||||
res = analyze_geo_extent(lats, lons)
|
||||
|
||||
assert res["n_points"] == 3
|
||||
regions = {entry["region"]: entry["count"] for entry in res["by_region"]}
|
||||
assert regions == {"España": 2, "Francia": 1}
|
||||
# Orden descendente por count: España (2) antes que Francia (1).
|
||||
assert res["by_region"][0]["region"] == "España"
|
||||
assert res["by_region"][0]["count"] == 2
|
||||
|
||||
# Madrid y Paris ambos hemisferio norte; Paris lon>0 -> 1 east, 2 west.
|
||||
assert res["hemisphere"]["north"] == 3
|
||||
assert res["hemisphere"]["east"] == 1
|
||||
assert res["hemisphere"]["west"] == 2
|
||||
|
||||
|
||||
def test_listas_vacias():
|
||||
"""Edge: listas vacias -> n_points 0, bbox None, sin lanzar."""
|
||||
res = analyze_geo_extent([], [])
|
||||
assert res["n_points"] == 0
|
||||
assert res["bbox"] is None
|
||||
assert res["centroid"] is None
|
||||
assert res["span_km"] == 0.0
|
||||
assert res["by_region"] == []
|
||||
assert res["hemisphere"] == {"north": 0, "south": 0, "east": 0, "west": 0}
|
||||
assert res["note"] == "sin coordenadas validas"
|
||||
|
||||
|
||||
def test_pares_invalidos_filtrados():
|
||||
"""Edge: None / NaN / fuera de rango se descartan, no lanza."""
|
||||
nan = float("nan")
|
||||
lats = [40.4, None, nan, 91.0, -200.0, 40.0]
|
||||
lons = [-3.7, -3.5, -3.0, 2.0, 5.0, -3.5]
|
||||
# Validos: indices 0 y 5 (lat 91 fuera de rango, lon -200 fuera de rango,
|
||||
# None y NaN descartados).
|
||||
res = analyze_geo_extent(lats, lons)
|
||||
assert res["n_points"] == 2
|
||||
assert res["by_region"][0]["region"] == "España"
|
||||
assert res["by_region"][0]["count"] == 2
|
||||
|
||||
|
||||
def test_longitudes_desbalanceadas():
|
||||
"""Edge: len(lats) != len(lons) usa el minimo comun sin lanzar."""
|
||||
lats = [40.4, 40.0, 41.0, 39.5] # 4 elementos
|
||||
lons = [-3.7, -3.5] # 2 elementos
|
||||
res = analyze_geo_extent(lats, lons)
|
||||
# Solo se emparejan los 2 primeros.
|
||||
assert res["n_points"] == 2
|
||||
assert res["bbox"]["lat_min"] == 40.0
|
||||
assert res["bbox"]["lat_max"] == 40.4
|
||||
|
||||
|
||||
def test_span_km_haversine_par_conocido():
|
||||
"""Edge: span_km coincide con haversine de la diagonal del bbox."""
|
||||
# Dos puntos: (0, 0) y (0, 1). bbox diagonal = mismos dos puntos.
|
||||
res = analyze_geo_extent([0.0, 0.0], [0.0, 1.0])
|
||||
# 1 grado de longitud en el ecuador ~ 111.19 km.
|
||||
expected = _haversine_km(0.0, 0.0, 0.0, 1.0)
|
||||
assert math.isclose(res["span_km"], expected, rel_tol=1e-9)
|
||||
assert math.isclose(res["span_km"], 111.19, abs_tol=0.5)
|
||||
|
||||
|
||||
def test_no_lanza_con_entradas_raras():
|
||||
"""Edge: tipos no-lista o None devuelven la forma vacia sin lanzar."""
|
||||
assert analyze_geo_extent(None, None)["n_points"] == 0
|
||||
assert analyze_geo_extent("foo", "bar")["n_points"] == 0
|
||||
# Strings dentro de las listas se descartan como invalidos.
|
||||
res = analyze_geo_extent(["x", 40.0], [None, -3.5])
|
||||
assert res["n_points"] == 1
|
||||
@@ -21,6 +21,9 @@ from .model import ( # noqa: F401
|
||||
Chapter,
|
||||
DataTable,
|
||||
Figure,
|
||||
GlossaryCollector,
|
||||
GlossaryEntry,
|
||||
Group,
|
||||
Heading,
|
||||
Image,
|
||||
KVTable,
|
||||
@@ -45,6 +48,9 @@ __all__ = [
|
||||
"Image",
|
||||
"Caption",
|
||||
"Note",
|
||||
"Group",
|
||||
"GlossaryEntry",
|
||||
"GlossaryCollector",
|
||||
"Chapter",
|
||||
"as_blocks",
|
||||
"as_chapters",
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Tests for inline-bold rendering (**bold**) in the AutomaticEDA engine.
|
||||
|
||||
Covers the pure helpers (parse_inline_bold / wrap_rich) and an end-to-end PPTX
|
||||
check that a ``**bold**`` span is rendered with NATIVE PowerPoint bold
|
||||
(``run.font.bold is True``) while no line overflows the wrap width (no-cut).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
# Make the engine importable as a package (datascience.automatic_eda).
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
|
||||
if _FUNCTIONS not in sys.path:
|
||||
sys.path.insert(0, _FUNCTIONS)
|
||||
|
||||
from datascience.automatic_eda import model # noqa: E402
|
||||
from datascience.automatic_eda import text_layout as tl # noqa: E402
|
||||
from datascience.automatic_eda import render_pptx # noqa: E402
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Pure helpers.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_parse_inline_bold_marks_spans_and_preserves_visible_text():
|
||||
src = "**Estacionariedad:** serie no estacionaria con `code` y normal."
|
||||
segs = tl.parse_inline_bold(src)
|
||||
# Visible text equals strip_inline_md (no characters lost, markers removed).
|
||||
visible = "".join(s for s, _ in segs)
|
||||
assert visible == tl.strip_inline_md(src)
|
||||
# The span "Estacionariedad:" is flagged bold; the rest is not.
|
||||
bold_text = "".join(s for s, b in segs if b)
|
||||
assert "Estacionariedad:" in bold_text
|
||||
assert "serie no estacionaria" not in bold_text
|
||||
|
||||
|
||||
def test_parse_inline_bold_handles_unbalanced_markers():
|
||||
# An unbalanced ** must not crash and must be stripped (matches strip_inline_md).
|
||||
segs = tl.parse_inline_bold("texto **sin cierre aqui")
|
||||
visible = "".join(s for s, _ in segs)
|
||||
assert visible == "texto sin cierre aqui"
|
||||
assert not any(b for _, b in segs) # nothing rendered bold.
|
||||
|
||||
|
||||
def test_wrap_rich_never_overflows_and_keeps_bold():
|
||||
text = ("**Segmento premium.** Clientes de alto gasto y baja frecuencia con "
|
||||
"ticket medio elevado y recurrencia anual estable a lo largo del año.")
|
||||
max_chars = 30
|
||||
lines = tl.wrap_rich(text, max_chars)
|
||||
# No visible line exceeds max_chars (no-cut: the renderer measures these).
|
||||
for ln in lines:
|
||||
visible = "".join(s for s, _ in ln)
|
||||
assert len(visible) <= max_chars, f"línea desborda: {visible!r}"
|
||||
# At least one segment is bold and it is the span content.
|
||||
bold_segs = [s for ln in lines for s, b in ln if b]
|
||||
assert any("Segmento premium." in s for s in bold_segs)
|
||||
|
||||
|
||||
def test_wrap_rich_hard_splits_long_token():
|
||||
long = "x" * 50
|
||||
lines = tl.wrap_rich(f"**{long}**", 20)
|
||||
for ln in lines:
|
||||
assert len("".join(s for s, _ in ln)) <= 20
|
||||
# The whole long token is preserved across the split lines.
|
||||
joined = "".join(s for ln in lines for s, _ in ln)
|
||||
assert joined == long
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# End-to-end: PPTX renders **bold** as a real bold run.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _has_pptx():
|
||||
try:
|
||||
import pptx # noqa: F401
|
||||
return True
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _has_pptx(), reason="python-pptx no instalado")
|
||||
def test_pptx_renders_bold_span_as_native_bold_run(tmp_path):
|
||||
from pptx import Presentation
|
||||
|
||||
doc = [model.Chapter(
|
||||
id="t", title="Negrita", version="1.0.0",
|
||||
blocks=[model.Markdown(
|
||||
text="Frase con **PALABRACLAVE** resaltada y texto normal después.")],
|
||||
)]
|
||||
out = str(tmp_path / "bold.pptx")
|
||||
res = render_pptx(doc, out, {"title": "T"})
|
||||
assert res.get("path") == out
|
||||
assert os.path.exists(out)
|
||||
|
||||
prs = Presentation(out)
|
||||
bold_texts = []
|
||||
all_text = []
|
||||
for slide in prs.slides:
|
||||
for shape in slide.shapes:
|
||||
if not shape.has_text_frame:
|
||||
continue
|
||||
for para in shape.text_frame.paragraphs:
|
||||
for run in para.runs:
|
||||
all_text.append(run.text)
|
||||
if run.font.bold:
|
||||
bold_texts.append(run.text)
|
||||
# The bold span text appears in a run with font.bold True (native bold).
|
||||
assert any("PALABRACLAVE" in t for t in bold_texts), \
|
||||
f"no se encontró run bold con el span; bold={bold_texts}"
|
||||
# And the surrounding plain text is NOT bold (markers did not bleed).
|
||||
assert any("resaltada" in t for t in all_text)
|
||||
assert not any("resaltada" in t for t in bold_texts)
|
||||
@@ -0,0 +1,221 @@
|
||||
"""LLM analysis chapter (ANÁLISIS LLM) — the interpretive layer, next to overview.
|
||||
|
||||
Third reference chapter for AutomaticEDA. Renders the ``llm`` block that the
|
||||
``eda`` group function ``eda_llm_insights`` already produced and stored in the
|
||||
``TableProfile`` — it does NOT call the LLM nor recompute anything. The block is
|
||||
turned into clean, markdown-style document blocks so it reads as a real chapter
|
||||
(table summary, row meaning, data dictionary, suggested analyses, cleaning
|
||||
suggestions, PII findings) and, crucially, **nothing is ever cut** in PDF or
|
||||
PPTX:
|
||||
|
||||
* Prose (summary, row meaning) → ``Markdown`` blocks the renderers wrap to whole
|
||||
lines, so no word is lost no matter how long the text is.
|
||||
* The data dictionary and PII findings → ``DataTable`` blocks the paginator
|
||||
splits by rows (repeating the header) and whose long cells wrap inside their
|
||||
column — wide, multi-row tables never overflow a page/slide.
|
||||
* Cleaning suggestions and suggested analyses → ``Markdown`` bullet lists; each
|
||||
item is a whole line the renderer wraps, never truncated mid-entry.
|
||||
|
||||
Position: this chapter is declared in ``chapters_registry.CHAPTER_ORDER`` right
|
||||
after ``overview`` so the interpretation sits next to the table preview, as the
|
||||
user asked ("va junto al overview").
|
||||
|
||||
Data source: the ``llm`` dict produced by ``eda_llm_insights`` (group ``eda``),
|
||||
read from ``profile['llm']`` (or ``ctx['llm']`` as a fallback). Shape::
|
||||
|
||||
{
|
||||
"summary": str, # what the table is, 2-3 sentences
|
||||
"row_meaning": str, # what one row represents / granularity
|
||||
"dictionary": [ {"column","description","business_meaning","unit"} ],
|
||||
"pii": [ {"column","kind","severity"} ],
|
||||
"cleaning": [str], # cleaning / transformation suggestions
|
||||
"analyses": [str], # suggested questions / analyses / hypotheses
|
||||
}
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
Reads everything defensively (``.get``) and NEVER raises; returns ``None`` when
|
||||
the profile carries no LLM block (e.g. ``profile_table`` ran without
|
||||
``run_llm``), so the chapter is simply omitted from the document.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "analisis_llm"
|
||||
CHAPTER_TITLE = "Análisis LLM"
|
||||
|
||||
# Key under which eda_llm_insights stores its interpretive block in the profile.
|
||||
LLM_KEY = "llm"
|
||||
|
||||
|
||||
def _clean_text(value) -> str:
|
||||
"""Coerce a value to a single trimmed line (collapse inner newlines).
|
||||
|
||||
Used for bullet items so each suggestion stays a single markdown bullet the
|
||||
renderer wraps; never drops content, only normalizes whitespace.
|
||||
"""
|
||||
text = model._safe_str(value).strip()
|
||||
if not text:
|
||||
return ""
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _para(value) -> str:
|
||||
"""Coerce a value to trimmed prose, preserving paragraph breaks."""
|
||||
text = model._safe_str(value).strip()
|
||||
if not text:
|
||||
return ""
|
||||
# Keep blank-line paragraph breaks; collapse runs of spaces/tabs per line.
|
||||
lines = [" ".join(ln.split()) for ln in text.splitlines()]
|
||||
out: list = []
|
||||
for ln in lines:
|
||||
if ln or (out and out[-1] != ""):
|
||||
out.append(ln)
|
||||
return "\n".join(out).strip()
|
||||
|
||||
|
||||
def _bullets(items) -> str:
|
||||
"""Build a markdown bullet list from a sequence of strings.
|
||||
|
||||
Each item becomes one ``- ...`` line (a whole, wrappable unit). Empty items
|
||||
and non-list inputs are handled gracefully; returns "" when there is nothing.
|
||||
"""
|
||||
if isinstance(items, str):
|
||||
items = [items]
|
||||
if not isinstance(items, (list, tuple)):
|
||||
return ""
|
||||
lines = []
|
||||
for it in items:
|
||||
text = _clean_text(it)
|
||||
if text:
|
||||
lines.append(f"- {text}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _summary_blocks(llm: dict) -> list:
|
||||
"""Heading + prose for the table summary, or [] if absent."""
|
||||
text = _para(llm.get("summary"))
|
||||
if not text:
|
||||
return []
|
||||
return [model.Heading(text="Resumen de la tabla", level=2),
|
||||
model.Markdown(text=text)]
|
||||
|
||||
|
||||
def _row_meaning_blocks(llm: dict) -> list:
|
||||
"""Heading + prose for what one row represents, or [] if absent."""
|
||||
text = _para(llm.get("row_meaning"))
|
||||
if not text:
|
||||
return []
|
||||
return [model.Heading(text="Significado de una fila", level=2),
|
||||
model.Markdown(text=text)]
|
||||
|
||||
|
||||
def _dictionary_block(llm: dict):
|
||||
"""DataTable for the data dictionary, or None if absent/empty.
|
||||
|
||||
Columns: Columna / Descripción / Significado de negocio / Unidad. The
|
||||
paginator splits this by rows repeating the header and wraps long cells, so a
|
||||
long dictionary (many columns) never gets cut.
|
||||
"""
|
||||
entries = llm.get("dictionary")
|
||||
if not isinstance(entries, (list, tuple)) or not entries:
|
||||
return None
|
||||
header = ["Columna", "Descripción", "Significado de negocio", "Unidad"]
|
||||
rows = []
|
||||
for e in entries:
|
||||
if not isinstance(e, dict):
|
||||
# Be tolerant: a bare string still shows up as a description row.
|
||||
rows.append(["—", _clean_text(e), "", ""])
|
||||
continue
|
||||
rows.append([
|
||||
_clean_text(e.get("column")) or "—",
|
||||
_clean_text(e.get("description")),
|
||||
_clean_text(e.get("business_meaning")),
|
||||
_clean_text(e.get("unit")),
|
||||
])
|
||||
if not rows:
|
||||
return None
|
||||
return model.DataTable(header=header, rows=rows, title="Diccionario de datos")
|
||||
|
||||
|
||||
def _analyses_blocks(llm: dict) -> list:
|
||||
"""Heading + bullet list of suggested analyses, or [] if absent."""
|
||||
bullets = _bullets(llm.get("analyses"))
|
||||
if not bullets:
|
||||
return []
|
||||
return [model.Heading(text="Análisis sugeridos", level=2),
|
||||
model.Markdown(text=bullets)]
|
||||
|
||||
|
||||
def _cleaning_blocks(llm: dict) -> list:
|
||||
"""Heading + bullet list of cleaning suggestions, or [] if absent."""
|
||||
bullets = _bullets(llm.get("cleaning"))
|
||||
if not bullets:
|
||||
return []
|
||||
return [model.Heading(text="Limpieza sugerida", level=2),
|
||||
model.Markdown(text=bullets)]
|
||||
|
||||
|
||||
def _pii_block(llm: dict):
|
||||
"""DataTable for PII/GDPR findings, or None if absent/empty."""
|
||||
entries = llm.get("pii")
|
||||
if not isinstance(entries, (list, tuple)) or not entries:
|
||||
return None
|
||||
header = ["Columna", "Tipo", "Severidad"]
|
||||
rows = []
|
||||
for e in entries:
|
||||
if not isinstance(e, dict):
|
||||
continue
|
||||
rows.append([
|
||||
_clean_text(e.get("column")) or "—",
|
||||
_clean_text(e.get("kind")),
|
||||
_clean_text(e.get("severity")),
|
||||
])
|
||||
if not rows:
|
||||
return None
|
||||
return model.DataTable(
|
||||
header=header, rows=rows, title="Datos personales (PII / RGPD)",
|
||||
note="detección automática orientativa — revisar antes de tratar los datos")
|
||||
|
||||
|
||||
def build_analisis_llm(profile: dict, ctx: dict):
|
||||
"""Build the LLM analysis Chapter, or None if there is no LLM block.
|
||||
|
||||
Consumes ``profile['llm']`` (the block produced by ``eda_llm_insights``,
|
||||
group ``eda``); falls back to ``ctx['llm']``. Returns ``None`` when no LLM
|
||||
block is present or it carries no usable content, so the chapter is omitted
|
||||
rather than rendering an empty section.
|
||||
"""
|
||||
profile = profile or {}
|
||||
ctx = ctx or {}
|
||||
|
||||
llm = profile.get(LLM_KEY)
|
||||
if not isinstance(llm, dict):
|
||||
llm = ctx.get(LLM_KEY)
|
||||
if not isinstance(llm, dict) or not llm:
|
||||
return None
|
||||
|
||||
blocks: list = []
|
||||
blocks += _summary_blocks(llm)
|
||||
blocks += _row_meaning_blocks(llm)
|
||||
|
||||
dict_block = _dictionary_block(llm)
|
||||
if dict_block is not None:
|
||||
blocks.append(model.Heading(text="Diccionario de datos", level=2))
|
||||
blocks.append(dict_block)
|
||||
|
||||
blocks += _analyses_blocks(llm)
|
||||
blocks += _cleaning_blocks(llm)
|
||||
|
||||
pii_block = _pii_block(llm)
|
||||
if pii_block is not None:
|
||||
blocks.append(model.Heading(text="Datos personales (PII / RGPD)", level=2))
|
||||
blocks.append(pii_block)
|
||||
|
||||
if not blocks:
|
||||
return None # LLM block present but every field empty → omit chapter.
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,190 @@
|
||||
"""Tests for the ANÁLISIS LLM chapter — DoD: golden + edges + anti-cut.
|
||||
|
||||
Self-contained: builds a synthetic TableProfile carrying an ``llm`` block (the
|
||||
shape ``eda_llm_insights`` produces) so the suite is fast and deterministic — no
|
||||
DuckDB and no LLM call. Verifies:
|
||||
|
||||
* golden — ``build_analisis_llm`` yields the chapter and the full document
|
||||
renders to PDF *and* PPTX with the summary, a suggested analysis, a cleaning
|
||||
suggestion and a dictionary column all present;
|
||||
* order — the chapter sits immediately after ``overview`` (user requirement);
|
||||
* edges — a profile with no ``llm`` block (or None/empty/malformed) returns
|
||||
``None`` and never raises;
|
||||
* anti-cut — a long dictionary (40 rows) and a 150-char cleaning suggestion are
|
||||
rendered to PDF and PPTX without losing a single row or word.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
from pptx import Presentation
|
||||
|
||||
from datascience.automatic_eda.chapters.analisis_llm import (
|
||||
build_analisis_llm, CHAPTER_VERSION)
|
||||
from datascience.automatic_eda.chapters_registry import build_document
|
||||
from datascience.automatic_eda.model import Chapter, DataTable
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
|
||||
|
||||
def _profile() -> dict:
|
||||
return {
|
||||
"table": "ventas",
|
||||
"source": "/data/ventas.csv",
|
||||
"profiled_at": "2026-06-30T10:00:00+00:00",
|
||||
"n_rows": 1000,
|
||||
"n_cols": 2,
|
||||
"quality_score": 92.5,
|
||||
"columns": [
|
||||
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.0,
|
||||
"null_count": 0,
|
||||
"numeric": {"mean": 42.5, "median": 40.0, "min": 1.0,
|
||||
"max": 100.0, "std": 12.3}},
|
||||
{"name": "categoria", "inferred_type": "categorical",
|
||||
"null_pct": 0.0, "null_count": 0,
|
||||
"categorical": {"top": [{"value": "neumaticos", "count": 500}]}},
|
||||
],
|
||||
"llm": {
|
||||
"summary": "Tabla de ventas por producto. Token SUMMARYTOKEN.",
|
||||
"row_meaning": "Cada fila es una venta. Token ROWTOKEN.",
|
||||
"dictionary": [
|
||||
{"column": "precio", "description": "Precio unitario DESCTOKEN",
|
||||
"business_meaning": "Ingreso por unidad", "unit": "EUR"},
|
||||
{"column": "categoria", "description": "Familia de producto",
|
||||
"business_meaning": "Segmento comercial", "unit": ""},
|
||||
],
|
||||
"pii": [{"column": "categoria", "kind": "ninguno", "severity": "low"}],
|
||||
"cleaning": ["Quitar nulos de precio CLEANTOKEN",
|
||||
"Normalizar mayusculas en categoria"],
|
||||
"analyses": ["Estudiar relacion precio-categoria ANALYSISTOKEN",
|
||||
"Detectar outliers de precio"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _pdf_text(path: str) -> str:
|
||||
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
|
||||
return re.sub(r"\s+", " ", txt)
|
||||
|
||||
|
||||
def _pptx_text(path: str) -> str:
|
||||
prs = Presentation(path)
|
||||
parts = []
|
||||
for sl in prs.slides:
|
||||
for sh in sl.shapes:
|
||||
if sh.has_text_frame:
|
||||
parts.append(sh.text_frame.text)
|
||||
if sh.has_table:
|
||||
tb = sh.table
|
||||
for r in range(len(tb.rows)):
|
||||
for c in range(len(tb.columns)):
|
||||
parts.append(tb.cell(r, c).text)
|
||||
return re.sub(r"\s+", " ", " ".join(parts))
|
||||
|
||||
|
||||
def test_golden_build_y_render_pdf_pptx():
|
||||
prof = _profile()
|
||||
ch = build_analisis_llm(prof, {})
|
||||
assert ch is not None
|
||||
assert ch.id == "analisis_llm"
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
assert ch.blocks # non-empty.
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
out_pdf = os.path.join(d, "eda.pdf")
|
||||
res = render_automatic_eda_pdf(prof, out_pdf, {"title": "EDA — ventas"})
|
||||
assert res["path"] == out_pdf and os.path.exists(out_pdf)
|
||||
ids = [c["id"] for c in res["chapters"]]
|
||||
assert "analisis_llm" in ids
|
||||
txt = _pdf_text(out_pdf)
|
||||
# The user's required content: summary, suggested analyses, cleaning.
|
||||
assert "SUMMARYTOKEN" in txt
|
||||
assert "ANALYSISTOKEN" in txt
|
||||
assert "CLEANTOKEN" in txt
|
||||
assert "DESCTOKEN" in txt # data dictionary cell.
|
||||
|
||||
out_pptx = os.path.join(d, "eda.pptx")
|
||||
res2 = render_automatic_eda_pptx(prof, out_pptx, {"title": "EDA — ventas"})
|
||||
assert res2["path"] == out_pptx and os.path.exists(out_pptx)
|
||||
ids2 = [c["id"] for c in res2["chapters"]]
|
||||
assert "analisis_llm" in ids2
|
||||
ptx = _pptx_text(out_pptx)
|
||||
assert "SUMMARYTOKEN" in ptx
|
||||
assert "ANALYSISTOKEN" in ptx
|
||||
assert "CLEANTOKEN" in ptx
|
||||
assert "DESCTOKEN" in ptx
|
||||
|
||||
|
||||
def test_orden_capitulo_junto_a_overview():
|
||||
chapters = build_document(_profile(), {})
|
||||
ids = [c.id for c in chapters]
|
||||
assert "overview" in ids and "analisis_llm" in ids
|
||||
# User requirement: the LLM chapter sits right after overview.
|
||||
assert ids.index("analisis_llm") == ids.index("overview") + 1
|
||||
|
||||
|
||||
def test_edge_sin_llm_devuelve_none():
|
||||
# No llm block at all.
|
||||
prof = {k: v for k, v in _profile().items() if k != "llm"}
|
||||
assert build_analisis_llm(prof, {}) is None
|
||||
# None / empty / malformed never raise and yield None.
|
||||
assert build_analisis_llm(None, None) is None
|
||||
assert build_analisis_llm({}, {}) is None
|
||||
assert build_analisis_llm({"llm": {}}, {}) is None
|
||||
assert build_analisis_llm({"llm": "not-a-dict"}, {}) is None
|
||||
# All-empty fields → omitted (no blocks).
|
||||
empty = {"llm": {"summary": "", "dictionary": [], "cleaning": [],
|
||||
"analyses": [], "pii": [], "row_meaning": ""}}
|
||||
assert build_analisis_llm(empty, {}) is None
|
||||
|
||||
|
||||
def test_edge_llm_via_ctx_fallback():
|
||||
# The block may arrive in ctx instead of the profile.
|
||||
prof = {k: v for k, v in _profile().items() if k != "llm"}
|
||||
ctx = {"llm": {"summary": "Resumen via ctx CTXTOKEN."}}
|
||||
ch = build_analisis_llm(prof, ctx)
|
||||
assert ch is not None and ch.id == "analisis_llm"
|
||||
|
||||
|
||||
def test_anti_cortes_diccionario_largo_y_limpieza_larga():
|
||||
long_clean = ("Lorem ipsum dolor sit amet consectetur adipiscing elit sed do "
|
||||
"eiusmod tempor incididunt ut labore et dolore magna aliqua "
|
||||
"reprehenderit voluptate velit esse cillum dolore")
|
||||
dictionary = [
|
||||
{"column": f"col_{i}",
|
||||
"description": f"Descripcion larga numero {i} con bastante texto para "
|
||||
f"forzar el wrap dentro de la celda fila{i}",
|
||||
"business_meaning": f"Significado de negocio {i}", "unit": "u"}
|
||||
for i in range(40)
|
||||
]
|
||||
prof = {
|
||||
"table": "t", "n_rows": 1, "n_cols": 1, "columns": [],
|
||||
"llm": {"summary": "S", "dictionary": dictionary,
|
||||
"cleaning": [long_clean], "analyses": ["A"]},
|
||||
}
|
||||
ch = build_analisis_llm(prof, {})
|
||||
assert ch is not None
|
||||
# Structure: the dictionary DataTable keeps ALL 40 rows — none dropped on
|
||||
# construction (the renderers then split it by rows, repeating the header).
|
||||
dts = [b for b in ch.blocks if isinstance(b, DataTable)]
|
||||
assert any(len(dt.rows) == 40 for dt in dts)
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
out_pdf = os.path.join(d, "x.pdf")
|
||||
render_automatic_eda_pdf([ch], out_pdf, {"write_manifest": False})
|
||||
# 40 wide rows + a long cleaning line cannot fit one page → it spills,
|
||||
# which is exactly the no-cut behaviour (paginate, never truncate).
|
||||
assert len(PdfReader(out_pdf).pages) > 1
|
||||
txt = _pdf_text(out_pdf)
|
||||
# The long cleaning suggestion is wrapped word-by-word, not truncated.
|
||||
for word in ("Lorem", "incididunt", "reprehenderit", "voluptate", "cillum"):
|
||||
assert word in txt
|
||||
|
||||
out_pptx = os.path.join(d, "x.pptx")
|
||||
res2 = render_automatic_eda_pptx([ch], out_pptx, {"write_manifest": False})
|
||||
assert res2["n_slides"] > 1 # table + long text spill across slides.
|
||||
ptx = _pptx_text(out_pptx)
|
||||
for word in ("Lorem", "reprehenderit", "voluptate"):
|
||||
assert word in ptx
|
||||
@@ -0,0 +1,427 @@
|
||||
"""Categorical distributions chapter (CAT DISTR).
|
||||
|
||||
Third reference chapter for AutomaticEDA. For every categorical column it shows,
|
||||
fulfilling the user's request:
|
||||
|
||||
1. A short opening explanation of **Shannon entropy** (what it measures, its 0
|
||||
and log2(k) bounds, the normalized 0–1 version) and the dataset row total used
|
||||
as a comparison baseline.
|
||||
2. Per column, a cardinality key/value table: distinct values, ``% distinct``
|
||||
(distinct / total rows), total dataset rows, singleton values (frequency 1),
|
||||
entropy with its theoretical maximum and the normalized ratio, mode, imbalance
|
||||
and string-length stats.
|
||||
3. A short note flagging problematic cardinality (id-like ≈100% distinct, or a
|
||||
single dominating category).
|
||||
4. A ``top-k`` table (value / count / %).
|
||||
5. A **donut pie chart** of the most common categories (top-k + an "Otros"
|
||||
bucket), drawn lazily so the renderers scale it to fit entirely.
|
||||
|
||||
Data comes from the ``eda`` group: each ``columns[i]['categorical']`` is the
|
||||
output of ``summarize_categorical`` (``top[{value,count,pct}]``, ``mode``,
|
||||
``n_distinct``, ``entropy``, ``imbalance``, ``len_min/mean/max``). The derived
|
||||
cardinality metrics and the pie figure are delegated to two registry functions
|
||||
(``categorical_cardinality_block`` and ``categorical_top_pie_figure``); both are
|
||||
imported lazily and degrade to a minimal inline fallback so this chapter never
|
||||
raises even if they are unavailable.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.1.0"
|
||||
CHAPTER_ID = "cat_distr"
|
||||
CHAPTER_TITLE = "Distribuciones categóricas"
|
||||
|
||||
# Glossary term this chapter explains. Registered in the shared collector and
|
||||
# marked clickable on its first appearance (end-to-end glossary example —
|
||||
# mejora 6). Other chapters hook their own terms the same way (see the contract).
|
||||
_TERM_ENTROPIA_KEY = "entropia"
|
||||
_TERM_ENTROPIA_LABEL = "Entropía (de Shannon)"
|
||||
_TERM_ENTROPIA_DEF = (
|
||||
"Medida, en bits, de cómo de repartidos están los valores de una columna "
|
||||
"categórica. Vale 0 cuando una sola categoría concentra todas las filas "
|
||||
"(máxima previsibilidad) y alcanza su máximo, log2(k) para k categorías "
|
||||
"distintas, cuando todas aparecen por igual (máxima diversidad). La entropía "
|
||||
"normalizada (entropía dividida por su máximo) la lleva al rango 0–1 para "
|
||||
"comparar columnas con distinto número de categorías.")
|
||||
|
||||
# Cap the number of categorical columns rendered to keep the document bounded;
|
||||
# the rest are summarized in a closing note (no silent truncation).
|
||||
MAX_COLS = 40
|
||||
# Rows shown in each top-k table and explicit slices in the pie.
|
||||
TOP_TABLE_ROWS = 15
|
||||
PIE_TOP_K = 6
|
||||
# Truncate very long category labels in tables (the renderer also wraps).
|
||||
LABEL_MAX = 48
|
||||
|
||||
|
||||
def _fmt_int(value) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
return f"{int(value):,}".replace(",", ".")
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
|
||||
|
||||
def _fmt_num(value, decimals: int = 3) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return str(value)
|
||||
if isinstance(value, int):
|
||||
return f"{value:,}".replace(",", ".")
|
||||
if isinstance(value, float):
|
||||
if value != value: # NaN
|
||||
return "NaN"
|
||||
if value in (float("inf"), float("-inf")):
|
||||
return str(value)
|
||||
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
|
||||
return text if text else "0"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _fmt_pct_value(value, decimals: int = 1) -> str:
|
||||
"""Format an already-in-percent value (0–100). None -> placeholder."""
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
return f"{float(value):.{decimals}f}%"
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
|
||||
|
||||
def _pct_from_maybe_fraction(value, decimals: int = 1) -> str:
|
||||
"""Format a percentage that may arrive as a 0–1 fraction or a 0–100 number."""
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
v = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
if v <= 1.0:
|
||||
v *= 100.0
|
||||
return f"{v:.{decimals}f}%"
|
||||
|
||||
|
||||
def _truncate(text: str, limit: int = LABEL_MAX) -> str:
|
||||
s = model._safe_str(text)
|
||||
if len(s) <= limit:
|
||||
return s
|
||||
return s[: max(1, limit - 1)].rstrip() + "…"
|
||||
|
||||
|
||||
def _is_categorical(col: dict) -> bool:
|
||||
"""A column is treated as categorical when it carries a non-empty top list
|
||||
and is not a pure numeric column (numeric columns may still expose a top)."""
|
||||
if not isinstance(col, dict):
|
||||
return False
|
||||
cat = col.get("categorical")
|
||||
if not (isinstance(cat, dict) and cat.get("top")):
|
||||
return False
|
||||
if col.get("inferred_type") == "numeric":
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _cardinality(cat: dict, n_rows) -> dict:
|
||||
"""Derive cardinality metrics for a column, via the registry function when
|
||||
available, otherwise a minimal inline fallback. Never raises."""
|
||||
try:
|
||||
from datascience.categorical_cardinality_block import (
|
||||
categorical_cardinality_block,
|
||||
)
|
||||
|
||||
out = categorical_cardinality_block(cat=cat, n_rows=n_rows)
|
||||
if isinstance(out, dict):
|
||||
return out
|
||||
except Exception: # noqa: BLE001 — fall back to the inline derivation.
|
||||
pass
|
||||
return _fallback_cardinality(cat, n_rows)
|
||||
|
||||
|
||||
def _fallback_cardinality(cat: dict, n_rows) -> dict:
|
||||
cat = cat or {}
|
||||
top = cat.get("top") or []
|
||||
n_distinct = cat.get("n_distinct")
|
||||
entropy = cat.get("entropy")
|
||||
try:
|
||||
nr = int(n_rows) if n_rows is not None else None
|
||||
except (TypeError, ValueError):
|
||||
nr = None
|
||||
pct_distinct = None
|
||||
if isinstance(n_distinct, (int, float)) and nr:
|
||||
pct_distinct = float(n_distinct) / nr * 100.0
|
||||
entropy_max = None
|
||||
if isinstance(n_distinct, (int, float)):
|
||||
entropy_max = math.log2(n_distinct) if n_distinct > 1 else 0.0
|
||||
entropy_norm = None
|
||||
if isinstance(entropy, (int, float)) and entropy_max:
|
||||
entropy_norm = max(0.0, min(1.0, float(entropy) / entropy_max))
|
||||
mode_pct = cat.get("mode_pct")
|
||||
if mode_pct is None and top and isinstance(top[0], dict):
|
||||
mode_pct = top[0].get("pct")
|
||||
# Normalize to a 0–100 scale: summarize_categorical emits a 0–1 fraction.
|
||||
if isinstance(mode_pct, (int, float)) and not isinstance(mode_pct, bool):
|
||||
mode_pct = float(mode_pct) * 100.0 if mode_pct <= 1.0 else float(mode_pct)
|
||||
else:
|
||||
mode_pct = None
|
||||
n_singletons = None
|
||||
if top:
|
||||
n_singletons = sum(
|
||||
1 for t in top if isinstance(t, dict) and t.get("count") == 1)
|
||||
return {
|
||||
"n_distinct": n_distinct,
|
||||
"n_rows": nr,
|
||||
"pct_distinct": pct_distinct,
|
||||
"entropy": entropy,
|
||||
"entropy_max": entropy_max,
|
||||
"entropy_norm": entropy_norm,
|
||||
"mode": cat.get("mode"),
|
||||
"mode_pct": mode_pct,
|
||||
"imbalance": cat.get("imbalance"),
|
||||
"n_singletons": n_singletons,
|
||||
"n_singletons_partial": (
|
||||
isinstance(n_distinct, (int, float)) and n_distinct > len(top)),
|
||||
"len_min": cat.get("len_min"),
|
||||
"len_mean": cat.get("len_mean"),
|
||||
"len_max": cat.get("len_max"),
|
||||
"id_like": pct_distinct is not None and pct_distinct >= 99.0,
|
||||
"dominated": mode_pct is not None and mode_pct >= 90.0,
|
||||
}
|
||||
|
||||
|
||||
def _pie_make(top, n_distinct, title, n_rows):
|
||||
"""Return a zero-arg callable that builds the donut figure lazily."""
|
||||
|
||||
def make():
|
||||
try:
|
||||
from datascience.categorical_top_pie_figure import (
|
||||
categorical_top_pie_figure,
|
||||
)
|
||||
|
||||
return categorical_top_pie_figure(
|
||||
top=top, n_distinct=n_distinct or 0, title=title,
|
||||
top_k=PIE_TOP_K, n_rows=n_rows)
|
||||
except Exception: # noqa: BLE001 — minimal local fallback figure.
|
||||
return _fallback_pie(top, title)
|
||||
|
||||
return make
|
||||
|
||||
|
||||
def _fallback_pie(top, title):
|
||||
"""Minimal donut figure used only if the registry function is unavailable."""
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
from matplotlib.figure import Figure
|
||||
|
||||
fig = Figure(figsize=(5.0, 3.2))
|
||||
ax = fig.add_subplot(111)
|
||||
items = [t for t in (top or [])
|
||||
if isinstance(t, dict) and isinstance(t.get("count"), (int, float))]
|
||||
items = sorted(items, key=lambda t: t.get("count") or 0, reverse=True)
|
||||
head = items[:PIE_TOP_K]
|
||||
rest = items[PIE_TOP_K:]
|
||||
labels = [_truncate(t.get("value"), 20) for t in head]
|
||||
sizes = [float(t.get("count") or 0) for t in head]
|
||||
if rest:
|
||||
labels.append(f"Otros ({len(rest)})")
|
||||
sizes.append(sum(float(t.get("count") or 0) for t in rest))
|
||||
if not sizes or sum(sizes) <= 0:
|
||||
ax.text(0.5, 0.5, "sin datos categóricos", ha="center", va="center")
|
||||
ax.axis("off")
|
||||
return fig
|
||||
ax.pie(sizes, labels=None, wedgeprops={"width": 0.42},
|
||||
autopct=lambda p: f"{p:.0f}%" if p >= 4 else "")
|
||||
ax.legend(labels, loc="center left", bbox_to_anchor=(1.0, 0.5),
|
||||
fontsize=7, frameon=False)
|
||||
ax.set_title(_truncate(title, 40))
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
|
||||
|
||||
def _normalize_card(card: dict) -> dict:
|
||||
"""Make the cardinality dict robust regardless of the upstream scale.
|
||||
|
||||
``summarize_categorical`` emits ``mode_pct`` as a 0–1 fraction; bring it to a
|
||||
0–100 scale and recompute the ``dominated`` flag here so the chapter is
|
||||
correct whether it consumed the registry function or the inline fallback.
|
||||
"""
|
||||
card = dict(card or {})
|
||||
mp = card.get("mode_pct")
|
||||
if isinstance(mp, (int, float)) and not isinstance(mp, bool):
|
||||
mp = float(mp) * 100.0 if mp <= 1.0 else float(mp)
|
||||
else:
|
||||
mp = None
|
||||
card["mode_pct"] = mp
|
||||
card["dominated"] = mp is not None and mp >= 90.0
|
||||
pd = card.get("pct_distinct")
|
||||
card["id_like"] = isinstance(pd, (int, float)) and pd >= 99.0
|
||||
return card
|
||||
|
||||
|
||||
def _cardinality_block(card: dict):
|
||||
"""KVTable with the cardinality / entropy metrics for one column."""
|
||||
n_singletons = card.get("n_singletons")
|
||||
if n_singletons is not None and card.get("n_singletons_partial"):
|
||||
singletons = f"≥{_fmt_int(n_singletons)} (en top mostrado)"
|
||||
elif n_singletons is not None:
|
||||
singletons = _fmt_int(n_singletons)
|
||||
else:
|
||||
singletons = "—"
|
||||
|
||||
entropy_ref = _fmt_num(card.get("entropy"))
|
||||
emax = card.get("entropy_max")
|
||||
if emax is not None:
|
||||
entropy_ref = f"{entropy_ref} (máx {_fmt_num(emax)})"
|
||||
|
||||
mode = card.get("mode")
|
||||
mode_pct = card.get("mode_pct")
|
||||
mode_str = "—" if mode is None else model._safe_str(mode)
|
||||
if mode is not None and mode_pct is not None:
|
||||
mode_str = f"{mode_str} ({_fmt_pct_value(mode_pct)})"
|
||||
|
||||
rows = [
|
||||
("Valores distintos", _fmt_int(card.get("n_distinct"))),
|
||||
("% distintos", _fmt_pct_value(card.get("pct_distinct"))),
|
||||
("Total filas (dataset)", _fmt_int(card.get("n_rows"))),
|
||||
("Valores únicos (frecuencia 1)", singletons),
|
||||
("Entropía (bits)", entropy_ref),
|
||||
("Entropía normalizada (0–1)", _fmt_num(card.get("entropy_norm"))),
|
||||
("Moda", mode_str),
|
||||
]
|
||||
imbalance = card.get("imbalance")
|
||||
if imbalance is not None:
|
||||
rows.append(("Desbalance", _fmt_num(imbalance)))
|
||||
lm = card.get("len_min")
|
||||
lmean = card.get("len_mean")
|
||||
lmax = card.get("len_max")
|
||||
if any(v is not None for v in (lm, lmean, lmax)):
|
||||
rows.append((
|
||||
"Longitud (mín/media/máx)",
|
||||
f"{_fmt_num(lm)} / {_fmt_num(lmean)} / {_fmt_num(lmax)}"))
|
||||
return model.KVTable(rows=rows, title="Cardinalidad")
|
||||
|
||||
|
||||
def _flag_note(card: dict):
|
||||
"""Return a Note flagging problematic cardinality, or None."""
|
||||
if card.get("id_like"):
|
||||
return model.Note(
|
||||
"Casi todos los valores son distintos (≈100% distintos): la columna "
|
||||
"se comporta como un identificador y aporta poco para agrupar o "
|
||||
"comparar categorías.")
|
||||
if card.get("dominated"):
|
||||
mp = card.get("mode_pct")
|
||||
mp_str = _fmt_pct_value(mp) if mp is not None else "muy alta"
|
||||
return model.Note(
|
||||
f"Una sola categoría domina la columna (moda {mp_str}): la "
|
||||
"distribución está muy desbalanceada.")
|
||||
return None
|
||||
|
||||
|
||||
def _topk_table(cat: dict):
|
||||
"""DataTable value / count / % for the top categories."""
|
||||
top = cat.get("top") or []
|
||||
n_distinct = cat.get("n_distinct")
|
||||
header = ["Valor", "Conteo", "%"]
|
||||
rows = []
|
||||
for t in top[:TOP_TABLE_ROWS]:
|
||||
if not isinstance(t, dict):
|
||||
continue
|
||||
rows.append([
|
||||
model._safe_str(t.get("value")),
|
||||
_fmt_int(t.get("count")),
|
||||
_pct_from_maybe_fraction(t.get("pct")),
|
||||
])
|
||||
if not rows:
|
||||
return None
|
||||
shown = len(rows)
|
||||
if isinstance(n_distinct, (int, float)) and n_distinct > shown:
|
||||
note = f"top {shown} de {_fmt_int(n_distinct)} categorías distintas"
|
||||
else:
|
||||
note = f"{shown} categorías"
|
||||
return model.DataTable(header=header, rows=rows, title="Top categorías",
|
||||
note=note)
|
||||
|
||||
|
||||
def _intro_blocks(n_rows, mark_term: bool = False):
|
||||
total = _fmt_int(n_rows)
|
||||
# Mark the first appearance of the term as a clickable glossary jump when the
|
||||
# term was registered (mark_term). The visible text is identical either way.
|
||||
entropia = ("[[term:entropia]]**entropía de Shannon**[[/term]]" if mark_term
|
||||
else "**entropía de Shannon**")
|
||||
text = (
|
||||
f"La {entropia} mide cómo de repartidos están los valores de "
|
||||
"una columna categórica, en bits. Vale 0 cuando una sola categoría "
|
||||
"concentra todas las filas (máxima previsibilidad) y alcanza su máximo, "
|
||||
"log2(k) para k categorías distintas, cuando todas aparecen por igual "
|
||||
"(máxima diversidad). La **entropía normalizada** (entropía dividida por "
|
||||
"su máximo) la lleva al rango 0–1 para comparar columnas con distinto "
|
||||
"número de categorías. Para cada columna se muestran los valores "
|
||||
"distintos, el porcentaje que representan sobre el total de filas, los "
|
||||
"valores únicos (que aparecen una sola vez), la tabla de las categorías "
|
||||
"más frecuentes y un gráfico de tarta (donut) de las más comunes."
|
||||
)
|
||||
if n_rows is not None:
|
||||
text += f" El dataset tiene {total} filas en total como referencia."
|
||||
return [
|
||||
model.Heading(text="Entropía y cardinalidad", level=2),
|
||||
model.Markdown(text=text),
|
||||
]
|
||||
|
||||
|
||||
def build_cat_distr(profile: dict, ctx: dict):
|
||||
"""Build the categorical-distributions Chapter, or None if the dataset has
|
||||
no categorical columns."""
|
||||
profile = profile or {}
|
||||
ctx = ctx or {}
|
||||
cols = profile.get("columns") or []
|
||||
cat_cols = [c for c in cols if _is_categorical(c)]
|
||||
if not cat_cols:
|
||||
return None
|
||||
|
||||
n_rows = profile.get("n_rows")
|
||||
# Register "entropía" in the shared glossary collector (if present) and mark
|
||||
# its first appearance clickable. End-to-end glossary example (mejora 6).
|
||||
glossary = ctx.get("glossary")
|
||||
mark_term = False
|
||||
if isinstance(glossary, model.GlossaryCollector):
|
||||
glossary.add(_TERM_ENTROPIA_KEY, _TERM_ENTROPIA_LABEL,
|
||||
_TERM_ENTROPIA_DEF)
|
||||
mark_term = True
|
||||
blocks = list(_intro_blocks(n_rows, mark_term=mark_term))
|
||||
|
||||
rendered = cat_cols[:MAX_COLS]
|
||||
for col in rendered:
|
||||
name = col.get("name") or "(columna)"
|
||||
cat = col.get("categorical") or {}
|
||||
card = _normalize_card(_cardinality(cat, n_rows))
|
||||
|
||||
blocks.append(model.Heading(text=str(name), level=2))
|
||||
blocks.append(_cardinality_block(card))
|
||||
note = _flag_note(card)
|
||||
if note is not None:
|
||||
blocks.append(note)
|
||||
topk = _topk_table(cat)
|
||||
if topk is not None:
|
||||
blocks.append(topk)
|
||||
blocks.append(model.Figure(
|
||||
make=_pie_make(cat.get("top") or [], card.get("n_distinct"),
|
||||
str(name), n_rows),
|
||||
caption=(f"Categorías más comunes de «{_truncate(name, 32)}» "
|
||||
"(donut: top-k + «Otros»)")))
|
||||
|
||||
if len(cat_cols) > len(rendered):
|
||||
omitted = len(cat_cols) - len(rendered)
|
||||
blocks.append(model.Note(
|
||||
f"Se muestran las primeras {len(rendered)} columnas categóricas; "
|
||||
f"quedan {omitted} sin mostrar para mantener acotado el informe."))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Tests for the CAT DISTR chapter — DoD: golden + edges + anti-cut.
|
||||
|
||||
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
|
||||
and deterministic. Verifies that ``build_cat_distr`` emits the blocks the user
|
||||
asked for (entropy intro, distinct/total/%-distinct/unique metrics, top-k table
|
||||
and a donut figure), that the chapter renders inside the full document to both
|
||||
PDF and PPTX showing that content, that a profile with no categorical columns
|
||||
yields ``None`` without raising, and that long labels / many columns are never
|
||||
cut in either output.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
from pptx import Presentation
|
||||
|
||||
from datascience.automatic_eda.model import (
|
||||
DataTable, Figure, Heading, KVTable, Note,
|
||||
)
|
||||
from datascience.automatic_eda.chapters.cat_distr import (
|
||||
CHAPTER_ID, CHAPTER_VERSION, build_cat_distr,
|
||||
)
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
|
||||
|
||||
def _profile() -> dict:
|
||||
return {
|
||||
"table": "productos",
|
||||
"source": "/data/productos.csv",
|
||||
"profiled_at": "2026-06-30T10:00:00+00:00",
|
||||
"n_rows": 1000,
|
||||
"n_cols": 3,
|
||||
"quality_score": 90.0,
|
||||
"columns": [
|
||||
{"name": "precio", "inferred_type": "numeric", "null_pct": 0.0,
|
||||
"null_count": 0,
|
||||
"numeric": {"mean": 42.5, "median": 40.0, "min": 1.0,
|
||||
"max": 100.0, "std": 12.3}},
|
||||
{"name": "categoria", "inferred_type": "categorical",
|
||||
"null_pct": 0.0, "null_count": 0, "distinct_count": 8,
|
||||
"categorical": {
|
||||
"top": [
|
||||
{"value": "neumaticos", "count": 500, "pct": 0.5},
|
||||
{"value": "aceite", "count": 300, "pct": 0.3},
|
||||
{"value": "filtros", "count": 120, "pct": 0.12},
|
||||
{"value": "frenos", "count": 80, "pct": 0.08},
|
||||
],
|
||||
"mode": "neumaticos", "n_distinct": 8, "entropy": 1.6,
|
||||
"imbalance": 6.25, "len_min": 6, "len_mean": 7.5,
|
||||
"len_max": 10}},
|
||||
{"name": "uuid", "inferred_type": "categorical",
|
||||
"null_pct": 0.0, "null_count": 0, "distinct_count": 1000,
|
||||
"categorical": {
|
||||
"top": [{"value": f"id-{i}", "count": 1} for i in range(5)],
|
||||
"mode": "id-0", "n_distinct": 1000, "entropy": 9.97,
|
||||
"imbalance": 1.0}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _pdf_text(path: str) -> str:
|
||||
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
|
||||
return re.sub(r"\s+", " ", txt)
|
||||
|
||||
|
||||
def _pptx_text(path: str) -> str:
|
||||
prs = Presentation(path)
|
||||
parts = []
|
||||
for sl in prs.slides:
|
||||
for sh in sl.shapes:
|
||||
if sh.has_text_frame:
|
||||
parts.append(sh.text_frame.text)
|
||||
if sh.has_table:
|
||||
tb = sh.table
|
||||
for r in range(len(tb.rows)):
|
||||
for c in range(len(tb.columns)):
|
||||
parts.append(tb.cell(r, c).text)
|
||||
return re.sub(r"\s+", " ", " ".join(parts))
|
||||
|
||||
|
||||
def _kinds(chapter):
|
||||
return [b.kind for b in chapter.blocks]
|
||||
|
||||
|
||||
def test_golden_build_cat_distr_emite_bloques_pedidos():
|
||||
ch = build_cat_distr(_profile(), {})
|
||||
assert ch is not None
|
||||
assert ch.id == CHAPTER_ID
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
kinds = _kinds(ch)
|
||||
# Entropy intro present.
|
||||
headings = [b.text for b in ch.blocks if isinstance(b, Heading)]
|
||||
assert any("Entrop" in h for h in headings)
|
||||
md = next(b for b in ch.blocks if b.kind == "markdown")
|
||||
assert "entropía" in md.text.lower() and "log2" in md.text
|
||||
# Cardinality metrics: distinct, total rows, %-distinct, unique values.
|
||||
kv = next(b for b in ch.blocks if isinstance(b, KVTable))
|
||||
labels = [r[0] for r in kv.rows]
|
||||
assert "Valores distintos" in labels
|
||||
assert "% distintos" in labels
|
||||
assert "Total filas (dataset)" in labels
|
||||
assert "Valores únicos (frecuencia 1)" in labels
|
||||
assert any("Entropía" in lbl for lbl in labels)
|
||||
# Top-k table + pie figure.
|
||||
dt = next(b for b in ch.blocks if isinstance(b, DataTable))
|
||||
assert dt.header == ["Valor", "Conteo", "%"]
|
||||
assert any("neumaticos" in str(cell) for row in dt.rows for cell in row)
|
||||
assert any(isinstance(b, Figure) for b in ch.blocks)
|
||||
# id-like column flagged with a Note.
|
||||
assert any(isinstance(b, Note) and "identificador" in b.text
|
||||
for b in ch.blocks)
|
||||
|
||||
|
||||
def test_golden_render_pdf_muestra_categoricas():
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
out = os.path.join(d, "eda.pdf")
|
||||
res = render_automatic_eda_pdf(_profile(), out, {"title": "EDA"})
|
||||
assert res["path"] == out and os.path.exists(out)
|
||||
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
|
||||
txt = _pdf_text(out)
|
||||
assert "Entrop" in txt
|
||||
assert "distintos" in txt
|
||||
assert "categoria" in txt and "neumaticos" in txt
|
||||
assert "donut" in txt # figure caption rendered as text.
|
||||
assert "identificador" in txt # id-like note rendered.
|
||||
|
||||
|
||||
def test_golden_render_pptx_muestra_categoricas():
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
out = os.path.join(d, "eda.pptx")
|
||||
res = render_automatic_eda_pptx(_profile(), out, {"title": "EDA"})
|
||||
assert res["path"] == out and os.path.exists(out)
|
||||
assert CHAPTER_ID in [c["id"] for c in res["chapters"]]
|
||||
txt = _pptx_text(out)
|
||||
assert "Entrop" in txt
|
||||
assert "categoria" in txt and "neumaticos" in txt
|
||||
assert "distintos" in txt
|
||||
|
||||
|
||||
def test_edge_sin_categoricas_devuelve_none():
|
||||
only_numeric = {
|
||||
"n_rows": 10, "columns": [
|
||||
{"name": "x", "inferred_type": "numeric",
|
||||
"numeric": {"mean": 1.0}}]}
|
||||
assert build_cat_distr(only_numeric, {}) is None
|
||||
# None / empty / no-columns never raise and yield None.
|
||||
assert build_cat_distr(None, None) is None
|
||||
assert build_cat_distr({}, {}) is None
|
||||
assert build_cat_distr({"columns": []}, {}) is None
|
||||
|
||||
|
||||
def test_anti_corte_label_largo_y_muchas_columnas():
|
||||
long_label = ("Lorem ipsum dolor sit amet consectetur adipiscing elit sed "
|
||||
"do eiusmod tempor incididunt ut labore reprehenderit voluptate")
|
||||
cols = []
|
||||
for i in range(30):
|
||||
cols.append({
|
||||
"name": f"cat_{i}", "inferred_type": "categorical",
|
||||
"distinct_count": 3,
|
||||
"categorical": {
|
||||
"top": [{"value": long_label, "count": 60},
|
||||
{"value": "b", "count": 30},
|
||||
{"value": "c", "count": 10}],
|
||||
"mode": long_label, "n_distinct": 3, "entropy": 1.2}})
|
||||
profile = {"table": "t", "source": "t.csv", "n_rows": 100,
|
||||
"n_cols": len(cols), "columns": cols}
|
||||
|
||||
ch = build_cat_distr(profile, {})
|
||||
assert ch is not None
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "anti.pdf")
|
||||
res = render_automatic_eda_pdf(profile, pdf, {"write_manifest": False})
|
||||
assert res["path"] == pdf
|
||||
assert res["n_pages"] > 1 # many columns spilled across pages, OK.
|
||||
txt = _pdf_text(pdf)
|
||||
# Long label wrapped (not truncated): every word survives.
|
||||
for word in ("Lorem", "incididunt", "reprehenderit", "voluptate"):
|
||||
assert word in txt
|
||||
# PPTX path must not raise either.
|
||||
pptx = os.path.join(d, "anti.pptx")
|
||||
res2 = render_automatic_eda_pptx(profile, pptx,
|
||||
{"write_manifest": False})
|
||||
assert res2["path"] == pptx and os.path.exists(pptx)
|
||||
@@ -0,0 +1,352 @@
|
||||
"""Correlation chapter — association matrix plus top positive/negative pairs.
|
||||
|
||||
Builds the CORRELACION chapter of an AutomaticEDA document from a TableProfile.
|
||||
It renders exactly what the user asked for:
|
||||
|
||||
1. A correlation/association **matrix** (heatmap) reconstructed from the evaluated
|
||||
pairs, signed for numeric-numeric pairs (Pearson/Spearman, ``[-1, 1]``) and as
|
||||
magnitude for the mixed-type metrics (Cramér's V, correlation ratio, mutual
|
||||
information, ``[0, 1]``). Labels are ordered by total connectivity so strong
|
||||
associations cluster together instead of being scattered alphabetically.
|
||||
2. The **TOP positive** pairs and the **TOP negative** pairs as two separate
|
||||
tables. Only numeric-numeric metrics carry a sign, so negative pairs are by
|
||||
construction Pearson/Spearman; positive pairs may use any method.
|
||||
3. The methods legend and the multiple-testing (FDR) summary, so the reader sees
|
||||
how many pairs survive the correction.
|
||||
4. A spuriousness caveat when the profile flags level-based correlations on
|
||||
non-stationary series (Granger–Newbold).
|
||||
|
||||
All data comes from ``profile['correlations']`` — the output of the ``eda`` group
|
||||
function ``association_matrix`` (optionally enriched by ``profile_table``). The
|
||||
chapter never recomputes any statistic; it only lays the existing values out as
|
||||
format-independent blocks. The renderers paginate tables (repeating the header)
|
||||
and scale the heatmap to fit entirely, so nothing is ever cut.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "correlacion"
|
||||
CHAPTER_TITLE = "Correlación"
|
||||
|
||||
# Methods whose value carries a sign (direction). Everything else is a magnitude
|
||||
# in [0, 1] and therefore only ever contributes to the positive side.
|
||||
_SIGNED_METHODS = ("pearson", "spearman")
|
||||
|
||||
# Cap the heatmap to the most-connected variables so it stays legible on a phone
|
||||
# screen / a slide. The renderer would scale a bigger matrix to fit, but the
|
||||
# cells become unreadable; we instead show the top-N and say so.
|
||||
_MAX_MATRIX_LABELS = 16
|
||||
|
||||
# How many pairs to show in each of the top-positive / top-negative tables.
|
||||
_TOP_N = 10
|
||||
|
||||
|
||||
def _is_num(v) -> bool:
|
||||
"""True for a real, finite int/float (not bool, not NaN/inf)."""
|
||||
return (
|
||||
isinstance(v, (int, float))
|
||||
and not isinstance(v, bool)
|
||||
and not (isinstance(v, float) and (math.isnan(v) or math.isinf(v)))
|
||||
)
|
||||
|
||||
|
||||
def _fmt_val(value, decimals: int = 2) -> str:
|
||||
"""Format an association value compactly, signed, with a fixed width feel."""
|
||||
if not _is_num(value):
|
||||
return "—"
|
||||
text = f"{float(value):+.{decimals}f}"
|
||||
# Strip a trailing -0.00 / +0.00 into a clean 0.00 for readability.
|
||||
if text in ("+0.00", "-0.00"):
|
||||
return "0.00"
|
||||
return text
|
||||
|
||||
|
||||
def _fmt_p(value) -> str:
|
||||
"""Format an adjusted p-value; tiny values collapse to a '<' threshold."""
|
||||
if not _is_num(value):
|
||||
return "—"
|
||||
p = float(value)
|
||||
if p < 0.001:
|
||||
return "<0.001"
|
||||
return f"{p:.3f}"
|
||||
|
||||
|
||||
def _is_signed(pair: dict) -> bool:
|
||||
"""True if the pair's method reports a directional (signed) value."""
|
||||
method = str(pair.get("method") or "").lower()
|
||||
return any(m in method for m in _SIGNED_METHODS)
|
||||
|
||||
|
||||
def _significant(pair: dict) -> bool:
|
||||
"""True if the pair is significant after FDR (or has no test to correct)."""
|
||||
if pair.get("significant") is True:
|
||||
return True
|
||||
# Pairs without an applicable test (p_value None) are not penalised: they are
|
||||
# admitted on magnitude alone upstream, so treat missing as "not rejected".
|
||||
return pair.get("p_value") is None and pair.get("significant") is None
|
||||
|
||||
|
||||
def _label(pair: dict) -> str:
|
||||
"""Human label for a pair, e.g. 'alcohol ↔ density'."""
|
||||
return f"{model._safe_str(pair.get('a'))} ↔ {model._safe_str(pair.get('b'))}"
|
||||
|
||||
|
||||
def _split_top(pairs: list, top_n: int = _TOP_N):
|
||||
"""Split evaluated pairs into ranked top-positive and top-negative lists.
|
||||
|
||||
Positive: any pair with a positive value, ranked by value descending.
|
||||
Negative: only signed (numeric-numeric) pairs with a negative value, ranked
|
||||
by value ascending (most negative first). Non-finite values are dropped.
|
||||
"""
|
||||
positive = []
|
||||
negative = []
|
||||
for pair in pairs:
|
||||
if not isinstance(pair, dict):
|
||||
continue
|
||||
value = pair.get("value")
|
||||
if not _is_num(value):
|
||||
continue
|
||||
if value > 0:
|
||||
positive.append(pair)
|
||||
elif value < 0 and _is_signed(pair):
|
||||
negative.append(pair)
|
||||
positive.sort(key=lambda p: float(p.get("value", 0.0)), reverse=True)
|
||||
negative.sort(key=lambda p: float(p.get("value", 0.0)))
|
||||
return positive[:top_n], negative[:top_n]
|
||||
|
||||
|
||||
def _top_table(pairs: list, title: str):
|
||||
"""Build a DataTable for a list of pairs, or None if there are none."""
|
||||
if not pairs:
|
||||
return None
|
||||
header = ["Par", "Método", "Valor", "p (FDR)", "Sig."]
|
||||
rows = []
|
||||
for pair in pairs:
|
||||
method = model._safe_str(pair.get("method")) or "—"
|
||||
rows.append([
|
||||
_label(pair),
|
||||
method,
|
||||
_fmt_val(pair.get("value")),
|
||||
_fmt_p(pair.get("p_value_adjusted")),
|
||||
"sí" if _significant(pair) else "no",
|
||||
])
|
||||
return model.DataTable(header=header, rows=rows, title=title)
|
||||
|
||||
|
||||
def _ordered_labels(pairs: list):
|
||||
"""Pick and order the matrix labels by total connectivity (descending).
|
||||
|
||||
Returns the list of variable names to place on the axes, capped at
|
||||
``_MAX_MATRIX_LABELS`` (the most-connected ones), plus a boolean saying
|
||||
whether the cap trimmed anything.
|
||||
"""
|
||||
strength = {}
|
||||
for pair in pairs:
|
||||
if not isinstance(pair, dict):
|
||||
continue
|
||||
value = pair.get("value")
|
||||
if not _is_num(value):
|
||||
continue
|
||||
mag = abs(float(value))
|
||||
for key in ("a", "b"):
|
||||
name = pair.get(key)
|
||||
if name is None:
|
||||
continue
|
||||
strength[name] = strength.get(name, 0.0) + mag
|
||||
if not strength:
|
||||
return [], False
|
||||
ordered = sorted(strength, key=lambda n: strength[n], reverse=True)
|
||||
trimmed = len(ordered) > _MAX_MATRIX_LABELS
|
||||
return ordered[:_MAX_MATRIX_LABELS], trimmed
|
||||
|
||||
|
||||
def _matrix_figure(pairs: list, labels: list):
|
||||
"""Return a Figure (lazy) with the signed association heatmap, or None.
|
||||
|
||||
The matplotlib figure is built lazily inside ``make`` so importing this
|
||||
module never requires matplotlib and a malformed plot degrades to nothing
|
||||
instead of aborting the chapter.
|
||||
"""
|
||||
if len(labels) < 2:
|
||||
return None
|
||||
|
||||
index = {name: i for i, name in enumerate(labels)}
|
||||
|
||||
def make():
|
||||
import numpy as np
|
||||
from matplotlib.figure import Figure
|
||||
|
||||
n = len(labels)
|
||||
grid = np.full((n, n), np.nan, dtype=float)
|
||||
for i in range(n):
|
||||
grid[i, i] = 1.0
|
||||
for pair in pairs:
|
||||
if not isinstance(pair, dict):
|
||||
continue
|
||||
a = pair.get("a")
|
||||
b = pair.get("b")
|
||||
value = pair.get("value")
|
||||
if a not in index or b not in index or not _is_num(value):
|
||||
continue
|
||||
v = float(value)
|
||||
# Mixed-type magnitudes are non-negative; keep them as-is on [0, 1].
|
||||
ia, ib = index[a], index[b]
|
||||
grid[ia, ib] = v
|
||||
grid[ib, ia] = v
|
||||
|
||||
import matplotlib
|
||||
|
||||
masked = np.ma.masked_invalid(grid)
|
||||
fig = Figure(figsize=(6.2, 5.6))
|
||||
ax = fig.add_subplot(111)
|
||||
cmap = matplotlib.colormaps["RdBu_r"].copy()
|
||||
cmap.set_bad(color="#eeeeee")
|
||||
im = ax.imshow(masked, cmap=cmap, vmin=-1.0, vmax=1.0, aspect="auto")
|
||||
ax.set_xticks(range(n))
|
||||
ax.set_yticks(range(n))
|
||||
short = [str(s)[:14] for s in labels]
|
||||
ax.set_xticks(range(n))
|
||||
ax.set_xticklabels(short, rotation=90, fontsize=7)
|
||||
ax.set_yticklabels(short, fontsize=7)
|
||||
# Annotate cells only when the matrix is small enough to stay legible.
|
||||
if n <= 8:
|
||||
for i in range(n):
|
||||
for j in range(n):
|
||||
cell = grid[i, j]
|
||||
if _is_num(cell):
|
||||
ax.text(j, i, f"{cell:+.2f}".replace("+", "") if cell < 0
|
||||
else f"{cell:.2f}",
|
||||
ha="center", va="center", fontsize=6,
|
||||
color="#222222")
|
||||
fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04,
|
||||
label="asociación (signo en num-num)")
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
|
||||
return model.Figure(make=make,
|
||||
caption="Matriz de asociación. Azul = positiva, rojo = "
|
||||
"negativa (sólo num-num lleva signo); gris = par "
|
||||
"no evaluado.")
|
||||
|
||||
|
||||
def _methods_block(corr: dict):
|
||||
"""Build a KVTable with the legend of the methods actually present."""
|
||||
legend = corr.get("methods_legend")
|
||||
if not isinstance(legend, dict) or not legend:
|
||||
return None
|
||||
rows = [(model._safe_str(k), model._safe_str(v)) for k, v in legend.items()]
|
||||
return model.KVTable(rows=rows, title="Métodos de asociación")
|
||||
|
||||
|
||||
def _fdr_text(corr: dict) -> str | None:
|
||||
"""One-line summary of the multiple-testing (FDR) correction, or None."""
|
||||
mt = corr.get("multiple_testing")
|
||||
if not isinstance(mt, dict) or not mt:
|
||||
return None
|
||||
method = model._safe_str(mt.get("method")).upper() or "FDR"
|
||||
alpha = mt.get("alpha")
|
||||
n_tests = mt.get("n_tests")
|
||||
n_rej = mt.get("n_rejected")
|
||||
parts = [f"Corrección por comparaciones múltiples ({method}"]
|
||||
if _is_num(alpha):
|
||||
parts[0] += f", α={float(alpha):g}"
|
||||
parts[0] += ")."
|
||||
if _is_num(n_tests):
|
||||
rej = n_rej if _is_num(n_rej) else "—"
|
||||
parts.append(
|
||||
f"De {int(n_tests)} pares con test, {rej} siguen siendo "
|
||||
f"significativos tras la corrección.")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def build_correlacion(profile: dict, ctx: dict):
|
||||
"""Build the Correlation Chapter, or None if there are no pairs to show.
|
||||
|
||||
Reads ``profile['correlations']`` (the ``association_matrix`` output). Returns
|
||||
``None`` when the dataset has fewer than two associable columns (no evaluated
|
||||
pairs), so the chapter is omitted instead of showing an empty section. Never
|
||||
raises: every access is defensive.
|
||||
|
||||
ctx keys consumed: none specific (presentation metadata is inherited from the
|
||||
document). The chapter reads everything it needs from the profile.
|
||||
"""
|
||||
profile = profile or {}
|
||||
ctx = ctx or {}
|
||||
|
||||
corr = profile.get("correlations")
|
||||
if not isinstance(corr, dict):
|
||||
return None
|
||||
pairs = corr.get("pairs")
|
||||
if not isinstance(pairs, list) or not pairs:
|
||||
return None
|
||||
|
||||
blocks: list = []
|
||||
|
||||
# Intro: what this chapter shows and how to read the sign.
|
||||
blocks.append(model.Markdown(text=(
|
||||
"Asociación entre columnas. Cada par se evalúa con la métrica adecuada a "
|
||||
"sus tipos (Pearson/Spearman entre numéricas — con **signo**; Cramér's V "
|
||||
"entre categóricas; razón de correlación num-categórica; información mutua "
|
||||
"como medida común no lineal). Sólo las correlaciones **num-num** tienen "
|
||||
"dirección: por eso los pares **negativos** son siempre num-num.")))
|
||||
|
||||
# 1) Association matrix (heatmap).
|
||||
labels, trimmed = _ordered_labels(pairs)
|
||||
fig = _matrix_figure(pairs, labels)
|
||||
if fig is not None:
|
||||
blocks.append(model.Heading(text="Matriz de asociación", level=2))
|
||||
blocks.append(fig)
|
||||
if trimmed:
|
||||
blocks.append(model.Note(text=(
|
||||
f"Se muestran las {len(labels)} variables más conectadas de la "
|
||||
"matriz para mantenerla legible; el resto de pares siguen en las "
|
||||
"tablas de abajo.")))
|
||||
|
||||
# 2) Top positive / top negative pairs.
|
||||
positive, negative = _split_top(pairs, _TOP_N)
|
||||
pos_table = _top_table(positive, f"Top {len(positive)} positivas")
|
||||
neg_table = _top_table(negative, f"Top {len(negative)} negativas")
|
||||
if pos_table is not None:
|
||||
blocks.append(model.Heading(text="Pares más correlacionados (positivos)",
|
||||
level=2))
|
||||
blocks.append(pos_table)
|
||||
if neg_table is not None:
|
||||
blocks.append(model.Heading(text="Pares más correlacionados (negativos)",
|
||||
level=2))
|
||||
blocks.append(neg_table)
|
||||
elif pos_table is not None:
|
||||
# No signed-negative pairs at all: say so honestly rather than omit.
|
||||
blocks.append(model.Note(text=(
|
||||
"No se han hallado correlaciones negativas significativas entre "
|
||||
"columnas numéricas.")))
|
||||
|
||||
# 3) Spuriousness caveat for level-based correlations (Granger–Newbold).
|
||||
caveat = corr.get("levels_caveat")
|
||||
if isinstance(caveat, str) and caveat.strip():
|
||||
blocks.append(model.Note(text=caveat.strip()))
|
||||
elif corr.get("levels_possible_spurious"):
|
||||
blocks.append(model.Note(text=(
|
||||
"Aviso: algunas correlaciones se calcularon sobre niveles de series "
|
||||
"no estacionarias y pueden ser espurias (Granger–Newbold). Compáralas "
|
||||
"sobre los retornos/diferencias antes de interpretarlas.")))
|
||||
|
||||
# 4) FDR summary + methods legend.
|
||||
fdr_text = _fdr_text(corr)
|
||||
if fdr_text:
|
||||
blocks.append(model.Markdown(text=fdr_text))
|
||||
methods = _methods_block(corr)
|
||||
if methods is not None:
|
||||
blocks.append(model.Heading(text="Métodos y leyenda", level=2))
|
||||
blocks.append(methods)
|
||||
|
||||
if not blocks:
|
||||
return None
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Tests for the CORRELACION chapter — DoD: golden + edges + error/anti-cut.
|
||||
|
||||
Self-contained: builds a synthetic TableProfile carrying a ``correlations`` block
|
||||
shaped exactly like ``association_matrix`` output (no DuckDB), so the suite is
|
||||
fast and deterministic. Verifies that the chapter emits the association-matrix
|
||||
figure plus separate top-positive / top-negative tables with the right pairs,
|
||||
that it returns None when the profile has no pairs, that a None/empty profile
|
||||
does not raise, and that a wide matrix with long labels renders to PDF *and* PPTX
|
||||
without cutting anything.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
|
||||
from datascience.automatic_eda.chapters.correlacion import (
|
||||
CHAPTER_VERSION,
|
||||
build_correlacion,
|
||||
)
|
||||
from datascience.automatic_eda.model import DataTable, Figure
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
|
||||
|
||||
def _pair(a, b, value, method, padj, sig, p=0.0001):
|
||||
return {
|
||||
"a": a, "b": b, "a_type": "numeric", "b_type": "numeric",
|
||||
"method": method, "value": value, "extra": {"mi": abs(value) * 0.5},
|
||||
"p_value": p, "p_value_adjusted": padj, "significant": sig,
|
||||
}
|
||||
|
||||
|
||||
def _profile() -> dict:
|
||||
"""Synthetic wine-like profile with signed and unsigned associations."""
|
||||
pairs = [
|
||||
_pair("alcohol", "quality", 0.48, "pearson/spearman", 0.0005, True),
|
||||
_pair("density", "alcohol", -0.78, "pearson/spearman", 0.0001, True),
|
||||
_pair("ph", "fixed_acidity", -0.68, "pearson/spearman", 0.0002, True),
|
||||
_pair("sulphates", "quality", 0.25, "pearson/spearman", 0.03, True),
|
||||
# Unsigned mixed-type metrics: only ever positive, never in the neg table.
|
||||
{"a": "region", "b": "type", "a_type": "categorical",
|
||||
"b_type": "categorical", "method": "cramers_v", "value": 0.55,
|
||||
"extra": {"mi": 0.3}, "p_value": 0.001, "p_value_adjusted": 0.004,
|
||||
"significant": True},
|
||||
]
|
||||
return {
|
||||
"table": "wine",
|
||||
"source": "/data/wine.csv",
|
||||
"n_rows": 1599,
|
||||
"n_cols": 12,
|
||||
"correlations": {
|
||||
"pairs": pairs,
|
||||
"strong": [p for p in pairs if abs(p["value"]) >= 0.5],
|
||||
"methods_legend": {
|
||||
"pearson": "num-num lineal (Pearson r), [-1, 1]",
|
||||
"cramers_v": "cat-cat simétrica (Cramér's V), [0, 1]",
|
||||
},
|
||||
"multiple_testing": {"method": "bh", "alpha": 0.05,
|
||||
"n_tests": 5, "n_rejected": 5},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _pdf_text(path: str) -> str:
|
||||
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
|
||||
return re.sub(r"\s+", " ", txt)
|
||||
|
||||
|
||||
def test_golden_chapter_tiene_matriz_y_top_positivos_y_negativos():
|
||||
ch = build_correlacion(_profile(), {})
|
||||
assert ch is not None
|
||||
assert ch.id == "correlacion"
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
kinds = [b.kind for b in ch.blocks]
|
||||
assert "figure" in kinds # association matrix heatmap.
|
||||
figs = [b for b in ch.blocks if isinstance(b, Figure)]
|
||||
assert figs and figs[0].make is not None # lazy figure.
|
||||
|
||||
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
|
||||
assert len(tables) >= 2 # top positive + top negative.
|
||||
flat = " ".join(str(c) for t in tables for r in t.rows for c in r)
|
||||
# Strongest positive present and signed +, strongest negative present and -.
|
||||
assert "alcohol" in flat and "quality" in flat
|
||||
assert "+0.48" in flat
|
||||
assert "density" in flat and "-0.78" in flat
|
||||
|
||||
|
||||
def test_golden_render_pdf_y_pptx_muestran_lo_exigido():
|
||||
prof = _profile()
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "corr.pdf")
|
||||
pptx = os.path.join(d, "corr.pptx")
|
||||
rp = render_automatic_eda_pdf(prof, pdf, {"title": "EDA — wine"})
|
||||
rx = render_automatic_eda_pptx(prof, pptx, {"title": "EDA — wine"})
|
||||
assert rp["path"] == pdf and rp["n_pages"] >= 1
|
||||
assert rx["path"] == pptx and rx["n_slides"] >= 1
|
||||
assert "correlacion" in [c["id"] for c in rp["chapters"]]
|
||||
assert "correlacion" in [c["id"] for c in rx["chapters"]]
|
||||
txt = _pdf_text(pdf)
|
||||
# The requirement: matrix + top positive/negative pairs, all visible.
|
||||
assert "Correlaci" in txt # chapter title (accents may vary in extract).
|
||||
assert "density" in txt and "alcohol" in txt and "quality" in txt
|
||||
assert "0.78" in txt and "0.48" in txt
|
||||
# Both signs surfaced as separate sections.
|
||||
assert "positiv" in txt.lower() and "negativ" in txt.lower()
|
||||
|
||||
|
||||
def test_edge_sin_pares_devuelve_none():
|
||||
# No correlations key, empty pairs, and wrong types all yield None, not error.
|
||||
assert build_correlacion({"table": "x"}, {}) is None
|
||||
assert build_correlacion({"correlations": {}}, {}) is None
|
||||
assert build_correlacion({"correlations": {"pairs": []}}, {}) is None
|
||||
assert build_correlacion({"correlations": {"pairs": "nope"}}, {}) is None
|
||||
assert build_correlacion(None, None) is None
|
||||
assert build_correlacion({}, {}) is None
|
||||
|
||||
|
||||
def test_edge_solo_positivos_emite_nota_sin_tabla_negativa():
|
||||
prof = {
|
||||
"correlations": {
|
||||
"pairs": [
|
||||
_pair("a", "b", 0.6, "pearson/spearman", 0.001, True),
|
||||
{"a": "c", "b": "d", "a_type": "categorical",
|
||||
"b_type": "categorical", "method": "cramers_v", "value": 0.7,
|
||||
"extra": {"mi": 0.4}, "p_value": 0.001,
|
||||
"p_value_adjusted": 0.003, "significant": True},
|
||||
],
|
||||
},
|
||||
}
|
||||
ch = build_correlacion(prof, {})
|
||||
assert ch is not None
|
||||
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
|
||||
assert len(tables) == 1 # only the positive table.
|
||||
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
|
||||
assert "negativas" in notes # honest "no negative correlations" note.
|
||||
|
||||
|
||||
def test_anticorte_matriz_ancha_y_etiquetas_largas_no_se_cortan():
|
||||
# 20 numeric vars with long names -> matrix trimmed to top-N + both renderers
|
||||
# must lay the chapter out without raising and keep a long label intact.
|
||||
long_a = "concentracion_de_dioxido_de_azufre_libre"
|
||||
long_b = "concentracion_de_dioxido_de_azufre_total"
|
||||
pairs = [_pair(long_a, long_b, -0.72, "pearson/spearman", 0.0001, True)]
|
||||
for i in range(20):
|
||||
pairs.append(_pair(f"variable_numerica_larga_{i:02d}",
|
||||
f"variable_numerica_larga_{(i + 1) % 20:02d}",
|
||||
0.55 - i * 0.02, "pearson/spearman", 0.01, True))
|
||||
prof = {"correlations": {"pairs": pairs,
|
||||
"multiple_testing": {"method": "bh", "alpha": 0.05,
|
||||
"n_tests": len(pairs),
|
||||
"n_rejected": len(pairs)}}}
|
||||
ch = build_correlacion(prof, {})
|
||||
assert ch is not None
|
||||
# A "showing top-N most connected" note appears when the matrix is trimmed.
|
||||
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
|
||||
assert "más conectadas" in notes
|
||||
# Anti-cut guarantee at the block level: the long pair reaches the renderer
|
||||
# whole (the block never truncates); the renderer then wraps the cell inside
|
||||
# its column. Both long labels are present, intact, in a table cell.
|
||||
tables = [b for b in ch.blocks if isinstance(b, DataTable)]
|
||||
cells = [str(c) for t in tables for r in t.rows for c in r]
|
||||
assert any(long_a in c and long_b in c for c in cells)
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "wide.pdf")
|
||||
pptx = os.path.join(d, "wide.pptx")
|
||||
rp = render_automatic_eda_pdf(prof, pdf, {"write_manifest": False})
|
||||
rx = render_automatic_eda_pptx(prof, pptx, {"write_manifest": False})
|
||||
# Both renderers lay the wide chapter out without raising and produce a
|
||||
# non-empty document (nothing dropped, just wrapped/scaled to fit).
|
||||
assert rp["path"] == pdf and os.path.exists(pdf) and rp["n_pages"] >= 1
|
||||
assert rx["path"] == pptx and os.path.exists(pptx) and rx["n_slides"] >= 1
|
||||
# A short, unbreakable fragment of the long label survives the wrap.
|
||||
assert "azufre" in _pdf_text(pdf)
|
||||
@@ -0,0 +1,477 @@
|
||||
"""Geospatial chapter (GEOSPATIAL) for AutomaticEDA.
|
||||
|
||||
When the dataset carries a coordinate pair (latitude/longitude), this chapter
|
||||
draws the points on a **geographic scatter** in an equirectangular projection
|
||||
(scaled so degrees of longitude are not stretched at the data's latitude) and
|
||||
analyses the **zone / country** the points fall in: bounding box, centroid,
|
||||
geographic span, and a per-region count. When there is **no** coordinate pair the
|
||||
chapter returns ``None`` — exactly the user requirement.
|
||||
|
||||
Detection and the heavy lifting are delegated to pure ``eda``-group registry
|
||||
functions, never reimplemented here:
|
||||
|
||||
- ``detect_latlon_columns`` — finds the (lat, lon) column pair by name + value
|
||||
range from the ``profile['columns']`` metadata.
|
||||
- ``analyze_geo_extent`` — bbox, centroid, haversine span, per-region counts and
|
||||
hemisphere from the raw coordinate arrays.
|
||||
- ``build_geo_scatter`` — deterministically down-sampled points + bbox + the
|
||||
aspect ratio for the equirectangular projection. This chapter only draws the
|
||||
matplotlib figure from that prepared data (same split as ``num_distr`` does
|
||||
with ``build_boxplot_stats``).
|
||||
|
||||
The raw coordinate arrays are **not** in a standard TableProfile (it stores only
|
||||
per-column aggregates), so — exactly like ``modelos`` reads ``raw_numeric`` from
|
||||
``ctx`` — this chapter looks for the coordinates in ``ctx`` (or ``profile``) and
|
||||
degrades honestly when they are absent: it still detects the columns and shows an
|
||||
approximate bounding box derived from the per-column ``numeric.min/max``, with a
|
||||
note that the raw points are needed for the map.
|
||||
|
||||
ctx keys this chapter consumes (all optional):
|
||||
geo_points : dict — ``{"lats": [...], "lons": [...]}`` raw coordinate arrays.
|
||||
Used directly when present (forward-compatible with a calculation phase
|
||||
that samples them from the table).
|
||||
raw_numeric : dict — ``{col: [values]}`` raw numeric columns; when present
|
||||
and ``geo_points`` is not, the detected lat/lon columns are read from it.
|
||||
run_geo_llm : bool — when True, call ``ask_llm`` for a one-line narrative of
|
||||
where the points concentrate (otherwise a derived note is used).
|
||||
geo_llm_model : str — model id for the optional live LLM call.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
Reads everything defensively (``.get``) and never raises.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from .. import model
|
||||
|
||||
# Pure registry functions (group ``eda``) delegated to. Imported defensively so
|
||||
# the chapter stays importable (degrading gracefully) if one is unavailable.
|
||||
try:
|
||||
from datascience.detect_latlon_columns import detect_latlon_columns
|
||||
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
|
||||
detect_latlon_columns = None # type: ignore[assignment]
|
||||
try:
|
||||
from datascience.analyze_geo_extent import analyze_geo_extent
|
||||
except Exception: # noqa: BLE001
|
||||
analyze_geo_extent = None # type: ignore[assignment]
|
||||
try:
|
||||
from datascience.build_geo_scatter import build_geo_scatter
|
||||
except Exception: # noqa: BLE001
|
||||
build_geo_scatter = None # type: ignore[assignment]
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "geospatial"
|
||||
CHAPTER_TITLE = "Análisis geoespacial"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Formatting helpers (mirror the other chapters' defensive style).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _fmt_num(value, decimals: int = 4) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return "sí" if value else "no"
|
||||
if isinstance(value, int):
|
||||
return f"{value:,}".replace(",", ".")
|
||||
if isinstance(value, float):
|
||||
if value != value: # NaN
|
||||
return "NaN"
|
||||
if value in (float("inf"), float("-inf")):
|
||||
return str(value)
|
||||
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
|
||||
return text if text else "0"
|
||||
return model._safe_str(value)
|
||||
|
||||
|
||||
def _fmt_coord(value, decimals: int = 4) -> str:
|
||||
"""Format a coordinate degree value, defensively."""
|
||||
try:
|
||||
return f"{float(value):.{decimals}f}°"
|
||||
except (TypeError, ValueError):
|
||||
return model._safe_str(value)
|
||||
|
||||
|
||||
def _fmt_km(value) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
v = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return model._safe_str(value)
|
||||
if v >= 100:
|
||||
return f"{v:,.0f} km".replace(",", ".")
|
||||
return f"{v:.1f} km"
|
||||
|
||||
|
||||
def _is_dict(v) -> bool:
|
||||
return isinstance(v, dict)
|
||||
|
||||
|
||||
def _clean_floats(seq) -> list:
|
||||
"""Return a list of floats from an arbitrary sequence (drop None/NaN)."""
|
||||
out = []
|
||||
if not isinstance(seq, (list, tuple)):
|
||||
return out
|
||||
for v in seq:
|
||||
try:
|
||||
f = float(v)
|
||||
except (TypeError, ValueError):
|
||||
out.append(None)
|
||||
continue
|
||||
out.append(f if f == f else None) # NaN -> None
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Resolve the (lat, lon) columns and the raw coordinate arrays.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _detect_columns(profile: dict) -> dict:
|
||||
"""Detect the lat/lon column pair from the profile metadata, or {}."""
|
||||
cols = profile.get("columns")
|
||||
if not isinstance(cols, list) or not cols or detect_latlon_columns is None:
|
||||
return {}
|
||||
try:
|
||||
det = detect_latlon_columns(cols)
|
||||
except Exception: # noqa: BLE001 — never break the chapter.
|
||||
return {}
|
||||
return det if _is_dict(det) else {}
|
||||
|
||||
|
||||
def _resolve_coords(profile: dict, ctx: dict, detected: dict):
|
||||
"""Return (lats, lons, source_label).
|
||||
|
||||
Order: ctx/profile['geo_points'] (explicit arrays) → ctx/profile
|
||||
['raw_numeric'] keyed by the detected lat/lon column names → (None, None).
|
||||
"""
|
||||
gp = ctx.get("geo_points") or profile.get("geo_points")
|
||||
if _is_dict(gp):
|
||||
lats = gp.get("lats")
|
||||
if lats is None:
|
||||
lats = gp.get("lat")
|
||||
lons = gp.get("lons")
|
||||
if lons is None:
|
||||
lons = gp.get("lon")
|
||||
if lats and lons:
|
||||
return list(lats), list(lons), "geo_points"
|
||||
|
||||
lat_col = (detected or {}).get("lat_col")
|
||||
lon_col = (detected or {}).get("lon_col")
|
||||
if lat_col and lon_col:
|
||||
raw = ctx.get("raw_numeric") or profile.get("raw_numeric")
|
||||
if _is_dict(raw):
|
||||
lats = raw.get(lat_col)
|
||||
lons = raw.get(lon_col)
|
||||
if lats and lons:
|
||||
return list(lats), list(lons), "raw_numeric"
|
||||
return None, None, "none"
|
||||
|
||||
|
||||
def _column_by_name(profile: dict, name):
|
||||
if not name:
|
||||
return None
|
||||
for col in profile.get("columns") or []:
|
||||
if isinstance(col, dict) and col.get("name") == name:
|
||||
return col
|
||||
return None
|
||||
|
||||
|
||||
def _bbox_from_profile(profile: dict, detected: dict):
|
||||
"""Approximate bbox from the per-column numeric.min/max (no raw points)."""
|
||||
lat_c = _column_by_name(profile, (detected or {}).get("lat_col"))
|
||||
lon_c = _column_by_name(profile, (detected or {}).get("lon_col"))
|
||||
lat_n = lat_c.get("numeric") if _is_dict(lat_c) else None
|
||||
lon_n = lon_c.get("numeric") if _is_dict(lon_c) else None
|
||||
if not _is_dict(lat_n) or not _is_dict(lon_n):
|
||||
return None
|
||||
try:
|
||||
return {
|
||||
"lat_min": float(lat_n.get("min")),
|
||||
"lat_max": float(lat_n.get("max")),
|
||||
"lon_min": float(lon_n.get("min")),
|
||||
"lon_max": float(lon_n.get("max")),
|
||||
}
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Figure builder (lazy: matplotlib only imported when the renderer draws it).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _make_geo_scatter(scatter: dict, lat_col: str, lon_col: str):
|
||||
"""Return a zero-arg callable drawing the geographic scatter, or None."""
|
||||
points = scatter.get("points") or []
|
||||
if not points:
|
||||
return None
|
||||
bbox = scatter.get("bbox") if _is_dict(scatter.get("bbox")) else {}
|
||||
aspect = scatter.get("aspect") or 1.0
|
||||
pad = scatter.get("pad") if _is_dict(scatter.get("pad")) else {}
|
||||
n_total = scatter.get("n_total")
|
||||
n_shown = scatter.get("n_shown")
|
||||
|
||||
def _draw():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
xs = [p[0] for p in points if isinstance(p, (list, tuple)) and len(p) >= 2]
|
||||
ys = [p[1] for p in points if isinstance(p, (list, tuple)) and len(p) >= 2]
|
||||
|
||||
fig, ax = plt.subplots(figsize=(6.6, 5.0))
|
||||
# More points -> smaller markers + lower alpha so dense clouds read as
|
||||
# density without saturating the page with ink (Tufte).
|
||||
n = max(len(xs), 1)
|
||||
size = 18 if n <= 200 else (8 if n <= 1000 else 4)
|
||||
alpha = 0.75 if n <= 200 else (0.5 if n <= 1000 else 0.35)
|
||||
ax.scatter(xs, ys, s=size, c="#2a6f97", alpha=alpha, linewidths=0,
|
||||
zorder=3)
|
||||
|
||||
# Bounding box rectangle for orientation.
|
||||
if bbox:
|
||||
try:
|
||||
lo_x, hi_x = float(bbox["lon_min"]), float(bbox["lon_max"])
|
||||
lo_y, hi_y = float(bbox["lat_min"]), float(bbox["lat_max"])
|
||||
ax.plot([lo_x, hi_x, hi_x, lo_x, lo_x],
|
||||
[lo_y, lo_y, hi_y, hi_y, lo_y],
|
||||
color="#e15759", linewidth=1.0, linestyle="--",
|
||||
alpha=0.8, zorder=4, label="Bounding box")
|
||||
px = float(pad.get("lon", 0.0) or 0.0)
|
||||
py = float(pad.get("lat", 0.0) or 0.0)
|
||||
ax.set_xlim(lo_x - px, hi_x + px)
|
||||
ax.set_ylim(lo_y - py, hi_y + py)
|
||||
except (TypeError, ValueError, KeyError):
|
||||
pass
|
||||
|
||||
# Equirectangular: scale Y/X so longitude is not stretched at this
|
||||
# latitude (integridad de proyección, Tufte). aspect = 1/cos(lat).
|
||||
try:
|
||||
ax.set_aspect(float(aspect))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
ax.set_xlabel(f"Longitud ({lon_col})", fontsize=8)
|
||||
ax.set_ylabel(f"Latitud ({lat_col})", fontsize=8)
|
||||
ax.tick_params(labelsize=7)
|
||||
ax.grid(color="#e6e6e6", linewidth=0.5, zorder=0)
|
||||
title = "Distribución geográfica de las coordenadas"
|
||||
if n_shown is not None and n_total is not None and n_shown < n_total:
|
||||
title += f"\n(mostrando {n_shown:,} de {n_total:,} puntos)".replace(",", ".")
|
||||
ax.set_title(title, fontsize=10)
|
||||
ax.legend(loc="best", fontsize=7, frameon=True, framealpha=0.9)
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
|
||||
return _draw
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Section builders.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _intro_block(detected: dict, lat_col: str, lon_col: str) -> list:
|
||||
conf = (detected or {}).get("confidence")
|
||||
reason = model._safe_str((detected or {}).get("reason"))
|
||||
conf_txt = ""
|
||||
if conf is not None:
|
||||
try:
|
||||
conf_txt = f" (confianza {float(conf) * 100:.0f}%)"
|
||||
except (TypeError, ValueError):
|
||||
conf_txt = ""
|
||||
text = (
|
||||
"Este dataset contiene **coordenadas geográficas**: se identificó el par "
|
||||
f"**latitud = «{lat_col}»** y **longitud = «{lon_col}»**{conf_txt}. La "
|
||||
"detección combina el nombre de la columna y el rango de sus valores "
|
||||
"(latitud en [−90, 90], longitud en [−180, 180])."
|
||||
)
|
||||
if reason:
|
||||
text += f"\n\n*Criterio de detección:* {reason}."
|
||||
return [model.Heading(text=CHAPTER_TITLE, level=1),
|
||||
model.Markdown(text=text)]
|
||||
|
||||
|
||||
def _extent_blocks(extent: dict) -> list:
|
||||
"""KVTable with bbox/centroid/span + DataTable with the per-region counts."""
|
||||
if not _is_dict(extent) or not extent.get("n_points"):
|
||||
return []
|
||||
blocks = []
|
||||
bbox = extent.get("bbox") if _is_dict(extent.get("bbox")) else {}
|
||||
centroid = extent.get("centroid") if _is_dict(extent.get("centroid")) else {}
|
||||
hemi = extent.get("hemisphere") if _is_dict(extent.get("hemisphere")) else {}
|
||||
|
||||
rows = [("Puntos con coordenadas", _fmt_num(extent.get("n_points")))]
|
||||
if bbox:
|
||||
rows.append(("Latitud (mín. / máx.)",
|
||||
f"{_fmt_coord(bbox.get('lat_min'))} a "
|
||||
f"{_fmt_coord(bbox.get('lat_max'))}"))
|
||||
rows.append(("Longitud (mín. / máx.)",
|
||||
f"{_fmt_coord(bbox.get('lon_min'))} a "
|
||||
f"{_fmt_coord(bbox.get('lon_max'))}"))
|
||||
if centroid:
|
||||
rows.append(("Centroide",
|
||||
f"{_fmt_coord(centroid.get('lat'))}, "
|
||||
f"{_fmt_coord(centroid.get('lon'))}"))
|
||||
if extent.get("span_km") is not None:
|
||||
rows.append(("Extensión (diagonal)", _fmt_km(extent.get("span_km"))))
|
||||
if hemi:
|
||||
n, s = hemi.get("north"), hemi.get("south")
|
||||
e, w = hemi.get("east"), hemi.get("west")
|
||||
rows.append(("Hemisferios",
|
||||
f"N {_fmt_num(n)} / S {_fmt_num(s)} · "
|
||||
f"E {_fmt_num(e)} / O {_fmt_num(w)}"))
|
||||
blocks.append(model.KVTable(rows=rows, title="Extensión geográfica"))
|
||||
|
||||
by_region = extent.get("by_region")
|
||||
if isinstance(by_region, list) and by_region:
|
||||
total = sum(r.get("count", 0) for r in by_region if _is_dict(r)) or 0
|
||||
rrows = []
|
||||
for r in by_region:
|
||||
if not _is_dict(r):
|
||||
continue
|
||||
cnt = r.get("count", 0)
|
||||
pct = (cnt / total) if total else None
|
||||
pct_txt = f"{pct * 100:.1f}%" if pct is not None else "—"
|
||||
rrows.append([model._safe_str(r.get("region")), _fmt_num(cnt),
|
||||
pct_txt])
|
||||
if rrows:
|
||||
blocks.append(model.DataTable(
|
||||
header=["Zona / país", "Puntos", "% del total"], rows=rrows,
|
||||
title="Distribución por zona",
|
||||
note="Asignación aproximada por bounding box de cada región "
|
||||
"(no es reverse-geocoding exacto de fronteras)."))
|
||||
return blocks
|
||||
|
||||
|
||||
def _narrative_block(profile: dict, ctx: dict, extent: dict) -> list:
|
||||
"""A one-line narrative of where the points concentrate.
|
||||
|
||||
Uses the derived ``note`` from analyze_geo_extent by default; optionally
|
||||
calls an LLM (ctx['run_geo_llm']) for a richer one-liner.
|
||||
"""
|
||||
note = model._safe_str((extent or {}).get("note"))
|
||||
if ctx.get("run_geo_llm"):
|
||||
by_region = (extent or {}).get("by_region") or []
|
||||
bbox = (extent or {}).get("bbox") or {}
|
||||
try:
|
||||
from core.ask_llm import ask_llm
|
||||
prompt = (
|
||||
"Eres un analista de datos. En UNA frase en español, describe "
|
||||
"dónde se concentran geográficamente estos puntos. Sé concreto "
|
||||
"y no inventes precisión que los datos no tienen.\n"
|
||||
f"Conteo por zona: {by_region}\nBounding box: {bbox}."
|
||||
)
|
||||
out = ask_llm(prompt,
|
||||
model=ctx.get("geo_llm_model",
|
||||
"claude-haiku-4-5-20251001"),
|
||||
echo=False)
|
||||
if out and isinstance(out, str) and out.strip():
|
||||
note = out.strip()
|
||||
except Exception: # noqa: BLE001 — degrade to the derived note.
|
||||
pass
|
||||
if not note:
|
||||
return []
|
||||
return [model.Markdown(text=f"**Interpretación.** {note}")]
|
||||
|
||||
|
||||
def _no_points_block(profile: dict, detected: dict) -> list:
|
||||
"""Degrade honestly when the raw coordinate arrays are not available."""
|
||||
blocks = []
|
||||
bbox = _bbox_from_profile(profile, detected)
|
||||
if bbox:
|
||||
rows = [
|
||||
("Latitud (mín. / máx.)",
|
||||
f"{_fmt_coord(bbox.get('lat_min'))} a "
|
||||
f"{_fmt_coord(bbox.get('lat_max'))}"),
|
||||
("Longitud (mín. / máx.)",
|
||||
f"{_fmt_coord(bbox.get('lon_min'))} a "
|
||||
f"{_fmt_coord(bbox.get('lon_max'))}"),
|
||||
]
|
||||
blocks.append(model.KVTable(
|
||||
rows=rows, title="Extensión geográfica (aproximada)"))
|
||||
blocks.append(model.Note(
|
||||
"No se incluyeron las coordenadas crudas en el contexto, por lo que el "
|
||||
"mapa y el análisis por zona no se han dibujado. El bounding box "
|
||||
"mostrado se deriva de los mínimos y máximos por columna. Para el "
|
||||
"scatter geográfico completo, pasa los arrays en "
|
||||
"ctx['geo_points'] = {'lats': [...], 'lons': [...]} o las columnas en "
|
||||
"ctx['raw_numeric']."))
|
||||
return blocks
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Entry point.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def build_geospatial(profile: dict, ctx: dict):
|
||||
"""Build the GEOSPATIAL Chapter, or None if the dataset has no coordinates.
|
||||
|
||||
Args:
|
||||
profile: the ``eda`` group TableProfile dict.
|
||||
ctx: presentation context; may carry ``geo_points``/``raw_numeric`` with
|
||||
the raw coordinate arrays and the ``run_geo_llm`` flag.
|
||||
|
||||
Returns:
|
||||
A ``model.Chapter`` with the geographic scatter + zone/country analysis,
|
||||
or ``None`` when no latitude/longitude column pair is detected.
|
||||
"""
|
||||
profile = profile or {}
|
||||
ctx = ctx or {}
|
||||
if not isinstance(profile, dict):
|
||||
return None
|
||||
|
||||
detected = _detect_columns(profile)
|
||||
lats, lons, source = _resolve_coords(profile, ctx, detected)
|
||||
|
||||
has_detection = bool((detected or {}).get("lat_col") and
|
||||
(detected or {}).get("lon_col"))
|
||||
has_points = bool(lats and lons)
|
||||
if not has_detection and not has_points:
|
||||
return None # chapter does not apply: no coordinates in this dataset.
|
||||
|
||||
# Labels for axes / intro. When only raw arrays were given (no detection),
|
||||
# fall back to generic names.
|
||||
lat_col = (detected or {}).get("lat_col") or "lat"
|
||||
lon_col = (detected or {}).get("lon_col") or "lon"
|
||||
|
||||
blocks = _intro_block(detected, lat_col, lon_col)
|
||||
|
||||
if has_points:
|
||||
clean_lats = _clean_floats(lats)
|
||||
clean_lons = _clean_floats(lons)
|
||||
|
||||
# Zone / country analysis.
|
||||
extent = {}
|
||||
if analyze_geo_extent is not None:
|
||||
try:
|
||||
extent = analyze_geo_extent(clean_lats, clean_lons) or {}
|
||||
except Exception: # noqa: BLE001
|
||||
extent = {}
|
||||
|
||||
# The geographic scatter figure (its own page/slide).
|
||||
scatter = {}
|
||||
if build_geo_scatter is not None:
|
||||
try:
|
||||
scatter = build_geo_scatter(clean_lats, clean_lons) or {}
|
||||
except Exception: # noqa: BLE001
|
||||
scatter = {}
|
||||
maker = _make_geo_scatter(scatter, lat_col, lon_col) if scatter else None
|
||||
if maker is not None:
|
||||
blocks.append(model.Figure(
|
||||
make=maker,
|
||||
caption="Cada punto es una observación situada por sus "
|
||||
"coordenadas; el recuadro rojo es el bounding box. La "
|
||||
"escala respeta la latitud (proyección equirectangular)."))
|
||||
else:
|
||||
blocks.append(model.Note(
|
||||
"No se pudo construir el scatter geográfico a partir de las "
|
||||
"coordenadas proporcionadas."))
|
||||
|
||||
blocks += _extent_blocks(extent)
|
||||
blocks += _narrative_block(profile, ctx, extent)
|
||||
else:
|
||||
# Columns detected but no raw points available — degrade honestly.
|
||||
blocks += _no_points_block(profile, detected)
|
||||
|
||||
if not blocks:
|
||||
return None
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,245 @@
|
||||
"""Tests for the GEOSPATIAL chapter — DoD: golden + edges + anti-cut.
|
||||
|
||||
Self-contained: builds synthetic TableProfiles (no DuckDB) so the suite is fast
|
||||
and deterministic. The raw coordinate arrays are passed through ``ctx`` exactly
|
||||
as the chapter's contract documents (``ctx['geo_points']`` / ``ctx['raw_numeric']``).
|
||||
|
||||
Verifies that the chapter detects the lat/lon pair, draws the geographic scatter
|
||||
figure, analyses the zone/country (bounding box + per-region counts), returns
|
||||
None when there are no coordinates, degrades honestly when the raw points are
|
||||
absent, and that a profile with long column names + many points + several
|
||||
regions renders to PDF and PPTX without cutting any text (long content wraps, it
|
||||
is never truncated).
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
from pptx import Presentation
|
||||
|
||||
from datascience.automatic_eda.chapters.geospatial import (
|
||||
build_geospatial,
|
||||
CHAPTER_VERSION,
|
||||
)
|
||||
from datascience.automatic_eda import build_document, render_pdf, render_pptx
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Synthetic data helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _grid(lat0: float, lon0: float, n: int, spread: float = 1.0):
|
||||
"""A small deterministic cloud of n points around (lat0, lon0)."""
|
||||
lats, lons = [], []
|
||||
for i in range(n):
|
||||
# deterministic pseudo-spread, no randomness.
|
||||
f = (i % 11) / 11.0 - 0.5
|
||||
g = (i % 7) / 7.0 - 0.5
|
||||
lats.append(lat0 + f * spread)
|
||||
lons.append(lon0 + g * spread)
|
||||
return lats, lons
|
||||
|
||||
|
||||
def _profile_with_coords(lat_name="lat", lon_name="lon", lats=None, lons=None):
|
||||
"""A profile carrying a lat/lon column pair with valid ranges."""
|
||||
lats = lats if lats is not None else [40.4, 41.0, 39.8, 40.1]
|
||||
lons = lons if lons is not None else [-3.7, -3.6, -4.0, -3.9]
|
||||
return {
|
||||
"table": "lugares",
|
||||
"columns": [
|
||||
{"name": lat_name, "inferred_type": "numeric",
|
||||
"numeric": {"min": min(lats), "max": max(lats),
|
||||
"mean": sum(lats) / len(lats)}},
|
||||
{"name": lon_name, "inferred_type": "numeric",
|
||||
"numeric": {"min": min(lons), "max": max(lons),
|
||||
"mean": sum(lons) / len(lons)}},
|
||||
{"name": "valor", "inferred_type": "numeric",
|
||||
"numeric": {"min": 0, "max": 100, "mean": 50}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _ctx_points(lats, lons):
|
||||
return {"geo_points": {"lats": lats, "lons": lons}}
|
||||
|
||||
|
||||
def _kinds(chapter):
|
||||
return [getattr(b, "kind", None) for b in chapter.blocks]
|
||||
|
||||
|
||||
def _tables(chapter):
|
||||
return [b for b in chapter.blocks if getattr(b, "kind", None) == "data_table"]
|
||||
|
||||
|
||||
def _figures(chapter):
|
||||
return [b for b in chapter.blocks if getattr(b, "kind", None) == "figure"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Golden
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_golden_estructura_y_version():
|
||||
lats, lons = [40.4, 41.0, 39.8, 40.1], [-3.7, -3.6, -4.0, -3.9]
|
||||
ch = build_geospatial(_profile_with_coords(lats=lats, lons=lons),
|
||||
_ctx_points(lats, lons))
|
||||
assert ch is not None
|
||||
assert ch.id == "geospatial"
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
kinds = _kinds(ch)
|
||||
# intro heading + markdown + scatter figure + extent kv + per-region table.
|
||||
assert "heading" in kinds
|
||||
assert "markdown" in kinds
|
||||
assert "figure" in kinds, "falta el scatter geográfico"
|
||||
assert "kv_table" in kinds, "falta la tabla de extensión"
|
||||
|
||||
|
||||
def test_golden_detecta_columnas_y_nombra_ejes():
|
||||
lats, lons = _grid(40.4, -3.7, 30, spread=0.8)
|
||||
prof = _profile_with_coords("latitude", "longitude", lats, lons)
|
||||
ch = build_geospatial(prof, _ctx_points(lats, lons))
|
||||
intro = [b for b in ch.blocks if b.kind == "markdown"][0].text
|
||||
assert "latitude" in intro and "longitude" in intro
|
||||
|
||||
|
||||
def test_golden_figura_es_perezosa_y_dibujable():
|
||||
lats, lons = _grid(40.4, -3.7, 50, spread=0.6)
|
||||
ch = build_geospatial(_profile_with_coords(lats=lats, lons=lons),
|
||||
_ctx_points(lats, lons))
|
||||
fig_block = _figures(ch)[0]
|
||||
assert fig_block.make is not None and fig_block.fig is None # lazy
|
||||
fig = fig_block.make() # must draw without raising
|
||||
assert fig is not None
|
||||
import matplotlib.pyplot as plt
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_golden_analisis_por_zona_espana():
|
||||
lats, lons = _grid(40.4, -3.7, 40, spread=0.5) # Madrid area
|
||||
ch = build_geospatial(_profile_with_coords(lats=lats, lons=lons),
|
||||
_ctx_points(lats, lons))
|
||||
tables = _tables(ch)
|
||||
region_tbl = [t for t in tables if "zona" in (t.title or "").lower()]
|
||||
assert region_tbl, "falta la tabla por zona/país"
|
||||
flat = " ".join(" ".join(str(c) for c in r) for r in region_tbl[0].rows)
|
||||
# Spain-area points must resolve to a Spain/European region, not empty.
|
||||
assert region_tbl[0].rows
|
||||
assert any(c for c in (region_tbl[0].rows[0]))
|
||||
|
||||
|
||||
def test_golden_raw_numeric_source():
|
||||
"""Coordinates can also come from ctx['raw_numeric'] keyed by detected cols."""
|
||||
lats, lons = _grid(48.85, 2.35, 25, spread=0.4) # Paris area
|
||||
prof = _profile_with_coords("lat", "lon", lats, lons)
|
||||
ctx = {"raw_numeric": {"lat": lats, "lon": lons}}
|
||||
ch = build_geospatial(prof, ctx)
|
||||
assert ch is not None
|
||||
assert _figures(ch), "el scatter debe construirse desde raw_numeric"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Edges
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_edge_sin_coordenadas_devuelve_none():
|
||||
prof = {
|
||||
"table": "ventas",
|
||||
"columns": [
|
||||
{"name": "precio", "inferred_type": "numeric",
|
||||
"numeric": {"min": 0, "max": 1000}},
|
||||
{"name": "categoria", "inferred_type": "text"},
|
||||
],
|
||||
}
|
||||
assert build_geospatial(prof, {}) is None
|
||||
|
||||
|
||||
def test_edge_none_y_vacio_no_rompen():
|
||||
assert build_geospatial(None, None) is None
|
||||
assert build_geospatial({}, {}) is None
|
||||
assert build_geospatial({"columns": []}, {}) is None
|
||||
assert build_geospatial("not a dict", {}) is None
|
||||
|
||||
|
||||
def test_edge_nombre_lat_pero_rango_invalido_no_aplica():
|
||||
"""A column named 'lat' whose values are out of [-90,90] is NOT a coordinate."""
|
||||
prof = {
|
||||
"table": "x",
|
||||
"columns": [
|
||||
{"name": "lat", "inferred_type": "numeric",
|
||||
"numeric": {"min": 1000, "max": 9999}},
|
||||
{"name": "lon", "inferred_type": "numeric",
|
||||
"numeric": {"min": 1000, "max": 9999}},
|
||||
],
|
||||
}
|
||||
assert build_geospatial(prof, {}) is None
|
||||
|
||||
|
||||
def test_edge_columnas_detectadas_sin_puntos_degrada():
|
||||
"""Detected lat/lon but no raw arrays -> honest note + approx bbox, no crash."""
|
||||
prof = _profile_with_coords(lats=[40.0, 41.0], lons=[-3.0, -4.0])
|
||||
ch = build_geospatial(prof, {}) # no geo_points / raw_numeric
|
||||
assert ch is not None
|
||||
assert not _figures(ch), "sin puntos no debe dibujarse el scatter"
|
||||
notes = [b for b in ch.blocks if b.kind == "note"]
|
||||
assert notes and "coordenadas crudas" in notes[0].text
|
||||
|
||||
|
||||
def test_edge_coordenadas_con_nan_se_filtran():
|
||||
lats = [40.4, float("nan"), 41.0, None, 39.8]
|
||||
lons = [-3.7, -3.6, float("nan"), -3.9, -4.0]
|
||||
ch = build_geospatial(_profile_with_coords(lats=[39.8, 41.0],
|
||||
lons=[-4.0, -3.6]),
|
||||
_ctx_points(lats, lons))
|
||||
assert ch is not None # must not raise on NaN/None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Anti-cut: long names + many points + several regions render without truncation
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _multiregion_points(per: int = 700):
|
||||
"""Points spread across Spain, France and the USA to fill the region table."""
|
||||
lats, lons = [], []
|
||||
for (la, lo) in ((40.4, -3.7), (48.85, 2.35), (39.0, -98.0)):
|
||||
gl, gn = _grid(la, lo, per, spread=2.0)
|
||||
lats += gl
|
||||
lons += gn
|
||||
return lats, lons
|
||||
|
||||
|
||||
def test_anticut_pdf_y_pptx_no_truncan():
|
||||
lat_name = "latitud_geografica_del_punto_de_observacion_registrado"
|
||||
lon_name = "longitud_geografica_del_punto_de_observacion_registrado"
|
||||
lats, lons = _multiregion_points(700)
|
||||
prof = _profile_with_coords(lat_name, lon_name, lats, lons)
|
||||
ctx = {"geo_points": {"lats": lats, "lons": lons}}
|
||||
|
||||
full = build_document(prof, ctx)
|
||||
assert any(c.id == "geospatial" for c in full)
|
||||
chapters = [c for c in full if c.id == "geospatial"]
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "g.pdf")
|
||||
pptx = os.path.join(d, "g.pptx")
|
||||
rp = render_pdf(chapters, pdf, {"title": "EDA"})
|
||||
rx = render_pptx(chapters, pptx, {"title": "EDA"})
|
||||
assert os.path.exists(pdf) and os.path.exists(pptx)
|
||||
assert (rp or {}).get("n_pages", 0) >= 1
|
||||
|
||||
# PDF: the long lat column name survives whole (wraps, not cut) and there
|
||||
# is no truncation marker in this chapter.
|
||||
pdf_txt = "".join((pg.extract_text() or "") for pg in PdfReader(pdf).pages)
|
||||
assert "…" not in pdf_txt and "..." not in pdf_txt
|
||||
norm = re.sub(r"\s+", "", pdf_txt)
|
||||
assert lat_name in norm, "el nombre largo de la columna se cortó en el PDF"
|
||||
|
||||
# PPTX: long name present in some shape/cell, untruncated.
|
||||
allt = []
|
||||
for s in Presentation(pptx).slides:
|
||||
for sh in s.shapes:
|
||||
if sh.has_text_frame:
|
||||
allt.append(sh.text_frame.text)
|
||||
if sh.has_table:
|
||||
for row in sh.table.rows:
|
||||
for c in row.cells:
|
||||
allt.append(c.text)
|
||||
joined = re.sub(r"\s+", "", "\n".join(allt))
|
||||
assert lat_name in joined, "el nombre largo de la columna se cortó en el PPTX"
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Glossary chapter (GLOSARIO) — always the last chapter, clickable terms.
|
||||
|
||||
Renders one entry per glossary term that the other chapters registered during
|
||||
the document build through ``ctx['glossary'].add(key, label, definition)`` (see
|
||||
``GlossaryCollector`` in ``model.py``). Each entry is a clickable destination:
|
||||
every in-text appearance a chapter marked with ``[[term:key]]texto[[/term]]``
|
||||
becomes a real jump to its entry here — PDF link annotations (PyMuPDF) and PPTX
|
||||
native slide jumps, both wired by the renderers.
|
||||
|
||||
Returns ``None`` when no term was registered (there is nothing to show), so the
|
||||
chapter simply disappears from documents that did not mark any term.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "glosario"
|
||||
CHAPTER_TITLE = "Glosario"
|
||||
|
||||
|
||||
def build_glosario(profile: dict, ctx: dict):
|
||||
"""Build the glossary Chapter from the shared collector, or None if empty."""
|
||||
ctx = ctx or {}
|
||||
glossary = ctx.get("glossary")
|
||||
if not isinstance(glossary, model.GlossaryCollector) or not glossary:
|
||||
return None
|
||||
|
||||
blocks = [
|
||||
model.Heading(text="Glosario de términos", level=1),
|
||||
model.Markdown(text=(
|
||||
"Definición de los términos técnicos que aparecen en el informe. "
|
||||
"Cada término va resaltado en el texto y, al pulsarlo, salta a su "
|
||||
"definición en esta sección.")),
|
||||
]
|
||||
# One clickable destination per term, alphabetically by visible label.
|
||||
for term in glossary.terms(by="label"):
|
||||
blocks.append(model.GlossaryEntry(
|
||||
key=model._safe_str(term.get("key")),
|
||||
label=model._safe_str(term.get("label")),
|
||||
definition=model._safe_str(term.get("definition"))))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -34,7 +34,7 @@ try:
|
||||
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
|
||||
build_boxplot_stats = None # type: ignore[assignment]
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_VERSION = "1.1.0"
|
||||
CHAPTER_ID = "num_distr"
|
||||
CHAPTER_TITLE = "Distribuciones numéricas"
|
||||
|
||||
@@ -278,12 +278,17 @@ def build_num_distr(profile: dict, ctx: dict):
|
||||
box = build_boxplot_stats(numeric) or {}
|
||||
except Exception: # noqa: BLE001 — degrade, never raise.
|
||||
box = {}
|
||||
blocks.append(model.Heading(text=str(name), level=2))
|
||||
blocks.append(model.Figure(
|
||||
make=_figure_maker(name, numeric, box),
|
||||
caption=f"Distribución de «{name}» — histograma (media/mediana/±σ) "
|
||||
f"y boxplot."))
|
||||
blocks.append(model.Markdown(text=_stats_note(name, numeric, box)))
|
||||
# Keep the column heading, its figure and its stats note together on the
|
||||
# same page/slide (mejora 3 — keep-together): the renderers measure the
|
||||
# whole Group and move it whole when it would not fit.
|
||||
blocks.append(model.Group(blocks=[
|
||||
model.Heading(text=str(name), level=2),
|
||||
model.Figure(
|
||||
make=_figure_maker(name, numeric, box),
|
||||
caption=f"Distribución de «{name}» — histograma "
|
||||
f"(media/mediana/±σ) y boxplot."),
|
||||
model.Markdown(text=_stats_note(name, numeric, box)),
|
||||
]))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
|
||||
@@ -65,19 +65,33 @@ def _pdf_text(path: str) -> str:
|
||||
return re.sub(r"\s+", " ", txt)
|
||||
|
||||
|
||||
def _flatten(blocks):
|
||||
"""Expand keep-together Groups so the per-column heading/figure/markdown are
|
||||
inspectable as a flat block list (the chapter wraps each column in a Group)."""
|
||||
out = []
|
||||
for b in blocks:
|
||||
if getattr(b, "kind", "") == "group":
|
||||
out.extend(_flatten(getattr(b, "blocks", []) or []))
|
||||
else:
|
||||
out.append(b)
|
||||
return out
|
||||
|
||||
|
||||
def test_golden_chapter_estructura_y_bloques():
|
||||
ch = build_num_distr(_profile(n_numeric=2), {})
|
||||
assert ch is not None
|
||||
assert ch.id == "num_distr"
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
kinds = [b.kind for b in ch.blocks]
|
||||
# Per-column blocks are wrapped in keep-together Groups: flatten to inspect.
|
||||
flat = _flatten(ch.blocks)
|
||||
kinds = [b.kind for b in flat]
|
||||
# Heading + intro Markdown, then per column: Heading + Figure + Markdown.
|
||||
assert kinds[0] == "heading"
|
||||
assert kinds[1] == "markdown"
|
||||
assert kinds.count("figure") == 2 # one figure per numeric column.
|
||||
assert kinds.count("heading") == 1 + 2 # chapter title + one per column.
|
||||
# Each figure has a lazy maker that produces a real matplotlib figure.
|
||||
figs = [b for b in ch.blocks if b.kind == "figure"]
|
||||
figs = [b for b in flat if b.kind == "figure"]
|
||||
fig = figs[0].make()
|
||||
assert fig is not None
|
||||
# Two stacked axes: histogram + boxplot share the figure.
|
||||
@@ -90,7 +104,8 @@ def test_golden_media_mediana_sigma_y_boxplot_presentes():
|
||||
# The intro documents the three reference lines and the Tukey boxplot; the
|
||||
# per-column note carries the actual mean/median/σ numbers and the shape.
|
||||
ch = build_num_distr(_profile(n_numeric=1, extra_categorical=False), {})
|
||||
md_texts = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
|
||||
md_texts = " ".join(b.text for b in _flatten(ch.blocks)
|
||||
if b.kind == "markdown")
|
||||
assert "media" in md_texts and "mediana" in md_texts
|
||||
assert "±1σ" in md_texts or "σ" in md_texts
|
||||
assert "boxplot" in md_texts.lower()
|
||||
@@ -126,7 +141,8 @@ def test_anti_corte_muchas_columnas_pdf_y_pptx():
|
||||
# 8 numeric columns + long note text: nothing may be cut. Every column
|
||||
# heading must survive in both the PDF text and the PPTX deck.
|
||||
ch = build_num_distr(_profile(n_numeric=8), {})
|
||||
names = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
|
||||
names = [b.text for b in _flatten(ch.blocks)
|
||||
if b.kind == "heading" and b.level == 2]
|
||||
assert len(names) == 8
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "num.pdf")
|
||||
|
||||
@@ -17,7 +17,7 @@ from datetime import datetime, timezone
|
||||
|
||||
from .. import model
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_VERSION = "1.1.0"
|
||||
CHAPTER_ID = "portada"
|
||||
CHAPTER_TITLE = "Portada"
|
||||
|
||||
@@ -67,6 +67,53 @@ def _fmt_int(v) -> str:
|
||||
return str(v)
|
||||
|
||||
|
||||
def _fmt_pct(value) -> str:
|
||||
"""Format a percentage that may arrive as a 0–1 fraction or a 0–100 number."""
|
||||
if value is None:
|
||||
return "—"
|
||||
try:
|
||||
v = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
if 0 < v <= 1.0:
|
||||
v *= 100.0
|
||||
return f"{v:.1f}%"
|
||||
|
||||
|
||||
def _summary_blocks(summary) -> list:
|
||||
"""Mini-summary of the rest of the analysis, shown on the cover (mejora 5).
|
||||
|
||||
The cover is built AFTER the body (``build_document`` passes the aggregated
|
||||
``ctx['document_summary']``), so it can reflect what the analysis found:
|
||||
shape, column types, quality flags and which chapters were included. Returns
|
||||
an empty list when there is no summary (the cover degrades to its metadata
|
||||
table only)."""
|
||||
if not isinstance(summary, dict) or not summary:
|
||||
return []
|
||||
rows = []
|
||||
n_num = summary.get("n_numeric")
|
||||
n_cat = summary.get("n_categorical")
|
||||
if n_num is not None or n_cat is not None:
|
||||
rows.append(("Columnas numéricas / categóricas",
|
||||
f"{_fmt_int(n_num)} / {_fmt_int(n_cat)}"))
|
||||
if summary.get("duplicate_pct") is not None:
|
||||
rows.append(("Filas duplicadas", _fmt_pct(summary.get("duplicate_pct"))))
|
||||
if summary.get("null_cell_pct") is not None:
|
||||
rows.append(("Celdas nulas", _fmt_pct(summary.get("null_cell_pct"))))
|
||||
titles = summary.get("chapter_titles") or []
|
||||
if titles:
|
||||
rows.append(("Capítulos del informe", _fmt_int(len(titles))))
|
||||
|
||||
blocks = [model.Heading(text="Resumen del análisis", level=2)]
|
||||
if rows:
|
||||
blocks.append(model.KVTable(rows=rows))
|
||||
if titles:
|
||||
bullets = "\n".join(f"- {model._safe_str(t)}" for t in titles)
|
||||
blocks.append(model.Markdown(
|
||||
text="Este informe incluye los siguientes capítulos:\n" + bullets))
|
||||
return blocks
|
||||
|
||||
|
||||
def _fmt_date_eu(value) -> str:
|
||||
"""Format a date/ISO string as European DD/MM/AAAA HH:mm (UI convention).
|
||||
|
||||
@@ -152,5 +199,8 @@ def build_portada(profile: dict, ctx: dict):
|
||||
model.Markdown(text=str(granularity)),
|
||||
]
|
||||
|
||||
# Mini-summary of the rest of the analysis (built last, shown on the cover).
|
||||
blocks.extend(_summary_blocks(ctx.get("document_summary")))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
|
||||
@@ -0,0 +1,613 @@
|
||||
"""Time-series chapter (TIMESERIES) for AutomaticEDA.
|
||||
|
||||
This chapter applies **only when the table has a date/datetime column**. When it
|
||||
does, it draws — exactly the user requirement — the evolution of the data over
|
||||
time (the value of each numeric column aggregated per period *and* the count of
|
||||
rows per period) plus the statistical analysis of the series (stationarity,
|
||||
autocorrelation, trend and seasonality). When there is no temporal column
|
||||
``build_timeseries`` returns ``None``.
|
||||
|
||||
Data sources, read defensively and never recomputed here:
|
||||
|
||||
- ``profile['columns']`` — to detect the time column and the numeric columns.
|
||||
Delegated to the pure registry function ``detect_time_column`` (group ``eda``).
|
||||
- ``profile['series'][col]`` — the per-column time-series analysis already
|
||||
produced by ``profile_table(run_series=True)``: ``stationarity`` (ADF+KPSS),
|
||||
``acf_pacf`` (ACF/PACF + Ljung-Box), ``stl`` (trend/seasonal/resid +
|
||||
Hyndman strengths) and the levels/returns suggestion.
|
||||
- ``ctx['timeseries_raw']`` (or ``profile['timeseries_raw']``) — the *raw* ordered
|
||||
series ``{time_col, t:[iso...], series:{col:[float|None]}}`` needed to draw the
|
||||
value-vs-time line and the per-period row count. Exactly like ``modelos`` reads
|
||||
``raw_numeric`` from ``ctx``, this chapter looks for the raw series there and
|
||||
degrades honestly when it is absent (it still renders the textual analysis).
|
||||
|
||||
The raw series is aggregated per period with the pure registry function
|
||||
``resample_timeseries`` and the datetime header is built with ``profile_datetime``
|
||||
(both group ``eda``). Every figure is emitted as a lazy ``Figure`` so the
|
||||
renderers rasterize and scale it to fit a whole page/slide; tables go through
|
||||
``DataTable``/``KVTable`` so the paginator splits them repeating the header. No
|
||||
content is ever cut.
|
||||
|
||||
ctx keys this chapter consumes (all optional):
|
||||
timeseries_raw : dict — ``{time_col, t:[...], series:{col:[...]}}`` raw
|
||||
ordered series used to draw the value-vs-time line and the row-count
|
||||
panel. When absent the chapter omits those figures (with a note) and
|
||||
renders only the analysis available in ``profile['series']``.
|
||||
|
||||
Contract: build_<id>(profile, ctx) -> Chapter | None ; CHAPTER_VERSION = "x.y.z".
|
||||
Reads everything defensively (``.get``) and never raises.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .. import model
|
||||
|
||||
# Pure/impure registry functions (group ``eda``) consumed by this chapter,
|
||||
# imported defensively so the chapter still builds (degrading the affected
|
||||
# section to a note) if any of them is somehow unavailable.
|
||||
try:
|
||||
from datascience.detect_time_column import detect_time_column
|
||||
except Exception: # noqa: BLE001 — keep the chapter importable no matter what.
|
||||
detect_time_column = None # type: ignore[assignment]
|
||||
try:
|
||||
from datascience.profile_datetime import profile_datetime
|
||||
except Exception: # noqa: BLE001
|
||||
profile_datetime = None # type: ignore[assignment]
|
||||
try:
|
||||
from datascience.resample_timeseries import resample_timeseries
|
||||
except Exception: # noqa: BLE001
|
||||
resample_timeseries = None # type: ignore[assignment]
|
||||
|
||||
CHAPTER_VERSION = "1.0.0"
|
||||
CHAPTER_ID = "timeseries"
|
||||
CHAPTER_TITLE = "Series temporales"
|
||||
|
||||
# Plain-Spanish gloss for the stationarity verdict of adf_kpss_stationarity.
|
||||
_VERDICT_GLOSS = {
|
||||
"stationary": "estacionaria: media y varianza estables en el tiempo; se "
|
||||
"puede modelar directamente.",
|
||||
"non_stationary": "no estacionaria: tiene tendencia o varianza cambiante "
|
||||
"(raíz unitaria). Correlacionar o modelar sus niveles "
|
||||
"produce relaciones espurias (Granger-Newbold); conviene "
|
||||
"diferenciar o pasar a retornos.",
|
||||
"inconclusive": "resultado no concluyente (ADF y KPSS discrepan): tratar con "
|
||||
"cautela, probablemente cerca de la no estacionariedad.",
|
||||
}
|
||||
|
||||
# OHLC-style name fragments used to collapse near-identical financial series.
|
||||
_OHLC_HINTS = ("open", "high", "low", "close", "adj", "price", "vwap")
|
||||
|
||||
|
||||
def _fmt_num(value, decimals: int = 3) -> str:
|
||||
"""Compact, defensive number formatting shared with the other chapters."""
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return "sí" if value else "no"
|
||||
if isinstance(value, int):
|
||||
return f"{value:,}".replace(",", ".")
|
||||
if isinstance(value, float):
|
||||
if value != value: # NaN
|
||||
return "NaN"
|
||||
if value in (float("inf"), float("-inf")):
|
||||
return str(value)
|
||||
text = f"{value:.{decimals}f}".rstrip("0").rstrip(".")
|
||||
return text if text else "0"
|
||||
return model._safe_str(value)
|
||||
|
||||
|
||||
def _is_dict(v) -> bool:
|
||||
return isinstance(v, dict)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Detection: which column is the time axis and which numeric columns to chart.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _detect(cols: list) -> dict:
|
||||
"""Return ``{time_col, numeric_cols, ...}`` via the registry function.
|
||||
|
||||
Falls back to an inline scan (datetime inferred_type / datetime semantic
|
||||
types) when ``detect_time_column`` is unavailable, so the chapter still works.
|
||||
"""
|
||||
if detect_time_column is not None:
|
||||
try:
|
||||
res = detect_time_column(cols)
|
||||
if _is_dict(res):
|
||||
return res
|
||||
except Exception: # noqa: BLE001 — degrade to the inline scan.
|
||||
pass
|
||||
time_col = None
|
||||
numeric_cols = []
|
||||
for c in cols or []:
|
||||
if not _is_dict(c):
|
||||
continue
|
||||
it = c.get("inferred_type")
|
||||
sem = c.get("semantic_type")
|
||||
if time_col is None and (
|
||||
it == "datetime" or sem in ("datetime_iso", "date_eu")):
|
||||
time_col = c.get("name")
|
||||
if it == "numeric":
|
||||
numeric_cols.append(c.get("name"))
|
||||
return {"time_col": time_col, "numeric_cols": numeric_cols,
|
||||
"time_semantic": "", "reason": "inline fallback"}
|
||||
|
||||
|
||||
def _raw_series_for(raw: dict, col: str):
|
||||
"""Return (t_list, v_list) for a column from the raw bundle, or (None, None)."""
|
||||
if not _is_dict(raw):
|
||||
return None, None
|
||||
t = raw.get("t")
|
||||
series = raw.get("series") if _is_dict(raw.get("series")) else {}
|
||||
v = series.get(col)
|
||||
if isinstance(t, list) and isinstance(v, list) and t and len(t) == len(v):
|
||||
return t, v
|
||||
return None, None
|
||||
|
||||
|
||||
def _ohlc_groups(numeric_cols: list, raw: dict) -> dict:
|
||||
"""Map each numeric column to a representative to collapse OHLC duplicates.
|
||||
|
||||
When several numeric columns are near-identical financial level series
|
||||
(open/high/low/close/adj close), charting each one repeats the same figure
|
||||
four times. We keep the first OHLC-looking column as the representative for
|
||||
the *figures* and list the collapsed ones in a note; the textual analysis is
|
||||
still produced for every column. Detection is by name only (cheap, no extra
|
||||
data dependency) and conservative: only collapses when >=2 OHLC-like names
|
||||
are present.
|
||||
"""
|
||||
ohlc = [c for c in numeric_cols
|
||||
if isinstance(c, str) and any(h in c.lower() for h in _OHLC_HINTS)]
|
||||
if len(ohlc) < 2:
|
||||
return {}
|
||||
representative = ohlc[0]
|
||||
return {c: representative for c in ohlc if c != representative}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Datetime header (MUST-9.3): range / frequency / regularity / gaps.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _datetime_header(time_col: str, raw: dict) -> list:
|
||||
"""Build the datetime profile header from the raw time axis, when present."""
|
||||
blocks: list = []
|
||||
t, _ = (raw.get("t"), None) if _is_dict(raw) else (None, None)
|
||||
if not (isinstance(t, list) and t and profile_datetime is not None):
|
||||
return blocks
|
||||
try:
|
||||
dt = profile_datetime(t)
|
||||
except Exception: # noqa: BLE001
|
||||
return blocks
|
||||
if not _is_dict(dt):
|
||||
return blocks
|
||||
|
||||
freq_gloss = {
|
||||
"daily": "diaria", "weekly": "semanal", "monthly": "mensual",
|
||||
"quarterly": "trimestral", "yearly": "anual",
|
||||
"irregular": "irregular", "unknown": "indeterminada",
|
||||
}
|
||||
rows = [
|
||||
("Columna de fecha", model._safe_str(time_col)),
|
||||
("Rango", f"{model._safe_str(dt.get('min'))} → "
|
||||
f"{model._safe_str(dt.get('max'))}"),
|
||||
("Observaciones", _fmt_num(dt.get("n"))),
|
||||
("Fechas distintas", _fmt_num(dt.get("n_distinct"))),
|
||||
("Frecuencia", freq_gloss.get(dt.get("freq"), model._safe_str(dt.get("freq")))),
|
||||
("Regular", "sí" if dt.get("is_regular") else "no"),
|
||||
]
|
||||
span = dt.get("span_days")
|
||||
if span is not None:
|
||||
rows.append(("Duración (días)", _fmt_num(span, 1)))
|
||||
n_gaps = dt.get("n_gaps")
|
||||
if n_gaps is not None:
|
||||
rows.append(("Huecos en la rejilla", _fmt_num(n_gaps)))
|
||||
blocks.append(model.KVTable(rows=rows, title="Perfil temporal"))
|
||||
note = dt.get("note")
|
||||
if note:
|
||||
blocks.append(model.Note(model._safe_str(note)))
|
||||
return blocks
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Figure builders (lazy: matplotlib only imported when the renderer draws them).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _parse_dates(labels: list):
|
||||
"""Parse a list of ISO-ish strings/dates to datetime, dropping unparseable.
|
||||
|
||||
Returns (dates, kept_index) so callers can align the values list.
|
||||
"""
|
||||
from datetime import date, datetime
|
||||
|
||||
out = []
|
||||
keep = []
|
||||
for i, lab in enumerate(labels):
|
||||
if isinstance(lab, datetime):
|
||||
out.append(lab)
|
||||
keep.append(i)
|
||||
continue
|
||||
if isinstance(lab, date):
|
||||
out.append(datetime(lab.year, lab.month, lab.day))
|
||||
keep.append(i)
|
||||
continue
|
||||
s = model._safe_str(lab).strip()
|
||||
if not s:
|
||||
continue
|
||||
s2 = s.replace("T", " ")
|
||||
parsed = None
|
||||
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%Y-%m-%d"):
|
||||
try:
|
||||
parsed = datetime.strptime(s2[:len(fmt) + 4] if False else s2, fmt)
|
||||
break
|
||||
except ValueError:
|
||||
continue
|
||||
if parsed is None:
|
||||
try:
|
||||
parsed = datetime.fromisoformat(s.replace("T", " "))
|
||||
except ValueError:
|
||||
continue
|
||||
out.append(parsed)
|
||||
keep.append(i)
|
||||
return out, keep
|
||||
|
||||
|
||||
def _make_evolution_figure(name: str, rs: dict):
|
||||
"""Lazy callable: value-vs-time line + per-period row-count panel (MUST-9.1)."""
|
||||
def _draw():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.dates as mdates
|
||||
|
||||
t_labels = rs.get("t") or []
|
||||
v = rs.get("v") or []
|
||||
counts = rs.get("count") or []
|
||||
dates, keep = _parse_dates(t_labels)
|
||||
vv = [v[i] if i < len(v) else None for i in keep]
|
||||
cc = [counts[i] if i < len(counts) else 0 for i in keep]
|
||||
|
||||
fig, (ax_v, ax_c) = plt.subplots(
|
||||
2, 1, figsize=(7.0, 4.6), sharex=True,
|
||||
gridspec_kw={"height_ratios": [3.0, 1.2], "hspace": 0.12})
|
||||
|
||||
# Top: value aggregated per period (line; gaps where the value is None).
|
||||
xs = [d for d, val in zip(dates, vv) if val is not None]
|
||||
ys = [val for val in vv if val is not None]
|
||||
if xs and ys:
|
||||
ax_v.plot(xs, ys, color="#4e79a7", linewidth=1.4, zorder=3)
|
||||
ax_v.fill_between(xs, ys, min(ys), color="#9ec6df", alpha=0.18,
|
||||
zorder=1)
|
||||
else:
|
||||
ax_v.text(0.5, 0.5, "(sin valores numéricos)", ha="center",
|
||||
va="center", fontsize=9, color="#8a8a8a",
|
||||
transform=ax_v.transAxes)
|
||||
ax_v.set_ylabel(name, fontsize=8)
|
||||
ax_v.tick_params(labelsize=7)
|
||||
ax_v.grid(axis="y", color="#eeeeee", linewidth=0.6)
|
||||
for spine in ("top", "right"):
|
||||
ax_v.spines[spine].set_visible(False)
|
||||
|
||||
# Bottom: number of observations per period (density / gaps).
|
||||
if dates and cc:
|
||||
# Bar width ~ median spacing so bars do not overlap nor leave gaps.
|
||||
width = 1.0
|
||||
if len(dates) > 1:
|
||||
deltas = sorted((dates[i + 1] - dates[i]).days
|
||||
for i in range(len(dates) - 1))
|
||||
width = max(deltas[len(deltas) // 2] * 0.8, 1.0)
|
||||
ax_c.bar(dates, cc, width=width, color="#59a14f", alpha=0.75,
|
||||
align="center")
|
||||
ax_c.set_ylabel("nº filas", fontsize=8)
|
||||
ax_c.tick_params(labelsize=7)
|
||||
ax_c.grid(axis="y", color="#eeeeee", linewidth=0.6)
|
||||
for spine in ("top", "right"):
|
||||
ax_c.spines[spine].set_visible(False)
|
||||
|
||||
ax_c.xaxis.set_major_locator(mdates.AutoDateLocator())
|
||||
ax_c.xaxis.set_major_formatter(mdates.ConciseDateFormatter(
|
||||
ax_c.xaxis.get_major_locator()))
|
||||
freq = rs.get("freq")
|
||||
suptitle = f"{name} — evolución temporal"
|
||||
if freq:
|
||||
suptitle += f" (agregado {freq})"
|
||||
fig.suptitle(suptitle, fontsize=10, fontweight="bold", x=0.02, ha="left")
|
||||
return fig
|
||||
|
||||
return _draw
|
||||
|
||||
|
||||
def _make_stl_figure(stl: dict):
|
||||
"""Lazy callable: the STL trend/seasonal/resid panels, or None if no values.
|
||||
|
||||
``stl_decompose`` only carries the component *values* for short series; for
|
||||
long ones it returns just summary stats (``note``). In that case there is
|
||||
nothing to plot and we return None (the caller renders the strengths as text).
|
||||
"""
|
||||
def _component_values(comp):
|
||||
if _is_dict(comp):
|
||||
vals = comp.get("values")
|
||||
if isinstance(vals, list) and vals:
|
||||
return [x for x in vals]
|
||||
return None
|
||||
|
||||
trend = _component_values(stl.get("trend"))
|
||||
seasonal = _component_values(stl.get("seasonal"))
|
||||
resid = _component_values(stl.get("resid"))
|
||||
if not any([trend, seasonal, resid]):
|
||||
return None
|
||||
|
||||
def _draw():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
panels = [("Tendencia", trend, "#4e79a7"),
|
||||
("Estacional", seasonal, "#59a14f"),
|
||||
("Resto", resid, "#e15759")]
|
||||
panels = [(lbl, vals, col) for lbl, vals, col in panels if vals]
|
||||
fig, axes = plt.subplots(len(panels), 1, figsize=(7.0, 1.4 * len(panels) + 0.6),
|
||||
sharex=True)
|
||||
if len(panels) == 1:
|
||||
axes = [axes]
|
||||
for ax, (lbl, vals, col) in zip(axes, panels):
|
||||
ax.plot(range(len(vals)), vals, color=col, linewidth=1.2)
|
||||
ax.set_ylabel(lbl, fontsize=8)
|
||||
ax.tick_params(labelsize=7)
|
||||
ax.grid(axis="y", color="#eeeeee", linewidth=0.6)
|
||||
for spine in ("top", "right"):
|
||||
ax.spines[spine].set_visible(False)
|
||||
axes[-1].set_xlabel("índice temporal", fontsize=8)
|
||||
fig.suptitle("Descomposición STL", fontsize=10, fontweight="bold",
|
||||
x=0.02, ha="left")
|
||||
fig.tight_layout(rect=(0, 0, 1, 0.96))
|
||||
return fig
|
||||
|
||||
return _draw
|
||||
|
||||
|
||||
def _make_acf_figure(acf_pacf: dict):
|
||||
"""Lazy callable: the ACF stem plot with ±1.96/√n bands, or None."""
|
||||
acf = acf_pacf.get("acf")
|
||||
n = acf_pacf.get("n")
|
||||
if not (isinstance(acf, list) and len(acf) > 1 and isinstance(n, int) and n > 0):
|
||||
return None
|
||||
|
||||
def _draw():
|
||||
import math
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
lags = list(range(len(acf)))
|
||||
fig, ax = plt.subplots(figsize=(7.0, 3.2))
|
||||
ax.vlines(lags, 0, acf, color="#4e79a7", linewidth=1.4)
|
||||
ax.plot(lags, acf, "o", color="#4e79a7", markersize=3)
|
||||
band = 1.96 / math.sqrt(n)
|
||||
ax.axhspan(-band, band, color="#cccccc", alpha=0.3,
|
||||
label="banda ±1.96/√n (ruido blanco)")
|
||||
ax.axhline(0, color="#888888", linewidth=0.8)
|
||||
ax.set_xlabel("retardo (lag)", fontsize=8)
|
||||
ax.set_ylabel("ACF", fontsize=8)
|
||||
ax.tick_params(labelsize=7)
|
||||
ax.legend(fontsize=7, loc="upper right", framealpha=0.85)
|
||||
ax.set_title("Autocorrelación (ACF): lags fuera de la banda = "
|
||||
"correlación significativa", fontsize=9)
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
|
||||
return _draw
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Per-column textual analysis from profile['series'][col].
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _analysis_markdown(sblock: dict) -> str:
|
||||
"""One markdown block summarizing stationarity / autocorrelation / STL."""
|
||||
parts: list = []
|
||||
|
||||
stat = sblock.get("stationarity") if _is_dict(sblock.get("stationarity")) else {}
|
||||
verdict = stat.get("verdict")
|
||||
if verdict:
|
||||
adf = stat.get("adf") if _is_dict(stat.get("adf")) else {}
|
||||
kpss = stat.get("kpss") if _is_dict(stat.get("kpss")) else {}
|
||||
line = (f"**Estacionariedad:** {_VERDICT_GLOSS.get(verdict, verdict)} "
|
||||
f"(ADF p={_fmt_num(adf.get('p_value'), 4)}, "
|
||||
f"KPSS p={_fmt_num(kpss.get('p_value'), 4)}).")
|
||||
warning = stat.get("warning")
|
||||
if warning:
|
||||
line += f" ⚠ {model._safe_str(warning)}"
|
||||
parts.append(line)
|
||||
|
||||
acf = sblock.get("acf_pacf") if _is_dict(sblock.get("acf_pacf")) else {}
|
||||
if acf:
|
||||
is_auto = acf.get("is_autocorrelated")
|
||||
lb = acf.get("ljung_box") if _is_dict(acf.get("ljung_box")) else {}
|
||||
sig = acf.get("significant_acf_lags") or []
|
||||
if is_auto is True:
|
||||
ac_line = ("**Autocorrelación:** la serie está autocorrelada "
|
||||
"(Ljung-Box rechaza independencia, "
|
||||
f"p={_fmt_num(lb.get('p_value'), 4)}): los valores dependen "
|
||||
"de su pasado, no es ruido blanco.")
|
||||
if sig:
|
||||
shown = ", ".join(str(x) for x in sig[:8])
|
||||
more = "…" if len(sig) > 8 else ""
|
||||
ac_line += f" Lags significativos: {shown}{more}."
|
||||
elif is_auto is False:
|
||||
ac_line = ("**Autocorrelación:** no se detecta autocorrelación "
|
||||
"significativa (compatible con ruido blanco, Ljung-Box "
|
||||
f"p={_fmt_num(lb.get('p_value'), 4)}).")
|
||||
else:
|
||||
ac_line = "**Autocorrelación:** no evaluable (datos insuficientes)."
|
||||
parts.append(ac_line)
|
||||
|
||||
stl = sblock.get("stl") if _is_dict(sblock.get("stl")) else {}
|
||||
if stl:
|
||||
ts = stl.get("trend_strength")
|
||||
ss = stl.get("seasonal_strength")
|
||||
if ts is not None or ss is not None:
|
||||
parts.append(
|
||||
"**Descomposición STL:** fuerza de tendencia "
|
||||
f"{_fmt_num(ts, 2)} y fuerza estacional {_fmt_num(ss, 2)} "
|
||||
"(escala 0–1 de Hyndman: cuanto más alto, más marcada la "
|
||||
"componente).")
|
||||
elif stl.get("note"):
|
||||
parts.append(f"**Descomposición STL:** {model._safe_str(stl.get('note'))}")
|
||||
|
||||
if sblock.get("levels_suggested"):
|
||||
reason = sblock.get("levels_reason")
|
||||
kind = sblock.get("levels_kind")
|
||||
tr = sblock.get("to_returns") if _is_dict(sblock.get("to_returns")) else None
|
||||
line = "**Transformación sugerida:** "
|
||||
line += "pasar a retornos" if kind == "returns" else "diferenciar la serie"
|
||||
if reason:
|
||||
line += f" — {model._safe_str(reason)}"
|
||||
if tr and tr.get("mean") is not None:
|
||||
line += (f" (retornos: media {_fmt_num(tr.get('mean'), 5)}, "
|
||||
f"σ {_fmt_num(tr.get('std'), 5)}).")
|
||||
parts.append(line)
|
||||
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Per-column section.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _column_section(name: str, sblock: dict, raw: dict, collapsed_into) -> list:
|
||||
"""Blocks for one numeric column: evolution figure + STL + ACF + analysis."""
|
||||
blocks = [model.Heading(text=model._safe_str(name), level=2)]
|
||||
|
||||
# --- Value-vs-time line + per-period row count (MUST-9.1). ---
|
||||
drew_evolution = False
|
||||
if collapsed_into is None: # skip the figure for collapsed OHLC duplicates.
|
||||
t, v = _raw_series_for(raw, name)
|
||||
if t is not None and resample_timeseries is not None:
|
||||
try:
|
||||
rs = resample_timeseries(t, v)
|
||||
except Exception: # noqa: BLE001
|
||||
rs = None
|
||||
if _is_dict(rs) and rs.get("t"):
|
||||
blocks.append(model.Figure(
|
||||
make=_make_evolution_figure(name, rs),
|
||||
caption=f"Evolución de «{name}» por periodo y nº de "
|
||||
f"observaciones (conteo de filas)."))
|
||||
drew_evolution = True
|
||||
else:
|
||||
blocks.append(model.Note(
|
||||
f"Serie casi idéntica a «{collapsed_into}» (grupo OHLC): se omite el "
|
||||
"gráfico para no repetirlo; el análisis estadístico se mantiene."))
|
||||
|
||||
if not drew_evolution and collapsed_into is None:
|
||||
blocks.append(model.Note(
|
||||
"Gráfico de evolución temporal no disponible: falta la serie cruda "
|
||||
"(pásala en ctx['timeseries_raw'] = {time_col, t, series}). Se "
|
||||
"muestra solo el análisis estadístico."))
|
||||
|
||||
# --- STL panels (MUST-9.2). ---
|
||||
stl = sblock.get("stl") if _is_dict(sblock.get("stl")) else {}
|
||||
if collapsed_into is None and stl:
|
||||
stl_fig = _make_stl_figure(stl)
|
||||
if stl_fig is not None:
|
||||
blocks.append(model.Figure(
|
||||
make=stl_fig,
|
||||
caption=f"Descomposición STL de «{name}»: tendencia, componente "
|
||||
f"estacional y resto."))
|
||||
|
||||
# --- ACF figure (autocorrelation structure). ---
|
||||
acf = sblock.get("acf_pacf") if _is_dict(sblock.get("acf_pacf")) else {}
|
||||
if collapsed_into is None and acf:
|
||||
acf_fig = _make_acf_figure(acf)
|
||||
if acf_fig is not None:
|
||||
blocks.append(model.Figure(
|
||||
make=acf_fig,
|
||||
caption=f"Función de autocorrelación de «{name}»."))
|
||||
|
||||
# --- Textual analysis (always, even for collapsed duplicates). ---
|
||||
analysis = _analysis_markdown(sblock)
|
||||
if analysis:
|
||||
blocks.append(model.Markdown(text=analysis))
|
||||
return blocks
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Entry point.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def build_timeseries(profile: dict, ctx: dict):
|
||||
"""Build the TIMESERIES Chapter, or ``None`` if the table has no date column.
|
||||
|
||||
Args:
|
||||
profile: the ``eda`` group TableProfile dict.
|
||||
ctx: presentation context; ``ctx['timeseries_raw']`` (optional) carries
|
||||
the raw ordered series used to draw the value-vs-time line and the
|
||||
per-period row count.
|
||||
|
||||
Returns:
|
||||
A ``model.Chapter`` with, per numeric column, the value-vs-time evolution
|
||||
+ row-count figure, the STL panels, the ACF figure and the statistical
|
||||
analysis; or ``None`` when there is no temporal column (the chapter does
|
||||
not apply).
|
||||
"""
|
||||
profile = profile or {}
|
||||
if not _is_dict(profile):
|
||||
profile = {}
|
||||
ctx = ctx or {}
|
||||
cols = profile.get("columns") or []
|
||||
|
||||
det = _detect(cols)
|
||||
time_col = det.get("time_col")
|
||||
if not time_col:
|
||||
return None # no date/datetime column -> chapter does not apply.
|
||||
|
||||
numeric_cols = det.get("numeric_cols") or []
|
||||
series_map = profile.get("series") if _is_dict(profile.get("series")) else {}
|
||||
raw = ctx.get("timeseries_raw") or profile.get("timeseries_raw")
|
||||
raw = raw if _is_dict(raw) else {}
|
||||
|
||||
# Which columns can the chapter say anything about: those with a series
|
||||
# analysis block and/or a raw series to chart. Preserve the profile order.
|
||||
chartable = []
|
||||
for name in numeric_cols:
|
||||
has_analysis = _is_dict(series_map.get(name))
|
||||
has_raw, _ = _raw_series_for(raw, name)
|
||||
if has_analysis or has_raw is not None:
|
||||
chartable.append(name)
|
||||
if not chartable:
|
||||
# A date column exists but nothing numeric to chart/analyse: still a
|
||||
# valid (small) chapter — show just the datetime header if we have it.
|
||||
header = _datetime_header(time_col, raw)
|
||||
if not header:
|
||||
return None
|
||||
intro = (
|
||||
f"La tabla tiene una columna temporal («{time_col}») pero no hay "
|
||||
"columnas numéricas con serie analizable.")
|
||||
blocks = [model.Heading(text=CHAPTER_TITLE, level=1),
|
||||
model.Markdown(text=intro)] + header
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
|
||||
collapsed = _ohlc_groups(chartable, raw)
|
||||
|
||||
intro = (
|
||||
"Este capítulo analiza la evolución de la tabla en el tiempo usando la "
|
||||
f"columna de fecha «{time_col}». Para cada columna numérica se muestra su "
|
||||
"**evolución por periodo** (valor agregado) junto al **número de filas por "
|
||||
"periodo** (densidad de observaciones), su **descomposición STL** "
|
||||
"(tendencia / estacionalidad / resto) y la **función de autocorrelación**; "
|
||||
"debajo, el análisis de la serie: estacionariedad (ADF + KPSS), "
|
||||
"autocorrelación (Ljung-Box) y, cuando procede, la transformación "
|
||||
"sugerida (retornos o diferencias) para evitar correlaciones espurias.")
|
||||
|
||||
blocks = [model.Heading(text=CHAPTER_TITLE, level=1),
|
||||
model.Markdown(text=intro)]
|
||||
blocks += _datetime_header(time_col, raw)
|
||||
|
||||
if collapsed:
|
||||
reps = sorted(set(collapsed.values()))
|
||||
collapsed_names = ", ".join(sorted(collapsed.keys()))
|
||||
blocks.append(model.Note(
|
||||
f"Series OHLC casi idénticas detectadas ({collapsed_names}): se "
|
||||
f"grafican consolidadas en «{', '.join(reps)}» para no repetir el "
|
||||
"mismo gráfico; cada columna conserva su análisis estadístico."))
|
||||
|
||||
for name in chartable:
|
||||
sblock = series_map.get(name) if _is_dict(series_map.get(name)) else {}
|
||||
blocks += _column_section(name, sblock, raw, collapsed.get(name))
|
||||
|
||||
return model.Chapter(id=CHAPTER_ID, title=CHAPTER_TITLE,
|
||||
version=CHAPTER_VERSION, blocks=blocks)
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Tests for the TIMESERIES chapter — DoD: golden + edges + anti-cut.
|
||||
|
||||
Self-contained: builds synthetic ``series`` blocks (shaped like
|
||||
``profile_table(run_series=True)`` output) and a raw ``timeseries_raw`` bundle,
|
||||
with no DuckDB, so the suite is fast and deterministic. Verifies that the chapter:
|
||||
|
||||
- returns ``None`` when there is no date/datetime column (the user requirement);
|
||||
- never raises on ``None``/empty/garbage input;
|
||||
- with a date column + raw series emits, per numeric column, the value-vs-time +
|
||||
row-count evolution figure, the STL panels, the ACF figure and the textual
|
||||
analysis (stationarity / autocorrelation / suggested transform);
|
||||
- collapses near-identical OHLC series into one chart while keeping every
|
||||
column's analysis;
|
||||
- renders without cutting anything in both PDF and PPTX (every column heading
|
||||
survives in the rendered output).
|
||||
"""
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
from pypdf import PdfReader
|
||||
|
||||
from datascience.automatic_eda.chapters.timeseries import (
|
||||
build_timeseries, CHAPTER_VERSION, _VERDICT_GLOSS,
|
||||
)
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Synthetic fixtures shaped like the real profile_table(run_series=True) output.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _dates(n: int) -> list:
|
||||
"""n consecutive daily ISO date strings starting 2021-01-01."""
|
||||
from datetime import date, timedelta
|
||||
|
||||
start = date(2021, 1, 1)
|
||||
return [(start + timedelta(days=i)).isoformat() for i in range(n)]
|
||||
|
||||
|
||||
def _series_block(n=120, verdict="non_stationary", autocorr=True, levels=True,
|
||||
with_stl_values=True):
|
||||
"""A synthetic ``series`` block like _build_series_block produces."""
|
||||
trend = [float(i) for i in range(n)]
|
||||
seasonal = [math.sin(i / 6.0) for i in range(n)]
|
||||
resid = [0.1 * ((-1) ** i) for i in range(n)]
|
||||
acf = [1.0] + [max(0.0, 0.9 - 0.05 * k) for k in range(1, 21)]
|
||||
block = {
|
||||
"order_col": "fecha",
|
||||
"ordered": True,
|
||||
"n": n,
|
||||
"stationarity": {
|
||||
"n": n, "verdict": verdict,
|
||||
"adf": {"p_value": 0.42, "stationary": False},
|
||||
"kpss": {"p_value": 0.01, "stationary": False},
|
||||
"warning": ("serie no estacionaria: riesgo de correlación espuria"
|
||||
if verdict != "stationary" else None),
|
||||
},
|
||||
"acf_pacf": {
|
||||
"n": n, "nlags": 20, "acf": acf,
|
||||
"significant_acf_lags": [1, 2, 3, 4, 5],
|
||||
"ljung_box": {"stat": 123.4, "p_value": 0.0 if autocorr else 0.7,
|
||||
"lags": 20},
|
||||
"is_autocorrelated": autocorr,
|
||||
},
|
||||
"period_source": "datetime_freq",
|
||||
"stl": {
|
||||
"n": n, "period": 7, "period_inferred": False, "robust": False,
|
||||
"trend": {"values": trend} if with_stl_values else {
|
||||
"note": "serie larga: solo estadisticos", "mean": 60.0},
|
||||
"seasonal": {"values": seasonal} if with_stl_values else {"mean": 0.0},
|
||||
"resid": {"values": resid} if with_stl_values else {"mean": 0.0},
|
||||
"trend_strength": 0.95, "seasonal_strength": 0.42,
|
||||
},
|
||||
}
|
||||
if levels:
|
||||
block["levels_suggested"] = True
|
||||
block["levels_kind"] = "returns"
|
||||
block["levels_reason"] = ("columna financiera no estacionaria: usar "
|
||||
"retornos evita correlación espuria.")
|
||||
block["to_returns"] = {"method": "log", "mean": 0.001, "std": 0.02}
|
||||
else:
|
||||
block["levels_suggested"] = False
|
||||
return block
|
||||
|
||||
|
||||
def _profile(numeric_names=("precio",), n=120, with_stl_values=True):
|
||||
cols = [{"name": "fecha", "inferred_type": "datetime",
|
||||
"semantic_type": "datetime_iso"}]
|
||||
series_map = {}
|
||||
for nm in numeric_names:
|
||||
cols.append({"name": nm, "inferred_type": "numeric",
|
||||
"numeric": {"min": 1.0, "max": 200.0, "mean": 100.0,
|
||||
"median": 95.0, "std": 40.0}})
|
||||
series_map[nm] = _series_block(n=n, with_stl_values=with_stl_values)
|
||||
return {"table": "cotizaciones", "n_rows": n, "n_cols": len(cols),
|
||||
"columns": cols, "series": series_map}
|
||||
|
||||
|
||||
def _ctx_raw(numeric_names=("precio",), n=120):
|
||||
t = _dates(n)
|
||||
series = {}
|
||||
for j, nm in enumerate(numeric_names):
|
||||
series[nm] = [float(100 + i + 5 * j) for i in range(n)]
|
||||
return {"timeseries_raw": {"time_col": "fecha", "t": t, "series": series}}
|
||||
|
||||
|
||||
def _pdf_text(path: str) -> str:
|
||||
txt = "".join((pg.extract_text() or "") for pg in PdfReader(path).pages)
|
||||
return re.sub(r"\s+", " ", txt)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Golden.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_golden_estructura_y_figuras():
|
||||
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
|
||||
assert ch is not None
|
||||
assert ch.id == "timeseries"
|
||||
assert ch.version == CHAPTER_VERSION
|
||||
kinds = [b.kind for b in ch.blocks]
|
||||
assert kinds[0] == "heading" # chapter title
|
||||
assert kinds[1] == "markdown" # intro
|
||||
assert "kv_table" in kinds # datetime profile header (MUST-9.3)
|
||||
# Per column: evolution figure + STL figure + ACF figure + analysis markdown.
|
||||
figs = [b for b in ch.blocks if b.kind == "figure"]
|
||||
assert len(figs) >= 3, "evolución + STL + ACF esperadas"
|
||||
# Lazy makers must produce real matplotlib figures.
|
||||
import matplotlib.pyplot as plt
|
||||
for f in figs:
|
||||
fig = f.make()
|
||||
assert fig is not None
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_golden_evolucion_tiene_dos_paneles_valor_y_conteo():
|
||||
# MUST-9.1: the evolution figure has a value panel + a row-count panel.
|
||||
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
|
||||
figs = [b for b in ch.blocks if b.kind == "figure"]
|
||||
import matplotlib.pyplot as plt
|
||||
fig = figs[0].make() # first figure is the evolution one.
|
||||
assert len(fig.axes) == 2, "panel de valor + panel de conteo de filas"
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_golden_analisis_textual_presente():
|
||||
ch = build_timeseries(_profile(("precio",)), _ctx_raw(("precio",)))
|
||||
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
|
||||
assert "Estacionariedad" in md
|
||||
assert "Autocorrelación" in md
|
||||
assert "STL" in md
|
||||
# Verdict gloss surfaced for the non-stationary preset.
|
||||
assert _VERDICT_GLOSS["non_stationary"].split(":")[0] in md
|
||||
# Levels/returns suggestion surfaced.
|
||||
assert "retornos" in md.lower()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Edges.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_edge_sin_columna_fecha_devuelve_none():
|
||||
prof = {"columns": [
|
||||
{"name": "precio", "inferred_type": "numeric", "numeric": {"mean": 1.0}},
|
||||
{"name": "ciudad", "inferred_type": "categorical",
|
||||
"categorical": {"top": []}},
|
||||
], "series": {"precio": _series_block()}}
|
||||
assert build_timeseries(prof, {}) is None
|
||||
|
||||
|
||||
def test_edge_none_y_vacio_no_revienta():
|
||||
assert build_timeseries(None, None) is None
|
||||
assert build_timeseries({}, {}) is None
|
||||
assert build_timeseries({"columns": []}, {}) is None
|
||||
# Date column but nothing numeric/series and no raw -> None (nothing to say).
|
||||
assert build_timeseries(
|
||||
{"columns": [{"name": "fecha", "inferred_type": "datetime"}]}, {}) is None
|
||||
|
||||
|
||||
def test_edge_sin_raw_degrada_pero_mantiene_analisis():
|
||||
# No ctx['timeseries_raw']: the chapter must still build (STL/ACF/analysis
|
||||
# from the profile) and note that the evolution chart is unavailable.
|
||||
ch = build_timeseries(_profile(("precio",)), {})
|
||||
assert ch is not None
|
||||
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
|
||||
assert "evolución temporal no disponible" in notes
|
||||
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
|
||||
assert "Estacionariedad" in md
|
||||
|
||||
|
||||
def test_edge_stl_solo_estadisticos_no_dibuja_panel_pero_no_revienta():
|
||||
# Long series: STL carries only stats (no 'values') -> no STL figure, but the
|
||||
# strengths still surface in the textual analysis.
|
||||
ch = build_timeseries(_profile(("precio",), with_stl_values=False),
|
||||
_ctx_raw(("precio",)))
|
||||
assert ch is not None
|
||||
md = " ".join(b.text for b in ch.blocks if b.kind == "markdown")
|
||||
assert "STL" in md
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# OHLC consolidation (MUST-9.3).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_ohlc_consolidacion():
|
||||
names = ("Open", "High", "Low", "Close")
|
||||
ch = build_timeseries(_profile(names), _ctx_raw(names))
|
||||
assert ch is not None
|
||||
notes = " ".join(b.text for b in ch.blocks if b.kind == "note")
|
||||
assert "OHLC" in notes
|
||||
# Only the representative draws the evolution figure; the other 3 are collapsed
|
||||
# so there are fewer evolution figures than columns.
|
||||
captions = [b.caption or "" for b in ch.blocks if b.kind == "figure"]
|
||||
evo = [c for c in captions if "Evolución" in c]
|
||||
assert len(evo) < len(names), "las series OHLC deben consolidarse"
|
||||
# Every column still has its analysis markdown (one heading per column).
|
||||
headings = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
|
||||
for nm in names:
|
||||
assert nm in headings
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Anti-cut: PDF + PPTX.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_anti_corte_pdf_y_pptx():
|
||||
names = tuple(f"serie_{i}" for i in range(6))
|
||||
prof = _profile(names, n=90)
|
||||
ctx = _ctx_raw(names, n=90)
|
||||
ch = build_timeseries(prof, ctx)
|
||||
col_headings = [b.text for b in ch.blocks if b.kind == "heading" and b.level == 2]
|
||||
assert len(col_headings) == 6
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pdf = os.path.join(d, "ts.pdf")
|
||||
res_pdf = render_automatic_eda_pdf(
|
||||
prof, pdf, {"ctx": ctx, "write_manifest": False})
|
||||
assert res_pdf["path"] == pdf
|
||||
txt = _pdf_text(pdf)
|
||||
for nm in col_headings:
|
||||
assert nm in txt, f"columna '{nm}' cortada/ausente en el PDF"
|
||||
pptx = os.path.join(d, "ts.pptx")
|
||||
res_pptx = render_automatic_eda_pptx(
|
||||
prof, pptx, {"ctx": ctx, "write_manifest": False})
|
||||
assert res_pptx["path"] == pptx
|
||||
assert res_pptx["n_slides"] >= 6
|
||||
@@ -26,19 +26,26 @@ from . import model
|
||||
# placeholders other agents will fill by creating chapters/<id>.py — they will
|
||||
# appear in this exact position automatically once their module exists.
|
||||
CHAPTER_ORDER = [
|
||||
"portada", # cover
|
||||
"portada", # cover — BUILT LAST, PLACED FIRST (see build_document).
|
||||
"overview", # df.head + columns/types/nulls/examples + describe
|
||||
"analisis_llm", # LLM interpretation — sits next to overview (user request)
|
||||
"num_distr", # numeric distributions
|
||||
"cat_distr", # categorical distributions
|
||||
"calidad", # data quality
|
||||
"correlacion", # correlations / associations
|
||||
"modelos", # cheap models (PCA/KMeans/outliers)
|
||||
"analisis_llm", # LLM interpretation
|
||||
"timeseries", # time-series analysis
|
||||
"geospatial", # geospatial
|
||||
"agregacion", # aggregations / pivots
|
||||
"glosario", # glossary — ALWAYS LAST; clickable term destinations.
|
||||
]
|
||||
|
||||
# Chapters whose position is special-cased by build_document: portada is built
|
||||
# last (so it can summarize the rest) but placed first; glosario is built and
|
||||
# placed last (it reads the terms every other chapter registered).
|
||||
_PORTADA = "portada"
|
||||
_GLOSARIO = "glosario"
|
||||
|
||||
|
||||
def build_chapter(chapter_id: str, profile: dict, ctx: dict):
|
||||
"""Build a single chapter by id, or None if absent/not-applicable/error.
|
||||
@@ -75,15 +82,72 @@ def build_document(profile: dict, ctx: dict = None) -> list:
|
||||
list[Chapter] in canonical order, containing only the chapters that are
|
||||
implemented and applicable. Never raises.
|
||||
"""
|
||||
if profile is None:
|
||||
profile = {}
|
||||
if not isinstance(profile, dict):
|
||||
profile = {}
|
||||
if ctx is None:
|
||||
ctx = {}
|
||||
chapters = []
|
||||
# Copy ctx so the shared collector / summary we add do not leak to the caller.
|
||||
ctx = dict(ctx) if isinstance(ctx, dict) else {}
|
||||
|
||||
# A single glossary collector is shared by every chapter via ctx['glossary'].
|
||||
# Chapters call ctx['glossary'].add(key, label, definition) and mark in-text
|
||||
# appearances with [[term:key]]…[[/term]]; the glosario chapter renders the
|
||||
# registered terms and the renderers wire the clickable links.
|
||||
glossary = ctx.get("glossary")
|
||||
if not isinstance(glossary, model.GlossaryCollector):
|
||||
glossary = model.GlossaryCollector()
|
||||
ctx["glossary"] = glossary
|
||||
|
||||
# 1) Body: every chapter except portada (built last) and glosario (placed
|
||||
# last), in canonical order. This also fills the glossary collector.
|
||||
body = []
|
||||
for cid in CHAPTER_ORDER:
|
||||
if cid in (_PORTADA, _GLOSARIO):
|
||||
continue
|
||||
ch = build_chapter(cid, profile, ctx)
|
||||
if ch is not None and ch.blocks:
|
||||
chapters.append(ch)
|
||||
body.append(ch)
|
||||
|
||||
# 2) Aggregated summary of the rest, for the cover (user decision: the cover
|
||||
# is BUILT after the body so it can reflect what the analysis found).
|
||||
ctx["document_summary"] = _summarize_document(profile, body)
|
||||
|
||||
# 3) Build the cover last, place it FIRST.
|
||||
portada = build_chapter(_PORTADA, profile, ctx)
|
||||
# 4) Build the glossary last (reads the terms the body registered), place LAST.
|
||||
glosario = build_chapter(_GLOSARIO, profile, ctx)
|
||||
|
||||
chapters = []
|
||||
if portada is not None and portada.blocks:
|
||||
chapters.append(portada)
|
||||
chapters.extend(body)
|
||||
if glosario is not None and glosario.blocks:
|
||||
chapters.append(glosario)
|
||||
return chapters
|
||||
|
||||
|
||||
def _summarize_document(profile: dict, body: list) -> dict:
|
||||
"""Aggregate a tiny findings summary of the body for the cover. Never raises.
|
||||
|
||||
Returns a dict with dataset shape, quality, column-type counts and the list
|
||||
of chapters actually included — enough for the cover to show a mini-summary
|
||||
of the analysis without re-deriving anything."""
|
||||
try:
|
||||
cols = profile.get("columns") or []
|
||||
n_num = sum(1 for c in cols if isinstance(c, dict)
|
||||
and c.get("inferred_type") == "numeric")
|
||||
n_cat = sum(1 for c in cols if isinstance(c, dict)
|
||||
and isinstance(c.get("categorical"), dict)
|
||||
and c.get("categorical", {}).get("top")
|
||||
and c.get("inferred_type") != "numeric")
|
||||
return {
|
||||
"n_chapters": len(body),
|
||||
"chapter_titles": [getattr(c, "title", "") for c in body],
|
||||
"n_rows": profile.get("n_rows"),
|
||||
"n_cols": profile.get("n_cols"),
|
||||
"quality_score": profile.get("quality_score"),
|
||||
"n_numeric": n_num,
|
||||
"n_categorical": n_cat,
|
||||
"duplicate_pct": profile.get("duplicate_pct"),
|
||||
"null_cell_pct": profile.get("null_cell_pct"),
|
||||
}
|
||||
except Exception: # noqa: BLE001 — the summary is best-effort.
|
||||
return {"n_chapters": len(body) if isinstance(body, list) else 0}
|
||||
|
||||
@@ -128,6 +128,39 @@ class Note:
|
||||
kind: str = field(default="note", init=False)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Group:
|
||||
"""A keep-together unit: its blocks render on the SAME page/slide.
|
||||
|
||||
Renderers measure the whole group first; if it does not fit in the remaining
|
||||
space they move it *whole* to the next page (PDF) or slide (PPTX) before
|
||||
drawing anything — so a heading never gets stranded apart from the figure and
|
||||
text it introduces. If the group is taller than a full page even on its own,
|
||||
it starts on a fresh page and flows (honest degradation, never cut). Use it to
|
||||
bind ``Heading`` + ``Markdown`` + ``Figure`` of one idea together (see the
|
||||
DISTR NUM / AGREGACION chapters).
|
||||
"""
|
||||
|
||||
blocks: list = field(default_factory=list)
|
||||
title: Optional[str] = None
|
||||
kind: str = field(default="group", init=False)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GlossaryEntry:
|
||||
"""One glossary term: a clickable destination at the end of the document.
|
||||
|
||||
Rendered as the term ``label`` (heading) plus its ``definition`` (markdown).
|
||||
The renderers register its page/slide position as the link target so every
|
||||
in-text appearance of the same ``key`` becomes a real clickable jump (PDF link
|
||||
annotation via PyMuPDF; PPTX internal slide jump)."""
|
||||
|
||||
key: str = ""
|
||||
label: str = ""
|
||||
definition: str = ""
|
||||
kind: str = field(default="glossary_entry", init=False)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Chapter:
|
||||
"""An ordered set of blocks with an id, a title and a generation version."""
|
||||
@@ -150,13 +183,17 @@ _BLOCK_BY_KIND = {
|
||||
"image": Image,
|
||||
"caption": Caption,
|
||||
"note": Note,
|
||||
"group": Group,
|
||||
"glossary_entry": GlossaryEntry,
|
||||
}
|
||||
|
||||
|
||||
def as_block(obj: Any):
|
||||
"""Coerce a value into a block dataclass. Unknown values become a Note."""
|
||||
if isinstance(obj, (Heading, Markdown, KVTable, DataTable, Figure, Image,
|
||||
Caption, Note)):
|
||||
Caption, Note, Group, GlossaryEntry)):
|
||||
if isinstance(obj, Group):
|
||||
obj.blocks = as_blocks(obj.blocks)
|
||||
return obj
|
||||
if isinstance(obj, dict):
|
||||
kind = obj.get("kind")
|
||||
@@ -189,6 +226,13 @@ def as_block(obj: Any):
|
||||
return Caption(text=_safe_str(obj.get("text")))
|
||||
if cls is Note:
|
||||
return Note(text=_safe_str(obj.get("text")))
|
||||
if cls is Group:
|
||||
return Group(blocks=as_blocks(obj.get("blocks")),
|
||||
title=obj.get("title"))
|
||||
if cls is GlossaryEntry:
|
||||
return GlossaryEntry(key=_safe_str(obj.get("key")),
|
||||
label=_safe_str(obj.get("label")),
|
||||
definition=_safe_str(obj.get("definition")))
|
||||
except Exception: # noqa: BLE001 — never raise on a malformed block.
|
||||
return Note(text=_safe_str(obj))
|
||||
return Note(text=_safe_str(obj))
|
||||
@@ -246,6 +290,67 @@ def _safe_str(v: Any) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Glossary collector — chapters register the terms they use; the glosario
|
||||
# chapter renders them at the end and the renderers wire the clickable links.
|
||||
# --------------------------------------------------------------------------- #
|
||||
class GlossaryCollector:
|
||||
"""Accumulates glossary terms registered by chapters during document build.
|
||||
|
||||
A single instance is created by :func:`build_document` and passed to every
|
||||
chapter via ``ctx['glossary']``. A chapter calls ``add(key, label,
|
||||
definition)`` to declare a term it explains (e.g. ``"entropia"`` →
|
||||
"Entropía"), and marks each in-text appearance with the inline span
|
||||
``[[term:key]]texto visible[[/term]]`` (see ``text_layout.parse_inline_rich``).
|
||||
The ``glosario`` chapter reads ``terms()`` to emit one :class:`GlossaryEntry`
|
||||
per term; the renderers turn every marked appearance into a real click that
|
||||
jumps to that entry. First registration of a key wins (idempotent); never
|
||||
raises."""
|
||||
|
||||
def __init__(self):
|
||||
self._terms: dict = {}
|
||||
self._order: list = []
|
||||
|
||||
def add(self, key: Any, label: Any = None, definition: Any = "") -> str:
|
||||
"""Register a term and return its normalized key (''. if invalid)."""
|
||||
try:
|
||||
k = _safe_str(key).strip()
|
||||
if not k:
|
||||
return ""
|
||||
if k not in self._terms:
|
||||
self._terms[k] = {
|
||||
"key": k,
|
||||
"label": _safe_str(label).strip() or k,
|
||||
"definition": _safe_str(definition),
|
||||
}
|
||||
self._order.append(k)
|
||||
return k
|
||||
except Exception: # noqa: BLE001 — collecting a term never breaks a build.
|
||||
return ""
|
||||
|
||||
def has(self, key: Any) -> bool:
|
||||
return _safe_str(key).strip() in self._terms
|
||||
|
||||
def get(self, key: Any) -> Optional[dict]:
|
||||
return self._terms.get(_safe_str(key).strip())
|
||||
|
||||
def terms(self, by: str = "label") -> list:
|
||||
"""Return the registered terms as dicts.
|
||||
|
||||
``by='label'`` (default) sorts alphabetically by visible label;
|
||||
``by='order'`` keeps first-appearance order."""
|
||||
if by == "order":
|
||||
return [self._terms[k] for k in self._order]
|
||||
return sorted(self._terms.values(),
|
||||
key=lambda t: _safe_str(t.get("label")).lower())
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._terms)
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return bool(self._terms)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Manifest — per-chapter versions and page/slide counts for tracking.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -0,0 +1,354 @@
|
||||
"""Tests for the AutomaticEDA engine features added in phase 4a.
|
||||
|
||||
Covers, with executable evidence, the six render-engine improvements:
|
||||
|
||||
1. Bold no longer overlaps the following text in the PDF (real width measured).
|
||||
2. Zebra striping on data tables (PDF Rectangle fills + PPTX cell fills).
|
||||
3. Keep-together: a Group moves whole to the next page/slide (heading never gets
|
||||
stranded from its figure).
|
||||
4. Every PPTX figure carries a visible caption/title (fallback to the heading).
|
||||
5. Cover is built last but placed first and reflects an aggregated summary.
|
||||
6. Glossary is the last chapter; the term "entropía" is a real clickable link in
|
||||
the PDF (PyMuPDF GOTO annotation) and in the PPTX (native slide-jump run).
|
||||
|
||||
Self-contained: synthetic profiles, no DuckDB. Heavy renderer checks (fitz/pptx)
|
||||
skip cleanly when the optional engine is missing.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..", "..")) # python/functions
|
||||
if _FUNCTIONS not in sys.path:
|
||||
sys.path.insert(0, _FUNCTIONS)
|
||||
|
||||
import matplotlib # noqa: E402
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.colors as mcolors # noqa: E402
|
||||
import matplotlib.pyplot as plt # noqa: E402
|
||||
from matplotlib.patches import Rectangle # noqa: E402
|
||||
|
||||
from datascience.automatic_eda import model # noqa: E402
|
||||
from datascience.automatic_eda import render_pdf_impl as RP # noqa: E402
|
||||
from datascience.automatic_eda import render_pptx_impl as RX # noqa: E402
|
||||
from datascience.automatic_eda import build_document # noqa: E402
|
||||
from datascience.render_automatic_eda_pdf import render_automatic_eda_pdf # noqa: E402
|
||||
from datascience.render_automatic_eda_pptx import render_automatic_eda_pptx # noqa: E402
|
||||
|
||||
|
||||
class _FakePdf:
|
||||
"""Stand-in for PdfPages so the placers can call _new_page in unit tests."""
|
||||
|
||||
def savefig(self, fig): # noqa: D401
|
||||
pass
|
||||
|
||||
|
||||
def _small_fig():
|
||||
fig = plt.figure(figsize=(4.0, 1.5))
|
||||
ax = fig.add_subplot(111)
|
||||
ax.plot([0, 1, 2], [1, 3, 2])
|
||||
return fig
|
||||
|
||||
|
||||
def _profile_with_cat_and_num():
|
||||
"""A tiny profile that triggers cat_distr (→ entropía term) and num_distr."""
|
||||
return {
|
||||
"table": "ventas", "n_rows": 120, "n_cols": 2, "quality_score": 91,
|
||||
"duplicate_pct": 1.5, "null_cell_pct": 0.8,
|
||||
"columns": [
|
||||
{"name": "region", "inferred_type": "categorical",
|
||||
"categorical": {
|
||||
"top": [{"value": "norte", "count": 50, "pct": 0.42},
|
||||
{"value": "sur", "count": 40, "pct": 0.33},
|
||||
{"value": "este", "count": 30, "pct": 0.25}],
|
||||
"mode": "norte", "n_distinct": 3, "entropy": 1.55,
|
||||
"imbalance": 0.1}},
|
||||
{"name": "importe", "inferred_type": "numeric",
|
||||
"numeric": {"mean": 50.0, "median": 48.0, "std": 10.0,
|
||||
"min": 10, "max": 99, "iqr": 15,
|
||||
"histogram": [{"lo": 0, "hi": 50, "count": 40},
|
||||
{"lo": 50, "hi": 100, "count": 80}]}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 1) Bold does not overlap the following text (PDF).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_pdf_bold_span_does_not_overlap_following_text():
|
||||
fig = plt.figure(figsize=(RP._W, RP._H))
|
||||
st = RP._PdfState(_FakePdf(), "t")
|
||||
st.fig = fig
|
||||
st.page = 1
|
||||
# A wide bold token immediately followed by normal text on the SAME line.
|
||||
rich = [[("PALABRAMUYANCHAENNEGRITA", True, None),
|
||||
(" texto normal justo después", False, None)]]
|
||||
RP._place_rich_lines(st, rich, RP._FS_BODY, RP._INK)
|
||||
|
||||
renderer = fig.canvas.get_renderer()
|
||||
boxes = sorted((t.get_window_extent(renderer) for t in fig.texts),
|
||||
key=lambda b: b.x0)
|
||||
assert len(boxes) == 2, "se esperaban dos spans dibujados"
|
||||
# The bold span ends before the normal span starts (no overlap). 1px slack.
|
||||
assert boxes[0].x1 <= boxes[1].x0 + 1.0, \
|
||||
"la negrita se solapa con el texto siguiente"
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 2) Zebra striping.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _facecolor_eq(artist, hexcolor) -> bool:
|
||||
want = mcolors.to_rgba(hexcolor)
|
||||
got = artist.get_facecolor()
|
||||
return all(abs(a - b) < 0.02 for a, b in zip(got[:3], want[:3]))
|
||||
|
||||
|
||||
def test_pdf_table_has_zebra_striping():
|
||||
fig = plt.figure(figsize=(RP._W, RP._H))
|
||||
st = RP._PdfState(_FakePdf(), "t")
|
||||
st.fig = fig
|
||||
st.page = 1
|
||||
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
|
||||
dt = model.DataTable(header=["A", "B"],
|
||||
rows=[["1", "x"], ["2", "y"], ["3", "z"], ["4", "w"]])
|
||||
RP._place_data_table(st, dt)
|
||||
zebra = [a for a in fig.findobj(Rectangle) if _facecolor_eq(a, RP._ZEBRA)]
|
||||
# 4 data rows → even rows (1-based 2 and 4) shaded = 2 zebra rectangles.
|
||||
assert len(zebra) == 2, f"esperadas 2 filas zebra, hay {len(zebra)}"
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_pptx_table_has_zebra_striping(tmp_path):
|
||||
pptx = pytest.importorskip("pptx")
|
||||
from pptx import Presentation
|
||||
from pptx.dml.color import RGBColor
|
||||
|
||||
doc = [model.Chapter(id="c", title="Tabla", version="1.0.0", blocks=[
|
||||
model.DataTable(header=["A", "B"],
|
||||
rows=[["1", "x"], ["2", "y"], ["3", "z"], ["4", "w"]])])]
|
||||
out = str(tmp_path / "zebra.pptx")
|
||||
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
|
||||
|
||||
prs = Presentation(out)
|
||||
table = None
|
||||
for slide in prs.slides:
|
||||
for sh in slide.shapes:
|
||||
if sh.has_table:
|
||||
table = sh.table
|
||||
break
|
||||
assert table is not None, "no se encontró la tabla en el deck"
|
||||
zebra = RGBColor(0xF6, 0xF8, 0xFA)
|
||||
white = RGBColor(0xFF, 0xFF, 0xFF)
|
||||
# Row 0 = header; data rows follow. Even data rows (table rows 2, 4) shaded.
|
||||
assert table.cell(1, 0).fill.fore_color.rgb == white
|
||||
assert table.cell(2, 0).fill.fore_color.rgb == zebra
|
||||
assert table.cell(4, 0).fill.fore_color.rgb == zebra
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 3) Keep-together (Group): heading + figure never split.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_pdf_group_moves_whole_to_next_page_when_it_does_not_fit():
|
||||
fig = plt.figure(figsize=(RP._W, RP._H))
|
||||
st = RP._PdfState(_FakePdf(), "t")
|
||||
st.fig = fig
|
||||
st.page = 1
|
||||
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
|
||||
grp = model.Group(blocks=[
|
||||
model.Heading(text="Sección con figura", level=2),
|
||||
model.Figure(make=_small_fig, caption="cap"),
|
||||
model.Markdown(text="Descripción breve de la figura."),
|
||||
])
|
||||
# Only ~0.4in left: the group does not fit here but fits on a fresh page.
|
||||
st.y = RP._CONTENT_BOTTOM - 0.4
|
||||
page_before = st.page
|
||||
RP._place_group(st, grp)
|
||||
# Exactly one page break: the whole group (heading+figure+text) stays
|
||||
# together on the new page — no second break inside it.
|
||||
assert st.page == page_before + 1
|
||||
plt.close(st.fig)
|
||||
|
||||
|
||||
def test_pdf_group_does_not_break_when_it_fits():
|
||||
fig = plt.figure(figsize=(RP._W, RP._H))
|
||||
st = RP._PdfState(_FakePdf(), "t")
|
||||
st.fig = fig
|
||||
st.page = 1
|
||||
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
|
||||
grp = model.Group(blocks=[
|
||||
model.Heading(text="Cabe entera", level=2),
|
||||
model.Figure(make=_small_fig, caption="cap"),
|
||||
])
|
||||
st.y = RP._CONTENT_TOP # empty page → fits, must not break.
|
||||
page_before = st.page
|
||||
RP._place_group(st, grp)
|
||||
assert st.page == page_before
|
||||
plt.close(st.fig)
|
||||
|
||||
|
||||
def test_pptx_group_moves_whole_to_next_slide(tmp_path):
|
||||
pytest.importorskip("pptx")
|
||||
from pptx import Presentation
|
||||
from pptx.util import Inches
|
||||
|
||||
prs = Presentation()
|
||||
prs.slide_width = Inches(RX._W)
|
||||
prs.slide_height = Inches(RX._H)
|
||||
st = RX._PptxState(prs, "t")
|
||||
st.chapter = model.Chapter(id="c", title="C", version="1.0.0")
|
||||
RX._new_slide(st, cont=False)
|
||||
grp = model.Group(blocks=[
|
||||
model.Heading(text="Sección con figura", level=2),
|
||||
model.Figure(make=_small_fig, caption="cap"),
|
||||
model.Markdown(text="Descripción breve."),
|
||||
])
|
||||
st.y = RX._CONTENT_BOTTOM - 0.4 # does not fit here.
|
||||
slide_before = st.slide_no
|
||||
RX._place_group(st, grp)
|
||||
assert st.slide_no == slide_before + 1 # one jump; group kept together.
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 4) Every PPTX figure carries a visible caption/title.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_pptx_figure_without_caption_gets_heading_title(tmp_path):
|
||||
pytest.importorskip("pptx")
|
||||
from pptx import Presentation
|
||||
from pptx.enum.shapes import MSO_SHAPE_TYPE
|
||||
|
||||
doc = [model.Chapter(id="c", title="Cap", version="1.0.0", blocks=[
|
||||
model.Heading(text="Mi sección gráfica", level=2),
|
||||
model.Figure(make=_small_fig), # NO caption provided.
|
||||
])]
|
||||
out = str(tmp_path / "cap.pptx")
|
||||
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
|
||||
|
||||
prs = Presentation(out)
|
||||
for slide in prs.slides:
|
||||
has_pic = any(sh.shape_type == MSO_SHAPE_TYPE.PICTURE
|
||||
for sh in slide.shapes)
|
||||
if not has_pic:
|
||||
continue
|
||||
italic = [r.text for sh in slide.shapes if sh.has_text_frame
|
||||
for p in sh.text_frame.paragraphs for r in p.runs
|
||||
if r.font.italic and r.text.strip()]
|
||||
assert italic, "la figura no lleva caption visible en su slide"
|
||||
assert any("Mi sección gráfica" in t for t in italic), \
|
||||
"el caption no cayó al título de la sección"
|
||||
return
|
||||
pytest.fail("no se encontró ningún slide con imagen")
|
||||
|
||||
|
||||
def test_pptx_no_figure_slide_is_ever_untitled(tmp_path):
|
||||
"""Invariant: across many figures (incl. tall ones), NO slide with an image
|
||||
lacks a visible caption — the caption never spills to the next slide."""
|
||||
pytest.importorskip("pptx")
|
||||
from pptx import Presentation
|
||||
from pptx.enum.shapes import MSO_SHAPE_TYPE
|
||||
|
||||
def _tall_fig():
|
||||
fig = plt.figure(figsize=(5.0, 4.6)) # nearly square → fills the slide.
|
||||
fig.add_subplot(111).bar([1, 2, 3], [4, 5, 6])
|
||||
return fig
|
||||
|
||||
blocks = []
|
||||
for i in range(6):
|
||||
blocks.append(model.Heading(text=f"Gráfico {i}", level=2))
|
||||
blocks.append(model.Figure(
|
||||
make=_tall_fig,
|
||||
caption=("Una descripción de la figura deliberadamente larga para "
|
||||
"que el caption ocupe más de una línea al envolverse en el "
|
||||
f"ancho del slide — figura número {i} del bloque.")))
|
||||
doc = [model.Chapter(id="c", title="Muchas figuras", version="1.0.0",
|
||||
blocks=blocks)]
|
||||
out = str(tmp_path / "many.pptx")
|
||||
assert render_automatic_eda_pptx(doc, out, {"write_manifest": False})["path"]
|
||||
|
||||
prs = Presentation(out)
|
||||
missing = []
|
||||
pics = 0
|
||||
for i, slide in enumerate(prs.slides):
|
||||
if not any(sh.shape_type == MSO_SHAPE_TYPE.PICTURE
|
||||
for sh in slide.shapes):
|
||||
continue
|
||||
pics += 1
|
||||
italic = [r.text for sh in slide.shapes if sh.has_text_frame
|
||||
for p in sh.text_frame.paragraphs for r in p.runs
|
||||
if r.font.italic and r.text.strip()]
|
||||
if not italic:
|
||||
missing.append(i)
|
||||
assert pics >= 6, f"esperadas >=6 figuras, hay {pics}"
|
||||
assert not missing, f"slides con imagen sin caption: {missing}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 5) Cover built last, placed first, with an aggregated summary.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_cover_first_glossary_last_with_summary():
|
||||
chs = build_document(_profile_with_cat_and_num(), ctx={"dataset_name": "v"})
|
||||
ids = [c.id for c in chs]
|
||||
assert ids[0] == "portada", f"la portada no es la primera: {ids}"
|
||||
assert ids[-1] == "glosario", f"el glosario no es el último: {ids}"
|
||||
cover = chs[0]
|
||||
headings = [b.text for b in cover.blocks if b.kind == "heading"]
|
||||
assert any("Resumen" in h for h in headings), \
|
||||
"la portada no incluye el resumen agregado"
|
||||
# The summary reflects the body chapters (e.g. the numeric/categorical ones).
|
||||
cover_text = " ".join(
|
||||
b.text for b in cover.blocks if getattr(b, "kind", "") == "markdown")
|
||||
assert "Distribuciones" in cover_text, \
|
||||
"el resumen de portada no menciona los capítulos del cuerpo"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 6) Glossary clickable in PDF (PyMuPDF GOTO) and PPTX (native slide jump).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_pdf_glossary_term_is_clickable(tmp_path):
|
||||
fitz = pytest.importorskip("fitz")
|
||||
out = str(tmp_path / "glos.pdf")
|
||||
res = render_automatic_eda_pdf(_profile_with_cat_and_num(), out,
|
||||
{"ctx": {"dataset_name": "v"},
|
||||
"write_manifest": False})
|
||||
assert res["path"] == out and os.path.exists(out)
|
||||
|
||||
doc = fitz.open(out)
|
||||
goto = [(pno, l) for pno in range(doc.page_count)
|
||||
for l in doc[pno].get_links() if l.get("kind") == fitz.LINK_GOTO]
|
||||
doc.close()
|
||||
assert goto, "no hay ningún enlace interno (entropía → glosario) en el PDF"
|
||||
# Destination must be a real page in the document (the glossary page).
|
||||
assert all(0 <= l.get("page", -1) for _p, l in goto)
|
||||
|
||||
|
||||
def test_pptx_glossary_term_is_clickable(tmp_path):
|
||||
pytest.importorskip("pptx")
|
||||
from pptx import Presentation
|
||||
from pptx.oxml.ns import qn
|
||||
|
||||
out = str(tmp_path / "glos.pptx")
|
||||
res = render_automatic_eda_pptx(_profile_with_cat_and_num(), out,
|
||||
{"ctx": {"dataset_name": "v"},
|
||||
"write_manifest": False})
|
||||
assert res["path"] == out and os.path.exists(out)
|
||||
|
||||
prs = Presentation(out)
|
||||
found = False
|
||||
for slide in prs.slides:
|
||||
for sh in slide.shapes:
|
||||
if not sh.has_text_frame:
|
||||
continue
|
||||
for p in sh.text_frame.paragraphs:
|
||||
for r in p.runs:
|
||||
rpr = r._r.find(qn("a:rPr"))
|
||||
if rpr is None:
|
||||
continue
|
||||
hl = rpr.find(qn("a:hlinkClick"))
|
||||
if hl is not None and \
|
||||
hl.get("action") == "ppaction://hlinksldjump":
|
||||
found = True
|
||||
assert found, "ningún término tiene hyperlink de salto a slide en el PPTX"
|
||||
@@ -60,6 +60,8 @@ _FS_BODY, _FS_CELL, _FS_NOTE = 10.5, 9.0, 9.0
|
||||
_GAP = 0.12 # vertical gap after a block, inches.
|
||||
_CELL_PAD = 0.06 # horizontal padding inside a table cell, inches.
|
||||
_ROW_VPAD = 0.05 # vertical padding inside a table row, inches.
|
||||
_ZEBRA = "#f6f8fa" # very light grey for zebra-striped (even) table rows.
|
||||
_LINK = "#2a6f97" # accent colour for clickable glossary terms.
|
||||
|
||||
|
||||
class _PdfState:
|
||||
@@ -73,6 +75,11 @@ class _PdfState:
|
||||
self.page = 0 # global page counter.
|
||||
self.chapter = None # current Chapter (for the footer).
|
||||
self.chapter_pages = 0 # pages produced for the current chapter.
|
||||
self.last_heading = "" # text of the most recent heading.
|
||||
# Glossary wiring (mejora 6). Pages are 0-based; rects/points are in PDF
|
||||
# points (1/72") with a top-left origin — same convention as PyMuPDF.
|
||||
self.term_sources = [] # [{key, page, rect:[x0,y0,x1,y1]}]
|
||||
self.term_dests = {} # key -> {page, point:[x,y]}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -121,6 +128,35 @@ def _draw_footer(st: _PdfState) -> None:
|
||||
transform=st.fig.transFigure, color=_RULE, lw=0.6))
|
||||
|
||||
|
||||
def _text_width_in(st: _PdfState, s: str, fs: float, bold: bool) -> float:
|
||||
"""Real rendered width (inches) of ``s`` at ``fs`` with the given weight.
|
||||
|
||||
Measured with the Agg renderer's own font metrics (the same TrueType the PDF
|
||||
backend embeds), so a **bold** span advances the cursor by its ACTUAL width —
|
||||
fixing the bug where bold text overlapped the following normal text because
|
||||
the cursor advanced by the normal-weight average-glyph estimate. Falls back to
|
||||
the deterministic character grid if the renderer is unavailable, so it never
|
||||
raises.
|
||||
"""
|
||||
if not s:
|
||||
return 0.0
|
||||
try:
|
||||
from matplotlib.font_manager import FontProperties
|
||||
renderer = st.fig.canvas.get_renderer()
|
||||
prop = FontProperties(family="sans-serif", size=fs,
|
||||
weight="bold" if bold else "normal")
|
||||
w_px, _h, _d = renderer.get_text_width_height_descent(s, prop, False)
|
||||
return w_px / float(st.fig.dpi)
|
||||
except Exception: # noqa: BLE001 — fall back to the conservative grid metric.
|
||||
return tl.avg_char_width_in(fs) * len(s)
|
||||
|
||||
|
||||
def _pt_rect(x0_in: float, y_top_in: float, x1_in: float,
|
||||
y_bottom_in: float) -> list:
|
||||
"""An inches box (top-left origin) → a PDF-points rect for PyMuPDF links."""
|
||||
return [x0_in * 72.0, y_top_in * 72.0, x1_in * 72.0, y_bottom_in * 72.0]
|
||||
|
||||
|
||||
def _remaining(st: _PdfState) -> float:
|
||||
return _CONTENT_BOTTOM - st.y
|
||||
|
||||
@@ -138,6 +174,7 @@ def _place_heading(st: _PdfState, block) -> None:
|
||||
level = max(1, min(3, int(getattr(block, "level", 1) or 1)))
|
||||
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
|
||||
text = tl.strip_inline_md(getattr(block, "text", ""))
|
||||
st.last_heading = text or st.last_heading
|
||||
max_chars = tl.chars_per_line(_USABLE_W, fs)
|
||||
lines = tl.wrap(text, max_chars)
|
||||
lh = tl.line_height_in(fs, leading=1.2)
|
||||
@@ -169,6 +206,49 @@ def _place_text_lines(st: _PdfState, lines: list, fs: float, color: str,
|
||||
st.y += lh
|
||||
|
||||
|
||||
def _place_rich_lines(st: _PdfState, rich_lines: list, fs: float, color: str,
|
||||
indent: float = 0.0, prefixes=None) -> None:
|
||||
"""Draw pre-wrapped lines of styled segments (bold + clickable term spans).
|
||||
|
||||
Each line is a list of ``(text, is_bold)`` or ``(text, is_bold, term_key)``
|
||||
segments. Segments are placed left-to-right, advancing x by the segment's
|
||||
REAL rendered width (measured with the renderer's font metrics for the actual
|
||||
weight) — this is what stops a bold span from overlapping the following text:
|
||||
the cursor no longer advances by the normal-weight estimate. A segment with a
|
||||
``term_key`` is drawn in the accent colour and its rectangle is recorded in
|
||||
``st.term_sources`` so it becomes a clickable jump to the glossary entry.
|
||||
``prefixes`` is an optional ``(first_line, other_lines)`` pair (e.g. a
|
||||
bullet) drawn before the segments.
|
||||
"""
|
||||
lh = tl.line_height_in(fs)
|
||||
for idx, segs in enumerate(rich_lines):
|
||||
_ensure_space(st, lh)
|
||||
x = _ML + indent
|
||||
if prefixes is not None:
|
||||
prefix = prefixes[0] if idx == 0 else prefixes[1]
|
||||
if prefix:
|
||||
st.fig.text(_xf(x), _yf(st.y), prefix, fontsize=fs, color=color,
|
||||
ha="left", va="top")
|
||||
x += _text_width_in(st, prefix, fs, False)
|
||||
for seg in segs:
|
||||
if len(seg) == 3:
|
||||
seg_text, is_bold, term = seg
|
||||
else:
|
||||
seg_text, is_bold, term = seg[0], seg[1], None
|
||||
if seg_text == "":
|
||||
continue
|
||||
w = _text_width_in(st, seg_text, fs, bool(is_bold))
|
||||
st.fig.text(_xf(x), _yf(st.y), seg_text, fontsize=fs,
|
||||
color=(_LINK if term else color), ha="left", va="top",
|
||||
fontweight="bold" if is_bold else "normal")
|
||||
if term:
|
||||
st.term_sources.append({
|
||||
"key": term, "page": st.page - 1,
|
||||
"rect": _pt_rect(x, st.y, x + w, st.y + lh)})
|
||||
x += w
|
||||
st.y += lh
|
||||
|
||||
|
||||
def _place_markdown(st: _PdfState, block) -> None:
|
||||
raw = getattr(block, "text", "") or ""
|
||||
md_lines = str(raw).split("\n")
|
||||
@@ -208,29 +288,26 @@ def _place_markdown(st: _PdfState, block) -> None:
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("- ") or stripped.startswith("* "):
|
||||
content = tl.strip_inline_md(stripped[2:])
|
||||
content = stripped[2:] # keep inline markers for bold rendering.
|
||||
bullet_chars = tl.chars_per_line(_USABLE_W - 0.22, _FS_BODY)
|
||||
wrapped = tl.wrap(content, bullet_chars)
|
||||
first = True
|
||||
for w in wrapped:
|
||||
prefix = "• " if first else " "
|
||||
_place_text_lines(st, [prefix + w], _FS_BODY, _INK,
|
||||
indent=0.0)
|
||||
first = False
|
||||
rich = tl.wrap_rich_terms(content, bullet_chars)
|
||||
_place_rich_lines(st, rich, _FS_BODY, _INK,
|
||||
prefixes=("• ", " "))
|
||||
i += 1
|
||||
continue
|
||||
# Plain paragraph (gather following plain lines into one paragraph).
|
||||
para = [tl.strip_inline_md(stripped)]
|
||||
para = [stripped] # keep inline markers; wrap_rich renders **bold**.
|
||||
j = i + 1
|
||||
while j < n:
|
||||
nxt = md_lines[j].strip()
|
||||
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
|
||||
break
|
||||
para.append(tl.strip_inline_md(nxt))
|
||||
para.append(nxt)
|
||||
j += 1
|
||||
text = " ".join(para)
|
||||
max_chars = tl.chars_per_line(_USABLE_W, _FS_BODY)
|
||||
_place_text_lines(st, tl.wrap(text, max_chars), _FS_BODY, _INK)
|
||||
_place_rich_lines(st, tl.wrap_rich_terms(text, max_chars), _FS_BODY,
|
||||
_INK)
|
||||
i = j
|
||||
st.y += _GAP
|
||||
|
||||
@@ -297,15 +374,18 @@ def _wrap_row(cells: list, widths: list, fs: float) -> list:
|
||||
|
||||
|
||||
def _draw_table_row(st: _PdfState, cells_lines: list, widths: list, fs: float,
|
||||
y0: float, header: bool) -> float:
|
||||
y0: float, header: bool, zebra: bool = False) -> float:
|
||||
lh = tl.line_height_in(fs)
|
||||
nlines = max((len(c) for c in cells_lines), default=1)
|
||||
row_h = lh * nlines + _ROW_VPAD * 2
|
||||
if header:
|
||||
# Background: header band, or a faint zebra fill for even data rows. Drawn
|
||||
# below the text/rule (zorder 0) so striping never hides cell content.
|
||||
bg = _HEAD_BG if header else (_ZEBRA if zebra else None)
|
||||
if bg is not None:
|
||||
st.fig.add_artist(Rectangle(
|
||||
(_xf(_ML), _yf(y0 + row_h)), _xf(_ML + _USABLE_W) - _xf(_ML),
|
||||
_yf(y0) - _yf(y0 + row_h), transform=st.fig.transFigure,
|
||||
color=_HEAD_BG, lw=0, zorder=0))
|
||||
color=bg, lw=0, zorder=0))
|
||||
x = _ML
|
||||
for c, lines in enumerate(cells_lines):
|
||||
for k, ln in enumerate(lines):
|
||||
@@ -350,14 +430,18 @@ def _place_data_table(st: _PdfState, block) -> None:
|
||||
+ _ROW_VPAD * 2
|
||||
_ensure_space(st, header_h() + max(first_row_h, lh))
|
||||
draw_header()
|
||||
for r in rows:
|
||||
# ``data_idx`` is the LOGICAL row index (not reset across page breaks) so the
|
||||
# zebra pattern stays coherent when a long table splits and repeats the
|
||||
# header: even rows (1-based) are shaded → 0-based odd indices.
|
||||
for data_idx, r in enumerate(rows):
|
||||
cells_lines = _wrap_row(r, widths, fs)
|
||||
row_h = lh * max((len(c) for c in cells_lines), default=1) \
|
||||
+ _ROW_VPAD * 2
|
||||
if _remaining(st) < row_h:
|
||||
_new_page(st)
|
||||
draw_header() # repeat header on the continuation page.
|
||||
st.y += _draw_table_row(st, cells_lines, widths, fs, st.y, header=False)
|
||||
st.y += _draw_table_row(st, cells_lines, widths, fs, st.y,
|
||||
header=False, zebra=(data_idx % 2 == 1))
|
||||
note = getattr(block, "note", None)
|
||||
if note:
|
||||
_place_text_lines(st, tl.wrap(model._safe_str(note),
|
||||
@@ -386,53 +470,98 @@ def _png_from_figure(fig) -> bytes:
|
||||
return buf.read()
|
||||
|
||||
|
||||
def _place_image_array(st: _PdfState, arr, caption) -> None:
|
||||
def _figure_png_cached(block):
|
||||
"""Rasterize a Figure to PNG bytes ONCE and cache (bytes, aspect).
|
||||
|
||||
Measuring (keep-together) and drawing must agree on the REAL aspect ratio:
|
||||
``bbox_inches='tight'`` changes it vs ``figsize``, so we rasterize once and
|
||||
reuse the bytes for both. Cached on the block; never raises."""
|
||||
cached = getattr(block, "_aeda_png", None)
|
||||
if cached is not None:
|
||||
return cached
|
||||
fig, owned = _resolve_figure(block)
|
||||
data = None
|
||||
if fig is not None:
|
||||
try:
|
||||
data = _png_from_figure(fig)
|
||||
finally:
|
||||
if owned:
|
||||
try:
|
||||
plt.close(fig)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
aspect = 0.66
|
||||
if data is not None:
|
||||
try:
|
||||
arr = mpimg.imread(io.BytesIO(data))
|
||||
aspect = (arr.shape[0] / arr.shape[1]) if arr.shape[1] else 0.66
|
||||
except Exception: # noqa: BLE001
|
||||
aspect = 0.66
|
||||
try:
|
||||
block._aeda_png = (data, aspect)
|
||||
return block._aeda_png
|
||||
except Exception: # noqa: BLE001 — block may reject attributes; degrade.
|
||||
return (data, aspect)
|
||||
|
||||
|
||||
def _image_aspect(block) -> float:
|
||||
"""Real aspect (h/w) of an Image block by path, for measurement."""
|
||||
path = getattr(block, "path", "")
|
||||
if path and os.path.exists(path):
|
||||
try:
|
||||
arr = mpimg.imread(path)
|
||||
return (arr.shape[0] / arr.shape[1]) if arr.shape[1] else 0.66
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return 0.66
|
||||
|
||||
|
||||
def _place_image_array(st: _PdfState, arr, caption, max_h_in=None) -> None:
|
||||
h_px, w_px = arr.shape[0], arr.shape[1]
|
||||
aspect = (h_px / w_px) if w_px else 1.0
|
||||
# Reserve the caption's REAL (possibly multi-line) height FIRST, then scale
|
||||
# the image to (max_h - cap_reserve) so figure + caption always fit the same
|
||||
# page. cap_reserve adds a cushion so the caption never spills to next page.
|
||||
cap_lines = (tl.wrap(model._safe_str(caption),
|
||||
tl.chars_per_line(_USABLE_W, _FS_NOTE))
|
||||
if caption else [])
|
||||
cap_real = tl.line_height_in(_FS_NOTE) * len(cap_lines) if caption else 0.0
|
||||
cap_reserve = (cap_real + 0.04 + 0.08) if caption else 0.0
|
||||
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
# height_in hint (model.Figure/Image): cap the height so a figure in a
|
||||
# keep-together Group shrinks to leave room for its heading and text.
|
||||
if isinstance(max_h_in, (int, float)) and max_h_in > 0:
|
||||
max_h = min(max_h, float(max_h_in))
|
||||
max_img_h = max(max_h - cap_reserve, 0.6)
|
||||
target_w = _USABLE_W
|
||||
target_h = target_w * aspect
|
||||
if target_h > max_h:
|
||||
target_h = max_h
|
||||
if target_h > max_img_h:
|
||||
target_h = max_img_h
|
||||
target_w = target_h / aspect if aspect else _USABLE_W
|
||||
cap_h = tl.line_height_in(_FS_NOTE) + 0.04 if caption else 0.0
|
||||
# Move whole image to next page if it does not fit in remaining space.
|
||||
if _remaining(st) < target_h + cap_h:
|
||||
if (max_h) >= target_h + cap_h:
|
||||
_new_page(st)
|
||||
else:
|
||||
# Taller than a full page even at min — already clamped to max_h.
|
||||
_new_page(st)
|
||||
if _remaining(st) < target_h + cap_reserve:
|
||||
_new_page(st)
|
||||
left_frac = _xf(_ML + (_USABLE_W - target_w) / 2.0)
|
||||
bottom_frac = _yf(st.y + target_h)
|
||||
ax = st.fig.add_axes([left_frac, bottom_frac, target_w / _W, target_h / _H])
|
||||
ax.imshow(arr)
|
||||
ax.axis("off")
|
||||
st.y += target_h + 0.04
|
||||
if caption:
|
||||
_place_text_lines(st, tl.wrap(model._safe_str(caption),
|
||||
tl.chars_per_line(_USABLE_W, _FS_NOTE)),
|
||||
_FS_NOTE, _MUTED, style="italic")
|
||||
if cap_lines:
|
||||
_place_text_lines(st, cap_lines, _FS_NOTE, _MUTED, style="italic")
|
||||
st.y += _GAP
|
||||
|
||||
|
||||
def _place_figure(st: _PdfState, block) -> None:
|
||||
fig, owned = _resolve_figure(block)
|
||||
if fig is None:
|
||||
png, _aspect = _figure_png_cached(block)
|
||||
if png is None:
|
||||
_place_text_lines(st, ["(figura no disponible)"], _FS_NOTE, _MUTED,
|
||||
style="italic")
|
||||
st.y += _GAP
|
||||
return
|
||||
try:
|
||||
png = _png_from_figure(fig)
|
||||
finally:
|
||||
if owned:
|
||||
try:
|
||||
plt.close(fig)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
arr = mpimg.imread(io.BytesIO(png))
|
||||
_place_image_array(st, arr, getattr(block, "caption", None))
|
||||
_place_image_array(st, arr, getattr(block, "caption", None),
|
||||
max_h_in=getattr(block, "height_in", None))
|
||||
|
||||
|
||||
def _place_image(st: _PdfState, block) -> None:
|
||||
@@ -443,7 +572,8 @@ def _place_image(st: _PdfState, block) -> None:
|
||||
st.y += _GAP
|
||||
return
|
||||
arr = mpimg.imread(path)
|
||||
_place_image_array(st, arr, getattr(block, "caption", None))
|
||||
_place_image_array(st, arr, getattr(block, "caption", None),
|
||||
max_h_in=getattr(block, "height_in", None))
|
||||
|
||||
|
||||
def _place_caption(st: _PdfState, block) -> None:
|
||||
@@ -460,6 +590,189 @@ def _place_note(st: _PdfState, block) -> None:
|
||||
st.y += _GAP
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Block measurement (mejora 3 — keep-together). These estimate a block's height
|
||||
# WITHOUT drawing it, so a Group can decide to move whole to the next page before
|
||||
# anything is drawn. Over-estimating is safe: it only triggers an earlier page
|
||||
# break, never a content cut (the placers keep their own no-cut pagination).
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _measure_heading_text(text: str, level: int) -> float:
|
||||
level = max(1, min(3, int(level or 1)))
|
||||
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
|
||||
lines = tl.wrap(tl.strip_inline_md(text), tl.chars_per_line(_USABLE_W, fs))
|
||||
h = tl.line_height_in(fs, leading=1.2) * len(lines) + 0.06
|
||||
if level == 1:
|
||||
h += 0.10
|
||||
return h + _GAP
|
||||
|
||||
|
||||
def _measure_markdown(block) -> float:
|
||||
raw = str(getattr(block, "text", "") or "")
|
||||
md_lines = raw.split("\n")
|
||||
h = 0.0
|
||||
i, n = 0, len(md_lines)
|
||||
while i < n:
|
||||
stripped = md_lines[i].strip()
|
||||
if stripped.startswith("|") and stripped.endswith("|"):
|
||||
j = i
|
||||
while j < n and md_lines[j].strip().startswith("|") \
|
||||
and md_lines[j].strip().endswith("|"):
|
||||
j += 1
|
||||
h += (tl.line_height_in(_FS_CELL) + _ROW_VPAD * 2) * (j - i) + _GAP
|
||||
i = j
|
||||
continue
|
||||
if stripped == "":
|
||||
h += tl.line_height_in(_FS_BODY) * 0.5
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("### "):
|
||||
h += _measure_heading_text(stripped[4:], 3)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("## "):
|
||||
h += _measure_heading_text(stripped[3:], 2)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("# "):
|
||||
h += _measure_heading_text(stripped[2:], 1)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("- ") or stripped.startswith("* "):
|
||||
lines = tl.wrap_rich_terms(
|
||||
stripped[2:], tl.chars_per_line(_USABLE_W - 0.22, _FS_BODY))
|
||||
h += tl.line_height_in(_FS_BODY) * len(lines)
|
||||
i += 1
|
||||
continue
|
||||
para = [stripped]
|
||||
j = i + 1
|
||||
while j < n:
|
||||
nxt = md_lines[j].strip()
|
||||
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
|
||||
break
|
||||
para.append(nxt)
|
||||
j += 1
|
||||
lines = tl.wrap_rich_terms(" ".join(para),
|
||||
tl.chars_per_line(_USABLE_W, _FS_BODY))
|
||||
h += tl.line_height_in(_FS_BODY) * len(lines)
|
||||
i = j
|
||||
return h + _GAP
|
||||
|
||||
|
||||
def _measure_figure_like(block) -> float:
|
||||
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
hint = getattr(block, "height_in", None)
|
||||
if isinstance(hint, (int, float)) and hint > 0:
|
||||
target_h = min(float(hint), max_h)
|
||||
else:
|
||||
# Real rasterized aspect (cached) so measuring matches drawing.
|
||||
if getattr(block, "kind", "") == "image":
|
||||
aspect = _image_aspect(block)
|
||||
else:
|
||||
_data, aspect = _figure_png_cached(block)
|
||||
target_h = min(_USABLE_W * aspect, max_h)
|
||||
cap = getattr(block, "caption", None)
|
||||
cap_h = tl.line_height_in(_FS_NOTE) + 0.04 if cap else 0.0
|
||||
return target_h + 0.04 + cap_h + _GAP
|
||||
|
||||
|
||||
def _measure_block(st: _PdfState, block) -> float:
|
||||
kind = getattr(block, "kind", "")
|
||||
try:
|
||||
if kind == "heading":
|
||||
return _measure_heading_text(getattr(block, "text", ""),
|
||||
getattr(block, "level", 1))
|
||||
if kind == "markdown":
|
||||
return _measure_markdown(block)
|
||||
if kind in ("figure", "image"):
|
||||
return _measure_figure_like(block)
|
||||
if kind in ("caption", "note"):
|
||||
lines = tl.wrap(getattr(block, "text", ""),
|
||||
tl.chars_per_line(_USABLE_W, _FS_NOTE))
|
||||
return tl.line_height_in(_FS_NOTE) * len(lines) + _GAP
|
||||
if kind == "kv_table":
|
||||
rows = getattr(block, "rows", []) or []
|
||||
return (tl.line_height_in(_FS_BODY) + _ROW_VPAD) * (len(rows) + 1) \
|
||||
+ _GAP
|
||||
if kind == "data_table":
|
||||
rows = getattr(block, "rows", []) or []
|
||||
return (tl.line_height_in(_FS_CELL) + _ROW_VPAD * 2) \
|
||||
* (len(rows) + 1) + _GAP
|
||||
if kind == "group":
|
||||
return sum(_measure_block(st, b)
|
||||
for b in (getattr(block, "blocks", []) or []))
|
||||
except Exception: # noqa: BLE001 — a measurement never aborts rendering.
|
||||
pass
|
||||
return tl.line_height_in(_FS_BODY)
|
||||
|
||||
|
||||
def _shrink_group_figures(st: _PdfState, blocks: list, avail_full: float) -> None:
|
||||
"""Cap each figure's height (via height_in) so the whole group fits a page.
|
||||
|
||||
The figure shrinks just enough to leave room for its heading, text and
|
||||
caption — keep-together puts the chart on the SAME page as its title and
|
||||
description instead of pushing it to the next page."""
|
||||
fig_blocks = [b for b in blocks
|
||||
if getattr(b, "kind", "") in ("figure", "image")]
|
||||
if not fig_blocks:
|
||||
return
|
||||
nonfig_h = sum(_measure_block(st, b) for b in blocks
|
||||
if getattr(b, "kind", "") not in ("figure", "image"))
|
||||
fig_overhead = tl.line_height_in(_FS_NOTE) + 0.04 + 0.04 + _GAP
|
||||
budget = avail_full - nonfig_h - 0.08 * len(fig_blocks)
|
||||
if budget <= 0.8:
|
||||
return
|
||||
per = budget / len(fig_blocks) - fig_overhead
|
||||
if per <= 0.6:
|
||||
return
|
||||
for fb in fig_blocks:
|
||||
cur = getattr(fb, "height_in", None)
|
||||
fb.height_in = (min(float(cur), per)
|
||||
if isinstance(cur, (int, float)) and cur > 0 else per)
|
||||
|
||||
|
||||
def _place_group(st: _PdfState, block) -> None:
|
||||
"""Render a keep-together Group: move it whole to the next page if needed."""
|
||||
blocks = getattr(block, "blocks", []) or []
|
||||
if not blocks:
|
||||
return
|
||||
avail_full = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
_shrink_group_figures(st, blocks, avail_full)
|
||||
total = sum(_measure_block(st, b) for b in blocks)
|
||||
if total <= avail_full:
|
||||
# Fits on one page: keep it together by moving whole when it won't fit.
|
||||
if total > _remaining(st):
|
||||
_new_page(st)
|
||||
elif st.y > _CONTENT_TOP + 1e-6:
|
||||
# Taller than a full page: at least start it on a fresh page, then flow.
|
||||
_new_page(st)
|
||||
for b in blocks:
|
||||
placer = _PLACERS.get(getattr(b, "kind", ""), _place_note)
|
||||
try:
|
||||
placer(st, b)
|
||||
except Exception: # noqa: BLE001 — a bad block never aborts the group.
|
||||
pass
|
||||
|
||||
|
||||
def _place_glossary_entry(st: _PdfState, block) -> None:
|
||||
"""Render one glossary term and register it as a clickable link target."""
|
||||
key = getattr(block, "key", "")
|
||||
label = getattr(block, "label", "") or key
|
||||
definition = getattr(block, "definition", "")
|
||||
# Reserve the term + its first definition line together, then anchor the
|
||||
# destination at the resolved page/position before drawing.
|
||||
_ensure_space(st, tl.line_height_in(_FS_H3, leading=1.2)
|
||||
+ tl.line_height_in(_FS_BODY) * 2)
|
||||
if key:
|
||||
st.term_dests[key] = {"page": st.page - 1,
|
||||
"point": [_ML * 72.0, st.y * 72.0]}
|
||||
_place_heading(st, model.Heading(text=str(label), level=3))
|
||||
if definition:
|
||||
_place_text_lines(st, tl.wrap(model._safe_str(definition),
|
||||
tl.chars_per_line(_USABLE_W, _FS_BODY)),
|
||||
_FS_BODY, _INK)
|
||||
st.y += _GAP * 0.5
|
||||
|
||||
|
||||
_PLACERS = {
|
||||
"heading": _place_heading,
|
||||
"markdown": _place_markdown,
|
||||
@@ -469,6 +782,8 @@ _PLACERS = {
|
||||
"image": _place_image,
|
||||
"caption": _place_caption,
|
||||
"note": _place_note,
|
||||
"group": _place_group,
|
||||
"glossary_entry": _place_glossary_entry,
|
||||
}
|
||||
|
||||
|
||||
@@ -525,8 +840,42 @@ def render_pdf(chapters: list, out_path: str, meta: dict = None) -> dict:
|
||||
return {"path": None, "n_pages": 0, "chapters": [],
|
||||
"note": f"fallo al escribir el PDF: {e}"}
|
||||
|
||||
# Mejora 6 — wire clickable glossary links now the PDF is closed on disk.
|
||||
# PdfPages cannot emit internal hyperlinks, so we post-process with PyMuPDF
|
||||
# (delegated registry function). Degrades silently if it is unavailable.
|
||||
n_links = _wire_glossary_links(st, out_path, notes)
|
||||
|
||||
note = f"{n_pages} páginas"
|
||||
if n_links:
|
||||
note += f" · {n_links} enlaces de glosario"
|
||||
if notes:
|
||||
note += " · " + "; ".join(notes)
|
||||
return {"path": out_path, "n_pages": n_pages, "chapters": chapters_meta,
|
||||
"note": note}
|
||||
|
||||
|
||||
def _wire_glossary_links(st: _PdfState, out_path: str, notes: list) -> int:
|
||||
"""Build {source rect → glossary dest} links and apply them via PyMuPDF.
|
||||
|
||||
Returns the number of links applied (0 if there is nothing to wire or the
|
||||
post-processor is unavailable). Never raises."""
|
||||
try:
|
||||
links = []
|
||||
for src in st.term_sources:
|
||||
dest = st.term_dests.get(src.get("key"))
|
||||
if not dest:
|
||||
continue
|
||||
links.append({
|
||||
"src_page": src["page"], "src_rect": src["rect"],
|
||||
"dst_page": dest["page"], "dst_point": dest["point"]})
|
||||
if not links:
|
||||
return 0
|
||||
from datascience.add_pdf_internal_links import add_pdf_internal_links
|
||||
res = add_pdf_internal_links(out_path, links)
|
||||
if isinstance(res, dict) and res.get("status") == "ok":
|
||||
return int(res.get("n_links") or 0)
|
||||
if isinstance(res, dict) and res.get("error"):
|
||||
notes.append(f"glosario sin enlaces: {res.get('error')}")
|
||||
except Exception as e: # noqa: BLE001 — links are best-effort.
|
||||
notes.append(f"glosario sin enlaces: {e}")
|
||||
return 0
|
||||
|
||||
@@ -43,6 +43,8 @@ _ACCENT = (0x2A, 0x6F, 0x97)
|
||||
_MUTED = (0x8A, 0x8A, 0x8A)
|
||||
_HEAD_BG = (0xEE, 0xF3, 0xF6)
|
||||
_WHITE = (0xFF, 0xFF, 0xFF)
|
||||
_ZEBRA = (0xF6, 0xF8, 0xFA) # faint grey for even (zebra) data rows.
|
||||
_LINK = (0x2A, 0x6F, 0x97) # accent colour for clickable glossary terms.
|
||||
|
||||
_FS_TITLE = 26
|
||||
_FS_H1, _FS_H2, _FS_H3 = 20, 16, 13
|
||||
@@ -59,6 +61,10 @@ class _PptxState:
|
||||
self.chapter = None
|
||||
self.slide_no = 0
|
||||
self.chapter_slides = 0
|
||||
self.last_heading = "" # text of the most recent heading.
|
||||
# Glossary wiring (mejora 6): runs to link and per-term target slide.
|
||||
self.term_runs = [] # [(key, run)]
|
||||
self.term_anchor_slide = {} # key -> Slide (glossary entry)
|
||||
|
||||
|
||||
def _rgb(c):
|
||||
@@ -151,10 +157,57 @@ def _add_text(st: _PptxState, lines: list, fs: float, color, bold=False,
|
||||
st.y += height
|
||||
|
||||
|
||||
def _add_rich_text(st: _PptxState, rich_lines: list, fs: float, color,
|
||||
indent=0.0, bullet=False) -> None:
|
||||
"""Add pre-wrapped lines of styled segments as one paragraph per line.
|
||||
|
||||
Each line is a list of ``(text, is_bold)`` or ``(text, is_bold, term_key)``
|
||||
segments; every segment becomes its own run so ``**bold**`` spans render with
|
||||
native PowerPoint bold (``run.font.bold``) without affecting the measured
|
||||
height (one paragraph per pre-wrapped line). A segment carrying a
|
||||
``term_key`` is drawn in the accent colour and its run is recorded in
|
||||
``st.term_runs`` so it later becomes a native hyperlink jumping to the
|
||||
glossary slide of that term.
|
||||
"""
|
||||
lh = tl.line_height_in(fs)
|
||||
height = lh * len(rich_lines) + 0.05
|
||||
_ensure(st, height)
|
||||
box = st.slide.shapes.add_textbox(
|
||||
Inches(_ML + indent), Inches(st.y), Inches(_USABLE_W - indent),
|
||||
Inches(height))
|
||||
tf = box.text_frame
|
||||
tf.word_wrap = True
|
||||
first = True
|
||||
for segs in rich_lines:
|
||||
p = tf.paragraphs[0] if first else tf.add_paragraph()
|
||||
first = False
|
||||
if bullet:
|
||||
r0 = p.add_run()
|
||||
r0.text = "• "
|
||||
r0.font.size = Pt(fs)
|
||||
r0.font.color.rgb = _rgb(color)
|
||||
for seg in segs:
|
||||
if len(seg) == 3:
|
||||
seg_text, is_bold, term = seg
|
||||
else:
|
||||
seg_text, is_bold, term = seg[0], seg[1], None
|
||||
if seg_text == "":
|
||||
continue
|
||||
run = p.add_run()
|
||||
run.text = seg_text
|
||||
run.font.size = Pt(fs)
|
||||
run.font.bold = bool(is_bold)
|
||||
run.font.color.rgb = _rgb(_LINK if term else color)
|
||||
if term:
|
||||
st.term_runs.append((term, run, st.slide))
|
||||
st.y += height
|
||||
|
||||
|
||||
def _place_heading(st: _PptxState, block) -> None:
|
||||
level = max(1, min(3, int(getattr(block, "level", 1) or 1)))
|
||||
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
|
||||
text = tl.strip_inline_md(getattr(block, "text", ""))
|
||||
st.last_heading = text or st.last_heading
|
||||
lines = tl.wrap(text, tl.chars_per_line(_USABLE_W, fs))
|
||||
_add_text(st, lines, fs, _INK, bold=True)
|
||||
st.y += 0.04
|
||||
@@ -196,22 +249,23 @@ def _place_markdown(st: _PptxState, block) -> None:
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("- ") or stripped.startswith("* "):
|
||||
content = tl.strip_inline_md(stripped[2:])
|
||||
lines = tl.wrap(content, tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
|
||||
_add_text(st, lines, _FS_BODY, _INK, bullet=True)
|
||||
content = stripped[2:] # keep inline markers for bold rendering.
|
||||
rich = tl.wrap_rich_terms(content,
|
||||
tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
|
||||
_add_rich_text(st, rich, _FS_BODY, _INK, bullet=True)
|
||||
i += 1
|
||||
continue
|
||||
para = [tl.strip_inline_md(stripped)]
|
||||
para = [stripped] # keep inline markers; wrap_rich_terms renders **bold**.
|
||||
j = i + 1
|
||||
while j < n:
|
||||
nxt = md_lines[j].strip()
|
||||
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
|
||||
break
|
||||
para.append(tl.strip_inline_md(nxt))
|
||||
para.append(nxt)
|
||||
j += 1
|
||||
text = " ".join(para)
|
||||
_add_text(st, tl.wrap(text, tl.chars_per_line(_USABLE_W, _FS_BODY)),
|
||||
_FS_BODY, _INK)
|
||||
_add_rich_text(st, tl.wrap_rich_terms(
|
||||
text, tl.chars_per_line(_USABLE_W, _FS_BODY)), _FS_BODY, _INK)
|
||||
i = j
|
||||
st.y += _GAP
|
||||
|
||||
@@ -258,7 +312,8 @@ def _row_height_in(cells, widths, fs) -> float:
|
||||
return lh * maxlines + 0.10
|
||||
|
||||
|
||||
def _emit_table(st: _PptxState, header, chunk, widths, fs) -> None:
|
||||
def _emit_table(st: _PptxState, header, chunk, widths, fs,
|
||||
start_index: int = 0) -> None:
|
||||
nrows = len(chunk) + (1 if header else 0)
|
||||
ncol = len(widths)
|
||||
# Pre-measure total height to size the shape (pptx still auto-grows rows).
|
||||
@@ -282,11 +337,14 @@ def _emit_table(st: _PptxState, header, chunk, widths, fs) -> None:
|
||||
cell.text = model._safe_str(header[c]) if c < len(header) else ""
|
||||
_style_cell(cell, fs, _INK, bold=True, fill=_HEAD_BG)
|
||||
ridx = 1
|
||||
for r in chunk:
|
||||
# Zebra striping: shade even data rows (1-based) using the GLOBAL row index
|
||||
# (start_index offset) so the pattern stays coherent across split chunks.
|
||||
for k, r in enumerate(chunk):
|
||||
fill = _ZEBRA if (start_index + k) % 2 == 1 else _WHITE
|
||||
for c in range(ncol):
|
||||
cell = gtable.cell(ridx, c)
|
||||
cell.text = model._safe_str(r[c]) if c < len(r) else ""
|
||||
_style_cell(cell, fs, _INK, bold=False, fill=_WHITE)
|
||||
_style_cell(cell, fs, _INK, bold=False, fill=fill)
|
||||
ridx += 1
|
||||
st.y += total_h + _GAP
|
||||
|
||||
@@ -330,6 +388,7 @@ def _place_data_table(st: _PptxState, block, shaded_header=True,
|
||||
avail = _remaining(st) - header_h
|
||||
chunk = []
|
||||
used = 0.0
|
||||
chunk_start = idx # global index of the first row in this chunk (zebra).
|
||||
while idx < n:
|
||||
rh = _row_height_in(rows[idx], widths, fs)
|
||||
if used + rh > avail and chunk:
|
||||
@@ -337,7 +396,7 @@ def _place_data_table(st: _PptxState, block, shaded_header=True,
|
||||
chunk.append(rows[idx])
|
||||
used += rh
|
||||
idx += 1
|
||||
_emit_table(st, header, chunk, widths, fs)
|
||||
_emit_table(st, header, chunk, widths, fs, start_index=chunk_start)
|
||||
note = getattr(block, "note", None)
|
||||
if note:
|
||||
_add_text(st, tl.wrap(model._safe_str(note),
|
||||
@@ -384,54 +443,97 @@ def _resolve_png(block):
|
||||
pass
|
||||
|
||||
|
||||
def _place_picture_bytes(st: _PptxState, data: bytes, caption) -> None:
|
||||
def _figure_bytes_cached(block):
|
||||
"""Rasterize a figure/image to PNG bytes ONCE and cache (bytes, aspect).
|
||||
|
||||
Measuring (keep-together) and drawing must agree on the real aspect ratio —
|
||||
``bbox_inches='tight'`` changes it vs ``figsize``, so we rasterize once and
|
||||
reuse the bytes for both. Cached on the block; never raises."""
|
||||
cached = getattr(block, "_aeda_png", None)
|
||||
if cached is not None:
|
||||
return cached
|
||||
kind = getattr(block, "kind", "")
|
||||
data = None
|
||||
if kind == "image":
|
||||
path = getattr(block, "path", "")
|
||||
if path and os.path.exists(path):
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = fh.read()
|
||||
except Exception: # noqa: BLE001
|
||||
data = None
|
||||
else:
|
||||
data = _resolve_png(block)
|
||||
aspect = 0.66
|
||||
if data is not None:
|
||||
w_px, h_px = _img_size_px(data)
|
||||
aspect = (h_px / w_px) if w_px else 0.66
|
||||
try:
|
||||
block._aeda_png = (data, aspect)
|
||||
return block._aeda_png
|
||||
except Exception: # noqa: BLE001 — block may reject attributes; degrade.
|
||||
return (data, aspect)
|
||||
|
||||
|
||||
def _place_picture_bytes(st: _PptxState, data: bytes, caption,
|
||||
max_h_in=None) -> None:
|
||||
# Mejora 4 — every figure on a slide carries a visible caption/title. If the
|
||||
# block has no caption, fall back to the current section heading, then to a
|
||||
# generic label, so no image is ever shown untitled.
|
||||
caption = (model._safe_str(caption).strip()
|
||||
or model._safe_str(st.last_heading).strip() or "Figura")
|
||||
w_px, h_px = _img_size_px(data)
|
||||
aspect = (h_px / w_px) if w_px else 0.66
|
||||
# Reserve the caption's REAL (possibly multi-line) height FIRST, then scale
|
||||
# the image to (max_h - cap_reserve): a figure never fills the whole slide,
|
||||
# so its caption always fits on the SAME slide and no image is untitled.
|
||||
# cap_real = what _add_text consumes; cap_reserve adds the post-image gap and
|
||||
# a small cushion so the caption never spills to the next slide.
|
||||
cap_lines = tl.wrap(caption, tl.chars_per_line(_USABLE_W, _FS_NOTE))
|
||||
cap_real = tl.line_height_in(_FS_NOTE) * len(cap_lines) + 0.05
|
||||
cap_reserve = cap_real + 0.05 + 0.10
|
||||
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
# height_in hint (model.Figure/Image): cap the target height so a figure in a
|
||||
# keep-together Group shrinks to leave room for its heading and text.
|
||||
if isinstance(max_h_in, (int, float)) and max_h_in > 0:
|
||||
max_h = min(max_h, float(max_h_in))
|
||||
max_img_h = max(max_h - cap_reserve, 0.6)
|
||||
target_w = _USABLE_W
|
||||
target_h = target_w * aspect
|
||||
if target_h > max_h:
|
||||
target_h = max_h
|
||||
if target_h > max_img_h:
|
||||
target_h = max_img_h
|
||||
target_w = target_h / aspect if aspect else _USABLE_W
|
||||
cap_h = tl.line_height_in(_FS_NOTE) + 0.05 if caption else 0.0
|
||||
if _remaining(st) < target_h + cap_h:
|
||||
# Keep the image and its caption together on the same slide.
|
||||
if _remaining(st) < target_h + cap_reserve:
|
||||
_new_slide(st, cont=True)
|
||||
left = _ML + (_USABLE_W - target_w) / 2.0
|
||||
st.slide.shapes.add_picture(io.BytesIO(data), Inches(left), Inches(st.y),
|
||||
width=Inches(target_w), height=Inches(target_h))
|
||||
st.y += target_h + 0.05
|
||||
if caption:
|
||||
_add_text(st, tl.wrap(model._safe_str(caption),
|
||||
tl.chars_per_line(_USABLE_W, _FS_NOTE)), _FS_NOTE, _MUTED,
|
||||
italic=True)
|
||||
_add_text(st, cap_lines, _FS_NOTE, _MUTED, italic=True)
|
||||
st.y += _GAP
|
||||
|
||||
|
||||
def _place_figure(st: _PptxState, block) -> None:
|
||||
png = _resolve_png(block)
|
||||
png, _aspect = _figure_bytes_cached(block)
|
||||
if png is None:
|
||||
_add_text(st, ["(figura no disponible)"], _FS_NOTE, _MUTED, italic=True)
|
||||
st.y += _GAP
|
||||
return
|
||||
_place_picture_bytes(st, png, getattr(block, "caption", None))
|
||||
_place_picture_bytes(st, png, getattr(block, "caption", None),
|
||||
max_h_in=getattr(block, "height_in", None))
|
||||
|
||||
|
||||
def _place_image(st: _PptxState, block) -> None:
|
||||
path = getattr(block, "path", "")
|
||||
if not path or not os.path.exists(path):
|
||||
data, _aspect = _figure_bytes_cached(block)
|
||||
if data is None:
|
||||
path = getattr(block, "path", "")
|
||||
_add_text(st, [f"(imagen no encontrada: {path})"], _FS_NOTE, _MUTED,
|
||||
italic=True)
|
||||
st.y += _GAP
|
||||
return
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = fh.read()
|
||||
except Exception as e: # noqa: BLE001
|
||||
_add_text(st, [f"(no se pudo leer la imagen: {e})"], _FS_NOTE, _MUTED,
|
||||
italic=True)
|
||||
st.y += _GAP
|
||||
return
|
||||
_place_picture_bytes(st, data, getattr(block, "caption", None))
|
||||
_place_picture_bytes(st, data, getattr(block, "caption", None),
|
||||
max_h_in=getattr(block, "height_in", None))
|
||||
|
||||
|
||||
def _place_caption(st: _PptxState, block) -> None:
|
||||
@@ -445,6 +547,170 @@ def _place_note(st: _PptxState, block) -> None:
|
||||
_place_caption(st, block)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Block measurement (mejora 3 — keep-together). Estimate a block's slide height
|
||||
# WITHOUT drawing it so a Group can move whole to the next slide before drawing.
|
||||
# Over-estimating only triggers an earlier slide break, never a content cut.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _measure_heading_text(text: str, level: int) -> float:
|
||||
level = max(1, min(3, int(level or 1)))
|
||||
fs = {1: _FS_H1, 2: _FS_H2, 3: _FS_H3}[level]
|
||||
lines = tl.wrap(tl.strip_inline_md(text), tl.chars_per_line(_USABLE_W, fs))
|
||||
return tl.line_height_in(fs) * len(lines) + 0.05 + 0.04
|
||||
|
||||
|
||||
def _measure_markdown(block) -> float:
|
||||
raw = str(getattr(block, "text", "") or "")
|
||||
md_lines = raw.split("\n")
|
||||
h = 0.0
|
||||
i, n = 0, len(md_lines)
|
||||
while i < n:
|
||||
stripped = md_lines[i].strip()
|
||||
if stripped.startswith("|") and stripped.endswith("|"):
|
||||
j = i
|
||||
while j < n and md_lines[j].strip().startswith("|") \
|
||||
and md_lines[j].strip().endswith("|"):
|
||||
j += 1
|
||||
h += (tl.line_height_in(_FS_CELL) + 0.10) * (j - i) + _GAP
|
||||
i = j
|
||||
continue
|
||||
if stripped == "":
|
||||
h += tl.line_height_in(_FS_BODY) * 0.4
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("### "):
|
||||
h += _measure_heading_text(stripped[4:], 3)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("## "):
|
||||
h += _measure_heading_text(stripped[3:], 2)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("# "):
|
||||
h += _measure_heading_text(stripped[2:], 1)
|
||||
i += 1
|
||||
continue
|
||||
if stripped.startswith("- ") or stripped.startswith("* "):
|
||||
lines = tl.wrap_rich_terms(
|
||||
stripped[2:], tl.chars_per_line(_USABLE_W - 0.3, _FS_BODY))
|
||||
h += tl.line_height_in(_FS_BODY) * len(lines) + 0.05
|
||||
i += 1
|
||||
continue
|
||||
para = [stripped]
|
||||
j = i + 1
|
||||
while j < n:
|
||||
nxt = md_lines[j].strip()
|
||||
if nxt == "" or nxt.startswith(("|", "#", "- ", "* ")):
|
||||
break
|
||||
para.append(nxt)
|
||||
j += 1
|
||||
lines = tl.wrap_rich_terms(" ".join(para),
|
||||
tl.chars_per_line(_USABLE_W, _FS_BODY))
|
||||
h += tl.line_height_in(_FS_BODY) * len(lines) + 0.05
|
||||
i = j
|
||||
return h + _GAP
|
||||
|
||||
|
||||
def _measure_figure_like(block) -> float:
|
||||
max_h = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
hint = getattr(block, "height_in", None)
|
||||
if isinstance(hint, (int, float)) and hint > 0:
|
||||
max_h = min(max_h, float(hint))
|
||||
# Use the REAL rasterized aspect (cached) so measuring matches drawing — this
|
||||
# is what keeps a figure together with its heading instead of splitting.
|
||||
_data, aspect = _figure_bytes_cached(block)
|
||||
target_h = min(_USABLE_W * aspect, max_h)
|
||||
# Caption is always emitted now (mejora 4), so always reserve its line.
|
||||
cap_h = tl.line_height_in(_FS_NOTE) + 0.05
|
||||
return target_h + 0.05 + cap_h + _GAP
|
||||
|
||||
|
||||
def _measure_block(st: _PptxState, block) -> float:
|
||||
kind = getattr(block, "kind", "")
|
||||
try:
|
||||
if kind == "heading":
|
||||
return _measure_heading_text(getattr(block, "text", ""),
|
||||
getattr(block, "level", 1))
|
||||
if kind == "markdown":
|
||||
return _measure_markdown(block)
|
||||
if kind in ("figure", "image"):
|
||||
return _measure_figure_like(block)
|
||||
if kind in ("caption", "note"):
|
||||
lines = tl.wrap(getattr(block, "text", ""),
|
||||
tl.chars_per_line(_USABLE_W, _FS_NOTE))
|
||||
return tl.line_height_in(_FS_NOTE) * len(lines) + 0.05 + _GAP
|
||||
if kind in ("kv_table", "data_table"):
|
||||
rows = getattr(block, "rows", []) or []
|
||||
return (tl.line_height_in(_FS_CELL) + 0.10) * (len(rows) + 1) + _GAP
|
||||
if kind == "group":
|
||||
return sum(_measure_block(st, b)
|
||||
for b in (getattr(block, "blocks", []) or []))
|
||||
except Exception: # noqa: BLE001 — a measurement never aborts rendering.
|
||||
pass
|
||||
return tl.line_height_in(_FS_BODY)
|
||||
|
||||
|
||||
def _shrink_group_figures(st: _PptxState, blocks: list, avail_full: float) -> None:
|
||||
"""Cap each figure's height (via height_in) so the whole group fits a slide.
|
||||
|
||||
The figure shrinks just enough to leave room for its heading, text and
|
||||
caption — that is how keep-together puts a chart on the SAME slide as its
|
||||
title and description instead of pushing it to the next slide."""
|
||||
fig_blocks = [b for b in blocks
|
||||
if getattr(b, "kind", "") in ("figure", "image")]
|
||||
if not fig_blocks:
|
||||
return
|
||||
nonfig_h = sum(_measure_block(st, b) for b in blocks
|
||||
if getattr(b, "kind", "") not in ("figure", "image"))
|
||||
fig_overhead = tl.line_height_in(_FS_NOTE) + 0.05 + 0.05 + _GAP
|
||||
budget = avail_full - nonfig_h - 0.10 * len(fig_blocks)
|
||||
if budget <= 1.0:
|
||||
return # not enough room to keep together; let it flow (degrade).
|
||||
per = budget / len(fig_blocks) - fig_overhead
|
||||
if per <= 0.8:
|
||||
return
|
||||
for fb in fig_blocks:
|
||||
cur = getattr(fb, "height_in", None)
|
||||
fb.height_in = (min(float(cur), per)
|
||||
if isinstance(cur, (int, float)) and cur > 0 else per)
|
||||
|
||||
|
||||
def _place_group(st: _PptxState, block) -> None:
|
||||
"""Render a keep-together Group: move it whole to the next slide if needed."""
|
||||
blocks = getattr(block, "blocks", []) or []
|
||||
if not blocks:
|
||||
return
|
||||
avail_full = _CONTENT_BOTTOM - _CONTENT_TOP
|
||||
_shrink_group_figures(st, blocks, avail_full)
|
||||
total = sum(_measure_block(st, b) for b in blocks)
|
||||
if total <= avail_full:
|
||||
if total > _remaining(st):
|
||||
_new_slide(st, cont=True)
|
||||
elif st.y > _CONTENT_TOP + 1e-6:
|
||||
_new_slide(st, cont=True)
|
||||
for b in blocks:
|
||||
placer = _PLACERS.get(getattr(b, "kind", ""), _place_note)
|
||||
try:
|
||||
placer(st, b)
|
||||
except Exception: # noqa: BLE001 — a bad block never aborts the group.
|
||||
pass
|
||||
|
||||
|
||||
def _place_glossary_entry(st: _PptxState, block) -> None:
|
||||
"""Render one glossary term and register its slide as the link target."""
|
||||
key = getattr(block, "key", "")
|
||||
label = getattr(block, "label", "") or key
|
||||
definition = getattr(block, "definition", "")
|
||||
_ensure(st, tl.line_height_in(_FS_H3) + tl.line_height_in(_FS_BODY) * 2)
|
||||
if key:
|
||||
st.term_anchor_slide[key] = st.slide
|
||||
_place_heading(st, model.Heading(text=str(label), level=3))
|
||||
if definition:
|
||||
_add_text(st, tl.wrap(model._safe_str(definition),
|
||||
tl.chars_per_line(_USABLE_W, _FS_BODY)), _FS_BODY, _INK)
|
||||
st.y += _GAP
|
||||
|
||||
|
||||
_PLACERS = {
|
||||
"heading": _place_heading,
|
||||
"markdown": _place_markdown,
|
||||
@@ -454,6 +720,8 @@ _PLACERS = {
|
||||
"image": _place_image,
|
||||
"caption": _place_caption,
|
||||
"note": _place_note,
|
||||
"group": _place_group,
|
||||
"glossary_entry": _place_glossary_entry,
|
||||
}
|
||||
|
||||
|
||||
@@ -505,6 +773,9 @@ def render_pptx(chapters: list, out_path: str, meta: dict = None) -> dict:
|
||||
_new_slide(st, cont=False)
|
||||
_place_note(st, model.Note(
|
||||
"(documento vacío — sin capítulos aplicables)"))
|
||||
# Mejora 6 — wire clickable glossary terms to their entry slide (native
|
||||
# PowerPoint slide-jump). Delegated registry function; degrades silently.
|
||||
n_links = _wire_glossary_links(st, notes)
|
||||
prs.save(out_path)
|
||||
n_slides = st.slide_no
|
||||
except Exception as e: # noqa: BLE001
|
||||
@@ -512,7 +783,35 @@ def render_pptx(chapters: list, out_path: str, meta: dict = None) -> dict:
|
||||
"note": f"fallo al escribir el PPTX: {e}"}
|
||||
|
||||
note = f"{n_slides} slides"
|
||||
if n_links:
|
||||
note += f" · {n_links} enlaces de glosario"
|
||||
if notes:
|
||||
note += " · " + "; ".join(notes)
|
||||
return {"path": out_path, "n_slides": n_slides, "chapters": chapters_meta,
|
||||
"note": note}
|
||||
|
||||
|
||||
def _wire_glossary_links(st: _PptxState, notes: list) -> int:
|
||||
"""Turn each recorded term run into a native jump to its glossary slide.
|
||||
|
||||
Returns the number of links applied. A term whose only appearance is inside
|
||||
its own glossary entry (source slide == target slide) is skipped. Never
|
||||
raises."""
|
||||
if not st.term_runs or not st.term_anchor_slide:
|
||||
return 0
|
||||
linked = 0
|
||||
try:
|
||||
from datascience.pptx_link_run_to_slide import pptx_link_run_to_slide
|
||||
except Exception as e: # noqa: BLE001
|
||||
notes.append(f"glosario sin enlaces: {e}")
|
||||
return 0
|
||||
for key, run, src_slide in st.term_runs:
|
||||
tgt = st.term_anchor_slide.get(key)
|
||||
if tgt is None or tgt is src_slide:
|
||||
continue
|
||||
try:
|
||||
if pptx_link_run_to_slide(run, src_slide, tgt):
|
||||
linked += 1
|
||||
except Exception: # noqa: BLE001 — links are best-effort.
|
||||
pass
|
||||
return linked
|
||||
|
||||
@@ -15,8 +15,22 @@ overflowing — that is wrapping, not loss: every character is still rendered.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import textwrap
|
||||
|
||||
# Inline span markers: ``**bold**`` / ``__bold__`` (rendered bold) and
|
||||
# `` `code` `` (markers removed, not styled). Matched non-greedily so the
|
||||
# shortest balanced pair wins. Unbalanced leftovers are stripped afterwards so
|
||||
# the visible text matches ``strip_inline_md`` exactly.
|
||||
_INLINE_SPAN_RE = re.compile(r"(\*\*.+?\*\*|__.+?__|`.+?`)")
|
||||
|
||||
# Glossary term span: ``[[term:key]]texto visible[[/term]]``. The visible text
|
||||
# (which may itself contain ``**bold**``) is kept and tagged with ``key`` so the
|
||||
# renderers can turn each appearance into a clickable jump to the glossary entry.
|
||||
_TERM_SPAN_RE = re.compile(r"\[\[term:([A-Za-z0-9_]+)\]\](.*?)\[\[/term\]\]",
|
||||
re.S)
|
||||
_TERM_OPEN_RE = re.compile(r"\[\[term:[A-Za-z0-9_]+\]\]")
|
||||
|
||||
|
||||
def avg_char_width_in(fontsize_pt: float) -> float:
|
||||
"""Approximate average glyph width in inches for a sans-serif font.
|
||||
@@ -79,11 +93,264 @@ def strip_inline_md(text: str) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
s = str(text)
|
||||
# Drop glossary term markers, keeping the visible inner text.
|
||||
s = _TERM_SPAN_RE.sub(lambda m: m.group(2), s)
|
||||
s = _TERM_OPEN_RE.sub("", s) # leftover unbalanced open marker.
|
||||
s = s.replace("[[/term]]", "") # leftover unbalanced close marker.
|
||||
for marker in ("**", "__", "`"):
|
||||
s = s.replace(marker, "")
|
||||
return s
|
||||
|
||||
|
||||
def _strip_term_markers(s: str) -> str:
|
||||
"""Remove any (balanced or leftover) glossary term markers, keeping text."""
|
||||
s = _TERM_OPEN_RE.sub("", s)
|
||||
return s.replace("[[/term]]", "")
|
||||
|
||||
|
||||
def _strip_leftover_markers(s: str) -> str:
|
||||
"""Drop any unbalanced inline markers from a plain (non-span) fragment.
|
||||
|
||||
Keeps the visible text identical to :func:`strip_inline_md` even when a
|
||||
``**`` / ``__`` / `` ` `` has no matching closing marker.
|
||||
"""
|
||||
for marker in ("**", "__", "`"):
|
||||
s = s.replace(marker, "")
|
||||
return s
|
||||
|
||||
|
||||
def parse_inline_bold(text: str):
|
||||
"""Split ``text`` into ``[(fragment, is_bold), ...]`` preserving order.
|
||||
|
||||
``**...**`` and ``__...__`` spans become bold fragments (markers removed);
|
||||
`` `code` `` keeps its text without the backticks and is not bold; any other
|
||||
text is emitted verbatim with unbalanced markers stripped. The concatenation
|
||||
of all fragment texts equals :func:`strip_inline_md` of the input — so the
|
||||
*visible* characters (and therefore line wrapping) are unchanged; only the
|
||||
bold flag is added. Adjacent fragments of the same weight are merged.
|
||||
"""
|
||||
s = "" if text is None else str(text)
|
||||
if not s:
|
||||
return []
|
||||
out = []
|
||||
|
||||
def _emit(fragment: str, bold: bool) -> None:
|
||||
if fragment == "":
|
||||
return
|
||||
if out and out[-1][1] == bold:
|
||||
out[-1] = (out[-1][0] + fragment, bold)
|
||||
else:
|
||||
out.append((fragment, bold))
|
||||
|
||||
pos = 0
|
||||
for m in _INLINE_SPAN_RE.finditer(s):
|
||||
if m.start() > pos:
|
||||
_emit(_strip_leftover_markers(s[pos:m.start()]), False)
|
||||
tok = m.group(0)
|
||||
if tok.startswith("**") and tok.endswith("**"):
|
||||
_emit(tok[2:-2], True)
|
||||
elif tok.startswith("__") and tok.endswith("__"):
|
||||
_emit(tok[2:-2], True)
|
||||
else: # `code`
|
||||
_emit(tok[1:-1], False)
|
||||
pos = m.end()
|
||||
if pos < len(s):
|
||||
_emit(_strip_leftover_markers(s[pos:]), False)
|
||||
return out
|
||||
|
||||
|
||||
def _hard_split(word: str, max_chars: int):
|
||||
"""Split a single long token into <= max_chars chunks (never loses chars)."""
|
||||
return [word[i:i + max_chars] for i in range(0, len(word), max_chars)] or [""]
|
||||
|
||||
|
||||
def wrap_rich(text: str, max_chars: int):
|
||||
"""Word-wrap ``text`` to ``max_chars`` while preserving inline bold spans.
|
||||
|
||||
Returns ``list[list[(fragment, is_bold)]]`` — one inner list of styled
|
||||
fragments per output line; concatenating an inner list's fragment texts is
|
||||
the visible line. Wrapping is word-aware and hard-splits over-long tokens, so
|
||||
no line exceeds ``max_chars`` (the renderers measure these very lines, so the
|
||||
no-cut guarantee holds). Bold spans never widen a line: only the bold flag is
|
||||
carried, the visible width is identical to :func:`wrap`.
|
||||
"""
|
||||
if max_chars < 1:
|
||||
max_chars = 1
|
||||
spans = parse_inline_bold(text)
|
||||
if not spans:
|
||||
return [[("", False)]]
|
||||
|
||||
# Flatten to (word, is_bold) tokens, honoring hard newlines as line breaks.
|
||||
# A token list of None marks a forced line break.
|
||||
tokens = [] # each: (word, bold) or ("\n", None)
|
||||
for frag, bold in spans:
|
||||
parts = frag.split("\n")
|
||||
for pi, part in enumerate(parts):
|
||||
if pi > 0:
|
||||
tokens.append(("\n", None))
|
||||
for word in part.split(" "):
|
||||
if word == "":
|
||||
continue
|
||||
tokens.append((word, bold))
|
||||
|
||||
lines = [] # list[list[(seg, bold)]]
|
||||
cur = [] # list[(word, bold)]
|
||||
cur_len = 0
|
||||
|
||||
def _flush():
|
||||
nonlocal cur, cur_len
|
||||
# Merge adjacent same-weight words (with separating spaces) into segments.
|
||||
merged = []
|
||||
for k, (word, bold) in enumerate(cur):
|
||||
piece = word if k == 0 else " " + word
|
||||
if merged and merged[-1][1] == bold:
|
||||
merged[-1] = (merged[-1][0] + piece, bold)
|
||||
else:
|
||||
merged.append((piece, bold))
|
||||
lines.append(merged or [("", False)])
|
||||
cur = []
|
||||
cur_len = 0
|
||||
|
||||
for word, bold in tokens:
|
||||
if bold is None: # forced newline
|
||||
_flush()
|
||||
continue
|
||||
if len(word) > max_chars:
|
||||
if cur:
|
||||
_flush()
|
||||
chunks = _hard_split(word, max_chars)
|
||||
for ci, chunk in enumerate(chunks):
|
||||
if ci < len(chunks) - 1:
|
||||
lines.append([(chunk, bold)])
|
||||
else:
|
||||
cur = [(chunk, bold)]
|
||||
cur_len = len(chunk)
|
||||
continue
|
||||
add = len(word) if cur_len == 0 else cur_len + 1 + len(word)
|
||||
if cur_len != 0 and add > max_chars:
|
||||
_flush()
|
||||
cur = [(word, bold)]
|
||||
cur_len = len(word)
|
||||
else:
|
||||
cur.append((word, bold))
|
||||
cur_len = add
|
||||
if cur:
|
||||
_flush()
|
||||
return lines or [[("", False)]]
|
||||
|
||||
|
||||
def parse_inline_rich(text: str):
|
||||
"""Split ``text`` into ``[(fragment, is_bold, term_key), ...]``.
|
||||
|
||||
Extends :func:`parse_inline_bold` with glossary term spans
|
||||
``[[term:key]]visible[[/term]]``: the inner ``visible`` text is parsed for
|
||||
``**bold**`` as usual and every resulting fragment carries ``term_key`` so the
|
||||
renderers can make it clickable. Text outside a term span gets ``term_key =
|
||||
None``. Unbalanced term markers are stripped (kept identical to
|
||||
:func:`strip_inline_md`). The concatenation of all fragment texts equals
|
||||
``strip_inline_md(text)`` — visible characters and wrapping are unchanged; only
|
||||
the bold flag and the term key are added. Adjacent fragments with the same
|
||||
(bold, term) are merged.
|
||||
"""
|
||||
s = "" if text is None else str(text)
|
||||
if not s:
|
||||
return []
|
||||
out = []
|
||||
|
||||
def _emit(fragment: str, bold: bool, term) -> None:
|
||||
if fragment == "":
|
||||
return
|
||||
if out and out[-1][1] == bold and out[-1][2] == term:
|
||||
out[-1] = (out[-1][0] + fragment, bold, term)
|
||||
else:
|
||||
out.append((fragment, bold, term))
|
||||
|
||||
def _emit_bolded(segment: str, term) -> None:
|
||||
# Reuse the bold parser on a term-marker-free segment.
|
||||
for frag, bold in parse_inline_bold(_strip_term_markers(segment)):
|
||||
_emit(frag, bold, term)
|
||||
|
||||
pos = 0
|
||||
for m in _TERM_SPAN_RE.finditer(s):
|
||||
if m.start() > pos:
|
||||
_emit_bolded(s[pos:m.start()], None)
|
||||
_emit_bolded(m.group(2), m.group(1))
|
||||
pos = m.end()
|
||||
if pos < len(s):
|
||||
_emit_bolded(s[pos:], None)
|
||||
return out
|
||||
|
||||
|
||||
def wrap_rich_terms(text: str, max_chars: int):
|
||||
"""Like :func:`wrap_rich` but preserving glossary term keys per fragment.
|
||||
|
||||
Returns ``list[list[(fragment, is_bold, term_key)]]`` — one inner list per
|
||||
output line. Wrapping is word-aware and hard-splits over-long tokens so no
|
||||
line exceeds ``max_chars`` (the renderers measure these very lines). Term and
|
||||
bold flags never widen a line: the visible width matches :func:`wrap`.
|
||||
"""
|
||||
if max_chars < 1:
|
||||
max_chars = 1
|
||||
spans = parse_inline_rich(text)
|
||||
if not spans:
|
||||
return [[("", False, None)]]
|
||||
|
||||
tokens = [] # each: (word, bold, term) or ("\n", None, None)
|
||||
for frag, bold, term in spans:
|
||||
parts = frag.split("\n")
|
||||
for pi, part in enumerate(parts):
|
||||
if pi > 0:
|
||||
tokens.append(("\n", None, None))
|
||||
for word in part.split(" "):
|
||||
if word == "":
|
||||
continue
|
||||
tokens.append((word, bold, term))
|
||||
|
||||
lines = []
|
||||
cur = []
|
||||
cur_len = 0
|
||||
|
||||
def _flush():
|
||||
nonlocal cur, cur_len
|
||||
merged = []
|
||||
for k, (word, bold, term) in enumerate(cur):
|
||||
piece = word if k == 0 else " " + word
|
||||
if merged and merged[-1][1] == bold and merged[-1][2] == term:
|
||||
merged[-1] = (merged[-1][0] + piece, bold, term)
|
||||
else:
|
||||
merged.append((piece, bold, term))
|
||||
lines.append(merged or [("", False, None)])
|
||||
cur = []
|
||||
cur_len = 0
|
||||
|
||||
for word, bold, term in tokens:
|
||||
if bold is None: # forced newline
|
||||
_flush()
|
||||
continue
|
||||
if len(word) > max_chars:
|
||||
if cur:
|
||||
_flush()
|
||||
chunks = _hard_split(word, max_chars)
|
||||
for ci, chunk in enumerate(chunks):
|
||||
if ci < len(chunks) - 1:
|
||||
lines.append([(chunk, bold, term)])
|
||||
else:
|
||||
cur = [(chunk, bold, term)]
|
||||
cur_len = len(chunk)
|
||||
continue
|
||||
add = len(word) if cur_len == 0 else cur_len + 1 + len(word)
|
||||
if cur_len != 0 and add > max_chars:
|
||||
_flush()
|
||||
cur = [(word, bold, term)]
|
||||
cur_len = len(word)
|
||||
else:
|
||||
cur.append((word, bold, term))
|
||||
cur_len = add
|
||||
if cur:
|
||||
_flush()
|
||||
return lines or [[("", False, None)]]
|
||||
|
||||
|
||||
def parse_md_table(lines: list):
|
||||
"""Parse consecutive ``| a | b |`` lines into ``(header, rows)`` or None.
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
name: build_eda_render_ctx
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: impure
|
||||
signature: "def build_eda_render_ctx(db_path: str, table: str, profile: dict, backend: str = 'duckdb', sample: int = 5000, base_ctx: dict = None) -> dict"
|
||||
description: "Constructor del `ctx` de datos crudos del motor AutomaticEDA. Dado un db_path+table (DuckDB o Postgres) y el TableProfile AGREGADO ya calculado por profile_table, produce el dict ctx que los renderers (render_automatic_eda_pdf/_pptx -> build_document(profile, ctx)) pasan a los capitulos que necesitan DATOS CRUDOS no presentes en el perfil agregado: modelos (project_clusters_2d en vivo), timeseries, geospatial y agregacion (groupby/pivot push-down). NO trae tablas enteras a RAM: muestrea con LIMIT sample y delega el push-down de la serie en extract_timeseries_raw. Construye el lector read-only query_fn(sql)->dict igual que profile_table (closure sobre duckdb_query_readonly / pg_query). Estilo dict-no-throw del grupo eda: NUNCA lanza; si una pieza falla, degrada esa clave a ausente/[] y sigue. Devuelve el ctx dict directamente (NO un wrapper {status,...}); se pasa tal cual como meta={'ctx': <ese dict>}. Claves de datos que produce: raw_numeric (muestra cruda alineada por fila), timeseries_raw (fechas+series), geo_points (lats/lons) y db_path+table para el push-down de agregacion. Respeta base_ctx: parte de una copia y solo AÑADE las claves de datos; las de presentacion (dataset_name, source_origin, ...) no se pisan."
|
||||
tags: [eda, datascience, automatic-eda, render, ctx, extraction, read-only, duckdb, postgres, python]
|
||||
uses_functions: [detect_time_column_py_datascience, extract_timeseries_raw_py_datascience, detect_latlon_columns_py_datascience, duckdb_query_readonly_py_infra, pg_query_py_infra]
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: "error_go_core"
|
||||
imports: []
|
||||
params:
|
||||
- name: db_path
|
||||
desc: "ruta al archivo DuckDB, o DSN PostgreSQL si backend='postgres'. Se guarda tal cual en ctx['db_path'] (el capitulo agregacion lo usa para el groupby/pivot push-down via DuckDB) y se inyecta en el closure query_fn. No se valida aqui: si la base no existe, las queries devuelven status error y las claves de datos se omiten."
|
||||
- name: table
|
||||
desc: "nombre de la tabla. Se escapa con comillas dobles en las queries (raw_numeric y timeseries) y se guarda en ctx['table']."
|
||||
- name: profile
|
||||
desc: "TableProfile AGREGADO producido por profile_table. Solo se lee su clave `columns` (lista de ColumnProfile dict con name / inferred_type / numeric.{min,max} / semantic_type). Lectura defensiva: si no es dict o no tiene columns, se trata como []. NO se traen las filas crudas de aqui — se muestrean de la base."
|
||||
- name: backend
|
||||
desc: "'duckdb' (default) o 'postgres'. Selecciona el lector read-only del registry (duckdb_query_readonly / pg_query). Cualquier otro valor devuelve el base_ctx tal cual, SIN añadir claves de datos (ni siquiera db_path/table)."
|
||||
- name: sample
|
||||
desc: "maximo de filas a muestrear (clausula LIMIT) tanto para raw_numeric (una sola query SELECT de las numericas) como para timeseries_raw (max_rows de extract_timeseries_raw). Default 5000. Acota memoria y tiempo de render."
|
||||
- name: base_ctx
|
||||
desc: "dict opcional con claves de PRESENTACION ya preparadas (dataset_name, source_origin, ...). Se parte de una copia y NO se pisan sus claves; solo se añaden las de datos. Default None -> {}."
|
||||
output: "El dict `ctx` directamente (NO un wrapper {status,...}); se pasa tal cual como meta={'ctx': <ese dict>} a render_automatic_eda_pdf/pptx. Nunca lanza. Para backends validos contiene SIEMPRE db_path + table, y opcionalmente: raw_numeric {col:[float|None,...]} (muestra cruda alineada por fila; omitida si no hay numericas o falla la query), timeseries_raw {time_col, t:[iso...], series:{col:[float|None,...]}} (solo si hay columna temporal + numericas y trae filas), geo_points {lats:[...], lons:[...]} (solo si se detecta par lat/lon y ambas estan en raw_numeric). Ante fallo global devuelve al menos {**base_ctx, 'db_path': db_path, 'table': table}. Backend desconocido -> base_ctx tal cual sin claves de datos."
|
||||
tested: true
|
||||
tests: ["test_db_path_y_table_en_ctx", "test_raw_numeric_con_columnas_numericas", "test_timeseries_raw_con_fecha", "test_geo_points_con_latlon", "test_sin_fecha_no_hay_timeseries", "test_base_ctx_preservado"]
|
||||
test_file_path: "python/functions/datascience/build_eda_render_ctx_test.py"
|
||||
file_path: "python/functions/datascience/build_eda_render_ctx.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience import build_eda_render_ctx, render_automatic_eda_pdf
|
||||
from datascience import profile_table # opcional: para obtener el TableProfile
|
||||
|
||||
# 1) Perfil agregado de la tabla (push-down, sin RAM).
|
||||
prof = profile_table("data/ventas.duckdb", "ventas_geo", write_report=False)["profile"]
|
||||
|
||||
# 2) ctx de datos crudos para los capitulos (muestrea con LIMIT, no carga todo).
|
||||
ctx = build_eda_render_ctx(
|
||||
"data/ventas.duckdb", "ventas_geo", prof,
|
||||
backend="duckdb", sample=5000,
|
||||
base_ctx={"dataset_name": "Ventas con geolocalizacion"},
|
||||
)
|
||||
# ctx == {
|
||||
# "dataset_name": "Ventas con geolocalizacion", # preservado del base_ctx
|
||||
# "db_path": "data/ventas.duckdb", "table": "ventas_geo",
|
||||
# "raw_numeric": {"ventas": [1200.5, ...], "lat": [40.41, ...], "lon": [-3.70, ...]},
|
||||
# "timeseries_raw": {"time_col": "fecha", "t": ["2024-01-01", ...], "series": {...}},
|
||||
# "geo_points": {"lats": [40.41, ...], "lons": [-3.70, ...]},
|
||||
# }
|
||||
|
||||
# 3) Se entrega tal cual a los renderers via meta={"ctx": ctx}.
|
||||
render_automatic_eda_pdf(prof, "reports/eda.pdf", meta={"ctx": ctx})
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Justo antes de renderizar un AutomaticEDA (PDF o PPTX), cuando ya tienes el
|
||||
TableProfile AGREGADO de `profile_table` pero los capitulos de modelos,
|
||||
timeseries, geospatial y agregacion necesitan DATOS CRUDOS que el perfil
|
||||
agregado no lleva (la muestra numerica alineada por fila, la serie cronologica,
|
||||
el par lat/lon, y el db_path/table para el push-down del groupby/pivot). Es el
|
||||
puente entre el perfil agregado y `build_document(profile, ctx)`: una sola
|
||||
llamada produce el `ctx` completo muestreando con `LIMIT` en vez de cargar la
|
||||
tabla entera en memoria.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Impura**: lee de la base de datos a traves de `query_fn` (closure sobre
|
||||
`duckdb_query_readonly` / `pg_query`). No abre conexiones fuera de esos
|
||||
wrappers del registry. Estilo dict-no-throw del grupo `eda`: NUNCA lanza; ante
|
||||
cualquier fallo (query, deteccion, render de una clave) degrada esa clave a
|
||||
ausente/`[]` y sigue. Ante un fallo global devuelve al menos
|
||||
`{**base_ctx, "db_path": db_path, "table": table}`.
|
||||
- **`error_type` en el frontmatter es `error_go_core` por convencion del
|
||||
registry** (toda funcion impura debe declararlo y el indexer lo exige), pero el
|
||||
codigo NO lanza esa excepcion: degrada al ctx parcial. Es metadata, no
|
||||
comportamiento.
|
||||
- **Devuelve el ctx dict directamente, NO un wrapper `{status,...}`**: a
|
||||
diferencia de `extract_timeseries_raw` / `profile_table`, esta funcion es el
|
||||
ultimo eslabon antes del render y su salida se pasa tal cual como
|
||||
`meta={"ctx": <ese dict>}`. No envuelvas su retorno.
|
||||
- **Backend desconocido**: con un `backend` que no sea `duckdb` ni `postgres`
|
||||
devuelve el `base_ctx` tal cual, SIN claves de datos (ni siquiera
|
||||
`db_path`/`table`). Comprueba el backend antes si dependes de esas claves.
|
||||
- **Alineacion por fila de `raw_numeric`**: `raw_numeric[col]` tiene una entrada
|
||||
por fila muestreada (un valor no convertible a float queda como `None`, no se
|
||||
descarta la fila) porque `project_clusters_2d` descarta filas listwise: todas
|
||||
las columnas deben tener la MISMA longitud. `geo_points` se construye desde
|
||||
`raw_numeric` para heredar esa alineacion.
|
||||
- **`geo_points` exige lat/lon en `raw_numeric`**: el par lat/lon solo se adjunta
|
||||
si ambas columnas se detectaron (nombre+rango) Y figuran en `raw_numeric`
|
||||
(es decir, son numericas en el perfil). Si la tabla guarda lat/lon como texto
|
||||
no promovido a numeric, no apareceran; el capitulo geospatial sabe degradar.
|
||||
- **`timeseries_raw` depende del orden del backend**: hereda el `ORDER BY
|
||||
"time_col"` de `extract_timeseries_raw`. Si la columna temporal esta guardada
|
||||
como texto no ordenable lexicograficamente (p.ej. `DD/MM/YYYY`), el orden no
|
||||
sera el cronologico real — normaliza la columna a date/timestamp antes.
|
||||
- **`LIMIT sample`**: con tablas grandes obtienes el primer tramo (raw_numeric
|
||||
por orden fisico, timeseries por orden cronologico), no un muestreo uniforme.
|
||||
Sube `sample` si necesitas mas cobertura.
|
||||
- **No loguear los datos crudos**: `raw_numeric` / `timeseries_raw` /
|
||||
`geo_points` pueden contener datos sensibles. En trazas usa solo conteos y
|
||||
nombres de columna, no el ctx completo.
|
||||
@@ -0,0 +1,200 @@
|
||||
"""build_eda_render_ctx — constructor del `ctx` de datos crudos del motor AutomaticEDA.
|
||||
|
||||
Funcion impura (lee de la base de datos) del grupo de capacidad `eda`. Dado un
|
||||
``db_path`` + ``table`` (DuckDB o PostgreSQL) y el ``TableProfile`` AGREGADO ya
|
||||
calculado por ``profile_table``, produce el dict ``ctx`` que los renderers
|
||||
(``render_automatic_eda_pdf`` / ``render_automatic_eda_pptx`` ->
|
||||
``build_document(profile, ctx)``) pasan a los capitulos que necesitan DATOS
|
||||
CRUDOS no presentes en el perfil agregado: modelos (``project_clusters_2d`` en
|
||||
vivo), timeseries, geospatial y agregacion (groupby/pivot push-down).
|
||||
|
||||
NO trae tablas enteras a RAM: muestrea con ``LIMIT sample`` y, para la serie
|
||||
temporal, delega el push-down en ``extract_timeseries_raw`` (una sola query
|
||||
ordenada). El lector read-only ``query_fn(sql) -> dict`` se construye igual que
|
||||
en ``profile_table`` (un closure sobre ``duckdb_query_readonly`` / ``pg_query``)
|
||||
y nunca abre conexiones fuera de esos wrappers.
|
||||
|
||||
Estilo dict-no-throw del grupo `eda`: la funcion NUNCA lanza. Si una pieza falla
|
||||
(query, deteccion, render de una clave), esa clave se degrada a ausente / lista
|
||||
vacia y el resto del ctx se construye igual. Ante un fallo global devuelve al
|
||||
menos ``{**base_ctx, "db_path": db_path, "table": table}``.
|
||||
|
||||
Claves de DATOS que produce (las consumen los capitulos):
|
||||
- ``raw_numeric`` : {col: [float|None, ...]} muestra cruda de las columnas
|
||||
numericas, ALINEADA POR FILA (una entrada por fila aunque
|
||||
sea None). La leen modelos (clustering 2D en vivo) y
|
||||
geospatial (lat/lon salen de aqui).
|
||||
- ``timeseries_raw`` : {time_col, t: [iso...], series: {col: [float|None, ...]}}.
|
||||
La lee el capitulo TIMESERIES.
|
||||
- ``geo_points`` : {lats: [...], lons: [...]} listas alineadas (ya floats).
|
||||
La lee el capitulo GEOSPATIAL.
|
||||
- ``db_path``, ``table`` : los usa el capitulo AGREGACION para el groupby/pivot
|
||||
push-down via DuckDB.
|
||||
|
||||
Las claves de PRESENTACION que traiga ``base_ctx`` (dataset_name, source_origin,
|
||||
...) NO se pisan: esta funcion solo AÑADE las claves de datos sobre una copia.
|
||||
"""
|
||||
|
||||
|
||||
def _to_float(value):
|
||||
"""Convierte un valor a float de forma defensiva. None si no es convertible.
|
||||
|
||||
Un bool es subclase de int en Python pero nunca es un valor numerico de
|
||||
serie/coordenada valido, asi que se trata como None (mismo criterio que
|
||||
extract_timeseries_raw / detect_latlon_columns).
|
||||
"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
s = str(value).strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return float(s)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def build_eda_render_ctx(db_path, table, profile, backend="duckdb", sample=5000, base_ctx=None):
|
||||
"""Construye el ctx de datos crudos para los renderers de AutomaticEDA.
|
||||
|
||||
Args:
|
||||
db_path: ruta al archivo DuckDB, o DSN PostgreSQL si backend="postgres".
|
||||
Se guarda tal cual en ctx["db_path"] (el capitulo agregacion lo usa
|
||||
para el push-down).
|
||||
table: nombre de la tabla. Se escapa con comillas dobles en las queries y
|
||||
se guarda en ctx["table"].
|
||||
profile: TableProfile agregado producido por profile_table. Solo se lee
|
||||
su clave ``columns`` (lista de ColumnProfile dict con name /
|
||||
inferred_type / numeric.{min,max} / semantic_type). Lectura
|
||||
defensiva: si no es dict o no tiene columns, se trata como [].
|
||||
backend: "duckdb" (default) o "postgres". Selecciona el lector read-only
|
||||
(duckdb_query_readonly / pg_query). Cualquier otro valor devuelve el
|
||||
base_ctx tal cual, sin añadir claves de datos.
|
||||
sample: maximo de filas a muestrear (clausula LIMIT) tanto para
|
||||
raw_numeric como para timeseries_raw. Default 5000.
|
||||
base_ctx: dict opcional con claves de presentacion ya preparadas
|
||||
(dataset_name, source_origin, ...). Se parte de una copia y NO se
|
||||
pisan sus claves; solo se añaden las de datos. Default None -> {}.
|
||||
|
||||
Returns:
|
||||
El dict ``ctx`` directamente (NO un wrapper {status,...}): se pasa tal
|
||||
cual como ``meta={"ctx": <ese dict>}`` a render_automatic_eda_pdf/pptx.
|
||||
Nunca lanza. Claves que puede contener: raw_numeric, timeseries_raw,
|
||||
geo_points (omitidas si no aplican o fallan), y siempre db_path + table
|
||||
para backends validos.
|
||||
"""
|
||||
# Copia de base_ctx: nunca mutamos el dict del caller. Las claves de
|
||||
# presentacion que ya traiga se conservan; las de datos se añaden encima.
|
||||
ctx = dict(base_ctx) if isinstance(base_ctx, dict) else {}
|
||||
|
||||
try:
|
||||
# 1) Lector read-only del backend activo, construido EXACTAMENTE como en
|
||||
# profile_table (closure sobre el wrapper del registry). Imports perezosos
|
||||
# dentro de la funcion: este modulo vive en el paquete `datascience`, asi
|
||||
# que importar sus hermanas a nivel de modulo crearia un ciclo al cargar
|
||||
# el __init__ del paquete. Lazy import rompe el ciclo y respeta el
|
||||
# contrato (imports explicitos, sin `import *`).
|
||||
if backend == "duckdb":
|
||||
from infra import duckdb_query_readonly
|
||||
|
||||
def query_fn(sql):
|
||||
return duckdb_query_readonly(db_path, sql)
|
||||
|
||||
elif backend == "postgres":
|
||||
from infra import pg_query
|
||||
|
||||
def query_fn(sql):
|
||||
return pg_query(db_path, sql)
|
||||
|
||||
else:
|
||||
# Backend desconocido: devolver base_ctx tal cual, sin claves de datos.
|
||||
return ctx
|
||||
|
||||
# 7) db_path + table SIEMPRE (para backends validos): el capitulo
|
||||
# agregacion los necesita para el groupby/pivot push-down via DuckDB.
|
||||
ctx["db_path"] = db_path
|
||||
ctx["table"] = table
|
||||
|
||||
# 2) Columnas del perfil agregado (lectura defensiva).
|
||||
cols = profile.get("columns") if isinstance(profile, dict) else None
|
||||
cols = cols or []
|
||||
|
||||
# 3) Deteccion temporal/numerica con la funcion PURA del registry.
|
||||
from datascience import detect_time_column
|
||||
|
||||
det = detect_time_column(cols)
|
||||
time_col = det.get("time_col")
|
||||
numeric_cols = det.get("numeric_cols") or []
|
||||
|
||||
# 4) raw_numeric: muestra de las columnas numericas CRUDAS, ALINEADAS POR
|
||||
# FILA en UNA sola query. Cada columna queda con una entrada por fila
|
||||
# (None si no parsea) para no desalinear filas: project_clusters_2d
|
||||
# descarta filas listwise, asi que las listas deben tener igual longitud.
|
||||
raw_numeric = {}
|
||||
if numeric_cols:
|
||||
try:
|
||||
cols_sql = ", ".join(f'"{c}"' for c in numeric_cols)
|
||||
sql = f'SELECT {cols_sql} FROM "{table}" LIMIT {int(sample)}'
|
||||
q = query_fn(sql)
|
||||
if isinstance(q, dict) and q.get("status") == "ok":
|
||||
rows = q.get("rows", []) or []
|
||||
raw_numeric = {c: [] for c in numeric_cols}
|
||||
for row in rows:
|
||||
for c in numeric_cols:
|
||||
raw_numeric[c].append(_to_float(row.get(c)))
|
||||
except Exception: # noqa: BLE001 - dict-no-throw: degradar la clave
|
||||
raw_numeric = {}
|
||||
if raw_numeric:
|
||||
ctx["raw_numeric"] = raw_numeric
|
||||
|
||||
# 5) timeseries_raw: SOLO si hay columna temporal y numericas. Se delega
|
||||
# el push-down en la funcion impura extract_timeseries_raw (una sola query
|
||||
# ordenada cronologicamente). Solo se adjunta si trae filas.
|
||||
if time_col and numeric_cols:
|
||||
try:
|
||||
from datascience import extract_timeseries_raw
|
||||
|
||||
ts = extract_timeseries_raw(
|
||||
query_fn, table, time_col, numeric_cols, max_rows=sample
|
||||
)
|
||||
if (
|
||||
isinstance(ts, dict)
|
||||
and ts.get("status") == "ok"
|
||||
and (ts.get("n") or 0) > 0
|
||||
):
|
||||
ctx["timeseries_raw"] = {
|
||||
"time_col": ts["time_col"],
|
||||
"t": ts["t"],
|
||||
"series": ts["series"],
|
||||
}
|
||||
except Exception: # noqa: BLE001 - dict-no-throw: omitir la clave
|
||||
pass
|
||||
|
||||
# 6) geo_points: detecta el par lat/lon con la funcion PURA del registry.
|
||||
# Solo se adjunta si AMBAS columnas estan en raw_numeric (ya floats,
|
||||
# alineadas por fila). Si no hay par o no estan, se omite: el capitulo
|
||||
# geospatial sabe degradar.
|
||||
try:
|
||||
from datascience import detect_latlon_columns
|
||||
|
||||
geo = detect_latlon_columns(cols)
|
||||
lat_col = geo.get("lat_col")
|
||||
lon_col = geo.get("lon_col")
|
||||
if lat_col and lon_col and lat_col in raw_numeric and lon_col in raw_numeric:
|
||||
ctx["geo_points"] = {
|
||||
"lats": raw_numeric[lat_col],
|
||||
"lons": raw_numeric[lon_col],
|
||||
}
|
||||
except Exception: # noqa: BLE001 - dict-no-throw: omitir la clave
|
||||
pass
|
||||
|
||||
return ctx
|
||||
except Exception: # noqa: BLE001 - dict-no-throw global: nunca reventar.
|
||||
# Fallback minimo: copia de base_ctx + db_path/table para que el capitulo
|
||||
# agregacion siga teniendo lo imprescindible.
|
||||
out = dict(base_ctx) if isinstance(base_ctx, dict) else {}
|
||||
out["db_path"] = db_path
|
||||
out["table"] = table
|
||||
return out
|
||||
@@ -0,0 +1,153 @@
|
||||
"""Tests para build_eda_render_ctx.
|
||||
|
||||
Self-contained: crea un DuckDB temporal pequeño con una columna fecha, varias
|
||||
numericas y un par lat/lon, construye un TableProfile minimo a mano (con la forma
|
||||
de columnas del grupo `eda`: name / inferred_type / numeric.{min,max} /
|
||||
semantic_type) y verifica que el ctx producido contiene las claves de datos que
|
||||
consumen los capitulos del AutomaticEDA.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
# El test importa funciones del registry como una app del registry: inserta el
|
||||
# directorio raiz `python/functions` en sys.path y luego `from datascience import`.
|
||||
_FUNCTIONS_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
if _FUNCTIONS_ROOT not in sys.path:
|
||||
sys.path.insert(0, _FUNCTIONS_ROOT)
|
||||
|
||||
import duckdb # noqa: E402
|
||||
|
||||
from datascience import build_eda_render_ctx # noqa: E402
|
||||
|
||||
_TABLE = "ventas_geo"
|
||||
# Filas: fecha creciente, 2 columnas numericas (ventas, unidades) y un par lat/lon
|
||||
# (Madrid -> lat ~40, lon ~-3, dentro de [-90,90] y [-180,180]).
|
||||
_ROWS = [
|
||||
("2024-01-01", 1200.5, 12, 40.41, -3.70),
|
||||
("2024-01-02", 980.0, 9, 41.38, 2.17),
|
||||
("2024-01-03", 1500.25, 15, 37.39, -5.99),
|
||||
("2024-01-04", 1100.0, 11, 39.47, -0.38),
|
||||
("2024-01-05", 1750.75, 18, 43.26, -2.93),
|
||||
]
|
||||
|
||||
|
||||
def _make_db(tmp_path):
|
||||
"""Crea un DuckDB temporal con la tabla de prueba y devuelve su ruta."""
|
||||
db_path = os.path.join(str(tmp_path), "eda_ctx.duckdb")
|
||||
con = duckdb.connect(db_path)
|
||||
try:
|
||||
con.execute(
|
||||
f'CREATE TABLE "{_TABLE}" '
|
||||
"(fecha DATE, ventas DOUBLE, unidades INTEGER, lat DOUBLE, lon DOUBLE)"
|
||||
)
|
||||
con.executemany(
|
||||
f'INSERT INTO "{_TABLE}" VALUES (?, ?, ?, ?, ?)', _ROWS
|
||||
)
|
||||
finally:
|
||||
con.close()
|
||||
return db_path
|
||||
|
||||
|
||||
def _profile_with_date():
|
||||
"""TableProfile minimo con columna fecha + numericas + lat/lon."""
|
||||
return {
|
||||
"columns": [
|
||||
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
|
||||
{
|
||||
"name": "ventas",
|
||||
"inferred_type": "numeric",
|
||||
"semantic_type": "decimal",
|
||||
"numeric": {"min": 980.0, "max": 1750.75},
|
||||
},
|
||||
{
|
||||
"name": "unidades",
|
||||
"inferred_type": "numeric",
|
||||
"semantic_type": "integer",
|
||||
"numeric": {"min": 9, "max": 18},
|
||||
},
|
||||
{
|
||||
"name": "lat",
|
||||
"inferred_type": "numeric",
|
||||
"semantic_type": "decimal",
|
||||
"numeric": {"min": 37.39, "max": 43.26},
|
||||
},
|
||||
{
|
||||
"name": "lon",
|
||||
"inferred_type": "numeric",
|
||||
"semantic_type": "decimal",
|
||||
"numeric": {"min": -5.99, "max": 2.17},
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def _profile_without_date():
|
||||
"""Mismo perfil pero SIN columna temporal (solo numericas)."""
|
||||
prof = _profile_with_date()
|
||||
prof["columns"] = [c for c in prof["columns"] if c["name"] != "fecha"]
|
||||
return prof
|
||||
|
||||
|
||||
def test_db_path_y_table_en_ctx(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
|
||||
assert ctx["db_path"] == db_path
|
||||
assert ctx["table"] == _TABLE
|
||||
|
||||
|
||||
def test_raw_numeric_con_columnas_numericas(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
|
||||
raw = ctx["raw_numeric"]
|
||||
# Las 4 columnas numericas (ventas, unidades, lat, lon), listas no vacias y
|
||||
# alineadas por fila (misma longitud == nº de filas).
|
||||
for col in ("ventas", "unidades", "lat", "lon"):
|
||||
assert col in raw
|
||||
assert len(raw[col]) == len(_ROWS)
|
||||
assert raw["ventas"][0] == 1200.5
|
||||
assert raw["unidades"][0] == 12.0 # int promovido a float
|
||||
|
||||
|
||||
def test_timeseries_raw_con_fecha(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
|
||||
ts = ctx["timeseries_raw"]
|
||||
assert ts["time_col"] == "fecha"
|
||||
assert len(ts["t"]) == len(_ROWS) # fechas ISO no vacias
|
||||
# Las numericas aparecen como series paralelas a t.
|
||||
for col in ("ventas", "unidades", "lat", "lon"):
|
||||
assert col in ts["series"]
|
||||
assert len(ts["series"][col]) == len(_ROWS)
|
||||
|
||||
|
||||
def test_geo_points_con_latlon(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date())
|
||||
geo = ctx["geo_points"]
|
||||
assert len(geo["lats"]) == len(_ROWS)
|
||||
assert len(geo["lons"]) == len(_ROWS)
|
||||
# Listas alineadas, ya floats, leidas de raw_numeric.
|
||||
assert geo["lats"][0] == 40.41
|
||||
assert geo["lons"][0] == -3.70
|
||||
|
||||
|
||||
def test_sin_fecha_no_hay_timeseries(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_without_date())
|
||||
assert "timeseries_raw" not in ctx
|
||||
# raw_numeric y geo_points siguen presentes (no dependen de la fecha).
|
||||
assert "raw_numeric" in ctx
|
||||
assert "geo_points" in ctx
|
||||
|
||||
|
||||
def test_base_ctx_preservado(tmp_path):
|
||||
db_path = _make_db(tmp_path)
|
||||
base = {"dataset_name": "ventas_geo_demo", "source_origin": "test"}
|
||||
ctx = build_eda_render_ctx(db_path, _TABLE, _profile_with_date(), base_ctx=base)
|
||||
# Las claves de presentacion del base_ctx no se pisan.
|
||||
assert ctx["dataset_name"] == "ventas_geo_demo"
|
||||
assert ctx["source_origin"] == "test"
|
||||
# Y las de datos se añaden encima.
|
||||
assert ctx["db_path"] == db_path
|
||||
assert "raw_numeric" in ctx
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: build_geo_scatter
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def build_geo_scatter(lats: list, lons: list, max_points: int = 2000) -> dict"
|
||||
description: "Prepara los datos de un scatter geografico en proyeccion equirectangular para el grupo eda. Empareja lats/lons por indice, descarta pares None/NaN/inf/bool o fuera de rango (lat en [-90,90], lon en [-180,180]) y aplica downsampling DETERMINISTA por paso fijo (pairs[::step]) cuando hay mas pares validos que max_points, para no saturar el PDF/PPTX en moviles. Devuelve los puntos en orden [lon, lat] listos para ax.scatter, el bbox, el aspect 1/cos(centroid_lat) clampado a [0.3,5.0] y un pad sugerido (~5% del rango con suelo minimo). Lectura defensiva; NUNCA lanza ni dibuja: el capitulo se encarga de matplotlib."
|
||||
tags: [eda, geospatial, datascience, scatter, map, downsample, equirectangular, profiling]
|
||||
params:
|
||||
- name: lats
|
||||
desc: "Lista (o tupla) de latitudes en grados, paralela a lons. Se empareja por indice. Un valor None, NaN, infinito, bool o fuera de [-90,90] descarta ese par. Lectura defensiva."
|
||||
- name: lons
|
||||
desc: "Lista (o tupla) de longitudes en grados, paralela a lats. Un valor None, NaN, infinito, bool o fuera de [-180,180] descarta ese par."
|
||||
- name: max_points
|
||||
desc: "Tope de puntos a devolver (default 2000). Si los pares validos superan el tope, se hace downsampling determinista por paso fijo step=ceil(n_total/max_points) tomando pairs[::step] (NO aleatorio, reproducible). Un valor no entero o <=0 desactiva el downsampling."
|
||||
output: "Dict listo para dibujar: {points: [[lon, lat], ...] en orden x=lon/y=lat para ax.scatter; n_total: pares validos antes del downsample (int); n_shown: puntos devueltos tras el downsample (int); downsampled: bool (n_shown<n_total); bbox: {lat_min, lat_max, lon_min, lon_max} o None si no hay puntos; aspect: 1/cos(centroid_lat) clampado a [0.3,5.0] para no estirar la proyeccion equirectangular; pad: {lon, lat} ~5% del rango respectivo con suelo minimo 0.01 grados}. Si no hay pares validos: points=[], n_total=0, n_shown=0, downsampled=False, bbox=None, aspect=1.0, pad={lon:0.0, lat:0.0}."
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: []
|
||||
tested: true
|
||||
tests: ["test_geo_scatter_nube_espana", "test_downsampling_determinista_y_reproducible", "test_listas_vacias_no_lanza", "test_un_solo_punto_pad_minimo_y_aspect_finito", "test_filtra_none_nan_y_fuera_de_rango", "test_latitud_alta_aspect_clamped"]
|
||||
test_file_path: "python/functions/datascience/build_geo_scatter_test.py"
|
||||
file_path: "python/functions/datascience/build_geo_scatter.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience.build_geo_scatter import build_geo_scatter
|
||||
|
||||
# Nube de coordenadas (lat, lon) alrededor de Madrid:
|
||||
lats = [40.0, 41.0, 39.0, 40.5]
|
||||
lons = [-3.7, -3.0, -4.0, -3.5]
|
||||
geo = build_geo_scatter(lats, lons, max_points=2000)
|
||||
|
||||
print(geo["points"][0]) # [-3.7, 40.0] -> orden [x=lon, y=lat]
|
||||
print(geo["bbox"]) # {'lat_min': 39.0, 'lat_max': 41.0, 'lon_min': -4.0, 'lon_max': -3.0}
|
||||
print(round(geo["aspect"], 3)) # 1.308 -> ensancha el eje x en latitudes medias
|
||||
print(geo["pad"]) # {'lon': 0.05, 'lat': 0.1} -> margen ~5%
|
||||
|
||||
# El capitulo dibuja con matplotlib (esta funcion NO dibuja):
|
||||
# xs = [p[0] for p in geo["points"]]; ys = [p[1] for p in geo["points"]]
|
||||
# ax.scatter(xs, ys); ax.set_aspect(geo["aspect"])
|
||||
# ax.set_xlim(geo["bbox"]["lon_min"] - geo["pad"]["lon"], geo["bbox"]["lon_max"] + geo["pad"]["lon"])
|
||||
# ax.set_ylim(geo["bbox"]["lat_min"] - geo["pad"]["lat"], geo["bbox"]["lat_max"] + geo["pad"]["lat"])
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
- Usala antes de dibujar un scatter geografico (mapa de puntos en proyeccion equirectangular) en el capitulo geospatial de `AutomaticEDA`: limpia los pares de coordenadas, los reduce a un tamano razonable para el PDF/PPTX y te da bbox, aspect y pad listos para fijar los ejes.
|
||||
- Cuando tengas dos columnas de lat/lon ya extraidas y quieras un punto de entrada determinista (mismo dataset -> mismo dibujo) que no sature el documento en moviles.
|
||||
- Cuando necesites el aspect correcto para que un grado de longitud no se vea estirado respecto a uno de latitud (integridad visual, Tufte) sin calcularlo a mano.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Funcion pura, sin I/O y determinista. NO dibuja: solo PREPARA los datos; el capitulo se encarga de matplotlib. Lectura defensiva: pares con None/NaN/inf/bool o coordenadas fuera de rango se descartan en silencio y NUNCA lanza.
|
||||
- El downsampling es DETERMINISTA por paso fijo (`step = ceil(n_total / max_points)`, `pairs[::step]`), NO aleatorio: la misma entrada produce siempre la misma salida (reproducible en tests). El primer punto mostrado es siempre el primer par valido. No es un muestreo uniforme aleatorio — es un barrido regular del orden de entrada.
|
||||
- `points` va en orden `[lon, lat]` (x, y), no `[lat, lon]`: pasalo directo a `ax.scatter(xs, ys)` sin invertir. Confundir el orden espeja el mapa.
|
||||
- `aspect = 1/cos(centroid_lat)` se clampa a `[0.3, 5.0]`. En latitudes altas `cos -> 0` y el valor real explota: por encima de ~78 grados el aspect queda fijado en 5.0. Si el centroide cae justo en un polo (`+-90`) se usa el clamp en vez de dividir por cero.
|
||||
- `pad` es ~5% del rango de cada eje con un suelo minimo de `0.01` grados: con un solo punto o todos iguales (rango 0) el pad cae al suelo para que el punto no quede en una linea. En el caso sin puntos validos el pad es `{lon:0.0, lat:0.0}` y `bbox` es `None`.
|
||||
- `bbox`, `aspect` y `pad` se calculan sobre los puntos YA mostrados (tras el downsample), de modo que los ejes encajan exactamente con lo que se dibuja.
|
||||
@@ -0,0 +1,153 @@
|
||||
"""build_geo_scatter — prepare points for a geographic scatter (EDA `geospatial`).
|
||||
|
||||
Pure function: no I/O, deterministic. Takes two parallel lists of latitudes and
|
||||
longitudes and returns the data a caller needs to draw a geographic scatter in an
|
||||
equirectangular projection: cleaned points in [lon, lat] order, a bounding box, a
|
||||
projection aspect ratio and a suggested axis padding.
|
||||
|
||||
It NEVER draws anything (no matplotlib) — the chapter that consumes this output is
|
||||
responsible for the rendering. Reading is defensive throughout and the function
|
||||
NEVER raises: malformed pairs (None, NaN, infinity or out-of-range coordinates)
|
||||
are silently dropped and an empty/valid result is always returned.
|
||||
|
||||
To keep the rendered PDF/PPTX light on phones, when the number of valid pairs
|
||||
exceeds `max_points` the points are down-sampled DETERMINISTICALLY by a fixed
|
||||
step (`pairs[::step]`), never randomly, so the result is reproducible.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
# Minimum axis padding (in degrees) so a single point or a zero-range cloud is
|
||||
# never drawn glued to the axis border (it would collapse to a line).
|
||||
_MIN_PAD = 0.01
|
||||
|
||||
# Aspect ratio clamp. 1/cos(lat) blows up near the poles; clamp keeps the render
|
||||
# sane (Tufte: do not let the projection stretch the cloud out of proportion).
|
||||
_ASPECT_MIN = 0.3
|
||||
_ASPECT_MAX = 5.0
|
||||
|
||||
|
||||
def _coord(value):
|
||||
"""Coerce to a finite float defensively; return None for invalid coordinates.
|
||||
|
||||
bool is a subclass of int, but a real latitude/longitude is never a bool, so
|
||||
True/False are treated as missing instead of coercing to 1.0/0.0. NaN and
|
||||
+/-infinity are never valid coordinates either.
|
||||
"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
coord = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if math.isnan(coord) or math.isinf(coord):
|
||||
return None
|
||||
return coord
|
||||
|
||||
|
||||
def build_geo_scatter(lats: list, lons: list, max_points: int = 2000) -> dict:
|
||||
"""Prepare the data for a geographic scatter in equirectangular projection.
|
||||
|
||||
Pairs `lats` and `lons` by index, drops invalid pairs, optionally
|
||||
down-samples deterministically, and derives the geometry (bbox, aspect, pad)
|
||||
a caller needs to draw the cloud. No raw rendering is performed.
|
||||
|
||||
Args:
|
||||
lats: List (or tuple) of latitudes in degrees. Paired by index with
|
||||
`lons`. A value that is None, NaN, infinite, bool or outside
|
||||
[-90, 90] discards that pair. Read defensively.
|
||||
lons: List (or tuple) of longitudes in degrees, parallel to `lats`. A
|
||||
value outside [-180, 180] (or None/NaN/inf/bool) discards that pair.
|
||||
max_points: Cap on the number of points returned. When the number of
|
||||
valid pairs exceeds this cap, the points are down-sampled by a fixed
|
||||
step `ceil(n_total / max_points)` taking `pairs[::step]` — DETERMINISTIC,
|
||||
not random, so the output is reproducible. A non-positive or non-int
|
||||
value disables down-sampling.
|
||||
|
||||
Returns:
|
||||
Dict ready for a caller's ax.scatter:
|
||||
{points: [[lon, lat], ...] (x=lon, y=lat order), n_total: valid pairs
|
||||
before down-sampling, n_shown: points returned, downsampled: bool,
|
||||
bbox: {lat_min, lat_max, lon_min, lon_max} or None, aspect: 1/cos(centroid
|
||||
lat) clamped to [0.3, 5.0], pad: {lon, lat} ~5% of each range with a small
|
||||
floor}. When there are no valid pairs returns points=[], n_total=0,
|
||||
n_shown=0, downsampled=False, bbox=None, aspect=1.0, pad={lon:0.0, lat:0.0}.
|
||||
"""
|
||||
pairs = [] # each item is (lon, lat) — already in [x, y] order
|
||||
if isinstance(lats, (list, tuple)) and isinstance(lons, (list, tuple)):
|
||||
n = min(len(lats), len(lons))
|
||||
for i in range(n):
|
||||
lat = _coord(lats[i])
|
||||
lon = _coord(lons[i])
|
||||
if lat is None or lon is None:
|
||||
continue
|
||||
if lat < -90.0 or lat > 90.0:
|
||||
continue
|
||||
if lon < -180.0 or lon > 180.0:
|
||||
continue
|
||||
pairs.append((lon, lat))
|
||||
|
||||
n_total = len(pairs)
|
||||
if n_total == 0:
|
||||
return {
|
||||
"points": [],
|
||||
"n_total": 0,
|
||||
"n_shown": 0,
|
||||
"downsampled": False,
|
||||
"bbox": None,
|
||||
"aspect": 1.0,
|
||||
"pad": {"lon": 0.0, "lat": 0.0},
|
||||
}
|
||||
|
||||
# Deterministic down-sampling by a fixed step. Reproducible: same input ->
|
||||
# same output, no randomness.
|
||||
if (
|
||||
isinstance(max_points, int)
|
||||
and not isinstance(max_points, bool)
|
||||
and max_points > 0
|
||||
and n_total > max_points
|
||||
):
|
||||
step = math.ceil(n_total / max_points)
|
||||
sampled = pairs[::step]
|
||||
else:
|
||||
sampled = pairs
|
||||
|
||||
points = [[lon, lat] for (lon, lat) in sampled]
|
||||
n_shown = len(points)
|
||||
downsampled = n_shown < n_total
|
||||
|
||||
lons_s = [p[0] for p in sampled]
|
||||
lats_s = [p[1] for p in sampled]
|
||||
lon_min, lon_max = min(lons_s), max(lons_s)
|
||||
lat_min, lat_max = min(lats_s), max(lats_s)
|
||||
bbox = {
|
||||
"lat_min": lat_min,
|
||||
"lat_max": lat_max,
|
||||
"lon_min": lon_min,
|
||||
"lon_max": lon_max,
|
||||
}
|
||||
|
||||
# Aspect for an equirectangular projection: stretch the x axis by 1/cos(lat)
|
||||
# at the cloud centroid so a degree of longitude reads at its real width.
|
||||
centroid_lat = sum(lats_s) / len(lats_s)
|
||||
cos_lat = math.cos(math.radians(centroid_lat))
|
||||
if cos_lat < 1e-12: # centroid at (or numerically at) a pole
|
||||
aspect = _ASPECT_MAX
|
||||
else:
|
||||
aspect = 1.0 / cos_lat
|
||||
aspect = max(_ASPECT_MIN, min(_ASPECT_MAX, aspect))
|
||||
|
||||
# Padding ~5% of each range, with a small floor so a zero-range cloud (single
|
||||
# point / all identical) still gets a non-zero margin.
|
||||
pad_lon = max(0.05 * (lon_max - lon_min), _MIN_PAD)
|
||||
pad_lat = max(0.05 * (lat_max - lat_min), _MIN_PAD)
|
||||
|
||||
return {
|
||||
"points": points,
|
||||
"n_total": n_total,
|
||||
"n_shown": n_shown,
|
||||
"downsampled": downsampled,
|
||||
"bbox": bbox,
|
||||
"aspect": aspect,
|
||||
"pad": {"lon": pad_lon, "lat": pad_lat},
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Tests para build_geo_scatter."""
|
||||
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from build_geo_scatter import build_geo_scatter
|
||||
|
||||
# Keys that a non-empty result dict must always contain.
|
||||
_EXPECTED_KEYS = {
|
||||
"points", "n_total", "n_shown", "downsampled", "bbox", "aspect", "pad",
|
||||
}
|
||||
|
||||
|
||||
def test_geo_scatter_nube_espana():
|
||||
"""Golden: nube en Espana -> points en orden [lon, lat], bbox, aspect>1, pad 5%."""
|
||||
# Cuatro puntos alrededor de Madrid (lat ~40, lon negativo).
|
||||
lats = [40.0, 41.0, 39.0, 40.5]
|
||||
lons = [-3.7, -3.0, -4.0, -3.5]
|
||||
r = build_geo_scatter(lats, lons)
|
||||
|
||||
assert set(r.keys()) == _EXPECTED_KEYS
|
||||
|
||||
# points en orden [x=lon, y=lat]: primer elemento lon (negativo), segundo lat (~40).
|
||||
assert r["points"] == [[-3.7, 40.0], [-3.0, 41.0], [-4.0, 39.0], [-3.5, 40.5]]
|
||||
for lon, lat in r["points"]:
|
||||
assert lon < 0.0 # longitudes de Espana son negativas
|
||||
assert 36.0 < lat < 44.0 # latitudes peninsulares
|
||||
|
||||
# Sin downsampling: 4 < 2000.
|
||||
assert r["n_total"] == 4
|
||||
assert r["n_shown"] == 4
|
||||
assert r["downsampled"] is False
|
||||
|
||||
# bbox correcto.
|
||||
assert r["bbox"] == {
|
||||
"lat_min": 39.0, "lat_max": 41.0,
|
||||
"lon_min": -4.0, "lon_max": -3.0,
|
||||
}
|
||||
|
||||
# aspect = 1/cos(centroid_lat); centroid = 40.125 -> ~1.31 > 1.
|
||||
centroid_lat = (40.0 + 41.0 + 39.0 + 40.5) / 4.0
|
||||
expected_aspect = 1.0 / math.cos(math.radians(centroid_lat))
|
||||
assert r["aspect"] > 1.0
|
||||
assert abs(r["aspect"] - expected_aspect) < 1e-9
|
||||
assert abs(r["aspect"] - 1.305) < 0.02 # cos(40) ~ 0.77
|
||||
|
||||
# pad 5% del rango (lon_range=1.0 -> 0.05 ; lat_range=2.0 -> 0.1).
|
||||
assert abs(r["pad"]["lon"] - 0.05) < 1e-9
|
||||
assert abs(r["pad"]["lat"] - 0.10) < 1e-9
|
||||
|
||||
|
||||
def test_downsampling_determinista_y_reproducible():
|
||||
"""Golden: 5000 puntos, max_points=2000 -> n_shown<=2000, downsampled, reproducible."""
|
||||
lats = [40.0 + (i % 100) * 0.01 for i in range(5000)]
|
||||
lons = [-3.0 - (i % 100) * 0.01 for i in range(5000)]
|
||||
|
||||
r1 = build_geo_scatter(lats, lons, max_points=2000)
|
||||
|
||||
assert r1["n_total"] == 5000
|
||||
assert r1["n_shown"] <= 2000
|
||||
assert r1["downsampled"] is True
|
||||
# step = ceil(5000/2000) = 3 -> len(pairs[::3]) = 1667.
|
||||
assert r1["n_shown"] == 1667
|
||||
|
||||
# Determinista: dos llamadas con la misma entrada dan exactamente lo mismo.
|
||||
r2 = build_geo_scatter(lats, lons, max_points=2000)
|
||||
assert r1 == r2
|
||||
assert r1["points"] == r2["points"]
|
||||
|
||||
# El primer punto del downsample es el primer par valido (step parte de 0).
|
||||
assert r1["points"][0] == [lons[0], lats[0]]
|
||||
|
||||
|
||||
def test_listas_vacias_no_lanza():
|
||||
"""Edge: listas vacias / None -> points [] sin lanzar."""
|
||||
r = build_geo_scatter([], [])
|
||||
assert r["points"] == []
|
||||
assert r["n_total"] == 0
|
||||
assert r["n_shown"] == 0
|
||||
assert r["downsampled"] is False
|
||||
assert r["bbox"] is None
|
||||
assert r["aspect"] == 1.0
|
||||
assert r["pad"] == {"lon": 0.0, "lat": 0.0}
|
||||
|
||||
# None como entrada tampoco lanza.
|
||||
assert build_geo_scatter(None, None)["points"] == []
|
||||
assert build_geo_scatter([40.0], None)["n_total"] == 0
|
||||
assert build_geo_scatter(None, [-3.0])["n_total"] == 0
|
||||
|
||||
|
||||
def test_un_solo_punto_pad_minimo_y_aspect_finito():
|
||||
"""Edge: un solo punto -> pad minimo no cero, bbox degenerado, aspect finito."""
|
||||
r = build_geo_scatter([40.0], [-3.7])
|
||||
|
||||
assert r["n_total"] == 1
|
||||
assert r["n_shown"] == 1
|
||||
assert r["points"] == [[-3.7, 40.0]]
|
||||
assert r["downsampled"] is False
|
||||
assert r["bbox"] == {
|
||||
"lat_min": 40.0, "lat_max": 40.0,
|
||||
"lon_min": -3.7, "lon_max": -3.7,
|
||||
}
|
||||
# rango 0 -> pad cae al floor minimo (no cero).
|
||||
assert r["pad"]["lon"] == 0.01
|
||||
assert r["pad"]["lat"] == 0.01
|
||||
# aspect finito y dentro del clamp.
|
||||
assert math.isfinite(r["aspect"])
|
||||
assert 0.3 <= r["aspect"] <= 5.0
|
||||
|
||||
|
||||
def test_filtra_none_nan_y_fuera_de_rango():
|
||||
"""Edge: pares con None/NaN/fuera de rango se descartan por indice."""
|
||||
nan = float("nan")
|
||||
inf = float("inf")
|
||||
# i=0 i=1 i=2 i=3 i=4 i=5 i=6
|
||||
lats = [40.0, None, nan, 200.0, 41.0, 39.0, inf]
|
||||
lons = [-3.0, -3.5, -3.6, -3.7, 999.0, -4.0, -2.0]
|
||||
r = build_geo_scatter(lats, lons)
|
||||
|
||||
# Validos solo i=0 (40,-3.0) e i=5 (39,-4.0):
|
||||
# i=1 lat None, i=2 lat NaN, i=3 lat 200 fuera de rango,
|
||||
# i=4 lon 999 fuera de rango, i=6 lat inf.
|
||||
assert r["n_total"] == 2
|
||||
assert r["points"] == [[-3.0, 40.0], [-4.0, 39.0]]
|
||||
assert r["bbox"] == {
|
||||
"lat_min": 39.0, "lat_max": 40.0,
|
||||
"lon_min": -4.0, "lon_max": -3.0,
|
||||
}
|
||||
|
||||
|
||||
def test_latitud_alta_aspect_clamped():
|
||||
"""Edge: latitudes ~85 -> aspect clamped <= 5.0."""
|
||||
r = build_geo_scatter([85.0, 85.0, 84.0], [10.0, 11.0, 9.0])
|
||||
# cos(~84.7) ~ 0.093 -> 1/0.093 ~ 10.7 -> clamp a 5.0.
|
||||
assert r["aspect"] <= 5.0
|
||||
assert r["aspect"] == 5.0
|
||||
assert math.isfinite(r["aspect"])
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
id: categorical_cardinality_block_py_datascience
|
||||
name: categorical_cardinality_block
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def categorical_cardinality_block(cat: dict, n_rows: int) -> dict"
|
||||
description: "Deriva métricas de cardinalidad listas para renderizar a partir de la salida de summarize_categorical para UNA columna categórica más el número total de filas. Calcula pct_distinct, entropy_max=log2(n_distinct), entropy_norm (recortada a [0,1]), n_singletons (sobre el top visible) y los flags id_like / dominated. NO recalcula la entropía ni reimplementa summarize_categorical: la consume. Estilo dict-no-throw del grupo eda — nunca lanza."
|
||||
tags: [eda, categorical, cardinality, entropy, profiling, datascience, pure]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: [math]
|
||||
example: |
|
||||
from categorical_cardinality_block import categorical_cardinality_block
|
||||
cat = {"top": [{"value": "a", "count": 5, "pct": 0.5}], "mode": "a",
|
||||
"mode_pct": 0.5, "n_distinct": 4, "entropy": 1.685, "imbalance": 5.0,
|
||||
"len_min": 1, "len_mean": 1.0, "len_max": 1}
|
||||
block = categorical_cardinality_block(cat, n_rows=10)
|
||||
tested: true
|
||||
tests:
|
||||
- "test_normal_case"
|
||||
- "test_empty_cat_does_not_raise"
|
||||
- "test_none_cat_does_not_raise"
|
||||
- "test_n_rows_zero_no_zero_division"
|
||||
- "test_id_like_when_distinct_near_rows"
|
||||
- "test_dominated_when_mode_pct_high"
|
||||
- "test_mode_pct_fallback_from_top_fraction"
|
||||
- "test_n_singletons_partial_when_top_truncated"
|
||||
- "test_single_distinct_value_entropy_norm_none"
|
||||
test_file_path: "python/functions/datascience/categorical_cardinality_block_test.py"
|
||||
file_path: "python/functions/datascience/categorical_cardinality_block.py"
|
||||
params:
|
||||
- name: cat
|
||||
desc: "Dict producido por summarize_categorical para UNA columna categórica. Claves leídas (todas opcionales, lectura defensiva): top (list de {value,count,pct}), mode, mode_pct (puede faltar), n_distinct, entropy (Shannon en bits), imbalance, len_min, len_mean, len_max. None o no-dict se tratan como {}."
|
||||
- name: n_rows
|
||||
desc: "Número total de filas del dataset. Usado para pct_distinct. Si es 0 o None, pct_distinct sale None (sin ZeroDivisionError)."
|
||||
output: "Dict con exactamente 16 claves, todas siempre presentes: n_distinct, n_rows, pct_distinct, entropy, entropy_max, entropy_norm, mode, mode_pct, imbalance, n_singletons, n_singletons_partial, len_min, len_mean, len_max, id_like, dominated. Valores None/False cuando no son derivables; la función nunca lanza. pct_distinct en escala 0-100. entropy_max=log2(n_distinct) (0.0 si n_distinct in {0,1}). entropy_norm=entropy/entropy_max recortada a [0,1]. n_singletons = nº de elementos de top con count==1 (None si top vacío). n_singletons_partial=True si n_distinct>len(top). id_like=pct_distinct>=99. dominated=mode_pct>=90."
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from categorical_cardinality_block import categorical_cardinality_block
|
||||
|
||||
# Salida típica de summarize_categorical para una columna, con n_rows del dataset.
|
||||
cat = {
|
||||
"top": [
|
||||
{"value": "a", "count": 5, "pct": 0.5},
|
||||
{"value": "b", "count": 3, "pct": 0.3},
|
||||
{"value": "c", "count": 1, "pct": 0.1},
|
||||
{"value": "d", "count": 1, "pct": 0.1},
|
||||
],
|
||||
"mode": "a",
|
||||
"mode_pct": 0.5,
|
||||
"n_distinct": 4,
|
||||
"entropy": 1.685, # Shannon en bits (<= log2(4) = 2.0)
|
||||
"imbalance": 5.0,
|
||||
"len_min": 1, "len_mean": 1.0, "len_max": 1,
|
||||
}
|
||||
|
||||
categorical_cardinality_block(cat, n_rows=10)
|
||||
# {
|
||||
# "n_distinct": 4, "n_rows": 10,
|
||||
# "pct_distinct": 40.0, # 4 / 10 * 100
|
||||
# "entropy": 1.685,
|
||||
# "entropy_max": 2.0, # log2(4)
|
||||
# "entropy_norm": 0.8425, # 1.685 / 2.0, recortado a [0,1]
|
||||
# "mode": "a", "mode_pct": 0.5,
|
||||
# "imbalance": 5.0,
|
||||
# "n_singletons": 2, # c y d con count == 1
|
||||
# "n_singletons_partial": False, # top cubre los 4 distintos
|
||||
# "len_min": 1, "len_mean": 1.0, "len_max": 1,
|
||||
# "id_like": False, # pct_distinct 40 < 99
|
||||
# "dominated": False, # mode_pct 0.5 < 90
|
||||
# }
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Úsala justo después de `summarize_categorical`, cuando vayas a renderizar el
|
||||
bloque de cardinalidad de una columna categórica en un EDA: necesitas el ratio
|
||||
de valores distintos (`pct_distinct`), la entropía normalizada al rango `[0,1]`
|
||||
para comparar columnas con cardinalidades distintas, el conteo de singletons, y
|
||||
las banderas heurísticas `id_like` (la columna parece un identificador) y
|
||||
`dominated` (una sola categoría domina). Pásale el dict crudo de
|
||||
`summarize_categorical` para esa columna y el `n_rows` total del dataset. No
|
||||
reimplementa nada: solo deriva métricas de presentación a partir de lo ya
|
||||
calculado.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`mode_pct` se pasa tal cual viene en `cat`.** `summarize_categorical`
|
||||
produce `mode_pct` como **fracción** (0–1), no como porcentaje. El flag
|
||||
`dominated` compara `mode_pct >= 90.0`, así que con la salida cruda de
|
||||
`summarize_categorical` (fracciones) `dominated` no se dispara: aliméntalo con
|
||||
`mode_pct` en escala 0–100 si quieres usar esa bandera. Solo el camino de
|
||||
*fallback* (cuando `cat` no trae `mode_pct` y se deriva de `top[0]['pct']`)
|
||||
normaliza una fracción `<= 1` multiplicándola por 100.
|
||||
- **`n_singletons` solo cubre el `top` visible.** Si `summarize_categorical` se
|
||||
llamó con `top_k` pequeño, hay valores fuera del top; en ese caso
|
||||
`n_singletons_partial` es `True` para avisar de que el conteo es parcial.
|
||||
- **`pct_distinct` es `None` si `n_rows` es 0 o `None`** (no lanza
|
||||
`ZeroDivisionError`); por tanto `id_like` queda `False` en ese caso.
|
||||
- **`entropy_norm` es `None` cuando `entropy_max <= 0`** (columna constante,
|
||||
`n_distinct in {0,1}`): no hay división por cero y no se inventa un 0/1.
|
||||
- **No recalcula la entropía.** Si `cat['entropy']` es incoherente con
|
||||
`n_distinct`, `entropy_norm` se recorta a `[0,1]` pero el valor de entrada no
|
||||
se corrige.
|
||||
- **`bool` no cuenta como número.** Un `True`/`False` en una clave numérica de
|
||||
`cat` se trata como ausente (`None`), por la guarda defensiva.
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Pure EDA helper: cardinality metrics block from a `summarize_categorical` output.
|
||||
|
||||
Part of the `eda` capability group. Consumes the per-column dict produced by
|
||||
``summarize_categorical`` (for a single categorical/text column) plus the total
|
||||
row count of the dataset and derives render-ready cardinality metrics: distinct
|
||||
ratio, normalized entropy, singleton count, and the ``id_like`` / ``dominated``
|
||||
flags.
|
||||
|
||||
It does NOT recompute the entropy nor reimplement ``summarize_categorical`` — it
|
||||
only reads that function's output. Dict-no-throw style of the `eda` group: it
|
||||
never raises. Missing or malformed inputs yield ``None``/``False``/``0`` for the
|
||||
affected keys, never an exception. Stdlib only (``math.log2``).
|
||||
"""
|
||||
|
||||
from math import log2
|
||||
|
||||
|
||||
def _num(value):
|
||||
"""Return ``value`` unchanged if it is a real (non-bool) number, else ``None``.
|
||||
|
||||
``bool`` is rejected on purpose: in Python ``True`` is an ``int`` but it is
|
||||
never a meaningful count/ratio here.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def categorical_cardinality_block(cat: dict, n_rows: int) -> dict:
|
||||
"""Derive cardinality metrics for one categorical column.
|
||||
|
||||
Args:
|
||||
cat: The per-column dict produced by ``summarize_categorical`` for a
|
||||
single categorical/text column. Expected (all optional, read
|
||||
defensively) keys: ``top`` (list of ``{value, count, pct}``),
|
||||
``mode``, ``mode_pct``, ``n_distinct``, ``entropy`` (Shannon, bits),
|
||||
``imbalance``, ``len_min``, ``len_mean``, ``len_max``. ``None`` or a
|
||||
non-dict is treated as ``{}``.
|
||||
n_rows: Total number of rows in the dataset (used for ``pct_distinct``).
|
||||
|
||||
Returns:
|
||||
Dict with exactly these keys, every one always present:
|
||||
``n_distinct``, ``n_rows``, ``pct_distinct``, ``entropy``,
|
||||
``entropy_max``, ``entropy_norm``, ``mode``, ``mode_pct``,
|
||||
``imbalance``, ``n_singletons``, ``n_singletons_partial``, ``len_min``,
|
||||
``len_mean``, ``len_max``, ``id_like``, ``dominated``. Values are
|
||||
``None``/``False`` when not derivable; the function never raises.
|
||||
"""
|
||||
cat = cat if isinstance(cat, dict) else {}
|
||||
|
||||
# --- passthroughs (numeric-validated, type preserved) ---
|
||||
n_distinct = _num(cat.get("n_distinct"))
|
||||
n_rows_out = _num(n_rows)
|
||||
entropy = _num(cat.get("entropy"))
|
||||
imbalance = _num(cat.get("imbalance"))
|
||||
len_min = _num(cat.get("len_min"))
|
||||
len_mean = _num(cat.get("len_mean"))
|
||||
len_max = _num(cat.get("len_max"))
|
||||
mode = cat.get("mode") # any value (or None); passthrough as-is
|
||||
|
||||
# --- pct_distinct ---
|
||||
if n_distinct is None or n_rows_out is None or n_rows_out == 0:
|
||||
pct_distinct = None
|
||||
else:
|
||||
pct_distinct = n_distinct / n_rows_out * 100.0
|
||||
|
||||
# --- entropy_max = log2(n_distinct) ---
|
||||
if n_distinct is None:
|
||||
entropy_max = None
|
||||
elif n_distinct > 1:
|
||||
entropy_max = log2(n_distinct)
|
||||
else: # n_distinct in {0, 1}
|
||||
entropy_max = 0.0
|
||||
|
||||
# --- entropy_norm = entropy / entropy_max, clipped to [0, 1] ---
|
||||
if entropy_max is not None and entropy_max > 0 and entropy is not None:
|
||||
entropy_norm = entropy / entropy_max
|
||||
entropy_norm = max(0.0, min(1.0, entropy_norm))
|
||||
else:
|
||||
entropy_norm = None
|
||||
|
||||
# --- mode_pct: prefer cat['mode_pct']; else derive from top[0].pct ---
|
||||
mode_pct = _num(cat.get("mode_pct"))
|
||||
top = cat.get("top")
|
||||
has_top = isinstance(top, (list, tuple)) and len(top) > 0
|
||||
if mode_pct is None and has_top:
|
||||
first = top[0]
|
||||
if isinstance(first, dict):
|
||||
first_pct = _num(first.get("pct"))
|
||||
if first_pct is not None:
|
||||
# Normalize to 0-100: a fraction (<= 1) becomes a percentage.
|
||||
mode_pct = first_pct * 100.0 if first_pct <= 1 else first_pct
|
||||
|
||||
# --- singletons (count == 1) within the visible top ---
|
||||
if has_top:
|
||||
n_singletons = sum(
|
||||
1
|
||||
for item in top
|
||||
if isinstance(item, dict) and _num(item.get("count")) == 1
|
||||
)
|
||||
else:
|
||||
n_singletons = None
|
||||
|
||||
# The singleton count only covers the visible top; there may be more
|
||||
# distinct values (and thus more singletons) outside it.
|
||||
top_len = len(top) if isinstance(top, (list, tuple)) else 0
|
||||
n_singletons_partial = bool(n_distinct is not None and n_distinct > top_len)
|
||||
|
||||
# --- derived flags ---
|
||||
id_like = pct_distinct is not None and pct_distinct >= 99.0
|
||||
dominated = mode_pct is not None and mode_pct >= 90.0
|
||||
|
||||
return {
|
||||
"n_distinct": n_distinct,
|
||||
"n_rows": n_rows_out,
|
||||
"pct_distinct": pct_distinct,
|
||||
"entropy": entropy,
|
||||
"entropy_max": entropy_max,
|
||||
"entropy_norm": entropy_norm,
|
||||
"mode": mode,
|
||||
"mode_pct": mode_pct,
|
||||
"imbalance": imbalance,
|
||||
"n_singletons": n_singletons,
|
||||
"n_singletons_partial": n_singletons_partial,
|
||||
"len_min": len_min,
|
||||
"len_mean": len_mean,
|
||||
"len_max": len_max,
|
||||
"id_like": id_like,
|
||||
"dominated": dominated,
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
"""Tests para categorical_cardinality_block."""
|
||||
|
||||
import sys
|
||||
import os
|
||||
from math import log2
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from categorical_cardinality_block import categorical_cardinality_block
|
||||
|
||||
|
||||
# Output contract: every call returns exactly these 16 keys.
|
||||
EXPECTED_KEYS = {
|
||||
"n_distinct",
|
||||
"n_rows",
|
||||
"pct_distinct",
|
||||
"entropy",
|
||||
"entropy_max",
|
||||
"entropy_norm",
|
||||
"mode",
|
||||
"mode_pct",
|
||||
"imbalance",
|
||||
"n_singletons",
|
||||
"n_singletons_partial",
|
||||
"len_min",
|
||||
"len_mean",
|
||||
"len_max",
|
||||
"id_like",
|
||||
"dominated",
|
||||
}
|
||||
|
||||
|
||||
def _sample_cat():
|
||||
"""A realistic summarize_categorical output for one column."""
|
||||
return {
|
||||
"top": [
|
||||
{"value": "a", "count": 5, "pct": 0.5},
|
||||
{"value": "b", "count": 3, "pct": 0.3},
|
||||
{"value": "c", "count": 1, "pct": 0.1},
|
||||
{"value": "d", "count": 1, "pct": 0.1},
|
||||
],
|
||||
"mode": "a",
|
||||
"mode_pct": 0.5,
|
||||
"n_distinct": 4,
|
||||
"entropy": 1.685, # <= log2(4) = 2.0
|
||||
"imbalance": 5.0,
|
||||
"len_min": 1,
|
||||
"len_mean": 1.0,
|
||||
"len_max": 1,
|
||||
}
|
||||
|
||||
|
||||
def test_normal_case():
|
||||
"""Caso normal: pct_distinct, entropy_max=log2(n_distinct), entropy_norm in [0,1], n_singletons."""
|
||||
cat = _sample_cat()
|
||||
result = categorical_cardinality_block(cat, n_rows=10)
|
||||
|
||||
assert set(result.keys()) == EXPECTED_KEYS
|
||||
|
||||
# passthroughs
|
||||
assert result["n_distinct"] == 4
|
||||
assert result["n_rows"] == 10
|
||||
assert result["entropy"] == 1.685
|
||||
assert result["imbalance"] == 5.0
|
||||
assert result["mode"] == "a"
|
||||
assert result["mode_pct"] == 0.5 # passthrough, not normalized
|
||||
assert result["len_min"] == 1
|
||||
assert result["len_max"] == 1
|
||||
|
||||
# pct_distinct = 4 / 10 * 100
|
||||
assert abs(result["pct_distinct"] - 40.0) < 1e-12
|
||||
|
||||
# entropy_max = log2(4) = 2.0
|
||||
assert abs(result["entropy_max"] - log2(4)) < 1e-12
|
||||
assert abs(result["entropy_max"] - 2.0) < 1e-12
|
||||
|
||||
# entropy_norm = 1.685 / 2.0 = 0.8425, within [0, 1]
|
||||
assert abs(result["entropy_norm"] - 1.685 / 2.0) < 1e-12
|
||||
assert 0.0 <= result["entropy_norm"] <= 1.0
|
||||
|
||||
# singletons: c and d have count == 1
|
||||
assert result["n_singletons"] == 2
|
||||
# top covers all distinct values (4 == 4)
|
||||
assert result["n_singletons_partial"] is False
|
||||
|
||||
# neither id-like (40%) nor dominated (mode_pct 0.5)
|
||||
assert result["id_like"] is False
|
||||
assert result["dominated"] is False
|
||||
|
||||
|
||||
def test_empty_cat_does_not_raise():
|
||||
"""Caso cat={}: no lanza, claves derivadas None y flags False."""
|
||||
result = categorical_cardinality_block({}, n_rows=100)
|
||||
|
||||
assert set(result.keys()) == EXPECTED_KEYS
|
||||
for key in (
|
||||
"n_distinct",
|
||||
"pct_distinct",
|
||||
"entropy",
|
||||
"entropy_max",
|
||||
"entropy_norm",
|
||||
"mode",
|
||||
"mode_pct",
|
||||
"imbalance",
|
||||
"n_singletons",
|
||||
"len_min",
|
||||
"len_mean",
|
||||
"len_max",
|
||||
):
|
||||
assert result[key] is None
|
||||
assert result["n_singletons_partial"] is False
|
||||
assert result["id_like"] is False
|
||||
assert result["dominated"] is False
|
||||
# n_rows is a passthrough of the argument, still coherent.
|
||||
assert result["n_rows"] == 100
|
||||
|
||||
|
||||
def test_none_cat_does_not_raise():
|
||||
"""Caso cat=None: tratado como {}, mismas garantias que el dict vacio."""
|
||||
result = categorical_cardinality_block(None, n_rows=None)
|
||||
assert set(result.keys()) == EXPECTED_KEYS
|
||||
assert result["n_distinct"] is None
|
||||
assert result["pct_distinct"] is None
|
||||
assert result["entropy_max"] is None
|
||||
assert result["entropy_norm"] is None
|
||||
assert result["id_like"] is False
|
||||
assert result["dominated"] is False
|
||||
|
||||
|
||||
def test_n_rows_zero_no_zero_division():
|
||||
"""Caso n_rows=0: pct_distinct None sin ZeroDivisionError."""
|
||||
cat = _sample_cat()
|
||||
result = categorical_cardinality_block(cat, n_rows=0)
|
||||
assert result["pct_distinct"] is None
|
||||
# n_distinct still passes through.
|
||||
assert result["n_distinct"] == 4
|
||||
assert result["id_like"] is False
|
||||
|
||||
|
||||
def test_id_like_when_distinct_near_rows():
|
||||
"""id_like True cuando n_distinct ~ n_rows (pct_distinct >= 99)."""
|
||||
cat = {"n_distinct": 99, "entropy": 6.6, "top": [], "mode": None}
|
||||
result = categorical_cardinality_block(cat, n_rows=100)
|
||||
assert abs(result["pct_distinct"] - 99.0) < 1e-12
|
||||
assert result["id_like"] is True
|
||||
|
||||
# exact identity column: 100 / 100 = 100%
|
||||
cat_full = {"n_distinct": 100, "top": []}
|
||||
result_full = categorical_cardinality_block(cat_full, n_rows=100)
|
||||
assert result_full["id_like"] is True
|
||||
|
||||
|
||||
def test_dominated_when_mode_pct_high():
|
||||
"""dominated True cuando mode_pct alto (>= 90)."""
|
||||
cat = {
|
||||
"n_distinct": 3,
|
||||
"entropy": 0.3,
|
||||
"mode": "x",
|
||||
"mode_pct": 95.0,
|
||||
"top": [
|
||||
{"value": "x", "count": 95, "pct": 0.95},
|
||||
{"value": "y", "count": 3, "pct": 0.03},
|
||||
{"value": "z", "count": 2, "pct": 0.02},
|
||||
],
|
||||
"imbalance": 47.5,
|
||||
}
|
||||
result = categorical_cardinality_block(cat, n_rows=100)
|
||||
assert result["mode_pct"] == 95.0
|
||||
assert result["dominated"] is True
|
||||
|
||||
|
||||
def test_mode_pct_fallback_from_top_fraction():
|
||||
"""Sin mode_pct: deriva del pct del primer top, fraccion <=1 escala a 0-100."""
|
||||
cat = {
|
||||
"n_distinct": 3,
|
||||
"top": [
|
||||
{"value": "x", "count": 95, "pct": 0.95},
|
||||
{"value": "y", "count": 5, "pct": 0.05},
|
||||
],
|
||||
}
|
||||
result = categorical_cardinality_block(cat, n_rows=100)
|
||||
# 0.95 (fraction) -> 95.0 (percentage)
|
||||
assert abs(result["mode_pct"] - 95.0) < 1e-12
|
||||
assert result["dominated"] is True
|
||||
|
||||
|
||||
def test_n_singletons_partial_when_top_truncated():
|
||||
"""n_distinct > len(top): n_singletons cubre solo el top visible, partial True."""
|
||||
cat = {
|
||||
"n_distinct": 10,
|
||||
"top": [
|
||||
{"value": "a", "count": 4, "pct": 0.4},
|
||||
{"value": "b", "count": 1, "pct": 0.1},
|
||||
{"value": "c", "count": 1, "pct": 0.1},
|
||||
],
|
||||
"entropy": 2.5,
|
||||
}
|
||||
result = categorical_cardinality_block(cat, n_rows=12)
|
||||
assert result["n_singletons"] == 2 # only b, c visible
|
||||
assert result["n_singletons_partial"] is True
|
||||
|
||||
|
||||
def test_single_distinct_value_entropy_norm_none():
|
||||
"""n_distinct=1: entropy_max=0.0 -> entropy_norm None (no division by zero)."""
|
||||
cat = {
|
||||
"n_distinct": 1,
|
||||
"entropy": 0.0,
|
||||
"mode": "only",
|
||||
"mode_pct": 1.0,
|
||||
"top": [{"value": "only", "count": 7, "pct": 1.0}],
|
||||
"imbalance": 1.0,
|
||||
}
|
||||
result = categorical_cardinality_block(cat, n_rows=7)
|
||||
assert result["entropy_max"] == 0.0
|
||||
assert result["entropy_norm"] is None
|
||||
assert result["n_singletons"] == 0
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
id: categorical_top_pie_figure_py_datascience
|
||||
name: categorical_top_pie_figure
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: impure
|
||||
signature: "def categorical_top_pie_figure(top: list, n_distinct: int = 0, title: str = \"\", top_k: int = 6, n_rows=None) -> \"matplotlib.figure.Figure\""
|
||||
description: "Construye una figura matplotlib tipo donut (pie con agujero central) de las top_k categorías más frecuentes de una columna categórica, agregando el resto en un sector gris \"Otros (N categorías)\". Consume el bloque `top` de summarize_categorical y devuelve un matplotlib.figure.Figure listo para rasterizar por el renderer del informe EDA. Backend Agg sin pyplot global; defensivo ante top vacío/None."
|
||||
tags: [eda, categorical, pie, donut, matplotlib, figure, visualization, datascience, impure]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: "error_go_core"
|
||||
imports: [matplotlib]
|
||||
example: |
|
||||
from categorical_top_pie_figure import categorical_top_pie_figure
|
||||
top = [
|
||||
{"value": "rojo", "count": 40, "pct": 0.4},
|
||||
{"value": "azul", "count": 30, "pct": 0.3},
|
||||
{"value": "verde", "count": 20, "pct": 0.2},
|
||||
]
|
||||
fig = categorical_top_pie_figure(top, n_distinct=12, title="color", top_k=6, n_rows=100)
|
||||
tested: true
|
||||
tests:
|
||||
- "test_returns_figure"
|
||||
- "test_ten_items_topk_six_yields_seven_wedges"
|
||||
- "test_empty_top_does_not_raise_and_returns_figure"
|
||||
- "test_long_value_truncated_in_legend"
|
||||
- "test_none_value_and_none_count_are_handled"
|
||||
- "test_n_rows_adds_exact_others_slice"
|
||||
test_file_path: "python/functions/datascience/categorical_top_pie_figure_test.py"
|
||||
file_path: "python/functions/datascience/categorical_top_pie_figure.py"
|
||||
params:
|
||||
- name: top
|
||||
desc: "Lista de dicts {value, count, pct} ordenada de mayor a menor por count (salida del bloque `top` de summarize_categorical). Puede venir vacía o con dicts incompletos: items no-dict, sin count, con count None o count <= 0 se descartan. value None se admite (sin etiqueta)."
|
||||
- name: n_distinct
|
||||
desc: "Nº total de categorías distintas de la columna. Etiqueta el sector agregado como \"Otros (n_distinct - top_k)\" (mínimo 0). Si no supera el nº de sectores mostrados, se usa el overflow real de `top` como nº de categorías agregadas. Default 0."
|
||||
- name: title
|
||||
desc: "Título de la figura (nombre de la columna). Se trunca a ~48 chars con elipsis si es muy largo. Default \"\" (sin título)."
|
||||
- name: top_k
|
||||
desc: "Nº máximo de sectores explícitos. Default 6. El sector \"Otros\" no cuenta contra este límite. Con top_k <= 0 se muestra al menos la categoría mayor."
|
||||
- name: n_rows
|
||||
desc: "Opcional. Total de filas del dataset. Si se da y la suma de counts mostrados < n_rows, el sector \"Otros\" usa (n_rows - suma_mostrada) como count para que los ángulos sean exactos respecto al total real. Si se omite, \"Otros\" usa la suma de counts fuera del top_k mostrado (solo cuando top trae más de top_k items). Default None."
|
||||
output: "Un matplotlib.figure.Figure (figsize 6.4x4.0, dpi 150) con un Axes donut (wedgeprops width 0.42) más una leyenda lateral con value truncado a 20 chars + count; el sector \"Otros\" en gris. Anotación central con el total n. Si no hay counts válidos, devuelve igualmente una Figure con un texto centrado \"sin datos categóricos\" (nunca lanza). El caller rasteriza/cierra la figura; la función no la muestra ni la guarda."
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from categorical_top_pie_figure import categorical_top_pie_figure
|
||||
|
||||
# `top` es la salida del bloque "top" de summarize_categorical (ya ordenado desc).
|
||||
top = [
|
||||
{"value": "rojo", "count": 40, "pct": 0.40},
|
||||
{"value": "azul", "count": 30, "pct": 0.30},
|
||||
{"value": "verde", "count": 20, "pct": 0.20},
|
||||
{"value": "amarillo", "count": 5, "pct": 0.05},
|
||||
]
|
||||
|
||||
fig = categorical_top_pie_figure(
|
||||
top,
|
||||
n_distinct=12, # 12 categorías distintas en total
|
||||
title="color_producto",
|
||||
top_k=6, # hasta 6 sectores explícitos
|
||||
n_rows=100, # "Otros" = 100 - 95 = 5, sobre 8 categorías agregadas
|
||||
)
|
||||
|
||||
# El renderer del informe lo rasteriza; aquí solo persistimos para inspección.
|
||||
fig.savefig("/tmp/donut_color.png")
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Úsala dentro de un informe EDA cuando quieras visualizar la composición de una
|
||||
columna categórica de un vistazo: cuántas filas caen en las categorías
|
||||
dominantes frente a la cola larga. Pásale directamente el bloque `top` de
|
||||
`summarize_categorical` (ya ordenado de mayor a menor) más `n_distinct` para que
|
||||
el sector "Otros" indique cuántas categorías quedan agrupadas. Es la pareja
|
||||
"composición" del gráfico de barras top-k: el donut comunica proporciones del
|
||||
total, las barras comunican magnitudes comparables.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Impura por matplotlib.** Toca la maquinaria de render. Usa el backend `Agg`
|
||||
y la API orientada a objetos `Figure`/`add_subplot` — NUNCA `pyplot.*` aquí,
|
||||
para no tocar el estado global ni filtrar figuras entre llamadas. `pyplot` NO
|
||||
es thread-safe; esta función evita ese riesgo construyendo el `Figure`
|
||||
directamente, así que es segura de llamar en bucle desde el renderer.
|
||||
- **El caller cierra la figura.** La función devuelve el `Figure` pero no lo
|
||||
muestra ni lo guarda. Quien la consume debe rasterizarla y luego liberarla
|
||||
(`fig.clf()` / `matplotlib.pyplot.close(fig)` si se usó pyplot en el caller)
|
||||
para no acumular memoria en lotes grandes de columnas.
|
||||
- **Pie engaña con muchos sectores.** Por eso `top_k` por defecto es 6 y el
|
||||
resto se agrega en "Otros": donuts con 15+ sectores son ilegibles. Para
|
||||
cardinalidad muy alta el donut solo muestra la cabeza de la distribución; la
|
||||
cola vive en el sector gris.
|
||||
- **Ángulos exactos solo con `n_rows`.** Sin `n_rows`, el sector "Otros" se
|
||||
calcula con el overflow presente en `top`; si `top` ya viene recortado a
|
||||
`top_k` por el productor, no habrá "Otros" aunque existan más categorías. Pasa
|
||||
`n_rows` (total de filas del dataset) para ángulos correctos respecto al total
|
||||
real.
|
||||
- **Defensiva, nunca lanza.** `top=[]`, `value=None`, `count=None` o counts no
|
||||
numéricos se manejan sin error: en el peor caso devuelve una `Figure` con
|
||||
"sin datos categóricos". No envuelvas la llamada en try/except por miedo a un
|
||||
raise — no lo hay.
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Impure EDA helper: donut figure of the most common categories (`eda` group).
|
||||
|
||||
Builds a matplotlib donut (pie with a central hole) of the ``top_k`` most
|
||||
frequent categories of a categorical column, folding everything else into a
|
||||
single "Otros (N categorías)" slice. Returns a ready-to-rasterize
|
||||
``matplotlib.figure.Figure``; it never shows nor saves it.
|
||||
|
||||
Impure because it touches matplotlib's rendering machinery. It uses the headless
|
||||
Agg backend and the object-oriented ``Figure`` API (no ``pyplot``) so it leaks no
|
||||
global state and is safe to call repeatedly from a report renderer.
|
||||
"""
|
||||
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
|
||||
from matplotlib.figure import Figure # noqa: E402
|
||||
|
||||
|
||||
# Gray reserved for the aggregated "Otros" slice.
|
||||
_OTHER_COLOR = "#9e9e9e"
|
||||
# Muted gray for secondary text (title fallback, center annotation, no-data).
|
||||
_MUTED_TEXT = "#5f6b7a"
|
||||
# Pleasant, colour-blind-friendly qualitative palette for the explicit slices.
|
||||
_PALETTE = [
|
||||
"#4C72B0",
|
||||
"#DD8452",
|
||||
"#55A868",
|
||||
"#C44E52",
|
||||
"#8172B3",
|
||||
"#937860",
|
||||
"#DA8BC3",
|
||||
"#8C8C8C",
|
||||
"#CCB974",
|
||||
"#64B5CD",
|
||||
]
|
||||
|
||||
|
||||
def _truncate(text, width: int = 20) -> str:
|
||||
"""Truncate ``text`` to ``width`` chars, appending an ellipsis if cut."""
|
||||
s = "" if text is None else str(text)
|
||||
if len(s) <= width:
|
||||
return s
|
||||
if width <= 1:
|
||||
return s[:width]
|
||||
return s[: width - 1] + "…"
|
||||
|
||||
|
||||
def categorical_top_pie_figure(
|
||||
top: list,
|
||||
n_distinct: int = 0,
|
||||
title: str = "",
|
||||
top_k: int = 6,
|
||||
n_rows=None,
|
||||
) -> "matplotlib.figure.Figure":
|
||||
"""Build a donut figure of the most common categories of a column.
|
||||
|
||||
Renders the ``top_k`` most frequent categories as explicit donut slices and
|
||||
aggregates every remaining category into a single gray "Otros (N
|
||||
categorías)" slice. Category names are not painted on the wedges; they are
|
||||
listed in a lateral legend (truncated value + count) to avoid overlap on
|
||||
narrow (mobile) figures.
|
||||
|
||||
The function is fully defensive: empty input, missing/``None`` values or
|
||||
counts never raise. When there is nothing valid to draw it still returns a
|
||||
``Figure`` carrying a centered "sin datos categóricos" message.
|
||||
|
||||
Args:
|
||||
top: List of ``{value, count, pct}`` dicts, already sorted by ``count``
|
||||
descending (the ``top`` block of ``summarize_categorical``). May be
|
||||
empty or carry incomplete/``None`` entries; non-dict items, items
|
||||
without a positive numeric ``count`` and ``None`` counts are skipped.
|
||||
n_distinct: Total number of distinct categories in the column. Used to
|
||||
label the aggregated slice as "Otros (n_distinct - top_k)" (floored
|
||||
at 0). Ignored when it does not exceed the number of shown slices.
|
||||
title: Figure title (the column name). Truncated when too long.
|
||||
top_k: Maximum number of explicit slices. Default 6. The "Otros" slice
|
||||
does not count against this limit.
|
||||
n_rows: Optional total row count of the dataset. When given and the sum
|
||||
of shown counts is below ``n_rows``, the "Otros" slice uses
|
||||
``n_rows - sum_shown`` as its count so the wedge angles are exact
|
||||
with respect to the real total. When omitted, "Otros" uses the sum
|
||||
of the counts that fall outside the shown ``top_k`` (only when
|
||||
``top`` carries more than ``top_k`` items).
|
||||
|
||||
Returns:
|
||||
A ``matplotlib.figure.Figure`` with a single donut Axes plus a lateral
|
||||
legend. The caller is responsible for rasterizing/closing it.
|
||||
"""
|
||||
fig = Figure(figsize=(6.4, 4.0), dpi=150)
|
||||
ax = fig.add_subplot(111)
|
||||
|
||||
safe_title = _truncate(title, 48)
|
||||
|
||||
# --- Defensive parse: keep only well-formed {value, count} with count > 0.
|
||||
cleaned = []
|
||||
if isinstance(top, list):
|
||||
for item in top:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
count = item.get("count")
|
||||
if count is None:
|
||||
continue
|
||||
try:
|
||||
count = float(count)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if count <= 0:
|
||||
continue
|
||||
cleaned.append((item.get("value"), count))
|
||||
|
||||
if not cleaned:
|
||||
ax.axis("off")
|
||||
ax.text(
|
||||
0.5,
|
||||
0.5,
|
||||
"sin datos categóricos",
|
||||
ha="center",
|
||||
va="center",
|
||||
fontsize=12,
|
||||
color=_MUTED_TEXT,
|
||||
transform=ax.transAxes,
|
||||
)
|
||||
if safe_title:
|
||||
ax.set_title(safe_title, fontsize=12, loc="center", pad=8)
|
||||
fig.tight_layout()
|
||||
return fig
|
||||
|
||||
# --- Split into shown slices and the aggregated remainder.
|
||||
shown = cleaned[: max(int(top_k), 0)]
|
||||
if not shown: # top_k <= 0 — show at least the largest category.
|
||||
shown = cleaned[:1]
|
||||
|
||||
sum_shown = sum(c for _, c in shown)
|
||||
overflow_count = sum(c for _, c in cleaned[len(shown):])
|
||||
|
||||
# How many categories are folded into "Otros".
|
||||
try:
|
||||
nd = int(n_distinct)
|
||||
except (TypeError, ValueError):
|
||||
nd = 0
|
||||
others_categories = max(nd - len(shown), 0)
|
||||
# If n_distinct is unknown/too small, fall back to the overflow we actually
|
||||
# have in `top` beyond the shown slices.
|
||||
overflow_items = len(cleaned) - len(shown)
|
||||
if others_categories == 0 and overflow_items > 0:
|
||||
others_categories = overflow_items
|
||||
|
||||
# Count attributed to the "Otros" slice for exact angles.
|
||||
others_count = 0.0
|
||||
if n_rows is not None:
|
||||
try:
|
||||
total_rows = float(n_rows)
|
||||
except (TypeError, ValueError):
|
||||
total_rows = None
|
||||
if total_rows is not None and total_rows > sum_shown:
|
||||
others_count = total_rows - sum_shown
|
||||
if others_count <= 0:
|
||||
others_count = overflow_count
|
||||
|
||||
labels = [v for v, _ in shown]
|
||||
values = [c for _, c in shown]
|
||||
colors = [_PALETTE[i % len(_PALETTE)] for i in range(len(shown))]
|
||||
|
||||
has_others = others_count > 0 and others_categories > 0
|
||||
if has_others:
|
||||
values.append(others_count)
|
||||
labels.append("Otros")
|
||||
colors.append(_OTHER_COLOR)
|
||||
|
||||
total = sum(values)
|
||||
|
||||
def _autopct(pct: float) -> str:
|
||||
# Hide tiny labels to avoid crowding the wedges.
|
||||
return f"{pct:.0f}%" if pct >= 5 else ""
|
||||
|
||||
wedges, _texts, autotexts = ax.pie(
|
||||
values,
|
||||
colors=colors,
|
||||
startangle=90,
|
||||
counterclock=False,
|
||||
wedgeprops={"width": 0.42, "edgecolor": "white", "linewidth": 1.0},
|
||||
autopct=_autopct,
|
||||
pctdistance=0.79,
|
||||
textprops={"fontsize": 8},
|
||||
)
|
||||
for at in autotexts:
|
||||
at.set_color("white")
|
||||
at.set_fontweight("bold")
|
||||
ax.set_aspect("equal")
|
||||
|
||||
# --- Lateral legend: truncated value + count (+ "(N categorías)" for Otros).
|
||||
legend_labels = []
|
||||
for idx, (lab, val) in enumerate(zip(labels, values)):
|
||||
if has_others and idx == len(labels) - 1:
|
||||
legend_labels.append(
|
||||
f"Otros ({others_categories} categorías) — {int(round(val))}"
|
||||
)
|
||||
else:
|
||||
legend_labels.append(f"{_truncate(lab, 20)} — {int(round(val))}")
|
||||
|
||||
ax.legend(
|
||||
wedges,
|
||||
legend_labels,
|
||||
title="Categorías",
|
||||
loc="center left",
|
||||
bbox_to_anchor=(1.02, 0.5),
|
||||
fontsize=8,
|
||||
title_fontsize=9,
|
||||
frameon=False,
|
||||
)
|
||||
|
||||
if safe_title:
|
||||
ax.set_title(safe_title, fontsize=13, loc="left", pad=10)
|
||||
|
||||
# Center annotation: total count covered by the donut.
|
||||
ax.text(
|
||||
0,
|
||||
0,
|
||||
f"n={int(round(total))}",
|
||||
ha="center",
|
||||
va="center",
|
||||
fontsize=11,
|
||||
color=_MUTED_TEXT,
|
||||
fontweight="bold",
|
||||
)
|
||||
|
||||
# Leave room on the right for the legend (avoid clipping it).
|
||||
fig.subplots_adjust(left=0.02, right=0.62, top=0.88, bottom=0.06)
|
||||
return fig
|
||||
@@ -0,0 +1,104 @@
|
||||
"""Tests para categorical_top_pie_figure (donut de categorías top, grupo eda).
|
||||
|
||||
Usa el backend Agg sin pyplot; no muestra ni guarda figuras. Cada test cierra
|
||||
explícitamente la Figure construida (matplotlib.pyplot.close) para no acumular
|
||||
estado entre tests.
|
||||
"""
|
||||
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
|
||||
import matplotlib.pyplot as plt # noqa: E402
|
||||
from matplotlib.figure import Figure # noqa: E402
|
||||
|
||||
from categorical_top_pie_figure import categorical_top_pie_figure
|
||||
|
||||
|
||||
def _make_top(n):
|
||||
"""n items {value, count, pct} ordenados desc por count."""
|
||||
return [
|
||||
{"value": f"cat_{i}", "count": n - i, "pct": (n - i) / sum(range(1, n + 1))}
|
||||
for i in range(n)
|
||||
]
|
||||
|
||||
|
||||
def _wedges(ax):
|
||||
"""Devuelve los wedges (sectores) de un Axes con un pie."""
|
||||
from matplotlib.patches import Wedge
|
||||
|
||||
return [p for p in ax.patches if isinstance(p, Wedge)]
|
||||
|
||||
|
||||
def test_returns_figure():
|
||||
fig = categorical_top_pie_figure(_make_top(3), n_distinct=3, title="col")
|
||||
assert isinstance(fig, Figure)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_ten_items_topk_six_yields_seven_wedges():
|
||||
top = _make_top(10)
|
||||
fig = categorical_top_pie_figure(top, n_distinct=10, title="muchas", top_k=6)
|
||||
ax = fig.axes[0]
|
||||
wedges = _wedges(ax)
|
||||
# 6 categorías explícitas + 1 sector "Otros".
|
||||
assert len(wedges) == 7
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_empty_top_does_not_raise_and_returns_figure():
|
||||
fig = categorical_top_pie_figure([], n_distinct=0, title="vacía")
|
||||
assert isinstance(fig, Figure)
|
||||
# Sin datos: no debe haber sectores de pie.
|
||||
assert len(_wedges(fig.axes[0])) == 0
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_long_value_truncated_in_legend():
|
||||
long_value = "una_categoria_con_un_nombre_larguisimo_que_excede_el_limite"
|
||||
top = [
|
||||
{"value": long_value, "count": 10, "pct": 0.5},
|
||||
{"value": "corta", "count": 10, "pct": 0.5},
|
||||
]
|
||||
fig = categorical_top_pie_figure(top, n_distinct=2, title="col", top_k=6)
|
||||
ax = fig.axes[0]
|
||||
legend = ax.get_legend()
|
||||
assert legend is not None
|
||||
texts = [t.get_text() for t in legend.get_texts()]
|
||||
# El valor largo aparece truncado con elipsis y NO en su forma completa.
|
||||
assert any("…" in t for t in texts)
|
||||
assert long_value not in " ".join(texts)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_none_value_and_none_count_are_handled():
|
||||
top = [
|
||||
{"value": None, "count": 5, "pct": 0.5},
|
||||
{"value": "b", "count": None, "pct": 0.0}, # count None -> se descarta
|
||||
{"value": "c", "count": 5, "pct": 0.5},
|
||||
]
|
||||
fig = categorical_top_pie_figure(top, n_distinct=2, title="con nones", top_k=6)
|
||||
assert isinstance(fig, Figure)
|
||||
# Solo 2 items válidos, sin overflow -> 2 wedges, sin "Otros".
|
||||
assert len(_wedges(fig.axes[0])) == 2
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def test_n_rows_adds_exact_others_slice():
|
||||
# 3 categorías mostradas suman 30, dataset real 100 -> "Otros" = 70.
|
||||
top = _make_top(3) # counts 3,2,1 -> reescalamos abajo
|
||||
top = [
|
||||
{"value": "a", "count": 15, "pct": 0.15},
|
||||
{"value": "b", "count": 10, "pct": 0.10},
|
||||
{"value": "c", "count": 5, "pct": 0.05},
|
||||
]
|
||||
fig = categorical_top_pie_figure(
|
||||
top, n_distinct=20, title="col", top_k=3, n_rows=100
|
||||
)
|
||||
ax = fig.axes[0]
|
||||
# 3 explícitas + Otros.
|
||||
assert len(_wedges(ax)) == 4
|
||||
legend_texts = [t.get_text() for t in ax.get_legend().get_texts()]
|
||||
# El sector Otros refleja n_distinct - top_k = 17 categorías y count 70.
|
||||
assert any("Otros (17 categorías)" in t and "70" in t for t in legend_texts)
|
||||
plt.close(fig)
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: detect_latlon_columns
|
||||
id: detect_latlon_columns_py_datascience
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def detect_latlon_columns(columns: list, samples: dict | None = None) -> dict"
|
||||
description: "Detecta un par (latitud, longitud) entre las columnas de un TableProfile del grupo eda combinando heuristica de nombre (latitude/longitude/lat/lon/lng + x/y debiles) con validacion de rango obligatoria (latitud en [-90,90], longitud en [-180,180]). Lee defensivamente con .get; NUNCA lanza. Usa el sub-bloque numeric.min/max o, si falta, la lista de samples opcional. Devuelve SIEMPRE un dict {lat_col, lon_col, confidence, reason}; si no hay par valido, las columnas van a None y confidence a 0.0."
|
||||
tags: [eda, geospatial, profiling, latlon, coordinates, detection, datascience]
|
||||
params:
|
||||
- name: columns
|
||||
desc: "Lista de dicts ColumnProfile (el campo `columns` de un TableProfile del grupo eda). Cada dict se lee con .get; solo `name` (str) es obligatorio. Se consultan `inferred_type` (p.ej. 'numeric') y el sub-dict `numeric` con `min`/`max` (floats) para validar el rango. Entradas no-dict o sin name se ignoran sin lanzar."
|
||||
- name: samples
|
||||
desc: "Opcional {nombre_columna: [valores...]} para validar el rango cuando una columna no trae numeric.min/max. Los valores nulos se ignoran; si algun valor no nulo no es numerico la columna no se considera coordenada. Si es None u omitido, solo se usa el bloque numeric."
|
||||
output: "Dict SIEMPRE presente con la forma {lat_col: str|None, lon_col: str|None, confidence: float en [0,1], reason: str en espanol}. En exito, lat_col y lon_col nombran columnas distintas; confidence ~1.0 para par con nombre fuerte (latitude/longitude/lat/lon/lng) + rango valido y ~0.7 para par debil (x/y) + rango. En fallo, ambas columnas None, confidence 0.0 y reason explica por que (sin columnas, nombre sin match, rango fuera de bounds, falta uno de los dos ejes...)."
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: []
|
||||
tested: true
|
||||
tests: ["test_par_latitude_longitude_fuerte", "test_par_lat_lon_abreviado", "test_par_x_y_debil_con_rango_valido", "test_nombre_lat_lon_pero_rango_fuera_no_detecta", "test_par_fuerte_prevalece_sobre_debil", "test_entradas_vacias_o_invalidas_no_lanzan", "test_solo_latitud_sin_longitud_no_detecta", "test_deteccion_por_samples_cuando_falta_numeric", "test_samples_fuera_de_rango_descarta"]
|
||||
test_file_path: "python/functions/datascience/detect_latlon_columns_test.py"
|
||||
file_path: "python/functions/datascience/detect_latlon_columns.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience.detect_latlon_columns import detect_latlon_columns
|
||||
|
||||
# Columnas tal y como vienen en profile['columns'] de un TableProfile del grupo eda:
|
||||
columns = [
|
||||
{"name": "id", "inferred_type": "numeric", "numeric": {"min": 1, "max": 9999}},
|
||||
{"name": "latitude", "inferred_type": "numeric", "numeric": {"min": -45.0, "max": 45.0}},
|
||||
{"name": "longitude", "inferred_type": "numeric", "numeric": {"min": -120.0, "max": 120.0}},
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
print(res["lat_col"], res["lon_col"], res["confidence"])
|
||||
# latitude longitude 1.0
|
||||
|
||||
# Sin bloque numeric, validando el rango con samples:
|
||||
cols2 = [{"name": "lat"}, {"name": "lon"}]
|
||||
samples = {"lat": [10.5, 20.0, 30.25], "lon": [-40.0, 50.5, 60.0]}
|
||||
print(detect_latlon_columns(cols2, samples)["lat_col"]) # lat
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
- Usala al perfilar una tabla en `AutomaticEDA` para decidir si tiene geometria de puntos: cuando `detect_latlon_columns` devuelve un par con `confidence` alta, el capitulo geospatial puede dibujar un mapa, calcular un bounding box o proponer un cluster espacial.
|
||||
- Antes de un analisis geoespacial (alpha shape, convex hull, joins por proximidad) para localizar automaticamente que columnas son la latitud y la longitud sin pedirlo al usuario.
|
||||
- Cuando recibas un `TableProfile` del grupo `eda` y quieras enrutar columnas a sub-analisis por tipo semantico: este es el detector del par lat/lon, complementario a `infer_semantic_type`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Funcion pura, sin I/O y determinista. Lectura defensiva con `.get`: NUNCA lanza. Cualquier input malformado (None, no-lista, entradas no-dict, claves ausentes) devuelve el dict de fallo con `lat_col`/`lon_col` en None y `confidence` 0.0.
|
||||
- **El nombre solo no basta**: una columna `latitude` cuyo rango se sale de `[-90, 90]` se descarta (no es coordenada real). Igual para `longitude` fuera de `[-180, 180]`. La validacion de rango es obligatoria.
|
||||
- El rango de latitud `[-90, 90]` es un subconjunto del de longitud `[-180, 180]`, por eso el nombre es necesario para desambiguar cual eje es cual; una columna numerica en `[-90, 90]` sin nombre que sugiera lat/lon no se detecta.
|
||||
- Los nombres genericos `x`/`y` (y `x_coord`/`y_coord`) son candidatos **debiles**: solo forman par si el rango encaja y existe la otra mitad (un `x`/`lon` para la `y`, un `y`/`lat` para la `x`). Un `y` suelto sin pareja devuelve None.
|
||||
- Requiere AMBOS ejes para considerar exito. Si solo encuentra latitud o solo longitud, devuelve el dict de fallo (no media coordenada).
|
||||
- `samples` solo se consulta cuando falta `numeric.min`/`numeric.max`. Si una columna trae el bloque numeric, ese manda aunque pases samples para ella.
|
||||
- El matching de nombre es por subcadena normalizada (se quitan `_`, `-` y espacios), asi que nombres como `plate` (contiene "lat") podrian marcarse como candidatos por nombre — pero solo pasarian si su rango cae en `[-90, 90]` y hay una longitud pareja, filtro que en la practica descarta los falsos positivos.
|
||||
@@ -0,0 +1,198 @@
|
||||
"""detect_latlon_columns — detect a (latitude, longitude) column pair in an EDA profile.
|
||||
|
||||
Pure function: no I/O, deterministic. Takes the `columns` list of a TableProfile
|
||||
(group `eda`) and decides whether two of its columns form a geographic coordinate
|
||||
pair (latitude + longitude), combining a name heuristic with a value-range check.
|
||||
|
||||
The detection is intentionally conservative: a name hint alone is never enough. A
|
||||
column is only accepted as latitude/longitude if its numeric range fits inside the
|
||||
valid coordinate bounds ([-90, 90] for latitude, [-180, 180] for longitude). When
|
||||
the `numeric` sub-block is absent the optional `samples` argument is used instead.
|
||||
|
||||
Reading is fully defensive (.get throughout) and the function NEVER raises: any
|
||||
malformed input (None, non-list, non-dict entries, missing keys) simply yields a
|
||||
no-pair result {"lat_col": None, "lon_col": None, "confidence": 0.0, "reason": ...}.
|
||||
"""
|
||||
|
||||
import re
|
||||
|
||||
# Collapse the separators a column name may use (snake_case, kebab-case, spaces)
|
||||
# so that "y_coord", "y-coord" and "y coord" all normalize to the same token.
|
||||
_SEP_RE = re.compile(r"[\s_\-]+")
|
||||
|
||||
# Name-match strengths: a strong, unambiguous coordinate name vs a weak generic
|
||||
# axis name (x / y) that only counts when the range also fits and a partner exists.
|
||||
_STRONG = 0.6
|
||||
_WEAK = 0.3
|
||||
_RANGE_BONUS = 0.4 # added once the mandatory range validation passes
|
||||
|
||||
|
||||
def _normalize(name):
|
||||
"""Lowercase a column name and strip separator chars (_, -, whitespace)."""
|
||||
if not isinstance(name, str):
|
||||
return ""
|
||||
return _SEP_RE.sub("", name.strip().lower())
|
||||
|
||||
|
||||
def _num(value):
|
||||
"""Coerce to float defensively; return None for None/bool/non-numeric."""
|
||||
# bool is a subclass of int; a coordinate value is never a real bool, so treat
|
||||
# True/False as missing instead of silently coercing to 1.0/0.0.
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _lat_name_strength(nn):
|
||||
"""Strength of a normalized name as a latitude candidate (0=no match)."""
|
||||
if not nn:
|
||||
return 0.0
|
||||
# "lat", "latitude", "latitud" all contain the "lat" stem.
|
||||
if "lat" in nn:
|
||||
return _STRONG
|
||||
# Weak generic axis name: only useful when paired with an x/lon partner.
|
||||
if nn in ("y", "ycoord", "ycoordinate", "ycoordinates"):
|
||||
return _WEAK
|
||||
return 0.0
|
||||
|
||||
|
||||
def _lon_name_strength(nn):
|
||||
"""Strength of a normalized name as a longitude candidate (0=no match)."""
|
||||
if not nn:
|
||||
return 0.0
|
||||
# "lon", "long", "longitude", "longitud" share the "lon" stem; "lng" is separate.
|
||||
if "lon" in nn or "lng" in nn:
|
||||
return _STRONG
|
||||
if nn in ("x", "xcoord", "xcoordinate", "xcoordinates"):
|
||||
return _WEAK
|
||||
return 0.0
|
||||
|
||||
|
||||
def _col_range(col, sample_values):
|
||||
"""Return (min, max) floats for a column, or (None, None) if not numeric.
|
||||
|
||||
Prefers the `numeric` sub-block min/max (the output of describe_numeric); falls
|
||||
back to the provided sample list. A column is only treated as numeric when both
|
||||
extremes are derivable: from the numeric block, or from samples whose every
|
||||
non-null value coerces to a number.
|
||||
"""
|
||||
if isinstance(col, dict):
|
||||
numeric = col.get("numeric")
|
||||
if isinstance(numeric, dict):
|
||||
mn = _num(numeric.get("min"))
|
||||
mx = _num(numeric.get("max"))
|
||||
if mn is not None and mx is not None:
|
||||
return mn, mx
|
||||
# Fall back to samples when the numeric block is missing or incomplete.
|
||||
if isinstance(sample_values, (list, tuple)):
|
||||
non_null = [v for v in sample_values if v is not None]
|
||||
if non_null:
|
||||
coerced = [_num(v) for v in non_null]
|
||||
# Any non-numeric sample means we cannot trust the column as numeric.
|
||||
if all(c is not None for c in coerced):
|
||||
return min(coerced), max(coerced)
|
||||
return None, None
|
||||
|
||||
|
||||
def _no_pair(reason):
|
||||
"""Canonical empty result: no coordinate pair detected."""
|
||||
return {"lat_col": None, "lon_col": None, "confidence": 0.0, "reason": reason}
|
||||
|
||||
|
||||
def detect_latlon_columns(columns: list, samples: dict | None = None) -> dict:
|
||||
"""Detect a (latitude, longitude) column pair from an eda TableProfile.
|
||||
|
||||
Combines a name heuristic (latitude/longitude/lat/lon/lng + weak x/y) with a
|
||||
mandatory range validation: the chosen latitude must sit in [-90, 90] and the
|
||||
longitude in [-180, 180]. A name hint whose range does not fit is discarded.
|
||||
Both sides are required for success; if only one is found, no pair is returned.
|
||||
|
||||
Args:
|
||||
columns: List of ColumnProfile dicts (the `columns` of a TableProfile).
|
||||
Each dict is read defensively with .get; only `name` is required.
|
||||
`numeric.min` / `numeric.max` (and optionally `inferred_type`) are used
|
||||
for the range check when present.
|
||||
samples: Optional {column_name: [values...]} used to validate the range
|
||||
when a column lacks `numeric.min`/`numeric.max`. If None/omitted, only
|
||||
the `numeric` sub-block is consulted.
|
||||
|
||||
Returns:
|
||||
Always a dict {"lat_col": str|None, "lon_col": str|None,
|
||||
"confidence": float, "reason": str}. On success lat_col and lon_col name
|
||||
the detected pair (distinct columns) and confidence is in [0, 1]: a pair
|
||||
validated by a strong name on both sides scores ~1.0, a weak x/y pair ~0.7.
|
||||
On failure both columns are None and confidence is 0.0.
|
||||
"""
|
||||
if not isinstance(columns, (list, tuple)) or len(columns) == 0:
|
||||
return _no_pair("sin columnas que inspeccionar")
|
||||
|
||||
sample_map = samples if isinstance(samples, dict) else {}
|
||||
|
||||
# (column_name, confidence) for each side. Confidence already includes the
|
||||
# range bonus because membership in the list implies the range was validated.
|
||||
lat_candidates = []
|
||||
lon_candidates = []
|
||||
|
||||
for col in columns:
|
||||
if not isinstance(col, dict):
|
||||
continue
|
||||
name = col.get("name")
|
||||
if not isinstance(name, str) or not name:
|
||||
continue
|
||||
|
||||
nn = _normalize(name)
|
||||
lat_strength = _lat_name_strength(nn)
|
||||
lon_strength = _lon_name_strength(nn)
|
||||
if lat_strength == 0.0 and lon_strength == 0.0:
|
||||
continue # name gives no coordinate hint; skip.
|
||||
|
||||
mn, mx = _col_range(col, sample_map.get(name))
|
||||
is_numeric = mn is not None and mx is not None
|
||||
if not is_numeric:
|
||||
continue # range cannot be validated -> not a coordinate.
|
||||
|
||||
if lat_strength > 0.0 and mn >= -90.0 and mx <= 90.0:
|
||||
lat_candidates.append((name, lat_strength + _RANGE_BONUS))
|
||||
if lon_strength > 0.0 and mn >= -180.0 and mx <= 180.0:
|
||||
lon_candidates.append((name, lon_strength + _RANGE_BONUS))
|
||||
|
||||
if not lat_candidates and not lon_candidates:
|
||||
return _no_pair("ninguna columna sugiere latitud ni longitud por nombre+rango")
|
||||
if not lat_candidates:
|
||||
return _no_pair("no se encontro columna de latitud valida (nombre+rango en [-90,90])")
|
||||
if not lon_candidates:
|
||||
return _no_pair("no se encontro columna de longitud valida (nombre+rango en [-180,180])")
|
||||
|
||||
# Pick the distinct pair with the highest combined confidence. First match wins
|
||||
# on ties to keep the result deterministic by input order.
|
||||
best = None # (combined, lat_name, lon_name, lat_c, lon_c)
|
||||
for lat_name, lat_c in lat_candidates:
|
||||
for lon_name, lon_c in lon_candidates:
|
||||
if lat_name == lon_name:
|
||||
continue # a column cannot be both axes of the same pair.
|
||||
combined = (lat_c + lon_c) / 2.0
|
||||
if best is None or combined > best[0]:
|
||||
best = (combined, lat_name, lon_name, lat_c, lon_c)
|
||||
|
||||
if best is None:
|
||||
return _no_pair("solo una columna sirve para ambos ejes; no hay par lat/lon distinto")
|
||||
|
||||
combined, lat_name, lon_name, lat_c, lon_c = best
|
||||
confidence = max(0.0, min(1.0, combined))
|
||||
|
||||
lat_label = "fuerte" if lat_c >= 0.9 else "debil"
|
||||
lon_label = "fuerte" if lon_c >= 0.9 else "debil"
|
||||
reason = (
|
||||
f"par lat='{lat_name}' (nombre {lat_label}) / lon='{lon_name}' "
|
||||
f"(nombre {lon_label}) con rango valido"
|
||||
)
|
||||
|
||||
return {
|
||||
"lat_col": lat_name,
|
||||
"lon_col": lon_name,
|
||||
"confidence": confidence,
|
||||
"reason": reason,
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Tests para detect_latlon_columns."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from detect_latlon_columns import detect_latlon_columns
|
||||
|
||||
# Keys that every result dict (success or failure) must expose.
|
||||
_EXPECTED_KEYS = {"lat_col", "lon_col", "confidence", "reason"}
|
||||
|
||||
|
||||
def _col(name, mn=None, mx=None, inferred="numeric"):
|
||||
"""Build a minimal ColumnProfile-like dict for the tests."""
|
||||
col = {"name": name, "inferred_type": inferred}
|
||||
if mn is not None or mx is not None:
|
||||
col["numeric"] = {"min": mn, "max": mx}
|
||||
return col
|
||||
|
||||
|
||||
def test_par_latitude_longitude_fuerte():
|
||||
"""Golden: nombres latitude/longitude con rango valido -> par con confianza alta."""
|
||||
columns = [
|
||||
_col("id", mn=1, mx=9999, inferred="numeric"),
|
||||
_col("latitude", mn=-45.0, mx=45.0),
|
||||
_col("longitude", mn=-120.0, mx=120.0),
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
|
||||
assert set(res.keys()) == _EXPECTED_KEYS
|
||||
assert res["lat_col"] == "latitude"
|
||||
assert res["lon_col"] == "longitude"
|
||||
# Nombre fuerte (0.6) + rango (0.4) en ambos lados -> 1.0.
|
||||
assert abs(res["confidence"] - 1.0) < 1e-9
|
||||
assert "rango valido" in res["reason"]
|
||||
|
||||
|
||||
def test_par_lat_lon_abreviado():
|
||||
"""Golden: nombres abreviados lat/lon tambien se detectan como fuertes."""
|
||||
columns = [
|
||||
_col("lat", mn=40.0, mx=43.0),
|
||||
_col("lon", mn=-4.0, mx=-1.0),
|
||||
_col("precio", mn=0.0, mx=500.0),
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
assert res["lat_col"] == "lat"
|
||||
assert res["lon_col"] == "lon"
|
||||
assert abs(res["confidence"] - 1.0) < 1e-9
|
||||
|
||||
|
||||
def test_par_x_y_debil_con_rango_valido():
|
||||
"""Edge: x/y genericos solo cuentan como par debil cuando el rango encaja."""
|
||||
columns = [
|
||||
_col("y_coord", mn=-10.0, mx=10.0), # debil latitud
|
||||
_col("x_coord", mn=-150.0, mx=150.0), # debil longitud
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
assert res["lat_col"] == "y_coord"
|
||||
assert res["lon_col"] == "x_coord"
|
||||
# Nombre debil (0.3) + rango (0.4) -> 0.7 en ambos lados.
|
||||
assert abs(res["confidence"] - 0.7) < 1e-9
|
||||
|
||||
|
||||
def test_nombre_lat_lon_pero_rango_fuera_no_detecta():
|
||||
"""Edge: nombre lat/lon con rango fuera de bounds -> NO es coordenada."""
|
||||
columns = [
|
||||
_col("latitude", mn=-200.0, mx=200.0), # fuera de [-90, 90]
|
||||
_col("longitude", mn=-120.0, mx=120.0), # valido, pero sin par lat
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
assert res["lat_col"] is None
|
||||
assert res["lon_col"] is None
|
||||
assert res["confidence"] == 0.0
|
||||
assert isinstance(res["reason"], str) and res["reason"]
|
||||
|
||||
|
||||
def test_par_fuerte_prevalece_sobre_debil():
|
||||
"""Edge: con candidatos fuertes y debiles, gana el par de mayor confianza."""
|
||||
columns = [
|
||||
_col("latitude", mn=-45.0, mx=45.0), # fuerte lat
|
||||
_col("y", mn=-30.0, mx=30.0), # debil lat
|
||||
_col("longitude", mn=-120.0, mx=120.0), # fuerte lon
|
||||
_col("x", mn=-100.0, mx=100.0), # debil lon
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
assert res["lat_col"] == "latitude"
|
||||
assert res["lon_col"] == "longitude"
|
||||
assert abs(res["confidence"] - 1.0) < 1e-9
|
||||
|
||||
|
||||
def test_entradas_vacias_o_invalidas_no_lanzan():
|
||||
"""Edge: sin columnas / vacio / no-lista / entradas no-dict -> dict None sin lanzar."""
|
||||
for bad in ([], None, "no soy lista", 42, [1, 2, 3], [{}], [{"foo": "bar"}]):
|
||||
res = detect_latlon_columns(bad)
|
||||
assert set(res.keys()) == _EXPECTED_KEYS
|
||||
assert res["lat_col"] is None
|
||||
assert res["lon_col"] is None
|
||||
assert res["confidence"] == 0.0
|
||||
assert isinstance(res["reason"], str)
|
||||
|
||||
|
||||
def test_solo_latitud_sin_longitud_no_detecta():
|
||||
"""Edge: solo hay latitud valida, falta la longitud -> sin par."""
|
||||
columns = [
|
||||
_col("latitude", mn=-45.0, mx=45.0),
|
||||
_col("temperatura", mn=-5.0, mx=40.0),
|
||||
]
|
||||
res = detect_latlon_columns(columns)
|
||||
assert res["lat_col"] is None
|
||||
assert res["lon_col"] is None
|
||||
assert res["confidence"] == 0.0
|
||||
|
||||
|
||||
def test_deteccion_por_samples_cuando_falta_numeric():
|
||||
"""Edge: sin bloque numeric, el rango se valida con samples."""
|
||||
columns = [
|
||||
{"name": "lat"}, # sin numeric ni inferred_type
|
||||
{"name": "lon"},
|
||||
]
|
||||
samples = {
|
||||
"lat": [10.5, 20.0, None, 30.25], # todos dentro de [-90, 90]
|
||||
"lon": [-40.0, 50.5, 60.0], # todos dentro de [-180, 180]
|
||||
}
|
||||
res = detect_latlon_columns(columns, samples)
|
||||
assert res["lat_col"] == "lat"
|
||||
assert res["lon_col"] == "lon"
|
||||
assert abs(res["confidence"] - 1.0) < 1e-9
|
||||
|
||||
|
||||
def test_samples_fuera_de_rango_descarta():
|
||||
"""Edge: samples fuera de bounds invalidan la columna pese al nombre fuerte."""
|
||||
columns = [{"name": "lat"}, {"name": "lon"}]
|
||||
samples = {
|
||||
"lat": [10.0, 95.0], # 95 > 90 -> latitud invalida
|
||||
"lon": [-40.0, 50.0],
|
||||
}
|
||||
res = detect_latlon_columns(columns, samples)
|
||||
assert res["lat_col"] is None
|
||||
assert res["lon_col"] is None
|
||||
assert res["confidence"] == 0.0
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: detect_time_column
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def detect_time_column(columns: list) -> dict"
|
||||
description: "Detecta, a partir de la lista de ColumnProfile de un TableProfile del grupo eda, cual es la columna de orden temporal y que columnas numericas hay para graficar una serie en el tiempo. Una columna es temporal si inferred_type=='datetime' o semantic_type in {datetime_iso, date_eu}; time_col es la primera temporal en orden. Es la pieza que usa el capitulo TIMESERIES del AutomaticEDA para decidir si aplica. Lectura defensiva dict-no-throw: nunca lanza, siempre devuelve las mismas claves."
|
||||
tags: [eda, timeseries, datetime, profiling, column-detection, automatic-eda, datascience, python]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: []
|
||||
params:
|
||||
- name: columns
|
||||
desc: "lista de ColumnProfile dict de un TableProfile del grupo eda. Cada elemento suele tener name, inferred_type, semantic_type y numeric. Elementos que no sean dict se ignoran; None/no-lista/vacia -> dict 'no aplica'."
|
||||
output: "dict SIEMPRE con: time_col (str|None, columna temporal elegida = primera temporal), time_semantic (str, semantic_type de la temporal o ''), numeric_cols (list[str], columnas con inferred_type=='numeric' en orden), n_datetime_cols (int), datetime_cols (list[str], todas las temporales en orden de aparicion), reason (str en espanol explicando la eleccion). Nunca lanza excepcion."
|
||||
tested: true
|
||||
tests: ["test_golden_datetime_y_numericas", "test_deteccion_por_semantic_type_date_eu", "test_sin_columna_temporal", "test_columns_none_no_revienta", "test_columns_vacia_no_revienta", "test_columns_no_lista_no_revienta", "test_elementos_basura_se_ignoran", "test_varias_datetime_elige_la_primera"]
|
||||
test_file_path: "python/functions/datascience/detect_time_column_test.py"
|
||||
file_path: "python/functions/datascience/detect_time_column.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from datascience import detect_time_column
|
||||
|
||||
columns = [
|
||||
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
|
||||
{"name": "ventas", "inferred_type": "numeric"},
|
||||
{"name": "unidades", "inferred_type": "numeric"},
|
||||
{"name": "region", "inferred_type": "text"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
res["time_col"] # -> "fecha"
|
||||
res["numeric_cols"] # -> ["ventas", "unidades"]
|
||||
res["n_datetime_cols"] # -> 1
|
||||
|
||||
# Sin columna temporal: el capitulo TIMESERIES no aplica.
|
||||
detect_time_column([{"name": "id", "inferred_type": "numeric"}])["time_col"] # -> None
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Cuando el capitulo TIMESERIES del AutomaticEDA recibe un TableProfile y necesita
|
||||
decidir si la tabla admite analisis de serie temporal: si `time_col` es None no
|
||||
hay eje de tiempo y el capitulo se salta; si hay `time_col` y `numeric_cols`,
|
||||
úsalas como eje X (orden cronologico) y series Y. Tambien sirve para enrutar el
|
||||
resto del pipeline (acf_pacf / stl_decompose / adf_kpss_stationarity) sobre las
|
||||
columnas numericas detectadas.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Es pura y stdlib-only (sin numpy ni DuckDB): segura de llamar en cualquier paso.
|
||||
- `time_col` se elige por ORDEN de aparicion en la lista, no por "mejor candidata".
|
||||
Si hay varias columnas datetime y quieres otra, filtra `datetime_cols` tu mismo.
|
||||
- Solo mira metadatos del perfil (`inferred_type`/`semantic_type`); no parsea ni
|
||||
valida los valores reales de la columna. La calidad de la deteccion depende de
|
||||
que el profiler (summarize_table_duckdb / infer_semantic_type) haya inferido bien.
|
||||
- Las claves del semantic_type son exactamente las del profiler: `datetime_iso`
|
||||
(ISO 8601) y `date_eu` (DD/MM/AAAA). Otros formatos de fecha no se detectan por
|
||||
semantic_type salvo que `inferred_type` ya sea `"datetime"`.
|
||||
- `numeric_cols` se basa en `inferred_type == "numeric"` (no en "integer"/"float");
|
||||
si tu profiler usa otra etiqueta, normalizala antes.
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Detecta la columna temporal y las columnas numericas de un TableProfile (grupo eda).
|
||||
|
||||
Funcion pura y determinista: a partir de la lista de columnas de un TableProfile
|
||||
producido por el grupo de capacidad `eda` (cada elemento es un ColumnProfile dict),
|
||||
decide cual es la columna de orden temporal y que columnas numericas hay disponibles
|
||||
para graficar una serie en el tiempo. Es la pieza que usa el capitulo TIMESERIES del
|
||||
AutomaticEDA para decidir si la tabla admite analisis de serie temporal.
|
||||
|
||||
Lectura 100% defensiva al estilo "dict-no-throw" del grupo eda: nunca lanza
|
||||
excepcion, siempre devuelve el mismo conjunto de claves.
|
||||
"""
|
||||
|
||||
# semantic_type que el profiler (infer_semantic_type) emite para fechas/datetimes.
|
||||
_DATETIME_SEMANTICS = ("datetime_iso", "date_eu")
|
||||
|
||||
|
||||
def detect_time_column(columns: list) -> dict:
|
||||
"""Detecta la columna temporal y las numericas de una lista de ColumnProfile.
|
||||
|
||||
Recorre los ColumnProfile de un TableProfile y clasifica cada columna como
|
||||
temporal o numerica leyendo de forma defensiva sus claves. Una columna es
|
||||
temporal si su ``inferred_type == "datetime"`` o si su ``semantic_type`` esta
|
||||
en {``"datetime_iso"``, ``"date_eu"``}. La columna temporal elegida
|
||||
(``time_col``) es la PRIMERA temporal en el orden de la lista. Las numericas
|
||||
(``numeric_cols``) son las de ``inferred_type == "numeric"``, en orden.
|
||||
|
||||
Funcion pura: no hace I/O, no muta el input, es determinista.
|
||||
|
||||
Args:
|
||||
columns: lista de ColumnProfile dict del grupo eda. Cada elemento suele
|
||||
tener claves como ``name``, ``inferred_type``, ``semantic_type`` y
|
||||
``numeric``. Los elementos que no sean dict se ignoran. Si ``columns``
|
||||
es None, no es lista o esta vacia, se devuelve el dict "no aplica".
|
||||
|
||||
Returns:
|
||||
Siempre un dict con las mismas claves::
|
||||
|
||||
{
|
||||
"time_col": str | None, # columna temporal elegida (None si no hay)
|
||||
"time_semantic": str, # semantic_type de la temporal ("" si no aplica)
|
||||
"numeric_cols": [str, ...], # columnas con inferred_type == "numeric"
|
||||
"n_datetime_cols": int, # nº de columnas temporales detectadas
|
||||
"datetime_cols": [str, ...],# todas las temporales, en orden de aparicion
|
||||
"reason": str, # frase corta (en espanol) que explica la eleccion
|
||||
}
|
||||
"""
|
||||
# Caso "no aplica": entrada invalida o vacia.
|
||||
if not isinstance(columns, list) or not columns:
|
||||
return {
|
||||
"time_col": None,
|
||||
"time_semantic": "",
|
||||
"numeric_cols": [],
|
||||
"n_datetime_cols": 0,
|
||||
"datetime_cols": [],
|
||||
"reason": "no se detecto columna de fecha/datetime",
|
||||
}
|
||||
|
||||
datetime_cols: list[str] = []
|
||||
datetime_semantics: list[str] = []
|
||||
numeric_cols: list[str] = []
|
||||
|
||||
for col in columns:
|
||||
# Ignora elementos que no sean dict sin fallar.
|
||||
if not isinstance(col, dict):
|
||||
continue
|
||||
|
||||
name = col.get("name")
|
||||
if name is None:
|
||||
name = ""
|
||||
else:
|
||||
name = str(name)
|
||||
|
||||
inferred_type = col.get("inferred_type") or ""
|
||||
semantic_type = col.get("semantic_type") or ""
|
||||
|
||||
is_datetime = inferred_type == "datetime" or semantic_type in _DATETIME_SEMANTICS
|
||||
if is_datetime:
|
||||
datetime_cols.append(name)
|
||||
datetime_semantics.append(semantic_type)
|
||||
|
||||
if inferred_type == "numeric":
|
||||
numeric_cols.append(name)
|
||||
|
||||
if not datetime_cols:
|
||||
return {
|
||||
"time_col": None,
|
||||
"time_semantic": "",
|
||||
"numeric_cols": numeric_cols,
|
||||
"n_datetime_cols": 0,
|
||||
"datetime_cols": [],
|
||||
"reason": "no se detecto columna de fecha/datetime",
|
||||
}
|
||||
|
||||
time_col = datetime_cols[0]
|
||||
time_semantic = datetime_semantics[0]
|
||||
|
||||
if len(datetime_cols) == 1:
|
||||
reason = f"columna temporal '{time_col}' detectada"
|
||||
else:
|
||||
reason = (
|
||||
f"{len(datetime_cols)} columnas temporales; se elige la primera "
|
||||
f"'{time_col}'"
|
||||
)
|
||||
|
||||
return {
|
||||
"time_col": time_col,
|
||||
"time_semantic": time_semantic,
|
||||
"numeric_cols": numeric_cols,
|
||||
"n_datetime_cols": len(datetime_cols),
|
||||
"datetime_cols": datetime_cols,
|
||||
"reason": reason,
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Tests para detect_time_column (grupo eda). Self-contained, sin DuckDB."""
|
||||
|
||||
from detect_time_column import detect_time_column
|
||||
|
||||
|
||||
def test_golden_datetime_y_numericas():
|
||||
columns = [
|
||||
{"name": "fecha", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
|
||||
{"name": "ventas", "inferred_type": "numeric"},
|
||||
{"name": "unidades", "inferred_type": "numeric"},
|
||||
{"name": "region", "inferred_type": "text"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
assert res["time_col"] == "fecha"
|
||||
assert res["time_semantic"] == "datetime_iso"
|
||||
assert res["numeric_cols"] == ["ventas", "unidades"]
|
||||
assert res["n_datetime_cols"] == 1
|
||||
assert res["datetime_cols"] == ["fecha"]
|
||||
assert isinstance(res["reason"], str) and res["reason"]
|
||||
|
||||
|
||||
def test_deteccion_por_semantic_type_date_eu():
|
||||
# inferred_type no es datetime, pero semantic_type date_eu => temporal.
|
||||
columns = [
|
||||
{"name": "id", "inferred_type": "numeric"},
|
||||
{"name": "dia", "inferred_type": "text", "semantic_type": "date_eu"},
|
||||
{"name": "importe", "inferred_type": "numeric"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
assert res["time_col"] == "dia"
|
||||
assert res["time_semantic"] == "date_eu"
|
||||
assert res["numeric_cols"] == ["id", "importe"]
|
||||
assert res["n_datetime_cols"] == 1
|
||||
assert res["datetime_cols"] == ["dia"]
|
||||
|
||||
|
||||
def test_sin_columna_temporal():
|
||||
columns = [
|
||||
{"name": "id", "inferred_type": "numeric"},
|
||||
{"name": "nombre", "inferred_type": "text"},
|
||||
{"name": "activo", "inferred_type": "boolean"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
assert res["time_col"] is None
|
||||
assert res["time_semantic"] == ""
|
||||
assert res["numeric_cols"] == ["id"]
|
||||
assert res["n_datetime_cols"] == 0
|
||||
assert res["datetime_cols"] == []
|
||||
assert res["reason"] == "no se detecto columna de fecha/datetime"
|
||||
|
||||
|
||||
def test_columns_none_no_revienta():
|
||||
res = detect_time_column(None)
|
||||
assert res["time_col"] is None
|
||||
assert res["time_semantic"] == ""
|
||||
assert res["numeric_cols"] == []
|
||||
assert res["n_datetime_cols"] == 0
|
||||
assert res["datetime_cols"] == []
|
||||
assert res["reason"] == "no se detecto columna de fecha/datetime"
|
||||
|
||||
|
||||
def test_columns_vacia_no_revienta():
|
||||
res = detect_time_column([])
|
||||
assert res["time_col"] is None
|
||||
assert res["numeric_cols"] == []
|
||||
assert res["n_datetime_cols"] == 0
|
||||
|
||||
|
||||
def test_columns_no_lista_no_revienta():
|
||||
# Un dict (no lista) tambien debe caer en el caso "no aplica".
|
||||
res = detect_time_column({"name": "fecha", "inferred_type": "datetime"})
|
||||
assert res["time_col"] is None
|
||||
assert res["numeric_cols"] == []
|
||||
|
||||
|
||||
def test_elementos_basura_se_ignoran():
|
||||
columns = [
|
||||
None,
|
||||
"no soy un dict",
|
||||
42,
|
||||
{"name": "ts", "inferred_type": "datetime"},
|
||||
{"name": "valor", "inferred_type": "numeric"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
assert res["time_col"] == "ts"
|
||||
assert res["numeric_cols"] == ["valor"]
|
||||
assert res["n_datetime_cols"] == 1
|
||||
|
||||
|
||||
def test_varias_datetime_elige_la_primera():
|
||||
columns = [
|
||||
{"name": "created_at", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
|
||||
{"name": "metric", "inferred_type": "numeric"},
|
||||
{"name": "updated_at", "inferred_type": "datetime", "semantic_type": "datetime_iso"},
|
||||
{"name": "fecha_baja", "inferred_type": "text", "semantic_type": "date_eu"},
|
||||
]
|
||||
res = detect_time_column(columns)
|
||||
assert res["time_col"] == "created_at"
|
||||
assert res["time_semantic"] == "datetime_iso"
|
||||
assert res["n_datetime_cols"] == 3
|
||||
assert res["datetime_cols"] == ["created_at", "updated_at", "fecha_baja"]
|
||||
assert res["numeric_cols"] == ["metric"]
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: extract_timeseries_raw
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: impure
|
||||
signature: "def extract_timeseries_raw(query_fn, table: str, time_col: str, value_cols: list, max_rows: int = 5000) -> dict"
|
||||
description: "Extrae la serie temporal CRUDA (fechas + una o varias columnas numericas) de una tabla, ordenada cronologicamente, para alimentar el render del capitulo TIMESERIES de AutomaticEDA (linea valor-vs-tiempo + conteo por periodo). Recibe un lector read-only inyectado `query_fn(sql) -> dict` (mismo contrato que duckdb_query_readonly / pg_query / el `_q` de profile_table) y NO abre ninguna conexion por su cuenta. Construye UNA sola query con identificadores escapados, ORDER BY por la columna temporal y LIMIT. Devuelve dict dict-no-throw: t (fechas ISO string), series (lista paralela float|None por columna) y n. El capitulo no toca la BD: recibe esto en ctx['timeseries_raw']. Reutilizable tambien por profile_table en una fase futura."
|
||||
tags: [eda, timeseries, datascience, automatic-eda, extraction, read-only, duckdb, postgres, python]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: "error_go_core"
|
||||
imports: [datetime]
|
||||
params:
|
||||
- name: query_fn
|
||||
desc: "callable lector read-only del backend activo. Recibe un string SQL y devuelve un dict {'status':'ok','rows':[{col:val,...},...]} (mismo contrato que duckdb_query_readonly o el `_q` de profile_table). NO se abre ninguna conexion dentro de la funcion: toda la lectura pasa por query_fn. Si es None -> error."
|
||||
- name: table
|
||||
desc: "nombre de la tabla de la que extraer la serie. Se escapa con comillas dobles en la query."
|
||||
- name: time_col
|
||||
desc: "nombre de la columna de orden temporal. Se usa en ORDER BY (cronologico ascendente) y se filtra IS NOT NULL. Sus valores se devuelven en `t` como string ISO."
|
||||
- name: value_cols
|
||||
desc: "lista de nombres de columnas numericas a extraer. Cada una produce una entrada en `series` con una lista paralela a `t`. Vacia o None -> status error."
|
||||
- name: max_rows
|
||||
desc: "limite de filas a leer (clausula LIMIT). Default 5000. Protege el render frente a tablas enormes."
|
||||
output: "dict (nunca lanza). En exito: {'status':'ok','time_col':str,'t':[str,...] (fechas ISO en orden),'series':{col:[float|None,...],...} (paralela a t por value_col, None si el valor no es convertible a float),'n':int}. En error (sin lanzar): {'status':'error','error':str,'time_col':str,'t':[],'series':{},'n':0}. Errores: query_fn None, value_cols vacia, table/time_col vacios, o query_fn devuelve status!='ok' (se propaga su error)."
|
||||
tested: true
|
||||
tests: ["test_golden_t_y_series_alineadas", "test_valor_no_convertible_da_none", "test_value_cols_vacia_status_error", "test_query_fn_status_error_propaga", "test_query_fn_none_da_error_sin_reventar", "test_sql_contiene_order_by_y_limit"]
|
||||
test_file_path: "python/functions/datascience/extract_timeseries_raw_test.py"
|
||||
file_path: "python/functions/datascience/extract_timeseries_raw.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience import extract_timeseries_raw
|
||||
from infra import duckdb_query_readonly
|
||||
|
||||
# El lector read-only se inyecta como closure (igual que el `_q` de profile_table).
|
||||
db = "data/ventas.duckdb"
|
||||
def _q(sql):
|
||||
return duckdb_query_readonly(db, sql)
|
||||
|
||||
res = extract_timeseries_raw(_q, "ventas_diarias", "fecha", ["importe", "unidades"])
|
||||
# res == {
|
||||
# "status": "ok",
|
||||
# "time_col": "fecha",
|
||||
# "t": ["2024-01-01", "2024-01-02", ...],
|
||||
# "series": {"importe": [1234.5, 980.0, ...], "unidades": [12.0, 9.0, ...]},
|
||||
# "n": 365,
|
||||
# }
|
||||
|
||||
# Se entrega al capitulo TIMESERIES sin que este toque la BD:
|
||||
ctx = {"timeseries_raw": res}
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Cuando el capitulo TIMESERIES de AutomaticEDA necesita pintar una serie
|
||||
valor-vs-tiempo (o conteo por periodo) y NO debe abrir la base de datos por su
|
||||
cuenta: extraes aqui las fechas + columnas numericas ordenadas y se las pasas en
|
||||
`ctx['timeseries_raw']`. Usala tambien siempre que quieras la secuencia cruda
|
||||
ordenada cronologicamente de una o varias columnas para alimentar otros
|
||||
contrastes de serie (ADF/KPSS, ACF/PACF, STL) reutilizando un unico lector
|
||||
read-only inyectado, en vez de hacer N muestreos a mano.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Impura**: lee de la base de datos a traves de `query_fn`. No abre conexiones
|
||||
por su cuenta — depende por completo del lector inyectado. Sigue el estilo
|
||||
dict-no-throw del grupo `eda`: nunca lanza; ante cualquier fallo devuelve
|
||||
`{"status":"error","error":...}` con `t=[]`, `series={}`, `n=0`.
|
||||
- **`error_type` en el frontmatter es `error_go_core` por convencion del registry**
|
||||
(toda funcion impura debe declararlo y el indexer lo exige), pero el codigo
|
||||
NO lanza esa excepcion: degrada al dict de error. Es metadata, no comportamiento.
|
||||
- **No loguear los datos crudos**: `t`/`series` pueden contener datos sensibles
|
||||
(igual que un HAR). No volcar el dict completo a logs ni a telemetria; en
|
||||
trazas usa solo `n` y los nombres de columna.
|
||||
- **Alineacion por fila**: `series[col][i]` corresponde a `t[i]`. Un valor no
|
||||
convertible a float se guarda como `None` (no se descarta la fila) para no
|
||||
romper la alineacion temporal.
|
||||
- **Orden**: el orden cronologico depende del `ORDER BY "time_col"` del backend.
|
||||
Si `time_col` esta guardada como texto con formato no lexicograficamente
|
||||
ordenable (p.ej. `DD/MM/YYYY`), el orden no sera el real — normaliza la columna
|
||||
a date/timestamp antes, o pasa una columna ya ordenable.
|
||||
- **`max_rows`**: con LIMIT, si la tabla supera `max_rows` obtienes solo el primer
|
||||
tramo cronologico, no un muestreo uniforme. Sube `max_rows` si necesitas el rango
|
||||
completo.
|
||||
@@ -0,0 +1,122 @@
|
||||
"""extract_timeseries_raw — extrae la serie temporal CRUDA de una tabla.
|
||||
|
||||
Lector read-only inyectado: recibe `query_fn(sql) -> dict` con el mismo contrato
|
||||
que duckdb_query_readonly / pg_query (y que el `_q` de profile_table):
|
||||
`{"status": "ok", "rows": [{col: val, ...}, ...]}`. Esta funcion NO abre ninguna
|
||||
conexion por su cuenta — solo usa `query_fn`. Construye UNA sola query ordenada
|
||||
por la columna temporal y devuelve las fechas (`t`) mas cada columna numerica en
|
||||
listas paralelas (`series`), listas para alimentar el render del capitulo
|
||||
TIMESERIES de AutomaticEDA (linea valor-vs-tiempo + conteo por periodo) sin que
|
||||
el capitulo toque la base de datos: recibe esto en `ctx['timeseries_raw']`.
|
||||
|
||||
Estilo dict-no-throw del grupo `eda`: nunca lanza; captura cualquier excepcion y
|
||||
degrada a `{"status": "error", "error": str, ...}`.
|
||||
"""
|
||||
|
||||
from datetime import date, datetime
|
||||
|
||||
|
||||
def _to_float(value):
|
||||
"""Convierte un valor a float de forma defensiva. None si no es convertible."""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
# Un bool es subclase de int en Python; no es un valor de serie valido.
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
s = str(value).strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return float(s)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _to_iso(value):
|
||||
"""Convierte un valor temporal a string ISO conservando el orden de la query.
|
||||
|
||||
date/datetime -> isoformat(); cualquier otro valor (string, etc.) -> str().
|
||||
None se preserva como None.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, (datetime, date)):
|
||||
return value.isoformat()
|
||||
return str(value)
|
||||
|
||||
|
||||
def extract_timeseries_raw(query_fn, table, time_col, value_cols, max_rows=5000):
|
||||
"""Extrae la serie temporal cruda (fechas + columnas numericas) de una tabla.
|
||||
|
||||
Args:
|
||||
query_fn: callable lector read-only del backend activo. Recibe un string
|
||||
SQL y devuelve un dict {"status": "ok", "rows": [{col: val, ...}]}
|
||||
(mismo contrato que duckdb_query_readonly / el `_q` de profile_table).
|
||||
No se abre ninguna conexion aqui: toda la lectura pasa por query_fn.
|
||||
table: nombre de la tabla.
|
||||
time_col: nombre de la columna de orden temporal.
|
||||
value_cols: lista de nombres de columnas numericas a extraer.
|
||||
max_rows: limite de filas (LIMIT). Default 5000.
|
||||
|
||||
Returns:
|
||||
dict (nunca lanza):
|
||||
{
|
||||
"status": "ok" | "error",
|
||||
"error": str, # solo si status == "error"
|
||||
"time_col": str,
|
||||
"t": [str, ...], # time_col como ISO string, en orden
|
||||
"series": {col: [float|None, ...], ...}, # paralela a t por columna
|
||||
"n": int # nº de filas devueltas
|
||||
}
|
||||
"""
|
||||
base = {"status": "ok", "time_col": time_col, "t": [], "series": {}, "n": 0}
|
||||
try:
|
||||
if query_fn is None:
|
||||
return {**base, "status": "error", "error": "query_fn es None"}
|
||||
if not value_cols:
|
||||
return {**base, "status": "error", "error": "value_cols vacío"}
|
||||
if not table or not time_col:
|
||||
return {
|
||||
**base,
|
||||
"status": "error",
|
||||
"error": "table y time_col son obligatorios",
|
||||
}
|
||||
|
||||
# Identificadores escapados con comillas dobles (como hace profile_table)
|
||||
# para tolerar nombres con mayusculas/espacios/palabras reservadas.
|
||||
cols_sql = ", ".join(f'"{c}"' for c in value_cols)
|
||||
sql = (
|
||||
f'SELECT "{time_col}", {cols_sql} FROM "{table}" '
|
||||
f'WHERE "{time_col}" IS NOT NULL '
|
||||
f'ORDER BY "{time_col}" '
|
||||
f"LIMIT {int(max_rows)}"
|
||||
)
|
||||
|
||||
q = query_fn(sql)
|
||||
if not isinstance(q, dict) or q.get("status") != "ok":
|
||||
err = (
|
||||
q.get("error", "query_fn fallo")
|
||||
if isinstance(q, dict)
|
||||
else "query_fn no devolvio un dict"
|
||||
)
|
||||
return {**base, "status": "error", "error": err}
|
||||
|
||||
rows = q.get("rows", []) or []
|
||||
t = []
|
||||
series = {c: [] for c in value_cols}
|
||||
for row in rows:
|
||||
t.append(_to_iso(row.get(time_col)))
|
||||
for c in value_cols:
|
||||
series[c].append(_to_float(row.get(c)))
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"time_col": time_col,
|
||||
"t": t,
|
||||
"series": series,
|
||||
"n": len(t),
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 - dict-no-throw: degradar, nunca lanzar
|
||||
return {**base, "status": "error", "error": str(e)}
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Tests para extract_timeseries_raw.
|
||||
|
||||
No usa DuckDB real: inyecta un query_fn FAKE (closure) que devuelve filas
|
||||
predefinidas y, opcionalmente, captura el SQL recibido para verificar la query
|
||||
generada (ORDER BY por la columna temporal + LIMIT). Asi el test es
|
||||
autocontenido y no depende de ningun backend.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from extract_timeseries_raw import extract_timeseries_raw
|
||||
|
||||
|
||||
def _fake_query(rows, captured=None, status="ok", error=None):
|
||||
"""Crea un query_fn FAKE.
|
||||
|
||||
`captured` (lista opcional) recibe el SQL ejecutado para poder inspeccionarlo.
|
||||
`status`/`error` permiten simular un fallo del backend.
|
||||
"""
|
||||
|
||||
def _q(sql):
|
||||
if captured is not None:
|
||||
captured.append(sql)
|
||||
if status != "ok":
|
||||
return {"status": "error", "error": error or "boom"}
|
||||
return {"status": "ok", "rows": rows}
|
||||
|
||||
return _q
|
||||
|
||||
|
||||
def test_golden_t_y_series_alineadas():
|
||||
"""Golden: t y series alineadas, floats convertidos, n correcto."""
|
||||
rows = [
|
||||
{"fecha": "2024-01-01", "ventas": "10", "stock": 5},
|
||||
{"fecha": "2024-01-02", "ventas": "20.5", "stock": 7},
|
||||
{"fecha": "2024-01-03", "ventas": 30, "stock": 9},
|
||||
]
|
||||
res = extract_timeseries_raw(_fake_query(rows), "t", "fecha", ["ventas", "stock"])
|
||||
assert res["status"] == "ok"
|
||||
assert res["n"] == 3
|
||||
assert res["time_col"] == "fecha"
|
||||
assert res["t"] == ["2024-01-01", "2024-01-02", "2024-01-03"]
|
||||
assert res["series"]["ventas"] == [10.0, 20.5, 30.0]
|
||||
assert res["series"]["stock"] == [5.0, 7.0, 9.0]
|
||||
|
||||
|
||||
def test_valor_no_convertible_da_none():
|
||||
"""Valor no convertible a float -> None en la serie (alineacion preservada)."""
|
||||
rows = [
|
||||
{"fecha": "2024-01-01", "ventas": "abc"},
|
||||
{"fecha": "2024-01-02", "ventas": None},
|
||||
{"fecha": "2024-01-03", "ventas": "12.5"},
|
||||
]
|
||||
res = extract_timeseries_raw(_fake_query(rows), "t", "fecha", ["ventas"])
|
||||
assert res["status"] == "ok"
|
||||
assert res["series"]["ventas"] == [None, None, 12.5]
|
||||
assert res["n"] == 3
|
||||
|
||||
|
||||
def test_value_cols_vacia_status_error():
|
||||
"""value_cols vacia -> status error con t/series/n vacios."""
|
||||
res = extract_timeseries_raw(_fake_query([]), "t", "fecha", [])
|
||||
assert res["status"] == "error"
|
||||
assert "value_cols" in res["error"]
|
||||
assert res["t"] == []
|
||||
assert res["series"] == {}
|
||||
assert res["n"] == 0
|
||||
|
||||
|
||||
def test_query_fn_status_error_propaga():
|
||||
"""query_fn que devuelve status != ok -> se propaga como error."""
|
||||
res = extract_timeseries_raw(
|
||||
_fake_query([], status="error", error="db locked"),
|
||||
"t",
|
||||
"fecha",
|
||||
["ventas"],
|
||||
)
|
||||
assert res["status"] == "error"
|
||||
assert "db locked" in res["error"]
|
||||
assert res["n"] == 0
|
||||
|
||||
|
||||
def test_query_fn_none_da_error_sin_reventar():
|
||||
"""query_fn None -> error degradado, sin excepcion."""
|
||||
res = extract_timeseries_raw(None, "t", "fecha", ["ventas"])
|
||||
assert res["status"] == "error"
|
||||
assert res["t"] == []
|
||||
assert res["n"] == 0
|
||||
|
||||
|
||||
def test_sql_contiene_order_by_y_limit():
|
||||
"""La query generada ordena por time_col y aplica el LIMIT sobre la tabla."""
|
||||
captured = []
|
||||
rows = [{"fecha": "2024-01-01", "ventas": 1}]
|
||||
extract_timeseries_raw(
|
||||
_fake_query(rows, captured),
|
||||
"ventas_tbl",
|
||||
"fecha",
|
||||
["ventas"],
|
||||
max_rows=123,
|
||||
)
|
||||
assert len(captured) == 1
|
||||
sql = captured[0]
|
||||
assert 'ORDER BY "fecha"' in sql
|
||||
assert "LIMIT 123" in sql
|
||||
assert 'FROM "ventas_tbl"' in sql
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: pptx_link_run_to_slide
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: impure
|
||||
signature: "def pptx_link_run_to_slide(run, source_slide, target_slide) -> bool"
|
||||
description: "Convierte un run de texto de python-pptx en un hyperlink INTERNO 'ir a la diapositiva'. python-pptx soporta run.hyperlink.address para URLs externas pero NO para saltar a otra slide del mismo deck; esta función crea ese salto manipulando el XML: añade una relación slide->slide (RT.SLIDE) y un <a:hlinkClick> con action='ppaction://hlinksldjump' y el r:id de la relación, insertado como primer hijo del <a:rPr> del run (orden del schema CT_TextCharacterProperties). Idempotente (elimina un hlinkClick previo antes de insertar). Al pulsar el texto en PowerPoint o visores compatibles se navega a target_slide. Motor python-pptx. No lanza nunca: cualquier excepción -> return False."
|
||||
tags: [eda, pptx, hyperlink, slide-jump, navigation, glossary, automatic-eda, python-pptx, xml, datascience, python]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: "error_go_core"
|
||||
imports: ["python-pptx"]
|
||||
params:
|
||||
- name: run
|
||||
desc: "el pptx.text.text._Run cuyo texto se vuelve clicable. Debe pertenecer a un run real (expone ._r, el elemento <a:r>). Un objeto sin ._r hace que la función devuelva False sin lanzar."
|
||||
- name: source_slide
|
||||
desc: "la Slide que contiene el run. Su part recibe la relación slide->slide (relate_to con RELATIONSHIP_TYPE.SLIDE); el r:id resultante se referencia en el hlinkClick."
|
||||
- name: target_slide
|
||||
desc: "la Slide de destino del salto. Debe pertenecer al MISMO Presentation que source_slide para que la relación interna sea válida."
|
||||
output: "bool. True si se aplicó el hyperlink interno (relación creada + <a:hlinkClick> insertado en el rPr del run); False si algo lo impidió (run inválido, slides de presentaciones distintas, etc.). Nunca lanza."
|
||||
tested: true
|
||||
tests: ["test_golden_run_se_vuelve_salto_a_otra_slide", "test_idempotente_reaplica_sin_duplicar_hlinkclick", "test_error_path_run_invalido_devuelve_false_sin_lanzar"]
|
||||
test_file_path: "python/functions/datascience/pptx_link_run_to_slide_test.py"
|
||||
file_path: "python/functions/datascience/pptx_link_run_to_slide.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from pptx import Presentation
|
||||
from pptx.util import Inches
|
||||
from pptx.oxml.ns import qn
|
||||
|
||||
from datascience.pptx_link_run_to_slide import pptx_link_run_to_slide
|
||||
|
||||
prs = Presentation()
|
||||
blank = prs.slide_layouts[6] # layout en blanco
|
||||
slide0 = prs.slides.add_slide(blank)
|
||||
slide1 = prs.slides.add_slide(blank) # destino del salto (p.ej. el glosario)
|
||||
|
||||
box = slide0.shapes.add_textbox(Inches(1), Inches(1), Inches(4), Inches(1))
|
||||
run = box.text_frame.paragraphs[0].add_run()
|
||||
run.text = "ir al glosario"
|
||||
|
||||
ok = pptx_link_run_to_slide(run, slide0, slide1)
|
||||
print(ok) # -> True
|
||||
|
||||
# El run quedó con <a:rPr><a:hlinkClick action="ppaction://hlinksldjump" r:id="rIdN"/></a:rPr>
|
||||
hlink = run._r.get_or_add_rPr().find(qn("a:hlinkClick"))
|
||||
print(hlink.get("action")) # -> ppaction://hlinksldjump
|
||||
prs.save("deck_con_salto.pptx")
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Cuando construyas un deck PPTX con **navegación interna** y quieras que un texto salte a
|
||||
otra diapositiva al pulsarlo: un **glosario clicable** (cada término enlaza a su slide de
|
||||
definición), un **índice/tabla de contenidos navegable**, botones "volver a la portada", o
|
||||
referencias cruzadas entre capítulos. Es la pieza que `python-pptx` no cubre de fábrica —
|
||||
úsala sobre los runs ya creados por renderers como `render_automatic_eda_pptx` del grupo
|
||||
`eda` para enriquecer el deck con saltos sin reescribir el XML a mano cada vez.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Impura**: muta el XML del run y crea una relación nueva en el part de `source_slide`.
|
||||
- **Solo navega en visores que respetan `ppaction://hlinksldjump`**: PowerPoint y la
|
||||
mayoría de visores compatibles lo siguen; algunos visores web/ligeros lo ignoran (el
|
||||
texto se ve igual pero no salta).
|
||||
- **Mismo Presentation**: `source_slide` y `target_slide` deben pertenecer al mismo deck.
|
||||
Si son de presentaciones distintas, la relación interna no es válida y el salto no
|
||||
funcionará (la función puede devolver True por crear la relación, pero el resultado en
|
||||
el visor no será el esperado).
|
||||
- **El `<a:hlinkClick>` vive en el `<a:rPr>` del run**, no como hijo directo del `<a:r>`.
|
||||
Para localizarlo: `run._r.get_or_add_rPr().find(qn("a:hlinkClick"))` (un `find` sobre
|
||||
`run._r` devuelve `None` porque solo mira hijos directos del `<a:r>`).
|
||||
- **Idempotente**: si el run ya tenía un `hlinkClick` (p.ej. una URL externa o un salto
|
||||
previo), se elimina antes de insertar el nuevo — un run tiene como mucho un click-link.
|
||||
- **Nunca lanza**: cualquier excepción (run sin `._r`, slides incompatibles, etc.) se
|
||||
traga y devuelve `False`. Comprobar el booleano si el salto es crítico.
|
||||
- **Dependencia python-pptx**: declarada en `python/pyproject.toml`. Tests con
|
||||
`~/fn_registry/python/.venv/bin/python3` (tiene `python-pptx` instalado).
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Convierte un run de texto de python-pptx en un hyperlink interno "ir a la diapositiva".
|
||||
|
||||
python-pptx expone ``run.hyperlink.address`` para URLs externas, pero NO ofrece una
|
||||
API pública para saltar a otra diapositiva del mismo deck. Esta función crea ese salto
|
||||
interno manipulando el XML: añade una relación ``slide -> slide`` y un
|
||||
``<a:hlinkClick>`` con la acción ``ppaction://hlinksldjump`` en el run, de modo que al
|
||||
pulsar el texto en PowerPoint (o en visores que respetan esa acción) se navega a la
|
||||
diapositiva de destino.
|
||||
"""
|
||||
|
||||
from pptx.opc.constants import RELATIONSHIP_TYPE as RT
|
||||
from pptx.oxml.ns import qn
|
||||
|
||||
|
||||
def pptx_link_run_to_slide(run, source_slide, target_slide) -> bool:
|
||||
"""Convierte un run de texto en un hyperlink interno "ir a la diapositiva".
|
||||
|
||||
Añade una relación ``slide -> slide`` desde la slide origen al part de la slide
|
||||
destino y crea un ``<a:hlinkClick>`` con ``action="ppaction://hlinksldjump"`` como
|
||||
primer hijo del ``<a:rPr>`` del run (orden válido del schema
|
||||
``CT_TextCharacterProperties``). La operación es idempotente: un ``hlinkClick``
|
||||
previo en el mismo run se elimina antes de insertar el nuevo.
|
||||
|
||||
Args:
|
||||
run: el ``pptx.text.text._Run`` cuyo texto se vuelve clicable.
|
||||
source_slide: la ``Slide`` que contiene el run.
|
||||
target_slide: la ``Slide`` de destino del salto.
|
||||
|
||||
Returns:
|
||||
True si se aplicó el hyperlink; False si algo impidió aplicarlo (no lanza).
|
||||
"""
|
||||
try:
|
||||
rId = source_slide.part.relate_to(target_slide.part, RT.SLIDE)
|
||||
rPr = run._r.get_or_add_rPr()
|
||||
# Elimina un hlinkClick previo si lo hubiera (idempotente).
|
||||
for existing in rPr.findall(qn("a:hlinkClick")):
|
||||
rPr.remove(existing)
|
||||
hlink = rPr.makeelement(
|
||||
qn("a:hlinkClick"),
|
||||
{
|
||||
qn("r:id"): rId,
|
||||
"action": "ppaction://hlinksldjump",
|
||||
},
|
||||
)
|
||||
# a:hlinkClick debe ir como primer hijo de rPr
|
||||
# (orden del schema CT_TextCharacterProperties).
|
||||
rPr.insert(0, hlink)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Tests for pptx_link_run_to_slide — salto interno run -> diapositiva.
|
||||
|
||||
Self-contained: construye una Presentation en memoria con dos slides en blanco,
|
||||
un textbox con un run en la slide 0, y verifica que la función inyecta un
|
||||
``<a:hlinkClick>`` con ``action="ppaction://hlinksldjump"`` y un ``r:id`` que
|
||||
resuelve al part de la slide 1.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("pptx")
|
||||
|
||||
from pptx import Presentation # noqa: E402
|
||||
from pptx.oxml.ns import qn # noqa: E402
|
||||
from pptx.util import Inches # noqa: E402
|
||||
|
||||
from datascience.pptx_link_run_to_slide import pptx_link_run_to_slide # noqa: E402
|
||||
|
||||
|
||||
def _two_slide_deck_with_run():
|
||||
prs = Presentation()
|
||||
blank = prs.slide_layouts[6] # layout en blanco
|
||||
slide0 = prs.slides.add_slide(blank)
|
||||
slide1 = prs.slides.add_slide(blank)
|
||||
|
||||
box = slide0.shapes.add_textbox(Inches(1), Inches(1), Inches(4), Inches(1))
|
||||
tf = box.text_frame
|
||||
para = tf.paragraphs[0]
|
||||
run = para.add_run()
|
||||
run.text = "ir al glosario"
|
||||
return prs, slide0, slide1, run
|
||||
|
||||
|
||||
def test_golden_run_se_vuelve_salto_a_otra_slide():
|
||||
prs, slide0, slide1, run = _two_slide_deck_with_run()
|
||||
|
||||
ok = pptx_link_run_to_slide(run, slide0, slide1)
|
||||
assert ok is True
|
||||
|
||||
# El hlinkClick es hijo del rPr del run (orden del schema
|
||||
# CT_TextCharacterProperties), no hijo directo del <a:r>.
|
||||
rPr = run._r.get_or_add_rPr()
|
||||
hlink = rPr.find(qn("a:hlinkClick"))
|
||||
assert hlink is not None
|
||||
assert hlink.get("action") == "ppaction://hlinksldjump"
|
||||
|
||||
rId = hlink.get(qn("r:id"))
|
||||
assert rId, "el hlinkClick debe llevar un r:id no vacío"
|
||||
|
||||
# El rId debe existir en las relaciones de la slide origen y apuntar
|
||||
# al part de la slide destino.
|
||||
rels = slide0.part.rels
|
||||
assert rId in rels
|
||||
assert rels[rId].target_part is slide1.part
|
||||
|
||||
|
||||
def test_idempotente_reaplica_sin_duplicar_hlinkclick():
|
||||
prs, slide0, slide1, run = _two_slide_deck_with_run()
|
||||
|
||||
assert pptx_link_run_to_slide(run, slide0, slide1) is True
|
||||
assert pptx_link_run_to_slide(run, slide0, slide1) is True
|
||||
|
||||
rPr = run._r.get_or_add_rPr()
|
||||
hlinks = rPr.findall(qn("a:hlinkClick"))
|
||||
assert len(hlinks) == 1
|
||||
|
||||
|
||||
def test_error_path_run_invalido_devuelve_false_sin_lanzar():
|
||||
prs, slide0, slide1, _run = _two_slide_deck_with_run()
|
||||
|
||||
# Un objeto sin ._r ni soporte de relación -> la función no lanza, devuelve False.
|
||||
ok = pptx_link_run_to_slide(object(), slide0, slide1)
|
||||
assert ok is False
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: profile_datetime
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def profile_datetime(values: list) -> dict"
|
||||
description: "Perfil minimo de una columna fecha/datetime para la cabecera del capitulo TIMESERIES de AutomaticEDA. Acepta datetime.date, datetime.datetime y strings ISO mezclados, parsea defensivamente e ignora lo no parseable (nunca lanza). Devuelve rango (min/max ISO), n, n_distinct, span_days, frecuencia inferida (daily/weekly/monthly/quarterly/yearly/irregular/unknown) a partir del paso mediano entre fechas distintas, is_regular (pasos ~constantes), n_gaps (huecos en la rejilla) y median_step_days. Solo stdlib (datetime + statistics)."
|
||||
tags: [statistics, timeseries, datetime, profiling, frequency, eda, automatic_eda, python]
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: [datetime, statistics]
|
||||
params:
|
||||
- name: values
|
||||
desc: "lista de valores fecha. Acepta datetime.date, datetime.datetime y strings ISO ('2021-06-28', '2021-06-28T00:00:00', '2021-06-28 12:00:00'). None, vacios y no parseables se ignoran; tz-aware se normaliza a naive. Si values es None o no iterable se trata como lista vacia."
|
||||
output: "dict SIEMPRE presente con: 'min'/'max' (ISO date YYYY-MM-DD o None), 'n' (valores parseables), 'n_distinct' (fechas unicas), 'span_days' (float o None), 'freq' (daily|weekly|monthly|quarterly|yearly|irregular|unknown), 'is_regular' (bool), 'n_gaps' (int), 'median_step_days' (float o None) y 'note' (str). Con <2 valores o una sola fecha distinta: freq='unknown', is_regular=False, n_gaps=0, median_step_days=None y nota. Nunca lanza."
|
||||
tested: true
|
||||
tests: ["test_serie_diaria_regular_golden", "test_serie_mensual_freq_monthly", "test_serie_con_hueco_cuenta_gaps", "test_strings_iso_mezclados_con_datetime", "test_lista_vacia_y_none_devuelve_unknown", "test_valores_no_parseables_ignorados", "test_span_days_correcto", "test_una_sola_fecha_es_coherente"]
|
||||
test_file_path: "python/functions/datascience/profile_datetime_test.py"
|
||||
file_path: "python/functions/datascience/profile_datetime.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from datascience import profile_datetime
|
||||
from datetime import date, datetime, timedelta
|
||||
|
||||
# Serie diaria regular de 30 dias
|
||||
fechas = [date(2021, 1, 1) + timedelta(days=i) for i in range(30)]
|
||||
res = profile_datetime(fechas)
|
||||
res["freq"] # -> "daily"
|
||||
res["is_regular"] # -> True
|
||||
res["n_gaps"] # -> 0
|
||||
res["min"], res["max"] # -> ("2021-01-01", "2021-01-30")
|
||||
res["span_days"] # -> 29.0
|
||||
|
||||
# Acepta strings ISO mezclados con objetos datetime/date; ignora lo no parseable
|
||||
profile_datetime(["2021-06-28", datetime(2021, 6, 29, 12), "basura", None])["n"] # -> 2
|
||||
|
||||
# Columna vacia o sin fechas validas
|
||||
profile_datetime([])["freq"] # -> "unknown" + note "datos insuficientes"
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Cuando construyes la cabecera del capitulo TIMESERIES de un EDA y necesitas
|
||||
caracterizar la columna de fecha antes de modelar: que rango cubre, cada cuanto
|
||||
llegan los datos (frecuencia), si la cadencia es regular y si hay huecos en la
|
||||
rejilla temporal. Es el complemento de fecha al perfil numerico/categorico del
|
||||
TableProfile (cierra el `datetime{}=None` pendiente). Pasale la columna de fechas
|
||||
en bruto (tal cual venga de la BD: dates, datetimes o strings ISO) y usa `freq` +
|
||||
`is_regular` + `n_gaps` para decidir si conviene resamplear, rellenar huecos o
|
||||
desestacionalizar mas adelante.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Es pura y stdlib-only, pero la inferencia de `freq` es heuristica por bandas
|
||||
sobre el **paso mediano entre fechas distintas** (se deduplica antes de medir).
|
||||
Cualquier paso fuera de las bandas conocidas (incluido sub-diario, p.ej. datos
|
||||
horarios) cae en `"irregular"`: no hay banda hourly.
|
||||
- El analisis de frecuencia/regularidad/huecos necesita **>=2 fechas distintas**.
|
||||
Con 0-1 valores parseables o una sola fecha unica, `freq="unknown"`,
|
||||
`median_step_days=None` y `n_gaps=0`, pero `min`/`max`/`span_days` siguen siendo
|
||||
coherentes si hay al menos una fecha.
|
||||
- `min`/`max` se reportan como ISO **date** (`YYYY-MM-DD`); la hora se conserva
|
||||
internamente para calcular `span_days` y `median_step_days` (que pueden ser
|
||||
fraccionarios con datetimes sub-diarios) pero no aparece en min/max.
|
||||
- Los datetime con zona horaria se normalizan a naive (se descarta el tzinfo) para
|
||||
poder mezclarlos con fechas naive sin que las restas lancen; esto puede desplazar
|
||||
la fecha en datetimes con offset grande. Para EDA es despreciable.
|
||||
- `is_regular` usa tolerancia ±25% sobre el paso mediano y umbral del 80% de los
|
||||
pasos dentro de banda; series de "primero de mes" (deltas 28-31) salen regulares.
|
||||
- `n_gaps` solo se calcula cuando `freq` es una rejilla regular conocida; con
|
||||
`freq` `"irregular"` o `"unknown"` siempre es 0.
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Perfil minimo de una columna fecha/datetime para la cabecera TIMESERIES (grupo eda).
|
||||
|
||||
Funcion pura y determinista que resume una columna temporal: rango (min/max),
|
||||
numero de fechas distintas, frecuencia inferida (daily/weekly/monthly/quarterly/
|
||||
yearly/irregular), regularidad de los pasos, huecos respecto a la rejilla inferida
|
||||
y paso mediano entre fechas consecutivas. Cierra el `datetime{}=None` que hoy deja
|
||||
pendiente el TableProfile de AutomaticEDA.
|
||||
|
||||
Acepta valores heterogeneos (``datetime.date``, ``datetime.datetime`` y strings
|
||||
ISO como ``"2021-06-28"``, ``"2021-06-28T00:00:00"`` o ``"2021-06-28 12:00:00"``),
|
||||
parsea de forma defensiva, ignora lo que no se puede parsear y NUNCA lanza.
|
||||
|
||||
Solo usa stdlib (``datetime`` + ``statistics``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
from datetime import date, datetime
|
||||
|
||||
|
||||
def _parse_one(v) -> datetime | None:
|
||||
"""Parsea un valor a ``datetime`` naive, o devuelve None si no es una fecha.
|
||||
|
||||
Acepta ``datetime.datetime``, ``datetime.date`` y strings ISO. Cualquier
|
||||
datetime con zona horaria se normaliza a naive (se descarta el tzinfo) para
|
||||
poder mezclarlo con fechas naive sin que las restas lancen ``TypeError``.
|
||||
"""
|
||||
if v is None or isinstance(v, bool):
|
||||
return None
|
||||
# datetime es subclase de date: comprobar datetime primero.
|
||||
if isinstance(v, datetime):
|
||||
return v.replace(tzinfo=None)
|
||||
if isinstance(v, date):
|
||||
return datetime(v.year, v.month, v.day)
|
||||
if isinstance(v, str):
|
||||
s = v.strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(s)
|
||||
except ValueError:
|
||||
return None
|
||||
return dt.replace(tzinfo=None)
|
||||
return None
|
||||
|
||||
|
||||
def _infer_freq(median_step_days: float) -> str:
|
||||
"""Clasifica la frecuencia a partir del paso mediano (en dias) entre fechas.
|
||||
|
||||
Bandas con tolerancia: ~1 dia -> daily, ~7 -> weekly, 28-31 -> monthly,
|
||||
89-92 -> quarterly, 360-366 -> yearly. Cualquier paso fuera de las bandas
|
||||
(incluido sub-diario) -> irregular.
|
||||
"""
|
||||
m = median_step_days
|
||||
if 0.5 <= m <= 1.5:
|
||||
return "daily"
|
||||
if 6.0 <= m <= 8.0:
|
||||
return "weekly"
|
||||
if 28.0 <= m <= 31.0:
|
||||
return "monthly"
|
||||
if 89.0 <= m <= 92.0:
|
||||
return "quarterly"
|
||||
if 360.0 <= m <= 366.0:
|
||||
return "yearly"
|
||||
return "irregular"
|
||||
|
||||
|
||||
def profile_datetime(values: list) -> dict:
|
||||
"""Perfila una columna de fechas para la cabecera del capitulo TIMESERIES.
|
||||
|
||||
Funcion pura y determinista: no hace I/O, no muta el input y nunca lanza.
|
||||
|
||||
El analisis de frecuencia, regularidad y huecos se hace sobre las **fechas
|
||||
distintas ordenadas** (se deduplica antes de calcular los pasos): los valores
|
||||
repetidos generarian pasos de 0 dias que distorsionarian el mediano y la
|
||||
inferencia. ``n`` cuenta los valores parseables (con duplicados) y
|
||||
``n_distinct`` las fechas unicas.
|
||||
|
||||
Args:
|
||||
values: lista de valores fecha. Acepta ``datetime.date``,
|
||||
``datetime.datetime`` y strings ISO (``"2021-06-28"``,
|
||||
``"2021-06-28T00:00:00"``, ``"2021-06-28 12:00:00"``). Los valores
|
||||
None, vacios o no parseables se ignoran. Si ``values`` es None o no
|
||||
iterable se trata como lista vacia.
|
||||
|
||||
Returns:
|
||||
Siempre un dict con esta forma::
|
||||
|
||||
{
|
||||
"min": str | None, # fecha minima ISO date (YYYY-MM-DD)
|
||||
"max": str | None, # fecha maxima ISO date
|
||||
"n": int, # nº de valores fecha parseables
|
||||
"n_distinct": int, # nº de fechas distintas
|
||||
"span_days": float | None, # (max - min) en dias
|
||||
"freq": str, # daily|weekly|monthly|quarterly|
|
||||
# yearly|irregular|unknown
|
||||
"is_regular": bool, # pasos ~constantes (tolerancia ±25%)
|
||||
"n_gaps": int, # saltos > ~1.5x el paso mediano
|
||||
"median_step_days": float | None, # paso mediano entre fechas
|
||||
"note": str # "" o nota corta
|
||||
}
|
||||
|
||||
Con menos de 2 valores parseables (o una sola fecha distinta) devuelve
|
||||
``freq="unknown"``, ``is_regular=False``, ``n_gaps=0``,
|
||||
``median_step_days=None`` y la nota correspondiente, manteniendo min/max
|
||||
y span_days coherentes cuando hay al menos una fecha.
|
||||
"""
|
||||
base = {
|
||||
"min": None,
|
||||
"max": None,
|
||||
"n": 0,
|
||||
"n_distinct": 0,
|
||||
"span_days": None,
|
||||
"freq": "unknown",
|
||||
"is_regular": False,
|
||||
"n_gaps": 0,
|
||||
"median_step_days": None,
|
||||
"note": "",
|
||||
}
|
||||
|
||||
if values is None:
|
||||
values = []
|
||||
try:
|
||||
iterator = list(values)
|
||||
except TypeError:
|
||||
iterator = []
|
||||
|
||||
parsed: list[datetime] = []
|
||||
for v in iterator:
|
||||
dt = _parse_one(v)
|
||||
if dt is not None:
|
||||
parsed.append(dt)
|
||||
|
||||
n = len(parsed)
|
||||
base["n"] = n
|
||||
|
||||
if n == 0:
|
||||
base["note"] = "datos insuficientes"
|
||||
return base
|
||||
|
||||
distinct = sorted(set(parsed))
|
||||
n_distinct = len(distinct)
|
||||
dt_min = min(parsed)
|
||||
dt_max = max(parsed)
|
||||
|
||||
base["n_distinct"] = n_distinct
|
||||
base["min"] = dt_min.date().isoformat()
|
||||
base["max"] = dt_max.date().isoformat()
|
||||
base["span_days"] = round((dt_max - dt_min).total_seconds() / 86400.0, 6)
|
||||
|
||||
# Sin al menos dos fechas distintas no hay pasos que medir.
|
||||
if n_distinct < 2:
|
||||
base["note"] = "datos insuficientes" if n < 2 else "una sola fecha distinta"
|
||||
return base
|
||||
|
||||
steps = [
|
||||
(distinct[i + 1] - distinct[i]).total_seconds() / 86400.0
|
||||
for i in range(n_distinct - 1)
|
||||
]
|
||||
median_step = float(statistics.median(steps))
|
||||
base["median_step_days"] = round(median_step, 6)
|
||||
|
||||
freq = _infer_freq(median_step)
|
||||
base["freq"] = freq
|
||||
|
||||
# Regularidad: >=80% de los pasos dentro de ±25% del paso mediano.
|
||||
if median_step > 0:
|
||||
tol = 0.25 * median_step
|
||||
within = sum(1 for s in steps if abs(s - median_step) <= tol)
|
||||
base["is_regular"] = (within / len(steps)) >= 0.8
|
||||
else:
|
||||
base["is_regular"] = False
|
||||
|
||||
# Huecos: pasos que superan ~1.5x el mediano. Solo tiene sentido cuando la
|
||||
# frecuencia es una rejilla regular conocida (no irregular/unknown).
|
||||
if freq not in ("unknown", "irregular") and median_step > 0:
|
||||
threshold = 1.5 * median_step
|
||||
base["n_gaps"] = sum(1 for s in steps if s > threshold)
|
||||
else:
|
||||
base["n_gaps"] = 0
|
||||
|
||||
return base
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Tests para profile_datetime."""
|
||||
|
||||
from datetime import date, datetime, timedelta
|
||||
|
||||
from profile_datetime import profile_datetime
|
||||
|
||||
|
||||
def test_serie_diaria_regular_golden():
|
||||
# 30 dias consecutivos: frecuencia diaria, regular, sin huecos.
|
||||
fechas = [date(2021, 1, 1) + timedelta(days=i) for i in range(30)]
|
||||
res = profile_datetime(fechas)
|
||||
assert res["n"] == 30
|
||||
assert res["n_distinct"] == 30
|
||||
assert res["min"] == "2021-01-01"
|
||||
assert res["max"] == "2021-01-30"
|
||||
assert res["span_days"] == 29.0
|
||||
assert res["freq"] == "daily"
|
||||
assert res["is_regular"] is True
|
||||
assert res["n_gaps"] == 0
|
||||
assert res["median_step_days"] == 1.0
|
||||
assert res["note"] == ""
|
||||
|
||||
|
||||
def test_serie_mensual_freq_monthly():
|
||||
# Primero de mes durante 14 meses: paso mediano ~30/31 dias -> monthly.
|
||||
fechas = []
|
||||
y, m = 2021, 1
|
||||
for _ in range(14):
|
||||
fechas.append(date(y, m, 1))
|
||||
m += 1
|
||||
if m > 12:
|
||||
m = 1
|
||||
y += 1
|
||||
res = profile_datetime(fechas)
|
||||
assert res["n"] == 14
|
||||
assert res["freq"] == "monthly"
|
||||
assert res["min"] == "2021-01-01"
|
||||
assert res["max"] == "2022-02-01"
|
||||
assert 28.0 <= res["median_step_days"] <= 31.0
|
||||
|
||||
|
||||
def test_serie_con_hueco_cuenta_gaps():
|
||||
# Serie diaria con un hueco de 3 dias (faltan i=7,8,9) -> n_gaps >= 1.
|
||||
fechas = [
|
||||
date(2021, 1, 1) + timedelta(days=i)
|
||||
for i in range(20)
|
||||
if i not in (7, 8, 9)
|
||||
]
|
||||
res = profile_datetime(fechas)
|
||||
assert res["freq"] == "daily"
|
||||
assert res["n_gaps"] >= 1
|
||||
assert res["median_step_days"] == 1.0
|
||||
|
||||
|
||||
def test_strings_iso_mezclados_con_datetime():
|
||||
# Mezcla de strings ISO (varios formatos) y objetos datetime/date.
|
||||
valores = [
|
||||
"2021-06-28",
|
||||
datetime(2021, 6, 29, 12, 0, 0),
|
||||
"2021-06-30T00:00:00",
|
||||
date(2021, 7, 1),
|
||||
]
|
||||
res = profile_datetime(valores)
|
||||
assert res["n"] == 4
|
||||
assert res["n_distinct"] == 4
|
||||
assert res["min"] == "2021-06-28"
|
||||
assert res["max"] == "2021-07-01"
|
||||
assert res["freq"] == "daily"
|
||||
assert res["note"] == ""
|
||||
|
||||
|
||||
def test_lista_vacia_y_none_devuelve_unknown():
|
||||
for entrada in ([], None):
|
||||
res = profile_datetime(entrada)
|
||||
assert res["n"] == 0
|
||||
assert res["n_distinct"] == 0
|
||||
assert res["min"] is None
|
||||
assert res["max"] is None
|
||||
assert res["span_days"] is None
|
||||
assert res["freq"] == "unknown"
|
||||
assert res["is_regular"] is False
|
||||
assert res["n_gaps"] == 0
|
||||
assert res["median_step_days"] is None
|
||||
assert res["note"] == "datos insuficientes"
|
||||
|
||||
|
||||
def test_valores_no_parseables_ignorados():
|
||||
# Strings basura, None, ints y un date valido mezclados: ignora lo no fecha.
|
||||
valores = [
|
||||
"no es una fecha",
|
||||
None,
|
||||
"2021-01-01",
|
||||
"2021-01-02",
|
||||
12345,
|
||||
"tampoco",
|
||||
date(2021, 1, 3),
|
||||
"",
|
||||
]
|
||||
res = profile_datetime(valores)
|
||||
assert res["n"] == 3 # solo 3 fechas parseables
|
||||
assert res["n_distinct"] == 3
|
||||
assert res["freq"] == "daily"
|
||||
assert res["min"] == "2021-01-01"
|
||||
assert res["max"] == "2021-01-03"
|
||||
|
||||
|
||||
def test_span_days_correcto():
|
||||
# Dos fechas a un anio de distancia: span 365 dias -> yearly.
|
||||
res = profile_datetime([date(2020, 1, 1), date(2020, 12, 31)])
|
||||
assert res["n"] == 2
|
||||
assert res["n_distinct"] == 2
|
||||
assert res["span_days"] == 365.0
|
||||
assert res["median_step_days"] == 365.0
|
||||
assert res["freq"] == "yearly"
|
||||
|
||||
|
||||
def test_una_sola_fecha_es_coherente():
|
||||
# Un unico valor: min == max, span 0, freq unknown, nota datos insuficientes.
|
||||
res = profile_datetime(["2021-06-28"])
|
||||
assert res["n"] == 1
|
||||
assert res["n_distinct"] == 1
|
||||
assert res["min"] == "2021-06-28"
|
||||
assert res["max"] == "2021-06-28"
|
||||
assert res["span_days"] == 0.0
|
||||
assert res["freq"] == "unknown"
|
||||
assert res["median_step_days"] is None
|
||||
assert res["note"] == "datos insuficientes"
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: resample_timeseries
|
||||
kind: function
|
||||
lang: py
|
||||
domain: datascience
|
||||
version: "1.0.0"
|
||||
purity: pure
|
||||
signature: "def resample_timeseries(t: list, v: list, freq: str = \"auto\", agg: str = \"mean\", max_points: int = 400) -> dict"
|
||||
description: "Agrega una serie temporal por periodo para graficar su evolucion y el CONTEO de observaciones por bucket. Nucleo del capitulo TIMESERIES de AutomaticEDA (grupo eda): recibe las fechas y los valores YA leidos (pura, sin tocar ninguna base de datos), empareja t[i] con v[i] por indice, parsea fechas defensivamente, trunca cada fecha al inicio de su bucket (daily/weekly/monthly/quarterly/yearly), y agrega los valores numericos validos por bucket mientras cuenta TODAS las observaciones con fecha valida (densidad temporal, incluida la fila cuyo valor es None). freq='auto' infiere del delta mediano entre fechas. Si hay mas buckets que max_points hace downsampling uniforme conservando primero y ultimo. Estilo dict-no-throw: NUNCA lanza; entrada vacia o longitudes incompatibles devuelve listas vacias + note='datos insuficientes'."
|
||||
tags: [eda, timeseries, resample, aggregate, profiling, datascience, time]
|
||||
params:
|
||||
- name: t
|
||||
desc: "Lista de fechas paralela a v. Acepta strings ISO ('YYYY-MM-DD' o 'YYYY-MM-DDTHH:MM:SS', con 'Z' opcional), datetime.date o datetime.datetime. Se parsea defensivamente; los pares cuya fecha no parsea se descartan junto con su valor."
|
||||
- name: v
|
||||
desc: "Lista de valores numericos (float/int) paralela a t. Puede contener None o valores no numericos: se ignoran en la agregacion pero la fila sigue contando en 'count' si su fecha es valida. bool, NaN e Inf se tratan como no numericos."
|
||||
- name: freq
|
||||
desc: "Granularidad del bucket: 'auto' (infiere del delta mediano en dias entre fechas: <=3 daily, <=16 weekly, <=75 monthly, <=200 quarterly, mayor yearly) o explicita en {daily, weekly, monthly, quarterly, yearly}. Una frecuencia desconocida cae a 'auto'."
|
||||
- name: agg
|
||||
desc: "Agregacion por bucket sobre los valores numericos validos: 'mean' | 'sum' | 'median' | 'last' (valor de la observacion cronologicamente mas reciente del bucket) | 'min' | 'max'. Una agregacion desconocida cae a 'mean'."
|
||||
- name: max_points
|
||||
desc: "Tope de buckets en la salida. Si n_buckets > max_points hace downsampling uniforme (1 de cada k buckets equiespaciados, conservando el primero y el ultimo) para no saturar el grafico del PDF/PPTX. max_points<=0 desactiva el limite."
|
||||
output: "Dict siempre con las mismas claves: t (lista de etiquetas ISO 'YYYY-MM-DD' por bucket, orden cronologico), v (lista paralela del valor agregado por bucket segun agg; None si el bucket no tiene ningun valor numerico valido), count (lista paralela del nº de observaciones con fecha valida por bucket), freq (frecuencia efectivamente usada), agg (agregacion usada), n_in (nº de pares (t,v) con fecha valida que entraron), n_buckets (nº de buckets antes del downsample), downsampled (bool, True si se aplico downsampling), note ('' o 'datos insuficientes' cuando no hay pares validos / longitudes incompatibles / listas vacias). Numericos de v en float, count en int."
|
||||
uses_functions: []
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: ""
|
||||
imports: []
|
||||
tested: true
|
||||
tests: ["test_daily_a_mensual_mean", "test_agg_sum_y_last", "test_count_cuenta_observacion_con_valor_none", "test_downsampling_respeta_max_points_y_extremos", "test_freq_auto_infiere_mensual", "test_edge_listas_vacias_o_desiguales"]
|
||||
test_file_path: "python/functions/datascience/resample_timeseries_test.py"
|
||||
file_path: "python/functions/datascience/resample_timeseries.py"
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.join("python", "functions"))
|
||||
from datascience.resample_timeseries import resample_timeseries
|
||||
|
||||
# Serie diaria agregada a buckets mensuales: media del valor + conteo de filas.
|
||||
t = ["2020-01-01", "2020-01-15", "2020-02-01", "2020-02-10", "2020-02-20"]
|
||||
v = [10.0, 20.0, 30.0, 40.0, 50.0]
|
||||
|
||||
r = resample_timeseries(t, v, freq="monthly", agg="mean")
|
||||
print(r["t"]) # ['2020-01-01', '2020-02-01']
|
||||
print(r["v"]) # [15.0, 40.0]
|
||||
print(r["count"]) # [2, 3] <- densidad: nº de observaciones por mes
|
||||
print(r["freq"], r["downsampled"]) # monthly False
|
||||
|
||||
# freq='auto' infiere la granularidad del delta mediano entre fechas.
|
||||
mensual = [f"2022-{m:02d}-01" for m in range(1, 13)]
|
||||
print(resample_timeseries(mensual, list(range(1, 13)))["freq"]) # monthly
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
- Usala en el capitulo TIMESERIES de `AutomaticEDA` para construir, a partir de una columna temporal (`detect_time_column`) y una columna numerica, la doble serie que el renderer dibuja: la EVOLUCION del valor agregado por periodo y el CONTEO de observaciones por periodo.
|
||||
- Cuando ya tengas las fechas y los valores leidos en memoria (de DuckDB, polars, CSV, etc.) y solo necesites agregarlos por dia/semana/mes/trimestre/año sin volver a tocar la base de datos — esta funcion es pura y recibe los datos por parametro.
|
||||
- Cuando quieras un downsampling controlado para que una serie muy larga (miles de fechas) quepa en un grafico de un PDF/PPTX sin saturarlo, conservando el primer y el ultimo punto.
|
||||
- Cuando no sepas la cadencia de la serie: pasa `freq="auto"` y deja que la infiera del delta mediano.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Funcion pura, sin I/O y determinista. NUNCA lanza: ante entrada invalida (listas vacias, longitudes distintas o todas las fechas no parseables) devuelve listas vacias + `note="datos insuficientes"`.
|
||||
- `count` cuenta OBSERVACIONES con fecha valida en el bucket (densidad temporal), aunque su valor numerico sea `None`/no numerico. `v` agrega SOLO los valores numericos validos del bucket; si no hay ninguno, `v` del bucket es `None` mientras `count` sigue reflejando las filas. No confundas `count` (filas) con el nº de valores agregados.
|
||||
- `bool`, `NaN` e `Inf` se tratan como NO numericos (se ignoran en `v`). Un string que no parsea a numero tambien se ignora en `v` pero su fila cuenta si la fecha es valida.
|
||||
- El truncado de bucket usa el inicio del periodo: semana = lunes ISO (`weekday()==0`), mes = dia 1, trimestre = primer dia del trimestre (ene/abr/jul/oct), año = 1 de enero. La etiqueta de cada bucket es esa fecha de inicio en ISO `YYYY-MM-DD`, no un rango.
|
||||
- El downsampling (`n_buckets > max_points`) reduce la salida a `<= max_points` puntos equiespaciados conservando primero y ultimo, pero `n_buckets` SIEMPRE reporta el conteo real previo al recorte. Si necesitas todos los buckets, sube `max_points` o ponlo `<=0`.
|
||||
- Las fechas con hora se truncan a su `date()` antes de agrupar: la granularidad minima es el dia (no hay buckets horarios).
|
||||
- `freq` desconocida o no-string cae a `"auto"`; `agg` desconocida cae a `"mean"`. El campo devuelto refleja la opcion efectivamente usada.
|
||||
@@ -0,0 +1,275 @@
|
||||
"""Agrega una serie temporal por periodo para el capitulo TIMESERIES (grupo eda).
|
||||
|
||||
Funcion pura y determinista: recibe las fechas y los valores YA leidos (nunca
|
||||
toca una base de datos ni hace I/O) y los agrega por bucket temporal para poder
|
||||
graficar la evolucion de la serie y, en paralelo, el CONTEO de observaciones por
|
||||
periodo (densidad temporal).
|
||||
|
||||
Estilo "dict-no-throw" del grupo eda: NUNCA lanza excepcion, siempre devuelve el
|
||||
mismo conjunto de claves. Lectura y parseo de fechas 100% defensivos. Solo usa la
|
||||
libreria estandar (``datetime``, ``statistics``, ``re``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import re
|
||||
import statistics
|
||||
|
||||
# Frecuencias soportadas, de mas fina a mas gruesa.
|
||||
_FREQS = ("daily", "weekly", "monthly", "quarterly", "yearly")
|
||||
|
||||
# Agregaciones soportadas.
|
||||
_AGGS = ("mean", "sum", "median", "last", "min", "max")
|
||||
|
||||
# Acepta el inicio de una fecha ISO con cualquier separador posterior
|
||||
# (incluido un caracter raro entre la fecha y la hora).
|
||||
_DATE_RE = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
|
||||
|
||||
|
||||
def _to_date(x) -> "datetime.date | None":
|
||||
"""Parsea defensivamente un valor a ``datetime.date``; devuelve None si falla."""
|
||||
if x is None:
|
||||
return None
|
||||
# datetime es subclase de date: comprobarlo primero.
|
||||
if isinstance(x, datetime.datetime):
|
||||
return x.date()
|
||||
if isinstance(x, datetime.date):
|
||||
return x
|
||||
s = str(x).strip()
|
||||
if not s:
|
||||
return None
|
||||
# Camino feliz: ISO completo (con o sin hora, con o sin 'Z' final).
|
||||
try:
|
||||
s2 = s[:-1] if s.endswith("Z") else s
|
||||
return datetime.datetime.fromisoformat(s2).date()
|
||||
except ValueError:
|
||||
pass
|
||||
# Fallback robusto: extrae el prefijo YYYY-MM-DD con cualquier separador.
|
||||
m = _DATE_RE.match(s)
|
||||
if m:
|
||||
try:
|
||||
return datetime.date(int(m.group(1)), int(m.group(2)), int(m.group(3)))
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _to_number(x) -> "float | None":
|
||||
"""Convierte a float si es numerico finito; devuelve None en otro caso."""
|
||||
if x is None:
|
||||
return None
|
||||
if isinstance(x, bool):
|
||||
# bool es subclase de int: lo tratamos como no-numerico para una serie.
|
||||
return None
|
||||
try:
|
||||
f = float(x)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
# Descarta NaN / Inf (no agregables de forma estable).
|
||||
if f != f or f in (float("inf"), float("-inf")):
|
||||
return None
|
||||
return f
|
||||
|
||||
|
||||
def _infer_freq(dates_sorted: list) -> str:
|
||||
"""Infiere la frecuencia desde el delta mediano (en dias) entre fechas."""
|
||||
if len(dates_sorted) < 2:
|
||||
return "daily"
|
||||
diffs = [
|
||||
(dates_sorted[i + 1] - dates_sorted[i]).days
|
||||
for i in range(len(dates_sorted) - 1)
|
||||
]
|
||||
diffs = [d for d in diffs if d > 0] # ignora duplicados del mismo dia
|
||||
if not diffs:
|
||||
return "daily"
|
||||
med = statistics.median(diffs)
|
||||
if med <= 3:
|
||||
return "daily"
|
||||
if med <= 16:
|
||||
return "weekly"
|
||||
if med <= 75:
|
||||
return "monthly"
|
||||
if med <= 200:
|
||||
return "quarterly"
|
||||
return "yearly"
|
||||
|
||||
|
||||
def _bucket_start(d: "datetime.date", freq: str) -> "datetime.date":
|
||||
"""Trunca una fecha al inicio de su bucket segun la frecuencia."""
|
||||
if freq == "weekly":
|
||||
return d - datetime.timedelta(days=d.weekday()) # lunes ISO
|
||||
if freq == "monthly":
|
||||
return datetime.date(d.year, d.month, 1)
|
||||
if freq == "quarterly":
|
||||
first_month = ((d.month - 1) // 3) * 3 + 1
|
||||
return datetime.date(d.year, first_month, 1)
|
||||
if freq == "yearly":
|
||||
return datetime.date(d.year, 1, 1)
|
||||
return d # daily (o cualquier otra cosa): la propia fecha
|
||||
|
||||
|
||||
def _downsample_indices(n: int, max_points: int) -> list:
|
||||
"""Indices equiespaciados conservando primero y ultimo (<= max_points)."""
|
||||
if max_points <= 0 or max_points >= n:
|
||||
return list(range(n))
|
||||
if max_points == 1:
|
||||
return [0]
|
||||
idx = sorted({round(i * (n - 1) / (max_points - 1)) for i in range(max_points)})
|
||||
return idx
|
||||
|
||||
|
||||
def _empty(freq_req: str, agg: str) -> dict:
|
||||
"""Resultado canonico cuando no hay datos suficientes."""
|
||||
eff_freq = freq_req if freq_req in _FREQS else "auto"
|
||||
return {
|
||||
"t": [],
|
||||
"v": [],
|
||||
"count": [],
|
||||
"freq": eff_freq,
|
||||
"agg": agg if agg in _AGGS else "mean",
|
||||
"n_in": 0,
|
||||
"n_buckets": 0,
|
||||
"downsampled": False,
|
||||
"note": "datos insuficientes",
|
||||
}
|
||||
|
||||
|
||||
def resample_timeseries(
|
||||
t: list,
|
||||
v: list,
|
||||
freq: str = "auto",
|
||||
agg: str = "mean",
|
||||
max_points: int = 400,
|
||||
) -> dict:
|
||||
"""Agrega una serie temporal por periodo (buckets) para graficarla.
|
||||
|
||||
Empareja ``t[i]`` con ``v[i]`` por indice, descarta los pares cuya fecha no
|
||||
parsea, trunca cada fecha al inicio de su bucket segun ``freq`` y agrupa. Por
|
||||
cada bucket devuelve el valor agregado (``agg`` sobre los valores numericos
|
||||
validos) y el CONTEO de observaciones con fecha valida (densidad temporal),
|
||||
independientemente de si su valor numerico es ``None``.
|
||||
|
||||
Funcion pura: no hace I/O, no muta los inputs, es determinista, NUNCA lanza.
|
||||
|
||||
Args:
|
||||
t: lista de fechas paralela a ``v``. Acepta strings ISO
|
||||
(``"YYYY-MM-DD"`` o ``"YYYY-MM-DDTHH:MM:SS"``, con ``Z`` opcional),
|
||||
``datetime.date`` o ``datetime.datetime``. Se parsea defensivamente;
|
||||
las fechas que no parsean se descartan junto con su valor.
|
||||
v: lista de valores numericos (float/int). Puede contener ``None`` o
|
||||
valores no numericos: estos se ignoran en la agregacion, pero la fila
|
||||
sigue contando en ``count`` (siempre que su fecha sea valida).
|
||||
freq: ``"auto"`` (infiere del delta mediano entre fechas) o uno de
|
||||
``"daily"``, ``"weekly"``, ``"monthly"``, ``"quarterly"``,
|
||||
``"yearly"``. Una frecuencia desconocida cae a ``"auto"``.
|
||||
agg: agregacion por bucket: ``"mean"``, ``"sum"``, ``"median"``,
|
||||
``"last"`` (valor de la observacion cronologicamente mas reciente),
|
||||
``"min"`` o ``"max"``. Una agregacion desconocida cae a ``"mean"``.
|
||||
max_points: si tras agregar hay mas buckets que este limite, se hace
|
||||
downsampling uniforme (1 de cada k buckets equiespaciados,
|
||||
conservando el primero y el ultimo) para no saturar el grafico.
|
||||
|
||||
Returns:
|
||||
Siempre un dict con las mismas claves::
|
||||
|
||||
{
|
||||
"t": [str, ...], # etiqueta ISO YYYY-MM-DD de cada bucket
|
||||
"v": [float|None, ...], # valor agregado por bucket (None si vacio)
|
||||
"count": [int, ...], # nº de observaciones con fecha valida
|
||||
"freq": str, # frecuencia efectivamente usada
|
||||
"agg": str, # agregacion usada
|
||||
"n_in": int, # nº de pares (t,v) con fecha valida
|
||||
"n_buckets": int, # nº de buckets antes del downsample
|
||||
"downsampled": bool, # True si se aplico downsampling
|
||||
"note": str, # "" o nota (p.ej. "datos insuficientes")
|
||||
}
|
||||
"""
|
||||
agg = agg if agg in _AGGS else "mean"
|
||||
freq_req = freq if isinstance(freq, str) else "auto"
|
||||
|
||||
# Validacion de entrada: deben ser listas de igual longitud y no vacias.
|
||||
if (
|
||||
not isinstance(t, list)
|
||||
or not isinstance(v, list)
|
||||
or len(t) == 0
|
||||
or len(t) != len(v)
|
||||
):
|
||||
return _empty(freq_req, agg)
|
||||
|
||||
# Empareja por indice y descarta fechas no parseables.
|
||||
parsed: list = [] # (date, original_index, number_or_None)
|
||||
for i, (ti, vi) in enumerate(zip(t, v)):
|
||||
d = _to_date(ti)
|
||||
if d is None:
|
||||
continue
|
||||
parsed.append((d, i, _to_number(vi)))
|
||||
|
||||
n_in = len(parsed)
|
||||
if n_in == 0:
|
||||
return _empty(freq_req, agg)
|
||||
|
||||
# Resuelve la frecuencia efectiva.
|
||||
if freq_req in _FREQS:
|
||||
eff_freq = freq_req
|
||||
else:
|
||||
dates_sorted = sorted(d for d, _, _ in parsed)
|
||||
eff_freq = _infer_freq(dates_sorted)
|
||||
|
||||
# Agrupa por bucket.
|
||||
buckets: dict = {}
|
||||
for d, idx, num in parsed:
|
||||
b = _bucket_start(d, eff_freq)
|
||||
slot = buckets.get(b)
|
||||
if slot is None:
|
||||
slot = {"count": 0, "vals": [], "last_key": None, "last_val": None}
|
||||
buckets[b] = slot
|
||||
slot["count"] += 1
|
||||
if num is not None:
|
||||
slot["vals"].append(num)
|
||||
key = (d, idx)
|
||||
if slot["last_key"] is None or key > slot["last_key"]:
|
||||
slot["last_key"] = key
|
||||
slot["last_val"] = num
|
||||
|
||||
ordered = sorted(buckets.items(), key=lambda kv: kv[0])
|
||||
n_buckets = len(ordered)
|
||||
|
||||
def _aggregate(vals: list, last_val) -> "float | None":
|
||||
if not vals:
|
||||
return None
|
||||
if agg == "sum":
|
||||
return float(sum(vals))
|
||||
if agg == "median":
|
||||
return float(statistics.median(vals))
|
||||
if agg == "last":
|
||||
return float(last_val) if last_val is not None else None
|
||||
if agg == "min":
|
||||
return float(min(vals))
|
||||
if agg == "max":
|
||||
return float(max(vals))
|
||||
return float(statistics.fmean(vals)) # mean (default)
|
||||
|
||||
t_out = [b.isoformat() for b, _ in ordered]
|
||||
v_out = [_aggregate(s["vals"], s["last_val"]) for _, s in ordered]
|
||||
c_out = [s["count"] for _, s in ordered]
|
||||
|
||||
downsampled = False
|
||||
if n_buckets > max_points > 0:
|
||||
keep = _downsample_indices(n_buckets, max_points)
|
||||
t_out = [t_out[i] for i in keep]
|
||||
v_out = [v_out[i] for i in keep]
|
||||
c_out = [c_out[i] for i in keep]
|
||||
downsampled = True
|
||||
|
||||
return {
|
||||
"t": t_out,
|
||||
"v": v_out,
|
||||
"count": c_out,
|
||||
"freq": eff_freq,
|
||||
"agg": agg,
|
||||
"n_in": n_in,
|
||||
"n_buckets": n_buckets,
|
||||
"downsampled": downsampled,
|
||||
"note": "",
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Tests para resample_timeseries (grupo eda)."""
|
||||
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
|
||||
from resample_timeseries import resample_timeseries
|
||||
|
||||
|
||||
def test_daily_a_mensual_mean():
|
||||
# Serie diaria agregada a buckets mensuales con agg="mean".
|
||||
t = [
|
||||
"2020-01-01", "2020-01-15",
|
||||
"2020-02-01", "2020-02-10", "2020-02-20",
|
||||
]
|
||||
v = [10.0, 20.0, 30.0, 40.0, 50.0]
|
||||
r = resample_timeseries(t, v, freq="monthly", agg="mean")
|
||||
|
||||
assert r["t"] == ["2020-01-01", "2020-02-01"]
|
||||
assert r["v"] == [15.0, 40.0] # (10+20)/2 ; (30+40+50)/3
|
||||
assert r["count"] == [2, 3]
|
||||
assert r["freq"] == "monthly"
|
||||
assert r["agg"] == "mean"
|
||||
assert r["n_in"] == 5
|
||||
assert r["n_buckets"] == 2
|
||||
assert r["downsampled"] is False
|
||||
assert r["note"] == ""
|
||||
|
||||
|
||||
def test_agg_sum_y_last():
|
||||
t = [
|
||||
"2020-01-01", "2020-01-15",
|
||||
"2020-02-01", "2020-02-10", "2020-02-20",
|
||||
]
|
||||
v = [10.0, 20.0, 30.0, 40.0, 50.0]
|
||||
|
||||
r_sum = resample_timeseries(t, v, freq="monthly", agg="sum")
|
||||
assert r_sum["v"] == [30.0, 120.0]
|
||||
assert r_sum["agg"] == "sum"
|
||||
|
||||
# last = valor de la observacion cronologicamente mas reciente del bucket,
|
||||
# aunque el orden de entrada este desordenado.
|
||||
t2 = ["2020-02-20", "2020-02-01", "2020-02-10", "2020-01-15", "2020-01-01"]
|
||||
v2 = [50.0, 30.0, 40.0, 20.0, 10.0]
|
||||
r_last = resample_timeseries(t2, v2, freq="monthly", agg="last")
|
||||
assert r_last["t"] == ["2020-01-01", "2020-02-01"]
|
||||
assert r_last["v"] == [20.0, 50.0] # Jan->2020-01-15=20 ; Feb->2020-02-20=50
|
||||
assert r_last["agg"] == "last"
|
||||
|
||||
|
||||
def test_count_cuenta_observacion_con_valor_none():
|
||||
# Un bucket con un valor None: count cuenta la fila, v ignora el None.
|
||||
t = ["2020-03-05", "2020-03-06", "2020-03-20"]
|
||||
v = [None, 7.0, 9.0]
|
||||
r = resample_timeseries(t, v, freq="monthly", agg="mean")
|
||||
|
||||
assert r["t"] == ["2020-03-01"]
|
||||
assert r["count"] == [3] # 3 filas con fecha valida
|
||||
assert r["v"] == [8.0] # media de los validos: (7+9)/2
|
||||
assert r["n_in"] == 3
|
||||
|
||||
# Bucket entero sin ningun valor numerico valido -> v = None, count sigue.
|
||||
r2 = resample_timeseries(
|
||||
["2020-04-01", "2020-04-02"], [None, "n/a"], freq="monthly"
|
||||
)
|
||||
assert r2["t"] == ["2020-04-01"]
|
||||
assert r2["count"] == [2]
|
||||
assert r2["v"] == [None]
|
||||
|
||||
|
||||
def test_downsampling_respeta_max_points_y_extremos():
|
||||
base = datetime.date(2021, 1, 1)
|
||||
t = [(base + datetime.timedelta(days=i)).isoformat() for i in range(500)]
|
||||
v = [float(i) for i in range(500)]
|
||||
r = resample_timeseries(t, v, freq="daily", agg="mean", max_points=400)
|
||||
|
||||
assert r["n_buckets"] == 500
|
||||
assert r["downsampled"] is True
|
||||
assert len(r["t"]) <= 400
|
||||
assert len(r["t"]) == len(r["v"]) == len(r["count"])
|
||||
# Primero y ultimo bucket conservados.
|
||||
assert r["t"][0] == "2021-01-01"
|
||||
assert r["t"][-1] == (base + datetime.timedelta(days=499)).isoformat()
|
||||
|
||||
|
||||
def test_freq_auto_infiere_mensual():
|
||||
# Fechas separadas ~1 mes -> auto infiere "monthly".
|
||||
t = [f"2022-{m:02d}-01" for m in range(1, 13)]
|
||||
v = [float(m) for m in range(1, 13)]
|
||||
r = resample_timeseries(t, v, freq="auto", agg="mean")
|
||||
|
||||
assert r["freq"] == "monthly"
|
||||
assert r["n_buckets"] == 12
|
||||
assert r["count"] == [1] * 12
|
||||
|
||||
# Fechas diarias consecutivas -> auto infiere "daily".
|
||||
base = datetime.date(2023, 1, 1)
|
||||
td = [(base + datetime.timedelta(days=i)).isoformat() for i in range(20)]
|
||||
rd = resample_timeseries(td, [float(i) for i in range(20)], freq="auto")
|
||||
assert rd["freq"] == "daily"
|
||||
|
||||
|
||||
def test_edge_listas_vacias_o_desiguales():
|
||||
vacio = resample_timeseries([], [])
|
||||
assert vacio["t"] == [] and vacio["v"] == [] and vacio["count"] == []
|
||||
assert vacio["note"] == "datos insuficientes"
|
||||
assert vacio["n_in"] == 0 and vacio["n_buckets"] == 0
|
||||
|
||||
desigual = resample_timeseries(["2020-01-01", "2020-01-02"], [1.0])
|
||||
assert desigual["note"] == "datos insuficientes"
|
||||
assert desigual["t"] == []
|
||||
|
||||
# Todas las fechas invalidas -> tambien insuficiente.
|
||||
invalidas = resample_timeseries(["no-fecha", "tampoco"], [1.0, 2.0])
|
||||
assert invalidas["note"] == "datos insuficientes"
|
||||
assert invalidas["n_in"] == 0
|
||||
@@ -5,7 +5,7 @@ lang: py
|
||||
domain: pipelines
|
||||
purity: impure
|
||||
version: "1.0.0"
|
||||
signature: "def profile_table(db_path: str, table: str, backend: str = \"duckdb\", sample: int = 5000, run_models: bool = False, run_llm: bool = False, run_series: bool = False, emit_pdf: bool = False, report_dir: str = \"reports\", write_report: bool = True) -> dict"
|
||||
signature: "def profile_table(db_path: str, table: str, backend: str = \"duckdb\", sample: int = 5000, run_models: bool = False, run_llm: bool = False, run_series: bool = False, emit_pdf: bool = False, emit_automatic: bool = False, report_dir: str = \"reports\", write_report: bool = True) -> dict"
|
||||
description: "Orquestador one-shot del grupo de capacidad eda: perfila UNA tabla (DuckDB o PostgreSQL) end-to-end componiendo las funciones del grupo (perfil base SQL + muestreo read-only + inferencia semantica + promocion de tipo + estadistica numerica/categorica + score de calidad + correlaciones con correccion FDR + re-expresion de Tukey + avisos exploratorios) y, opcional, modelos baratos (run_models), interpretacion LLM (run_llm) y analisis de serie temporal por columna (run_series: estacionariedad ADF+KPSS, ACF/PACF, STL, retornos). Emite el TableProfile completo mas (opcional) report markdown + JSON sidecar + PDF movil (emit_pdf). Es la composicion canonica para hazme un EDA de esta tabla."
|
||||
tags: [eda, duckdb, postgres, profiling, data-quality, pipeline, dataops, timeseries]
|
||||
uses_functions:
|
||||
@@ -26,6 +26,9 @@ uses_functions:
|
||||
- exploratory_caveats_py_datascience
|
||||
- render_eda_markdown_py_datascience
|
||||
- render_eda_pdf_py_datascience
|
||||
- build_eda_render_ctx_py_datascience
|
||||
- render_automatic_eda_pdf_py_datascience
|
||||
- render_automatic_eda_pptx_py_datascience
|
||||
- duckdb_query_readonly_py_infra
|
||||
- pg_query_py_infra
|
||||
uses_types: []
|
||||
@@ -55,11 +58,13 @@ params:
|
||||
desc: "Si True (default False) calcula por columna numerica un bloque de serie temporal (estacionariedad ADF+KPSS, ACF/PACF, STL y, si parece de niveles, retornos). Ordena por la primera columna datetime si existe; si no, por el orden fisico. Guardado en col['series'] y agregado en prof['series']."
|
||||
- name: emit_pdf
|
||||
desc: "Si True (default False) renderiza un PDF multipagina vertical (legible en movil) del perfil junto al report markdown y devuelve su ruta en pdf_path."
|
||||
- name: emit_automatic
|
||||
desc: "Si True (default False) emite ADEMAS el informe AutomaticEDA completo en PDF (A5 movil) Y PPTX (16:9) con los 11 capitulos del motor; construye el ctx de datos crudos con build_eda_render_ctx para que modelos/timeseries/geospatial/agregacion salgan poblados. Aditivo: no sustituye a emit_pdf. Rutas en aeda_pdf_path / aeda_pptx_path / aeda_manifest_path."
|
||||
- name: report_dir
|
||||
desc: "Directorio donde escribir los reports si write_report (y el PDF si emit_pdf). Default 'reports'. Se crea si no existe."
|
||||
- name: write_report
|
||||
desc: "Si True (default) escribe report markdown + JSON sidecar timestamped en report_dir; si False no toca disco y los paths markdown/json del retorno son None (emit_pdf es independiente)."
|
||||
output: "dict {status:'ok', profile:<TableProfile enriquecido con quality_score, key_candidates, type_breakdown recalculado, correlaciones con FDR, reexpression por columna numerica, caveats, y (con run_series) series>, report_md_path:str|None, report_json_path:str|None, pdf_path:str|None} o {status:'error', error:str} (dict-no-throw)."
|
||||
output: "dict {status:'ok', profile:<TableProfile enriquecido con quality_score, key_candidates, type_breakdown recalculado, correlaciones con FDR, reexpression por columna numerica, caveats, y (con run_series) series>, report_md_path:str|None, report_json_path:str|None, pdf_path:str|None, aeda_pdf_path:str|None, aeda_pptx_path:str|None, aeda_manifest_path:str|None (estos tres solo con emit_automatic)} o {status:'error', error:str} (dict-no-throw)."
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
@@ -32,11 +32,14 @@ from datascience import (
|
||||
acf_pacf,
|
||||
adf_kpss_stationarity,
|
||||
association_matrix,
|
||||
build_eda_render_ctx,
|
||||
column_quality_score,
|
||||
describe_numeric,
|
||||
eda_llm_insights,
|
||||
exploratory_caveats,
|
||||
infer_semantic_type,
|
||||
render_automatic_eda_pdf,
|
||||
render_automatic_eda_pptx,
|
||||
render_eda_markdown,
|
||||
render_eda_pdf,
|
||||
run_eda_models,
|
||||
@@ -385,6 +388,7 @@ def profile_table(
|
||||
run_llm: bool = False,
|
||||
run_series: bool = False,
|
||||
emit_pdf: bool = False,
|
||||
emit_automatic: bool = False,
|
||||
report_dir: str = "reports",
|
||||
write_report: bool = True,
|
||||
) -> dict:
|
||||
@@ -412,6 +416,15 @@ def profile_table(
|
||||
emit_pdf: si True (default False) renderiza un PDF multipagina vertical
|
||||
(legible en movil) del perfil junto al report markdown y devuelve su
|
||||
ruta en pdf_path.
|
||||
emit_automatic: si True (default False) emite ademas el informe
|
||||
AutomaticEDA COMPLETO en sus dos formatos (PDF A5 movil + PPTX 16:9)
|
||||
con los 11 capitulos del motor por capitulos. Construye el contexto
|
||||
de datos crudos con build_eda_render_ctx (raw_numeric para modelos/
|
||||
geo, timeseries_raw para series, geo_points para el mapa, db_path/
|
||||
table para la agregacion push-down) para que los capitulos modelos/
|
||||
timeseries/geospatial/agregacion salgan poblados, no degradados. Es
|
||||
ADITIVO: no sustituye a emit_pdf (render_eda_pdf). Sus rutas vuelven
|
||||
en aeda_pdf_path / aeda_pptx_path / aeda_manifest_path.
|
||||
report_dir: directorio donde escribir los reports si write_report.
|
||||
Default "reports". Se crea si no existe.
|
||||
write_report: si True (default), escribe un report markdown + un JSON
|
||||
@@ -727,12 +740,51 @@ def profile_table(
|
||||
except Exception: # noqa: BLE001
|
||||
pdf_path = None
|
||||
|
||||
# Informe AutomaticEDA completo (PDF + PPTX por capitulos). Aditivo:
|
||||
# convive con emit_pdf (render_eda_pdf) sin sustituirlo. Construye el ctx
|
||||
# con los datos crudos para que modelos/timeseries/geospatial/agregacion
|
||||
# salgan poblados; degrada por clave si build_eda_render_ctx falla.
|
||||
aeda_pdf_path = None
|
||||
aeda_pptx_path = None
|
||||
aeda_manifest_path = None
|
||||
if emit_automatic:
|
||||
try:
|
||||
os.makedirs(report_dir, exist_ok=True)
|
||||
base_ctx = {
|
||||
"dataset_name": table,
|
||||
"source_origin": db_path,
|
||||
"storage": "DuckDB" if backend == "duckdb" else (
|
||||
"PostgreSQL" if backend == "postgres" else backend),
|
||||
}
|
||||
if run_llm:
|
||||
base_ctx.update({"run_cluster_llm": True,
|
||||
"run_geo_llm": True, "run_agg_llm": True})
|
||||
ctx = build_eda_render_ctx(
|
||||
db_path, table, prof, backend=backend, sample=sample,
|
||||
base_ctx=base_ctx)
|
||||
meta = {"title": f"EDA — {table}", "ctx": ctx}
|
||||
aeda_pdf_target = os.path.join(report_dir,
|
||||
f"aeda_{table}_{ts}.pdf")
|
||||
aeda_pptx_target = os.path.join(report_dir,
|
||||
f"aeda_{table}_{ts}.pptx")
|
||||
rpdf = render_automatic_eda_pdf(prof, aeda_pdf_target, meta) or {}
|
||||
rpptx = render_automatic_eda_pptx(
|
||||
prof, aeda_pptx_target, meta) or {}
|
||||
aeda_pdf_path = rpdf.get("path")
|
||||
aeda_pptx_path = rpptx.get("path")
|
||||
aeda_manifest_path = rpdf.get("manifest_path")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"profile": prof,
|
||||
"report_md_path": report_md_path,
|
||||
"report_json_path": report_json_path,
|
||||
"pdf_path": pdf_path,
|
||||
"aeda_pdf_path": aeda_pdf_path,
|
||||
"aeda_pptx_path": aeda_pptx_path,
|
||||
"aeda_manifest_path": aeda_manifest_path,
|
||||
}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: render_automatic_eda
|
||||
kind: pipeline
|
||||
lang: py
|
||||
domain: pipelines
|
||||
purity: impure
|
||||
version: "1.0.0"
|
||||
signature: "def render_automatic_eda(db_path: str, table: str, backend: str = \"duckdb\", sample: int = 5000, run_models: bool = True, run_series: bool = True, run_llm: bool = False, out_dir: str = \"reports\", basename: str = None, ctx_extra: dict = None) -> dict"
|
||||
description: "Informe AutomaticEDA COMPLETO one-shot de una tabla DuckDB/PostgreSQL: perfila con profile_table, construye el ctx con los datos crudos (build_eda_render_ctx: raw_numeric para modelos/geo, timeseries_raw para series, geo_points para el mapa, db_path/table para la agregacion push-down) y emite PDF (A5 movil) Y PPTX (16:9) del mismo documento por capitulos, con los 11 capitulos POBLADOS de verdad (clusters pintados sobre el PCA, evolucion temporal, mapa geografico y tablas de agregacion), no degradados. Devuelve las rutas de PDF/PPTX y el manifiesto de versiones por capitulo."
|
||||
tags: [eda, duckdb, postgres, profiling, pipeline, dataops, report, pdf, pptx]
|
||||
uses_functions:
|
||||
- profile_table_py_pipelines
|
||||
- build_eda_render_ctx_py_datascience
|
||||
- render_automatic_eda_pdf_py_datascience
|
||||
- render_automatic_eda_pptx_py_datascience
|
||||
uses_types: []
|
||||
returns: []
|
||||
returns_optional: false
|
||||
error_type: error_go_core
|
||||
imports: []
|
||||
tested: true
|
||||
tests:
|
||||
- "render end-to-end sobre DuckDB sintetico con categoricas + fecha + lat/lon emite PDF y PPTX con paginas/slides"
|
||||
test_file_path: "python/functions/pipelines/render_automatic_eda_test.py"
|
||||
file_path: "python/functions/pipelines/render_automatic_eda.py"
|
||||
params:
|
||||
- name: db_path
|
||||
desc: "Ruta al archivo DuckDB (read-only, debe existir) o DSN PostgreSQL si backend='postgres'."
|
||||
- name: table
|
||||
desc: "Nombre de la tabla a perfilar e informar."
|
||||
- name: backend
|
||||
desc: "'duckdb' (default) o 'postgres'. Selecciona el motor de perfilado y muestreo."
|
||||
- name: sample
|
||||
desc: "Maximo de filas/valores muestreados por columna para el perfil y para los datos crudos del ctx (LIMIT). Default 5000."
|
||||
- name: run_models
|
||||
desc: "Si True (default) corre los modelos baratos (PCA/KMeans/IsolationForest/normalidad); necesario para que el capitulo modelos pinte los clusters sobre el plano PCA."
|
||||
- name: run_series
|
||||
desc: "Si True (default) calcula el analisis de serie temporal por columna numerica; necesario para el analisis del capitulo timeseries (la grafica de evolucion sale de los datos crudos del ctx aunque sea False)."
|
||||
- name: run_llm
|
||||
desc: "Si True (default False) hace la interpretacion LLM del perfil y ACTIVA la narrativa LLM de los capitulos modelos/geospatial/agregacion (titulos de segmento, descripcion de zona, seleccion de agregaciones). Con False usan su derivacion cuantitativa sin red."
|
||||
- name: out_dir
|
||||
desc: "Directorio de salida (se crea si no existe). Default 'reports'."
|
||||
- name: basename
|
||||
desc: "Nombre base de los archivos sin extension. Default 'aeda_<table>_<timestamp>'."
|
||||
- name: ctx_extra
|
||||
desc: "Dict opcional con claves de presentacion/contexto extra que se mezclan en el ctx (dataset_name, description, source_origin, ...); no pisan las claves de datos calculadas por build_eda_render_ctx."
|
||||
output: "dict {status:'ok', pdf_path:str, pptx_path:str, manifest_path:str|None, n_pages:int, n_slides:int, pdf_note:str, pptx_note:str, profile:<TableProfile>} o {status:'error', error:str} (dict-no-throw)."
|
||||
---
|
||||
|
||||
## Ejemplo
|
||||
|
||||
```python
|
||||
from pipelines.render_automatic_eda import render_automatic_eda
|
||||
|
||||
# Tabla DuckDB con categoricas + fecha + numericas: informe completo a reports/.
|
||||
r = render_automatic_eda("/tmp/ventas.duckdb", "ventas",
|
||||
run_models=True, run_series=True, out_dir="reports")
|
||||
print(r["status"], r["pdf_path"], r["pptx_path"], r["n_pages"], r["n_slides"])
|
||||
# ok reports/aeda_ventas_20260630-120500.pdf reports/aeda_ventas_20260630-120500.pptx 14 16
|
||||
|
||||
# Con narrativa LLM (titulos de segmento, descripcion geografica, etc.):
|
||||
r = render_automatic_eda("/tmp/ventas.duckdb", "ventas", run_llm=True)
|
||||
```
|
||||
|
||||
## Cuando usarla
|
||||
|
||||
Cuando quieras el informe AutomaticEDA COMPLETO (PDF + PPTX) de una tabla en una
|
||||
sola llamada, con los capitulos de modelos, series, geoespacial y agregacion ya
|
||||
poblados (no degradados). Es el reemplazo de "perfila + monta el ctx a mano +
|
||||
llama a los dos renderers": este pipeline orquesta `profile_table` ->
|
||||
`build_eda_render_ctx` -> `render_automatic_eda_pdf`/`_pptx`. Usalo como
|
||||
entregable para compartir un EDA, o como el motor detras de `profile_table(
|
||||
emit_automatic=True)` y del skill `/eda`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Impura: ESCRIBE el PDF, el PPTX y `automatic_eda_manifest.json` en `out_dir`.
|
||||
- `db_path` debe existir: DuckDB read-only no crea la base.
|
||||
- `run_models=True` y `run_series=True` por defecto encarecen el perfil (PCA/
|
||||
KMeans/IsolationForest + ADF/KPSS/STL por columna). Para un informe mas barato
|
||||
ponlos a False: los capitulos modelos/timeseries se omiten o se reducen, pero
|
||||
el resto del informe sale igual.
|
||||
- `run_llm=True` hace llamadas de red (interpretacion del perfil + narrativa por
|
||||
capitulo). Sin red, dejalo en False: los capitulos siguen completos con su
|
||||
derivacion cuantitativa (titulos de segmento derivados, nota geografica
|
||||
derivada, seleccion de agregaciones cuantitativa).
|
||||
- El PPTX requiere `python-pptx`; si no esta instalado, `pptx_path` sera None y
|
||||
`pptx_note` lo explica (el PDF se emite igual).
|
||||
- Los datos crudos del ctx se muestrean con `sample` (LIMIT), no se trae la tabla
|
||||
entera a RAM; con tablas enormes sube `sample` si quieres mas representatividad
|
||||
(coste: mas memoria).
|
||||
@@ -0,0 +1,157 @@
|
||||
"""render_automatic_eda — EDA completo one-shot: perfil → ctx → PDF + PPTX.
|
||||
|
||||
Pipeline impuro del grupo de capacidad `eda`. Dada UNA tabla DuckDB (o
|
||||
PostgreSQL), produce el informe AutomaticEDA COMPLETO en sus dos formatos a la
|
||||
vez (PDF móvil A5 + PPTX 16:9) con los 11 capítulos POBLADOS, en una sola
|
||||
llamada. Compone, sin reimplementar su lógica, cuatro funciones del registry:
|
||||
|
||||
- profile_table : perfila la tabla end-to-end (TableProfile agregado),
|
||||
opcionalmente con modelos baratos y análisis de serie.
|
||||
- build_eda_render_ctx : construye el `ctx` con los DATOS CRUDOS que el
|
||||
TableProfile agregado no incluye (raw_numeric para
|
||||
modelos/geo, timeseries_raw para series, geo_points
|
||||
para el mapa, db_path/table para la agregación
|
||||
push-down). Sin él, esos capítulos degradan.
|
||||
- render_automatic_eda_pdf : renderiza el documento por capítulos a PDF.
|
||||
- render_automatic_eda_pptx : renderiza el mismo documento a PPTX.
|
||||
|
||||
El TableProfile agregado basta para portada/overview/distribuciones/calidad/
|
||||
correlación, pero los capítulos `modelos`, `timeseries`, `geospatial` y
|
||||
`agregacion` necesitan datos crudos (los clusters proyectados sobre el PCA, la
|
||||
serie valor-vs-tiempo, los puntos lat/lon, las filas para el groupby/pivot).
|
||||
`build_eda_render_ctx` los muestrea (LIMIT + push-down, sin traer la tabla
|
||||
entera a RAM) y los entrega en `ctx`; este pipeline los pasa como `meta['ctx']`
|
||||
a ambos renderers para que el informe salga completo.
|
||||
|
||||
Estilo dict-no-throw del grupo `eda`: nunca lanza; captura cualquier error y
|
||||
degrada a `{"status": "error", "error": str}`.
|
||||
"""
|
||||
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from datascience import (
|
||||
build_eda_render_ctx,
|
||||
render_automatic_eda_pdf,
|
||||
render_automatic_eda_pptx,
|
||||
)
|
||||
from pipelines.profile_table import profile_table
|
||||
|
||||
# Tokens de almacenamiento por backend (para la portada del informe).
|
||||
_STORAGE = {"duckdb": "DuckDB", "postgres": "PostgreSQL"}
|
||||
|
||||
|
||||
def render_automatic_eda(
|
||||
db_path: str,
|
||||
table: str,
|
||||
backend: str = "duckdb",
|
||||
sample: int = 5000,
|
||||
run_models: bool = True,
|
||||
run_series: bool = True,
|
||||
run_llm: bool = False,
|
||||
out_dir: str = "reports",
|
||||
basename: str = None,
|
||||
ctx_extra: dict = None,
|
||||
) -> dict:
|
||||
"""Perfila una tabla y emite el informe AutomaticEDA completo (PDF + PPTX).
|
||||
|
||||
Args:
|
||||
db_path: ruta al archivo DuckDB, o DSN PostgreSQL si backend="postgres".
|
||||
table: nombre de la tabla a perfilar.
|
||||
backend: "duckdb" (default) o "postgres".
|
||||
sample: máximo de filas/valores muestreados por columna para el perfil
|
||||
y para los datos crudos del ctx (LIMIT). Default 5000.
|
||||
run_models: si True (default) corre los modelos baratos
|
||||
(PCA/KMeans/IsolationForest/normalidad). Necesario para que el
|
||||
capítulo `modelos` pinte los clusters sobre el plano PCA.
|
||||
run_series: si True (default) calcula el análisis de serie temporal por
|
||||
columna numérica. Necesario para el análisis del capítulo
|
||||
`timeseries` (la gráfica de evolución sale de los datos crudos del
|
||||
ctx aunque run_series sea False).
|
||||
run_llm: si True (default False) hace la interpretación LLM del perfil y
|
||||
ACTIVA además la narrativa LLM de los capítulos modelos/geospatial/
|
||||
agregacion (títulos de segmento, descripción de la zona, selección de
|
||||
agregaciones). Con False esos capítulos usan su derivación
|
||||
cuantitativa (siguen completos, sin llamadas de red).
|
||||
out_dir: directorio de salida (se crea si no existe). Default "reports".
|
||||
basename: nombre base de los archivos sin extensión. Default
|
||||
"aeda_<table>_<timestamp>".
|
||||
ctx_extra: dict opcional con claves de presentación/contexto extra que se
|
||||
mezclan en el ctx (p.ej. dataset_name, description, source_origin).
|
||||
No pisan las claves de datos calculadas por build_eda_render_ctx.
|
||||
|
||||
Returns:
|
||||
dict (nunca lanza). En éxito::
|
||||
|
||||
{"status": "ok", "pdf_path": str, "pptx_path": str,
|
||||
"manifest_path": str|None, "n_pages": int, "n_slides": int,
|
||||
"pdf_note": str, "pptx_note": str, "profile": <TableProfile>}
|
||||
|
||||
En error: {"status": "error", "error": str}.
|
||||
"""
|
||||
try:
|
||||
# 1) Perfil base + modelos/serie opcionales. No escribe report propio
|
||||
# (write_report=False): este pipeline emite su propio par PDF/PPTX.
|
||||
pres = profile_table(
|
||||
db_path,
|
||||
table,
|
||||
backend=backend,
|
||||
sample=sample,
|
||||
run_models=run_models,
|
||||
run_llm=run_llm,
|
||||
run_series=run_series,
|
||||
emit_pdf=False,
|
||||
write_report=False,
|
||||
)
|
||||
if pres.get("status") != "ok":
|
||||
return {"status": "error",
|
||||
"error": f"profile_table falló: {pres.get('error')}"}
|
||||
prof = pres.get("profile") or {}
|
||||
|
||||
# 2) Contexto de presentación + datos crudos para los 4 capítulos que los
|
||||
# necesitan. Las claves de presentación van en base_ctx; build_eda_render_ctx
|
||||
# añade raw_numeric / timeseries_raw / geo_points / db_path / table.
|
||||
base_ctx = {
|
||||
"dataset_name": table,
|
||||
"source_origin": db_path,
|
||||
"storage": _STORAGE.get(backend, backend),
|
||||
}
|
||||
if run_llm:
|
||||
# Activa la narrativa LLM de los capítulos que la soportan.
|
||||
base_ctx.update({
|
||||
"run_cluster_llm": True,
|
||||
"run_geo_llm": True,
|
||||
"run_agg_llm": True,
|
||||
})
|
||||
if ctx_extra:
|
||||
base_ctx.update(ctx_extra)
|
||||
|
||||
ctx = build_eda_render_ctx(
|
||||
db_path, table, prof, backend=backend, sample=sample,
|
||||
base_ctx=base_ctx,
|
||||
)
|
||||
|
||||
# 3) Render a ambos formatos desde el MISMO documento por capítulos.
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
|
||||
base = basename or f"aeda_{table}_{ts}"
|
||||
pdf_path = os.path.join(out_dir, base + ".pdf")
|
||||
pptx_path = os.path.join(out_dir, base + ".pptx")
|
||||
meta = {"title": f"EDA — {table}", "ctx": ctx}
|
||||
|
||||
rpdf = render_automatic_eda_pdf(prof, pdf_path, meta) or {}
|
||||
rpptx = render_automatic_eda_pptx(prof, pptx_path, meta) or {}
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"pdf_path": rpdf.get("path"),
|
||||
"pptx_path": rpptx.get("path"),
|
||||
"manifest_path": rpdf.get("manifest_path"),
|
||||
"n_pages": rpdf.get("n_pages"),
|
||||
"n_slides": rpptx.get("n_slides"),
|
||||
"pdf_note": rpdf.get("note"),
|
||||
"pptx_note": rpptx.get("note"),
|
||||
"profile": prof,
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 — dict-no-throw: degradar, nunca lanzar.
|
||||
return {"status": "error", "error": str(e)}
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Test del pipeline render_automatic_eda — EDA completo a PDF + PPTX.
|
||||
|
||||
Self-contained: crea un DuckDB temporal pequeño con categóricas + fecha + lat/lon
|
||||
+ varias numéricas, corre el pipeline (sin LLM) y verifica que emite PDF y PPTX
|
||||
con páginas/slides, manifest, y que los capítulos dependientes de ctx quedan
|
||||
POBLADOS (sin la nota de degradación).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_FUNCTIONS = os.path.abspath(os.path.join(_HERE, "..", "..")) # python/functions
|
||||
if _FUNCTIONS not in sys.path:
|
||||
sys.path.insert(0, _FUNCTIONS)
|
||||
|
||||
import duckdb # noqa: E402
|
||||
|
||||
from pipelines.render_automatic_eda import render_automatic_eda # noqa: E402
|
||||
|
||||
|
||||
def _make_db(path):
|
||||
con = duckdb.connect(path)
|
||||
con.execute(
|
||||
"CREATE TABLE sales (d DATE, region VARCHAR, channel VARCHAR, "
|
||||
"amount DOUBLE, units INTEGER, lat DOUBLE, lon DOUBLE)"
|
||||
)
|
||||
from datetime import date, timedelta
|
||||
|
||||
regions = ["norte", "sur", "este"]
|
||||
channels = ["web", "tienda"]
|
||||
centers = {"norte": (43.0, -3.0), "sur": (37.0, -5.0), "este": (39.5, -0.4)}
|
||||
rows = []
|
||||
d0 = date(2024, 1, 1)
|
||||
for i in range(180):
|
||||
r = regions[i % 3]
|
||||
ch = channels[i % 2]
|
||||
clat, clon = centers[r]
|
||||
rows.append((
|
||||
d0 + timedelta(days=i), r, ch,
|
||||
round(100 + (i % 7) * 13.5 + (5 if ch == "web" else 0), 2),
|
||||
10 + (i % 5),
|
||||
round(clat + (i % 3) * 0.1, 4),
|
||||
round(clon + (i % 4) * 0.1, 4),
|
||||
))
|
||||
con.executemany("INSERT INTO sales VALUES (?,?,?,?,?,?,?)", rows)
|
||||
con.close()
|
||||
|
||||
|
||||
def test_pipeline_emits_pdf_and_pptx_with_chapters(tmp_path):
|
||||
db = str(tmp_path / "sales.duckdb")
|
||||
_make_db(db)
|
||||
out = str(tmp_path / "out")
|
||||
|
||||
r = render_automatic_eda(db, "sales", run_models=True, run_series=True,
|
||||
run_llm=False, out_dir=out, basename="test_sales")
|
||||
assert r["status"] == "ok", r.get("error")
|
||||
|
||||
# Both formats produced.
|
||||
assert r["pdf_path"] and os.path.exists(r["pdf_path"])
|
||||
assert r["pptx_path"] and os.path.exists(r["pptx_path"])
|
||||
assert (r["n_pages"] or 0) > 0
|
||||
assert (r["n_slides"] or 0) > 0
|
||||
# Per-chapter manifest written next to the output.
|
||||
assert r["manifest_path"] and os.path.exists(r["manifest_path"])
|
||||
|
||||
|
||||
def test_pipeline_chapters_populated_not_degraded(tmp_path):
|
||||
"""The 4 ctx-dependent chapters build with real data (no degradation note)."""
|
||||
import json
|
||||
|
||||
db = str(tmp_path / "sales.duckdb")
|
||||
_make_db(db)
|
||||
out = str(tmp_path / "out")
|
||||
r = render_automatic_eda(db, "sales", run_models=True, run_series=True,
|
||||
run_llm=False, out_dir=out, basename="t2")
|
||||
assert r["status"] == "ok"
|
||||
|
||||
# The manifest lists the ctx-dependent chapters as actually rendered.
|
||||
with open(r["manifest_path"], encoding="utf-8") as fh:
|
||||
man = json.load(fh)
|
||||
chapters = man.get("chapters") or {}
|
||||
for cid in ("modelos", "timeseries", "geospatial", "agregacion"):
|
||||
assert cid in chapters, f"capítulo {cid} ausente del manifest: {list(chapters)}"
|
||||
|
||||
|
||||
def test_pipeline_bad_db_degrades_without_raising(tmp_path):
|
||||
r = render_automatic_eda(str(tmp_path / "nope.duckdb"), "ghost",
|
||||
out_dir=str(tmp_path / "o"))
|
||||
assert r["status"] == "error"
|
||||
assert "error" in r
|
||||
@@ -25,6 +25,7 @@ dependencies = [
|
||||
"polars>=1.40.1",
|
||||
"pymeshlab>=2025.7.post1",
|
||||
"pymssql>=2.3.13",
|
||||
"pymupdf>=1.28.0",
|
||||
"pypdf>=6.10.0",
|
||||
"pyproj>=3.7.2",
|
||||
"python-docx>=1.2.0",
|
||||
|
||||
Reference in New Issue
Block a user