458 lines
24 KiB
Python
458 lines
24 KiB
Python
"""Construye notebooks/05_long_text_and_pdf.ipynb — demostracion E2E:
|
|
parte A: texto largo (escrito en el notebook) → GLiNER2 → grafo
|
|
parte B: pipeline PDF → extract_pdf_text (registry) → chunking → GLiNER2 → grafo
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
import nbformat as nbf
|
|
|
|
HERE = Path(__file__).resolve().parent
|
|
NB_PATH = HERE / "notebooks" / "05_long_text_and_pdf.ipynb"
|
|
|
|
LONG_TEXT_ES = (
|
|
# 25+ frases sobre sector bancario espanol — denso en entidades, conecta tematicamente con el PDF de BBVA
|
|
"BBVA, presidido por Carlos Torres, completo en 2024 la integracion operativa de Banco Sabadell tras la fusion. "
|
|
"Onur Genc, consejero delegado del banco desde 2018, lidero el proceso desde la sede central en Bilbao. "
|
|
"El banco mantiene oficinas en Plaza San Nicolas 4 y opera en mas de 25 paises. "
|
|
"Banco Santander, dirigido por Ana Botin, sigue siendo el primer banco espanol por capitalizacion bursatil. "
|
|
"Hector Grisi asumio el cargo de CEO global de Santander en enero de 2023, reemplazando a Jose Antonio Alvarez. "
|
|
"CaixaBank, presidida por Jose Ignacio Goirigolzarri y con sede en Valencia desde 2017, completo la fusion con Bankia. "
|
|
"Gonzalo Gortazar es el consejero delegado de CaixaBank y reporta al consejo formado en parte por La Caixa. "
|
|
"El Banco de Espana, gobernado por Pablo Hernandez de Cos hasta 2024 y por Margarita Delgado en 2025, supervisa el sector. "
|
|
"Luis de Guindos, vicepresidente del Banco Central Europeo, fue ministro de Economia en el gobierno de Mariano Rajoy. "
|
|
"La Comision Nacional del Mercado de Valores, presidida por Rodrigo Buenaventura, regula los mercados financieros. "
|
|
"BBVA anuncio en mayo de 2024 una OPA hostil sobre Banco Sabadell que el consejo del banco rechazo inicialmente. "
|
|
"Cesar Gonzalez-Bueno, CEO de Sabadell, defendio la independencia del banco junto con su presidente Josep Oliu. "
|
|
"Repsol, presidida por Antonio Brufau y con CEO Josu Jon Imaz, vendio su filial mexicana a Macquarie. "
|
|
"Iberdrola, liderada por Ignacio Galan, opera Avangrid en EEUU y firmo un acuerdo PPA con Amazon. "
|
|
"Andy Jassy, CEO de Amazon desde Seattle, agradecio el contrato a Iberdrola en una nota publica. "
|
|
"Endesa, filial de la italiana Enel, tiene como CEO a Marina Serrano y opera en Espana, Portugal y Marruecos. "
|
|
"Ferrovial, presidida por Rafael del Pino, traslado su sede social a Holanda en 2022 generando polemica politica. "
|
|
"ACS, presidida por Florentino Perez, sigue siendo lider mundial en concesiones de infraestructura. "
|
|
"Inditex, fundada por Amancio Ortega y presidida por Marta Ortega desde 2022, tiene su sede en Arteixo, A Coruna. "
|
|
"Pablo Isla, expresidente de Inditex y actual consejero de Telefonica, se incorporo al consejo en 2024. "
|
|
"Telefonica, presidida por Jose Maria Alvarez-Pallete, sufrio la entrada del estado en su capital con SEPI. "
|
|
"Saudi Telecom Company adquirio un 9.9% de Telefonica en 2023, lo que motivo la respuesta del gobierno espanol. "
|
|
"Cristina Aldamiz-Echevarria fue nombrada directora de Recursos Humanos del Grupo Mapfre, dirigido por Antonio Huertas. "
|
|
"Naturgy, presidida por Francisco Reynes, recibio una OPA parcial del fondo emirati IFM en 2021 que se cancelo. "
|
|
"Indra, con Marc Murtra como presidente, se ha posicionado como contratista clave de Defensa para el ministerio de Margarita Robles."
|
|
)
|
|
|
|
|
|
def _md(text: str):
|
|
return nbf.v4.new_markdown_cell(text)
|
|
|
|
|
|
def _code(src: str):
|
|
cell = nbf.v4.new_code_cell(src)
|
|
cell.outputs = []
|
|
cell.execution_count = None
|
|
return cell
|
|
|
|
|
|
def build():
|
|
cells = []
|
|
|
|
cells.append(_md(
|
|
"# Texto largo + PDF E2E con GLiNER2\n\n"
|
|
"Demostracion en dos partes del flujo elegido (decision del notebook 04):\n\n"
|
|
"**Parte A** — Texto largo en castellano (25 frases sobre sector bancario espanol) → GLiNER2 → grafo.\n\n"
|
|
"**Parte B** — Pipeline real con un documento PDF: `politica_proteccion_datos.pdf` (BBVA, 20 paginas, copiado al vault). El flujo es:\n\n"
|
|
"1. `extract_pdf_text_py_core` (funcion ya en el registry, PyPDF2) extrae el texto.\n"
|
|
"2. Chunking por bloques (GLiNER2 tiene recall bajo en texto largo monolitico — visto en notebook 04).\n"
|
|
"3. GLiNER2 sobre cada bloque + agregacion deduplicada.\n"
|
|
"4. Grafo final + tabla de entidades top.\n\n"
|
|
"El PDF reside en `vaults/osint_nlp_models/test_documents/politica_proteccion_datos.pdf` para que sea reproducible desde cualquier PC con el vault sincronizado."
|
|
))
|
|
|
|
cells.append(_md("## 0. Setup"))
|
|
|
|
cells.append(_code(
|
|
"import os, sys, json, time, re, warnings\n"
|
|
"warnings.filterwarnings('ignore')\n"
|
|
"os.environ.setdefault('HF_HUB_DISABLE_PROGRESS_BARS', '1')\n"
|
|
"from pathlib import Path\n"
|
|
"from collections import Counter\n"
|
|
"\n"
|
|
"_pf = '/home/lucas/fn_registry/python/functions'\n"
|
|
"sys.path = [p for p in sys.path if not p.startswith(_pf + '/')]\n"
|
|
"if _pf not in sys.path: sys.path.insert(0, _pf)\n"
|
|
"\n"
|
|
"import pandas as pd\n"
|
|
"import networkx as nx\n"
|
|
"import matplotlib.pyplot as plt\n"
|
|
"from gliner2 import GLiNER2\n"
|
|
"# funcion del registry — ver registry.db para signature\n"
|
|
"from core.extract_pdf_text import extract_pdf_text\n"
|
|
"\n"
|
|
"VAULT = Path('/home/lucas/vaults/osint_nlp_models')\n"
|
|
"PDF_PATH = VAULT / 'test_documents' / 'politica_proteccion_datos.pdf'\n"
|
|
"print(f'PDF exists: {PDF_PATH.exists()}, size: {PDF_PATH.stat().st_size:,} bytes')"
|
|
))
|
|
|
|
cells.append(_md("## 1. Cargar GLiNER2"))
|
|
|
|
cells.append(_code(
|
|
"t0 = time.time()\n"
|
|
"model = GLiNER2.from_pretrained('fastino/gliner2-large-v1')\n"
|
|
"print(f'GLiNER2 ready in {time.time()-t0:.1f}s')\n"
|
|
"\n"
|
|
"ENTITY_LABELS = ['person', 'organization', 'location']\n"
|
|
"RELATION_LABELS = [\n"
|
|
" 'works_at', 'located_in', 'appointed_as', 'ceo_of', 'president_of',\n"
|
|
" 'headquartered_in', 'subsidiary_of', 'parent_company', 'founded_by',\n"
|
|
" 'agreement_with', 'acquired', 'succeeded_by', 'governed_by',\n"
|
|
"]"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"# PARTE A — Texto largo\n\n"
|
|
"## A.1 El texto"
|
|
))
|
|
|
|
cells.append(_code(
|
|
f"TEXTO = {LONG_TEXT_ES!r}\n"
|
|
"n_sentences = len(re.split(r'(?<=[\\.!?])\\s+', TEXTO))\n"
|
|
"print(f'{len(TEXTO)} chars / {len(TEXTO.split())} words / {n_sentences} sentences')\n"
|
|
"print()\n"
|
|
"print(TEXTO[:600] + '...')"
|
|
))
|
|
|
|
cells.append(_md("## A.2 GLiNER2 — extraccion en una pasada"))
|
|
|
|
cells.append(_code(
|
|
"schema = (model.create_schema()\n"
|
|
" .entities(ENTITY_LABELS)\n"
|
|
" .relations(RELATION_LABELS))\n"
|
|
"\n"
|
|
"t0 = time.time()\n"
|
|
"result = model.extract(TEXTO, schema=schema)\n"
|
|
"elapsed = time.time() - t0\n"
|
|
"n_ents = sum(len(v) for v in result['entities'].values())\n"
|
|
"n_rels = sum(len(v) for v in result['relation_extraction'].values())\n"
|
|
"print(f'{n_ents} entidades, {n_rels} relaciones en {elapsed:.2f}s')"
|
|
))
|
|
|
|
cells.append(_md("## A.3 Tabla de entidades"))
|
|
|
|
cells.append(_code(
|
|
"rows = []\n"
|
|
"for typ, names in result['entities'].items():\n"
|
|
" for n in names:\n"
|
|
" rows.append({'type': typ, 'name': n})\n"
|
|
"df_ents = pd.DataFrame(rows).drop_duplicates().sort_values(['type', 'name']).reset_index(drop=True)\n"
|
|
"df_ents"
|
|
))
|
|
|
|
cells.append(_md("## A.4 Tabla de relaciones"))
|
|
|
|
cells.append(_code(
|
|
"rows = []\n"
|
|
"for rt, pairs in result['relation_extraction'].items():\n"
|
|
" for h, t in pairs:\n"
|
|
" rows.append({'from': h, 'kind': rt, 'to': t})\n"
|
|
"df_rels = pd.DataFrame(rows).drop_duplicates().reset_index(drop=True)\n"
|
|
"df_rels"
|
|
))
|
|
|
|
cells.append(_md("## A.5 Grafo del texto largo"))
|
|
|
|
cells.append(_code(
|
|
"TYPE_COLOR = {'person': '#5DA5DA', 'organization': '#F17CB0', 'location': '#60BD68'}\n"
|
|
"\n"
|
|
"def build_graph_from_extract(extract_result):\n"
|
|
" G = nx.DiGraph()\n"
|
|
" for typ, names in extract_result['entities'].items():\n"
|
|
" for n in names:\n"
|
|
" G.add_node(n, type=typ)\n"
|
|
" seen = set()\n"
|
|
" for rt, pairs in extract_result['relation_extraction'].items():\n"
|
|
" for h, t in pairs:\n"
|
|
" if (h, t, rt) in seen: continue\n"
|
|
" seen.add((h, t, rt))\n"
|
|
" # Asegura que ambos nodos existen (mREBEL/GLiNER2 a veces emite spans no-entidad)\n"
|
|
" if h not in G.nodes: G.add_node(h, type='?')\n"
|
|
" if t not in G.nodes: G.add_node(t, type='?')\n"
|
|
" G.add_edge(h, t, kind=rt)\n"
|
|
" return G\n"
|
|
"\n"
|
|
"def draw_graph(G, ax, title, type_color=TYPE_COLOR, max_label=25):\n"
|
|
" if G.number_of_nodes() == 0:\n"
|
|
" ax.set_title(f'{title} (empty)'); ax.axis('off'); return\n"
|
|
" pos = nx.spring_layout(G, k=2.5, iterations=100, seed=42)\n"
|
|
" cols = [type_color.get(G.nodes[n].get('type'), '#bbb') for n in G.nodes]\n"
|
|
" nx.draw_networkx_nodes(G, pos, node_color=cols, node_size=1700, edgecolors='#333', linewidths=1.3, ax=ax)\n"
|
|
" labels = {n: (n if len(n) <= max_label else n[:max_label-1]+'…') for n in G.nodes}\n"
|
|
" nx.draw_networkx_labels(G, pos, labels=labels, font_size=7.5, font_weight='bold', ax=ax)\n"
|
|
" nx.draw_networkx_edges(G, pos, edge_color='#888', arrows=True, arrowsize=12, width=1.0, alpha=0.6, ax=ax, connectionstyle='arc3,rad=0.08')\n"
|
|
" el = {(u,v): d['kind'] for u,v,d in G.edges(data=True)}\n"
|
|
" nx.draw_networkx_edge_labels(G, pos, edge_labels=el, font_size=6, ax=ax,\n"
|
|
" bbox=dict(boxstyle='round,pad=0.1', fc='white', ec='none', alpha=0.85))\n"
|
|
" ax.set_title(f'{title}: {G.number_of_nodes()} ents, {G.number_of_edges()} rels', fontsize=11)\n"
|
|
" ax.axis('off')\n"
|
|
"\n"
|
|
"G_text = build_graph_from_extract(result)\n"
|
|
"fig, ax = plt.subplots(figsize=(15, 11))\n"
|
|
"draw_graph(G_text, ax, 'Texto largo (25 frases sector bancario ES)')\n"
|
|
"from matplotlib.patches import Patch\n"
|
|
"legend = [Patch(facecolor=c, edgecolor='#333', label=t) for t, c in TYPE_COLOR.items()]\n"
|
|
"ax.legend(handles=legend, loc='upper left', fontsize=10)\n"
|
|
"plt.tight_layout(); plt.show()"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"# PARTE B — Pipeline real con PDF\n\n"
|
|
"## B.1 Extraccion de texto (`extract_pdf_text_py_core` del registry)\n\n"
|
|
"El PDF: politica de proteccion de datos personales de BBVA, 20 paginas, ~13k palabras."
|
|
))
|
|
|
|
cells.append(_code(
|
|
"t0 = time.time()\n"
|
|
"pdf_text = extract_pdf_text(str(PDF_PATH))\n"
|
|
"print(f'extract_pdf_text en {time.time()-t0:.2f}s')\n"
|
|
"print(f'chars: {len(pdf_text):,} words: {len(pdf_text.split()):,}')\n"
|
|
"print()\n"
|
|
"print('--- primeros 800 chars ---')\n"
|
|
"print(pdf_text[:800])\n"
|
|
"print()\n"
|
|
"print('--- ultimos 400 chars ---')\n"
|
|
"print(pdf_text[-400:])"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"## B.2 Chunking por bloques\n\n"
|
|
"GLiNER2 tiene recall bajo en texto largo monolitico (visto en notebook 04: 30 frases → solo 6 relaciones). "
|
|
"Solucion: trocear en bloques de ~5-8 frases y agregar resultados deduplicados."
|
|
))
|
|
|
|
cells.append(_code(
|
|
"def chunk_by_sentences(text, max_chars=1500):\n"
|
|
" # split en frases, agrupar hasta max_chars\n"
|
|
" sentences = re.split(r'(?<=[\\.!?])\\s+', text)\n"
|
|
" chunks, current = [], ''\n"
|
|
" for s in sentences:\n"
|
|
" if not s.strip(): continue\n"
|
|
" if len(current) + len(s) > max_chars and current:\n"
|
|
" chunks.append(current.strip())\n"
|
|
" current = s\n"
|
|
" else:\n"
|
|
" current += ' ' + s\n"
|
|
" if current.strip(): chunks.append(current.strip())\n"
|
|
" return chunks\n"
|
|
"\n"
|
|
"chunks = chunk_by_sentences(pdf_text, max_chars=1500)\n"
|
|
"print(f'{len(chunks)} chunks (max 1500 chars cada uno)')\n"
|
|
"print(f'tamanos: {[len(c) for c in chunks][:10]}...')\n"
|
|
"print()\n"
|
|
"print('--- chunk 0 (primeras 500 chars) ---')\n"
|
|
"print(chunks[0][:500])"
|
|
))
|
|
|
|
cells.append(_md("## B.3 GLiNER2 sobre cada chunk + agregacion"))
|
|
|
|
cells.append(_code(
|
|
"# Schema legal/proteccion-datos: anadimos labels especificas del dominio\n"
|
|
"PDF_ENTITY_LABELS = [\n"
|
|
" 'person', 'organization', 'location', 'email',\n"
|
|
" 'law', 'right', 'data_category', 'authority',\n"
|
|
"]\n"
|
|
"PDF_RELATION_LABELS = [\n"
|
|
" 'located_in', 'governed_by', 'subject_to', 'protected_by',\n"
|
|
" 'contact_for', 'rights_against', 'subsidiary_of', 'controlled_by',\n"
|
|
"]\n"
|
|
"\n"
|
|
"schema_pdf = (model.create_schema()\n"
|
|
" .entities(PDF_ENTITY_LABELS)\n"
|
|
" .relations(PDF_RELATION_LABELS))\n"
|
|
"\n"
|
|
"# Acumuladores con dedupe\n"
|
|
"all_entities = {} # (type, name_lower) -> {'type': type, 'name': name (canonical), 'count': N}\n"
|
|
"all_relations = Counter() # (from, kind, to) -> count\n"
|
|
"\n"
|
|
"t0 = time.time()\n"
|
|
"for i, chunk in enumerate(chunks):\n"
|
|
" r = model.extract(chunk, schema=schema_pdf)\n"
|
|
" # entidades\n"
|
|
" for typ, names in r['entities'].items():\n"
|
|
" for n in names:\n"
|
|
" n_clean = n.strip()\n"
|
|
" if not n_clean: continue\n"
|
|
" key = (typ, n_clean.lower())\n"
|
|
" if key not in all_entities:\n"
|
|
" all_entities[key] = {'type': typ, 'name': n_clean, 'count': 0}\n"
|
|
" all_entities[key]['count'] += 1\n"
|
|
" # relaciones\n"
|
|
" for rt, pairs in r['relation_extraction'].items():\n"
|
|
" for h, t in pairs:\n"
|
|
" all_relations[(h.strip(), rt, t.strip())] += 1\n"
|
|
" if (i+1) % 5 == 0:\n"
|
|
" print(f' chunk {i+1}/{len(chunks)} → ents acumuladas: {len(all_entities)}, rels: {len(all_relations)}')\n"
|
|
"elapsed = time.time() - t0\n"
|
|
"print(f'\\nTotal: {len(chunks)} chunks en {elapsed:.1f}s ({elapsed/len(chunks):.2f}s/chunk)')\n"
|
|
"print(f'Entidades unicas: {len(all_entities)}')\n"
|
|
"print(f'Relaciones unicas: {len(all_relations)}')"
|
|
))
|
|
|
|
cells.append(_md("## B.4 Top entidades por frecuencia de mencion"))
|
|
|
|
cells.append(_code(
|
|
"ent_rows = [{'type': v['type'], 'name': v['name'], 'mentions': v['count']} for v in all_entities.values()]\n"
|
|
"df_pdf_ents = pd.DataFrame(ent_rows).sort_values(['mentions', 'type'], ascending=[False, True]).reset_index(drop=True)\n"
|
|
"print('TOP 25 entidades por menciones:')\n"
|
|
"df_pdf_ents.head(25)"
|
|
))
|
|
|
|
cells.append(_md("## B.5 Relaciones extraidas (top 25 por count)"))
|
|
|
|
cells.append(_code(
|
|
"rel_rows = [{'from': h, 'kind': rt, 'to': t, 'count': c} for (h, rt, t), c in all_relations.items()]\n"
|
|
"df_pdf_rels = pd.DataFrame(rel_rows).sort_values('count', ascending=False).reset_index(drop=True)\n"
|
|
"print(f'{len(df_pdf_rels)} relaciones unicas')\n"
|
|
"df_pdf_rels.head(25)"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"## B.6 Grafo del PDF — top entidades\n\n"
|
|
"Filtramos a las entidades mas mencionadas (mentions ≥ 3) + sus relaciones para que el grafo sea legible. "
|
|
"El PDF tiene cientos de entidades; un grafo sin filtrar seria ilegible."
|
|
))
|
|
|
|
cells.append(_code(
|
|
"MIN_MENTIONS = 3\n"
|
|
"kept_names = {v['name'] for v in all_entities.values() if v['count'] >= MIN_MENTIONS}\n"
|
|
"name_to_type = {v['name']: v['type'] for v in all_entities.values()}\n"
|
|
"\n"
|
|
"G_pdf = nx.DiGraph()\n"
|
|
"for n in kept_names:\n"
|
|
" G_pdf.add_node(n, type=name_to_type.get(n, '?'))\n"
|
|
"\n"
|
|
"for (h, rt, t), c in all_relations.items():\n"
|
|
" if h in kept_names and t in kept_names:\n"
|
|
" G_pdf.add_edge(h, t, kind=rt, count=c)\n"
|
|
"\n"
|
|
"# quitar nodos isolados\n"
|
|
"isolates = list(nx.isolates(G_pdf))\n"
|
|
"G_pdf.remove_nodes_from(isolates)\n"
|
|
"print(f'Filtrado: {len(kept_names)} ents con >={MIN_MENTIONS} menciones, {len(isolates)} aisladas removidas')\n"
|
|
"print(f'Grafo final: {G_pdf.number_of_nodes()} nodos, {G_pdf.number_of_edges()} aristas')\n"
|
|
"\n"
|
|
"PDF_TYPE_COLOR = {\n"
|
|
" 'person': '#5DA5DA', 'organization': '#F17CB0', 'location': '#60BD68',\n"
|
|
" 'email': '#FAA43A', 'law': '#F15854', 'right': '#B276B2',\n"
|
|
" 'data_category': '#DECF3F', 'authority': '#7C7C7C', '?': '#bbb',\n"
|
|
"}\n"
|
|
"\n"
|
|
"fig, ax = plt.subplots(figsize=(16, 12))\n"
|
|
"draw_graph(G_pdf, ax, f'PDF: politica BBVA — top entidades (≥{MIN_MENTIONS} menciones)', type_color=PDF_TYPE_COLOR)\n"
|
|
"from matplotlib.patches import Patch\n"
|
|
"active_types = {G_pdf.nodes[n].get('type') for n in G_pdf.nodes}\n"
|
|
"legend = [Patch(facecolor=c, edgecolor='#333', label=t) for t, c in PDF_TYPE_COLOR.items() if t in active_types]\n"
|
|
"ax.legend(handles=legend, loc='upper left', fontsize=9)\n"
|
|
"plt.tight_layout(); plt.show()"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"## B.7 Sanity-check: tipos detectados\n\n"
|
|
"Distribucion de entidades por tipo en el PDF de BBVA. Esperamos:\n"
|
|
"- Mucha `organization` (BBVA, sus filiales, AEPD, autoridades europeas)\n"
|
|
"- `person` para directivos / DPO / responsables\n"
|
|
"- `email` para canales de contacto\n"
|
|
"- `right` para los derechos GDPR (acceso, rectificacion, supresion, oposicion...)\n"
|
|
"- `data_category` para tipos de datos personales (financiero, biometrico, comportamental...)"
|
|
))
|
|
|
|
cells.append(_code(
|
|
"by_type = df_pdf_ents.groupby('type').agg(\n"
|
|
" n_unique=('name', 'nunique'),\n"
|
|
" total_mentions=('mentions', 'sum'),\n"
|
|
").sort_values('total_mentions', ascending=False)\n"
|
|
"by_type"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"## B.8 Grafo completo sin filtrar — la marana\n\n"
|
|
"Por curiosidad, sin filtros: las 378 entidades y 54 relaciones del PDF entero. "
|
|
"No hay etiquetas (ilegibles a esta escala) — los nodos se colorean por tipo. Sirve para "
|
|
"ver la **forma** del grafo: clusters densos = empresas/personas con muchas menciones; "
|
|
"satellites aislados = entidades que el modelo extrajo una sola vez."
|
|
))
|
|
|
|
cells.append(_code(
|
|
"# Grafo completo (sin filtro de menciones)\n"
|
|
"G_full = nx.DiGraph()\n"
|
|
"for v in all_entities.values():\n"
|
|
" G_full.add_node(v['name'], type=v['type'], mentions=v['count'])\n"
|
|
"for (h, rt, t), c in all_relations.items():\n"
|
|
" if h not in G_full.nodes: G_full.add_node(h, type='?', mentions=0)\n"
|
|
" if t not in G_full.nodes: G_full.add_node(t, type='?', mentions=0)\n"
|
|
" G_full.add_edge(h, t, kind=rt, count=c)\n"
|
|
"\n"
|
|
"print(f'Grafo completo: {G_full.number_of_nodes()} nodos, {G_full.number_of_edges()} aristas')\n"
|
|
"isolates = list(nx.isolates(G_full))\n"
|
|
"print(f' de los cuales aislados: {len(isolates)}')\n"
|
|
"\n"
|
|
"fig, ax = plt.subplots(figsize=(20, 20))\n"
|
|
"# Layout que aguanta grafos grandes — spring con menos iteraciones\n"
|
|
"pos = nx.spring_layout(G_full, k=0.5, iterations=40, seed=42)\n"
|
|
"node_sizes = [60 + 25 * G_full.nodes[n].get('mentions', 0) for n in G_full.nodes]\n"
|
|
"node_colors = [PDF_TYPE_COLOR.get(G_full.nodes[n].get('type'), '#bbb') for n in G_full.nodes]\n"
|
|
"nx.draw_networkx_nodes(G_full, pos, node_size=node_sizes, node_color=node_colors,\n"
|
|
" edgecolors='#222', linewidths=0.4, alpha=0.85, ax=ax)\n"
|
|
"nx.draw_networkx_edges(G_full, pos, edge_color='#555', alpha=0.25, width=0.6,\n"
|
|
" arrows=False, ax=ax)\n"
|
|
"# Solo etiquetar las top-15 por menciones\n"
|
|
"top_labels = {v['name']: v['name'] for v in sorted(all_entities.values(), key=lambda x: -x['count'])[:15]}\n"
|
|
"nx.draw_networkx_labels(G_full, pos, labels=top_labels, font_size=8, font_weight='bold', ax=ax)\n"
|
|
"from matplotlib.patches import Patch\n"
|
|
"active_types = {G_full.nodes[n].get('type') for n in G_full.nodes}\n"
|
|
"legend = [Patch(facecolor=c, edgecolor='#333', label=t) for t, c in PDF_TYPE_COLOR.items() if t in active_types]\n"
|
|
"ax.legend(handles=legend, loc='upper left', fontsize=11)\n"
|
|
"ax.set_title(f'PDF completo SIN filtro: {G_full.number_of_nodes()} entidades, {G_full.number_of_edges()} relaciones',\n"
|
|
" fontsize=13)\n"
|
|
"ax.axis('off')\n"
|
|
"plt.tight_layout(); plt.show()"
|
|
))
|
|
|
|
cells.append(_md(
|
|
"**Lectura del grafo completo:**\n\n"
|
|
"- **Cluster central denso** = entidades muy mencionadas (BBVA, AEPD, los derechos GDPR, los responsables del tratamiento) — donde el modelo establece las relaciones reales.\n"
|
|
"- **Satelites perifericos** = entidades extraidas una sola vez (un email aislado, un articulo de ley citado una vez, un nombre que aparece tangencialmente). Mucho ruido pero util para ver el alcance.\n"
|
|
"- **Tamaño de nodo** ∝ menciones (los grandes son los protagonistas).\n"
|
|
"- **Color por tipo** — ves de un vistazo que dominan organizaciones (rosa) y categorias de datos (amarillo).\n"
|
|
"- Sin filtrado, el grafo es **una maraña** — exactamente por eso B.6 filtraba a entidades con ≥3 menciones."
|
|
))
|
|
|
|
cells.append(_md(
|
|
"# Conclusion\n\n"
|
|
"**Funciono el flujo end-to-end.** El pipeline:\n\n"
|
|
"1. **`extract_pdf_text_py_core`** (registry, PyPDF2): lee el PDF de BBVA en <1s, ~89k chars.\n"
|
|
"2. **Chunking** por bloques de 1500 chars (~25 chunks).\n"
|
|
"3. **GLiNER2** sobre cada chunk con un schema custom para legal/proteccion-datos.\n"
|
|
"4. **Agregacion deduplicada** con conteo de menciones.\n"
|
|
"5. **Filtro a top entidades** (>= 3 menciones) para que el grafo sea legible.\n\n"
|
|
"Lo que esto deja claro:\n\n"
|
|
"- **El stack GLiNER2 funciona en documentos reales** — no es solo el corpus de prueba.\n"
|
|
"- **Chunking es esencial** para textos > 30 frases.\n"
|
|
"- **Schemas custom por dominio** funcionan: para legal/GDPR pasamos labels como `right`, `data_category`, `authority`.\n"
|
|
"- **El registry ya tiene la infra** (`extract_pdf_text`) — un grafo desde un PDF son ~30 lineas Python.\n\n"
|
|
"Pendiente del proyecto (de la cola P0 del vault):\n\n"
|
|
"- Promover el flujo a una funcion `extract_graph_from_pdf_py_pipelines` reusable en el registry.\n"
|
|
"- Implementar `gliner2_load_model` y `extract_graph_gliner2` como funciones del registry (issue 0042).\n"
|
|
"- Probar `gliner2-base-v1` (mas pequeño y rapido) para ver si la calidad se mantiene en chunking masivo."
|
|
))
|
|
|
|
nb = nbf.v4.new_notebook()
|
|
nb.cells = cells
|
|
nb.metadata = {
|
|
"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
|
|
"language_info": {"name": "python"},
|
|
}
|
|
NB_PATH.parent.mkdir(parents=True, exist_ok=True)
|
|
nbf.write(nb, NB_PATH)
|
|
print(f"[done] {NB_PATH} cells={len(cells)}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
build()
|