feat(enrichers): vendoring de funciones Python por enricher (issue 0033b)

Cada enricher con `lang: python` y `uses_functions` no vacio ahora
puede empaquetar las funciones del registry que necesita en
`<enricher>/_vendored/`. El run.py importa de ahi en lugar de
`<registry_root>/python/functions/`, lo que hace al binario
distribuible sin dependencia de un fn_registry montado.

Cambios:

1. tools/vendor_enricher_python.sh
   - Lee `uses_functions` del manifest (filtrando IDs `*_py_*`).
   - Resuelve `file_path` desde registry.db.
   - Copia recursivamente con expansion transitiva: si un fichero
     vendorizado importa siblings del mismo dominio, los siblings
     tambien se copian (resuelve el caso `extract_iocs.py` que
     importa 7 modulos hermanos).
   - Genera `.vendor.lock` con `<id>  <sha256>  <src_path>` por
     funcion declarada para auditoria.
   - Idempotente — si todos los hashes coinciden, no rehace nada.

2. Manifests actualizados con `uses_functions`:
   - fetch_webpage:        normalize_url + html_to_markdown
   - extract_links:        extract_urls
   - extract_text_entities: extract_iocs

3. run.py de los 3 enrichers afectados: importan de `_vendored/`
   si existe, fallback a `<registry_root>/python/functions/` en
   modo dev (mantiene los tests pytest funcionando).

4. app.md: anade `cryptography` a python_runtime_deps porque el
   blob `cybersecurity.cybersecurity` lo importa al top.

5. Tests:
   - test_vendor_script.py — 6 tests del script: layout correcto,
     transitive siblings, lock con SHA256, idempotencia, modulos
     importables en aislamiento.
   - 16 tests de enrichers existentes pasan via vendoring (no usan
     registry_root porque _vendored/ tiene prioridad).

6. Issue 0033b movido a issues/completed/.

Tests: 32/32 verde (16 enrichers + 6 dispatcher + 4 runtime + 6
vendor).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-03 00:20:41 +02:00
parent 4ef6a5f7db
commit ee0d26ce2d
17 changed files with 368 additions and 18 deletions
+2
View File
@@ -4,5 +4,7 @@ description: "Lee la markdown cacheada de un Webpage (metadata.markdown_path) y
applies_to: [Webpage]
emits: [Url]
relations: [LINKS_TO]
uses_functions:
- extract_urls_py_cybersecurity
params:
- { name: max_links, type: int, default: 50 }
+10 -3
View File
@@ -69,9 +69,16 @@ def main() -> int:
text = open(abs_md, "r", encoding="utf-8", errors="replace").read()
progress(0.45, "extracting")
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
# Prefiere _vendored/ (issue 0033b) si existe; si no, fallback al
# registry_root para modo dev local.
vendored = os.path.join(os.path.dirname(__file__), "_vendored")
if os.path.isdir(vendored):
if vendored not in sys.path:
sys.path.insert(0, vendored)
elif registry_root:
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
from cybersecurity.cybersecurity import extract_urls # type: ignore
urls = extract_urls(text)
@@ -4,6 +4,8 @@ description: "Lee la markdown cacheada de un Webpage y extrae IoCs (IPs, emails,
applies_to: [Webpage]
emits: [Email, IPAddress, Domain, FileHash, CryptoWallet, CVE, MACAddress, Phone]
relations: [EXTRACTED_FROM]
uses_functions:
- extract_iocs_py_cybersecurity
params:
- { name: types, type: string, default: "" }
- { name: max_entities, type: int, default: 200 }
+10 -3
View File
@@ -98,9 +98,16 @@ def main() -> int:
text = open(abs_md, "r", encoding="utf-8", errors="replace").read()
progress(0.30, "extracting iocs")
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
# Prefiere _vendored/ (issue 0033b) si existe; si no, fallback al
# registry_root para modo dev local.
vendored = os.path.join(os.path.dirname(__file__), "_vendored")
if os.path.isdir(vendored):
if vendored not in sys.path:
sys.path.insert(0, vendored)
elif registry_root:
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
from cybersecurity.extract_iocs import extract_iocs # type: ignore
iocs = extract_iocs(text, types_list)
+3
View File
@@ -4,5 +4,8 @@ description: "Descarga HTML de una URL, extrae markdown limpio (readabilipy) y g
applies_to: [Url, Webpage]
emits: [Domain]
relations: [BELONGS_TO]
uses_functions:
- normalize_url_py_cybersecurity
- html_to_markdown_py_core
params:
- { name: timeout_s, type: int, default: 15 }
+10 -4
View File
@@ -38,10 +38,16 @@ def log(msg: str) -> None:
def load_registry_funcs(registry_root: str):
"""Anade el registry al sys.path e importa funciones que usamos."""
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
"""Importa funciones del registry. Prefiere `_vendored/` (issue 0033b);
si no existe, fallback a `<registry_root>/python/functions/` (modo dev)."""
vendored = os.path.join(os.path.dirname(__file__), "_vendored")
if os.path.isdir(vendored):
if vendored not in sys.path:
sys.path.insert(0, vendored)
elif registry_root:
py_funcs = os.path.join(registry_root, "python", "functions")
if py_funcs not in sys.path:
sys.path.insert(0, py_funcs)
from cybersecurity.cybersecurity import normalize_url # type: ignore
from core.html_to_markdown import html_to_markdown # type: ignore
return normalize_url, html_to_markdown