# functions

id: parse_cron_expr_py_core
name: parse_cron_expr
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: parse_cron_expr(expr: str) -> dict
tags: ["cron","scheduling","parsing","time","pure"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/parse_cron_expr_test.py
file_path: python/functions/core/parse_cron_expr.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 8d6c5ba0e54e616c0487cf75b6f6d88f995fd77ffd20938cafa96a2dcc8921b3
description:
  Parsea una expresion cron estandar de 5 campos. Soporta *, rangos (1-5), listas (1,3,5), pasos (*/15) y aliases (@hourly, @daily, @weekly, @monthly, @yearly). Retorna dict con claves minute, hour, day_of_month, month, day_of_week con listas de enteros validos.

example:
  ```python
  from parse_cron_expr import parse_cron_expr

  sched = parse_cron_expr("*/15 * * * *")
  # sched["minute"] = [0, 15, 30, 45]
  # sched["hour"]   = list(range(24))

  sched2 = parse_cron_expr("@daily")
  # sched2["minute"] = [0]
  # sched2["hour"]   = [0]

  sched3 = parse_cron_expr("0 9 * * 1-5")
  # sched3["day_of_week"] = [1, 2, 3, 4, 5]
  ```

tests:
  ["*/15 expande minutos a [0 15 30 45]","@daily resuelve a minuto 0 hora 0","0 9 1,15 * * expande dias a [1 15]","0 9 * * 1-5 expande dia de semana a [1 2 3 4 5]","expresion con 4 campos lanza ValueError","minuto fuera de rango lanza ValueError"]

notes:
  Misma semantica que la version Go (parse_cron_expr_go_core). Usa dict en lugar de struct. Limites de campos: minute [0,59], hour [0,23], day_of_month [1,31], month [1,12], day_of_week [0,6] (0=domingo). Lanza ValueError con mensaje descriptivo para entradas invalidas.

code:
  """Parser de expresiones cron estandar de 5 campos."""

  from __future__ import annotations

  _ALIASES: dict[str, str] = {
      "@yearly": "0 0 1 1 *",
      "@annually": "0 0 1 1 *",
      "@monthly": "0 0 1 * *",
      "@weekly": "0 0 * * 0",
      "@daily": "0 0 * * *",
      "@midnight": "0 0 * * *",
      "@hourly": "0 * * * *",
  }

  # (lo, hi) inclusive for each field: minute, hour, day_of_month, month, day_of_week
  _FIELD_LIMITS: list[tuple[int, int]] = [
      (0, 59),
      (0, 23),
      (1, 31),
      (1, 12),
      (0, 6),
  ]
  _FIELD_NAMES = ["minute", "hour", "day_of_month", "month", "day_of_week"]


  def parse_cron_expr(expr: str) -> dict[str, list[int]]:
      """Parsea una expresion cron estandar de 5 campos.

      Soporta *, rangos (1-5), listas (1,3,5), pasos (*/15) y aliases
      (@hourly, @daily, @weekly, @monthly, @yearly).

      Args:
          expr: Expresion cron de 5 campos o alias.

      Returns:
          Dict con claves minute, hour, day_of_month, month, day_of_week,
          cada una con la lista de enteros validos para ese campo.

      Raises:
          ValueError: Si la expresion es invalida o un valor esta fuera de rango.
      """
      expr = expr.strip()
      expr = _ALIASES.get(expr, expr)
      fields = expr.split()
      if len(fields) != 5:
          raise ValueError(
              f"parse_cron_expr: expected 5 fields, got {len(fields)} in {expr!r}"
          )
      result: dict[str, list[int]] = {}
      for i, field in enumerate(fields):
          lo, hi = _FIELD_LIMITS[i]
          key = _FIELD_NAMES[i]
          result[key] = _parse_field(field, lo, hi, key)
      return result


  def _parse_field(field: str, lo: int, hi: int, name: str) -> list[int]:
      if field == "*":
          return list(range(lo, hi + 1))
      seen: set[int] = set()
      values: list[int] = []
      for part in field.split(","):
          for v in _parse_part(part, lo, hi):
              if v not in seen:
                  seen.add(v)
                  values.append(v)
      values.sort()
      return values


  def _parse_part(part: str, lo: int, hi: int) -> list[int]:
      if "/" in part:
          base, step_str = part.split("/", 1)
          try:
              step = int(step_str)
          except ValueError:
              raise ValueError(f"invalid step {step_str!r}")
          if step <= 0:
              raise ValueError(f"step must be positive, got {step}")
          if base == "*":
              start, end = lo, hi
          elif "-" in base:
              start, end = _parse_range(base, lo, hi)
          else:
              start = _parse_value(base, lo, hi)
              end = hi
          return list(range(start, end + 1, step))
      if "-" in part:
          start, end = _parse_range(part, lo, hi)
          return list(range(start, end + 1))
      return [_parse_value(part, lo, hi)]


  def _parse_range(s: str, lo: int, hi: int) -> tuple[int, int]:
      parts = s.split("-", 1)
      if len(parts) != 2:
          raise ValueError(f"invalid range {s!r}")
      start = _parse_value(parts[0], lo, hi)
      end = _parse_value(parts[1], lo, hi)
      if start > end:
          raise ValueError(f"range start {start} > end {end}")
      return start, end


  def _parse_value(s: str, lo: int, hi: int) -> int:
      try:
          v = int(s)
      except ValueError:
          raise ValueError(f"invalid value {s!r}: not an integer")
      if v < lo or v > hi:
          raise ValueError(f"value {v} out of range [{lo}, {hi}]")
      return v

params_schema:
  {"params":[{"name":"expr","desc":"Expresión cron estándar de 5 campos o alias como @daily, @hourly"}],"output":"Diccionario con claves minute, hour, day_of_month, month, day_of_week conteniendo listas de enteros válidos"}


id: parse_git_url_py_core
name: parse_git_url
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def parse_git_url(url: str, known_hosts: list[str] | None = None) -> str | None
tags: ["git","url","parsing","github","gitlab","repository"]
uses_functions: []
uses_types: []
returns: []
imports: ["urllib.parse"]
test_file_path: python/functions/core/parse_git_url_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 4ee5e80ade9730ccf6f0a9ba3c7b2713bdc6ac39ab322eceebdca817b27ea0f5
description:
  Parsea una URL de code hosting (GitHub, GitLab, etc.) y retorna el path org/repo. Soporta HTTPS, HTTP, SSH (git@), git:// y ssh://.

example:
  ```python
  parse_git_url("https://github.com/psf/requests")
  # "psf/requests"

  parse_git_url("git@github.com:psf/requests.git")
  # "psf/requests"

  parse_git_url("https://github.com/psf/requests/issues/123")
  # "psf/requests"  (ignora segmentos extra)

  parse_git_url("https://bitbucket.org/org/repo")
  # None  (host desconocido)
  ```

tests:
  ["URL HTTPS GitHub","URL SSH git@","URL con .git suffix","URL con path extra (issues/123)","URL no reconocida"]

notes:
  Funcion pura. Sanitiza org y repo dejando solo `[a-zA-Z0-9_-]`. Los hosts por defecto son `github.com` y `gitlab.com`; se pueden extender via `known_hosts`. No hace requests de red.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"url","desc":"URL de repositorio de código (HTTPS, SSH, git://, ssh://)"},{"name":"known_hosts","desc":"Lista opcional de hosts conocidos a reconocer"}],"output":"String con formato 'org/repo' si la URL es válida, None si no se puede parsear"}


id: parse_iso_datetime_py_core
name: parse_iso_datetime
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: parse_iso_datetime(value: str) -> datetime
tags: ["datetime","iso8601","parse","time","windows"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","datetime"]
test_file_path: python/functions/core/parse_iso_datetime_test.py
file_path: python/functions/core/parse_iso_datetime.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 9c003264cccefa59999cc793dbfaef318b9fc7b0f01c6853bdc50b1c0a58a823
description:
  Parsea un datetime ISO 8601 tolerando fracciones de segundo con mas de 6 digitos. Windows produce timestamps como 2026-02-21T13:20:23.1470042+08:00 donde la fraccion excede los 6 digitos que acepta datetime.fromisoformat.

example:
  ```python
  from parse_iso_datetime import parse_iso_datetime

  # Timestamp normal
  dt = parse_iso_datetime("2026-02-21T13:20:23+08:00")

  # Timestamp Windows con 7 digitos en fraccion
  dt = parse_iso_datetime("2026-02-21T13:20:23.1470042+08:00")

  # UTC con Z
  dt = parse_iso_datetime("2026-02-21T13:20:23Z")
  ```

tests:
  ["ISO normal sin fraccion","ISO con Z como UTC","ISO con fraccion de mas de 6 digitos","ISO con timezone offset"]

notes:
  Algoritmo:
  1. Regex `(\.\d{6})\d+` trunca la fraccion de segundo a exactamente 6 digitos.
  2. Reemplaza `Z` final por `+00:00` para compatibilidad con `fromisoformat`.
  3. Llama a `datetime.fromisoformat(normalized)`.

  Funcion pura. No hace I/O ni tiene efectos secundarios.

code:
  """Parsea un datetime ISO 8601 tolerando fracciones de segundo con mas de 6 digitos."""

  import re
  from datetime import datetime

  _FRAC_RE = re.compile(r"(\.\d{6})\d+")


  def parse_iso_datetime(value: str) -> datetime:
      """Parsea un datetime ISO 8601 tolerando fracciones de segundo con >6 digitos.

      Windows produce timestamps como ``2026-02-21T13:20:23.1470042+08:00`` donde
      la fraccion de segundo excede los 6 digitos que acepta ``datetime.fromisoformat``.
      Esta funcion trunca la fraccion a 6 digitos antes de parsear.

      Args:
          value: String ISO 8601. Puede contener 'Z' como offset UTC o
                 fracciones de segundo con mas de 6 digitos.

      Returns:
          datetime parseado, con tzinfo si el string incluia offset.
      """
      normalized = _FRAC_RE.sub(r"\1", value)
      normalized = normalized.replace("Z", "+00:00")
      return datetime.fromisoformat(normalized)

params_schema:
  {"params":[{"name":"value","desc":"String en formato ISO 8601 (posiblemente con fracciones de segundo excesivas)"}],"output":"Objeto datetime parseado y normalizado"}


id: parse_llm_json_py_core
name: parse_llm_json
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def parse_llm_json(response: str) -> dict
tags: ["llm","json","parsing","cleaning","think","codeblock"]
uses_functions: ["strip_think_tags_py_core","strip_markdown_codeblock_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["json"]
test_file_path: python/functions/core/parse_llm_json_test.py
file_path: python/functions/core/parse_llm_json.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: d37286401cf3f83416587c6bf4e06938e6e3224663de3fe1b25da4c7650568fb
description:
  Parsea una respuesta LLM como JSON, limpiando primero think tags y markdown codeblocks. Combina strip_think_tags + strip_markdown_codeblock + json.loads.

example:
  ```python
  # Respuesta limpia
  result = parse_llm_json('{"status": "ok"}')
  # {"status": "ok"}

  # Respuesta con think tags y codeblock
  raw = '<think>I need to return JSON.</think>\n```json\n{"status": "ok"}\n```'
  result = parse_llm_json(raw)
  # {"status": "ok"}

  # JSON invalido lanza ValueError
  parse_llm_json("not json at all")
  # ValueError: parse_llm_json: invalid JSON after cleaning. Cleaned text: 'not json at all'
  ```

tests:
  ["JSON limpio es parseado correctamente","JSON con think tags y codeblock es parseado correctamente","JSON invalido lanza ValueError"]

notes:
  Funcion impura porque puede lanzar ValueError. El mensaje de error incluye el texto ya limpiado (post strip_think_tags y strip_markdown_codeblock) para facilitar el debugging. Composicion de tres funciones: strip_think_tags → strip_markdown_codeblock → json.loads.

code:
  """Parse JSON from LLM responses, cleaning think tags and markdown code blocks."""

  import json

  from strip_think_tags import strip_think_tags
  from strip_markdown_codeblock import strip_markdown_codeblock


  def parse_llm_json(response: str) -> dict:
      """Parse an LLM response as JSON, cleaning think tags and markdown code blocks first.

      Combines strip_think_tags + strip_markdown_codeblock + json.loads into a
      single pipeline. Raises ValueError with the cleaned text included in the
      message to aid debugging.

      Args:
          response: Raw LLM response string.

      Returns:
          Parsed JSON as a dict.

      Raises:
          ValueError: If the cleaned text is not valid JSON.
      """
      cleaned = strip_think_tags(response)
      cleaned = strip_markdown_codeblock(cleaned)
      try:
          return json.loads(cleaned)
      except json.JSONDecodeError as exc:
          raise ValueError(
              f"parse_llm_json: invalid JSON after cleaning. "
              f"Cleaned text: {cleaned!r}"
          ) from exc

params_schema:
  {"params":[{"name":"response","desc":"String con respuesta del LLM (posiblemente con think tags y markdown codeblocks)"}],"output":"Diccionario parseado del JSON limpio"}


id: parse_page_range_py_core
name: parse_page_range
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def parse_page_range(pages: str) -> list[int]
tags: ["parsing","range","pages","string"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Lanza ValueError si un rango tiene start > end.
content_hash: 4d47b364c066da092db85fe81c6fcfa0dc1367cdd27110338e346d246e32a027
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/retrieve.py
description:
  Parsea string de rangos de paginas ('5-7', '3,8', '12') a lista de enteros ordenada y sin duplicados.

example:
  ```python
  parse_page_range("1-3,7,10-12")
  # [1, 2, 3, 7, 10, 11, 12]

  parse_page_range("5")
  # [5]
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"pages","desc":"String con especificación de páginas (ej: '5-7,3,10-12')"}],"output":"Lista ordenada de números de página sin duplicados"}


id: parser_registry_py_core
name: parser_registry
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["parser","registry","plugin","extensible","dispatch","pattern","singleton"]
uses_functions: []
uses_types: ["base_parser_py_core","parse_result_py_core"]
returns: ["parse_result_py_core"]
error_type: error_go_core
imports: []
test_file_path: python/functions/core/parser_registry_test.py
file_path: python/functions/core/parser_registry.py
created_at: 2026-04-05T11:25:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: cd508c918240be87ff06ece09e27f1c82b70106995dee9e41371f5eaed31ad86
params_schema: {"output":"ParseResult con contenido parseado y metadatos del parsing"}
signature:
  class ParserRegistry: register(name: str, parser: BaseParser) -> None; unregister(name: str) -> None; get_parser(name: str) -> BaseParser | None; get_parser_for_file(path: str) -> BaseParser | None; async parse(source: str, **kwargs) -> ParseResult; list_parsers() -> list[str]; list_supported_extensions() -> list[str]

description:
  Registry extensible que despacha parsing de archivos al parser correcto basado en extension. Patron plugin: registrar parsers por nombre y extensiones, resolver automaticamente. Mantiene estado mutable (mapa extension→parser). Singleton global disponible via get_registry().

example:
  ```python
  from parser_registry import ParserRegistry, get_registry

  # Usar instancia propia
  registry = ParserRegistry()
  registry.register("markdown", MarkdownParser())
  registry.register("pdf", PDFParser())
  registry.register("text", TextParser())

  result = await registry.parse("documento.md")
  result2 = await registry.parse("Contenido directo\ncon varias lineas")

  # Usar singleton global
  global_registry = get_registry()
  global_registry.register("markdown", MarkdownParser())
  ```

tests:
  ["registrar parser custom","resolver parser por extension de archivo","parsear archivo existente con extension registrada","parsear string de contenido sin path","desregistrar parser elimina nombre y extensiones","extension desconocida hace fallback al parser text"]

notes:
  Patron plugin extensible: cada parser se registra por nombre y declara sus extensiones. El mapa extension→nombre se construye automaticamente al registrar. Unregister limpia el mapa solo si el parser registrado sigue siendo el mismo (evita colisiones si dos parsers comparten extension).

  El singleton `get_registry()` usa lazy initialization thread-unsafe — suficiente para la mayoria de casos. Para uso multi-threaded, crear instancias propias de ParserRegistry.

  Los parsers especiales `"text"`, `"directory"` y `"code"` son convencion — el registry no los crea automaticamente, deben registrarse explicitamente.

documentation:
  ## Algoritmo de despacho

  `parse(source)` sigue este orden de decision:

  1. Si `source` es URL de repositorio de codigo (github.com, gitlab.com, bitbucket.org) → delega al parser `"code"`.
  2. Si `source` parece path de archivo (longitud ≤ 512, sin newlines) y existe en disco:
     - Si es directorio → delega al parser `"directory"`.
     - Resuelve parser por extension → delega.
     - Sin parser para la extension → fallback al parser `"text"` via `parse_content`.
  3. Si no parece path (contenido largo o con newlines) → parsea como string con parser `"text"` via `parse_content`.

code:
  """ParserRegistry — registry extensible que despacha parsing por extension de archivo."""

  from __future__ import annotations

  import re
  from pathlib import Path
  from typing import TYPE_CHECKING

  if TYPE_CHECKING:
      from base_parser import BaseParser
      from parse_result import ParseResult

  # Patron para detectar URLs de repositorios de codigo.
  _CODE_REPO_PATTERN = re.compile(
      r"^https?://(github\.com|gitlab\.com|bitbucket\.org)/[\w.\-]+/[\w.\-]+"
  )

  # Longitud maxima que puede tener un path de archivo valido (heuristica).
  _MAX_PATH_LEN = 512


  def _looks_like_path(source: str) -> bool:
      """Heuristica: True si source parece un path de archivo, no contenido."""
      return len(source) <= _MAX_PATH_LEN and "\n" not in source


  class ParserRegistry:
      """Registry extensible que despacha parsing al parser correcto segun extension.

      Patron plugin: registrar parsers por nombre y lista de extensiones,
      resolver automaticamente al parsear un archivo. El registry mantiene
      un mapa de extension→nombre para O(1) lookup.

      Uso tipico:
          registry = ParserRegistry()
          registry.register("markdown", MarkdownParser())
          registry.register("pdf", PDFParser())
          result = await registry.parse("doc.md")
      """

      def __init__(self) -> None:
          # nombre → instancia de parser
          self._parsers: dict[str, "BaseParser"] = {}
          # extension → nombre del parser
          self._ext_map: dict[str, str] = {}

      # ------------------------------------------------------------------
      # Registro
      # ------------------------------------------------------------------

      def register(self, name: str, parser: "BaseParser") -> None:
          """Registra un parser con el nombre dado.

          Sobreescribe cualquier parser previamente registrado con el mismo nombre.
          Actualiza el mapa de extensiones automaticamente.

          Args:
              name:   Nombre unico del parser (ej: "markdown", "pdf").
              parser: Instancia de BaseParser a registrar.
          """
          self._parsers[name] = parser
          for ext in parser.supported_extensions:
              self._ext_map[ext.lower()] = name

      def unregister(self, name: str) -> None:
          """Elimina un parser del registry.

          Tambien elimina sus extensiones del mapa. Si el nombre no existe,
          no hace nada.

          Args:
              name: Nombre del parser a eliminar.
          """
          parser = self._parsers.pop(name, None)
          if parser is None:
              return
          for ext in parser.supported_extensions:
              key = ext.lower()
              if self._ext_map.get(key) == name:
                  del self._ext_map[key]

      # ------------------------------------------------------------------
      # Resolucion
      # ------------------------------------------------------------------

      def get_parser(self, name: str) -> "BaseParser | None":
          """Retorna el parser registrado bajo el nombre dado.

          Args:
              name: Nombre del parser.

          Returns:
              Instancia de BaseParser o None si no existe.
          """
          return self._parsers.get(name)

      def get_parser_for_file(self, path: str) -> "BaseParser | None":
          """Resuelve el parser correcto para un archivo dado su extension.

          Args:
              path: Path del archivo (puede ser relativo o absoluto).

          Returns:
              Instancia de BaseParser o None si no hay parser para esa extension.
          """
          ext = Path(path).suffix.lower()
          name = self._ext_map.get(ext)
          if name is None:
              return None
          return self._parsers.get(name)

      # ------------------------------------------------------------------
      # Parseo
      # ------------------------------------------------------------------

      async def parse(self, source: str, **kwargs) -> "ParseResult":
          """Parsea source despachando al parser correcto segun el tipo de entrada.

          Algoritmo de despacho:
          1. Si source es URL de repo de codigo → delega al parser "code".
          2. Si source parece path de archivo (corto, sin newlines) y existe en disco:
             - Si es directorio → delega al parser "directory".
             - Resuelve parser por extension → delega.
             - Si no hay parser para la extension → fallback al parser "text".
          3. Si no parece path (contenido largo o con newlines) → parsea como
             string de contenido con el parser "text".

          Args:
              source:   Path de archivo, directorio, URL o string de contenido.
              **kwargs: Argumentos adicionales pasados al parser seleccionado.

          Returns:
              ParseResult con el arbol de nodos del documento.

          Raises:
              ValueError: Si el parser requerido no esta registrado.
          """
          # 1. URL de repositorio de codigo
          if _CODE_REPO_PATTERN.match(source):
              return await self._dispatch("code", source, **kwargs)

          # 2. Parece un path de archivo
          if _looks_like_path(source):
              p = Path(source)
              if p.exists():
                  if p.is_dir():
                      return await self._dispatch("directory", source, **kwargs)
                  parser = self.get_parser_for_file(source)
                  if parser is not None:
                      return await parser.parse(source, **kwargs)
                  # Fallback a text si no hay parser para la extension
                  return await self._dispatch_content("text", source, **kwargs)

          # 3. Es contenido directo (string largo o con newlines)
          text_parser = self._parsers.get("text")
          if text_parser is None:
              raise ValueError(
                  "No hay parser 'text' registrado. Registra un TextParser primero."
              )
          return await text_parser.parse_content(source, **kwargs)

      # ------------------------------------------------------------------
      # Introspeccion
      # ------------------------------------------------------------------

      def list_parsers(self) -> list[str]:
          """Retorna los nombres de todos los parsers registrados.

          Returns:
              Lista de nombres de parsers, ordenada alfabeticamente.
          """
          return sorted(self._parsers.keys())

      def list_supported_extensions(self) -> list[str]:
          """Retorna todas las extensiones soportadas por parsers registrados.

          Returns:
              Lista de extensiones (con punto, lowercase), ordenada alfabeticamente.
          """
          return sorted(self._ext_map.keys())

      # ------------------------------------------------------------------
      # Helpers internos
      # ------------------------------------------------------------------

      async def _dispatch(self, name: str, source: str, **kwargs) -> "ParseResult":
          """Despacha a un parser por nombre via parse()."""
          parser = self._parsers.get(name)
          if parser is None:
              raise ValueError(
                  f"Parser '{name}' no esta registrado. "
                  f"Parsers disponibles: {self.list_parsers()}"
              )
          return await parser.parse(source, **kwargs)

      async def _dispatch_content(
          self, name: str, content: str, **kwargs
      ) -> "ParseResult":
          """Despacha a un parser por nombre via parse_content()."""
          parser = self._parsers.get(name)
          if parser is None:
              raise ValueError(
                  f"Parser '{name}' no esta registrado. "
                  f"Parsers disponibles: {self.list_parsers()}"
              )
          return await parser.parse_content(content, **kwargs)


  # ------------------------------------------------------------------
  # Singleton global
  # ------------------------------------------------------------------

  _registry: ParserRegistry | None = None


  def get_registry() -> ParserRegistry:
      """Retorna la instancia global del ParserRegistry (lazy-initialized).

      Returns:
          Instancia singleton de ParserRegistry.
      """
      global _registry
      if _registry is None:
          _registry = ParserRegistry()
      return _registry


id: partition_py_core
name: partition
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def partition(xs: list, pred: callable) -> tuple
description: Divide una lista en dos: (elementos que cumplen el predicado, elementos que no).
tags: ["list","functional","partition","split","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Retorna tupla de dos listas: (matches, non_matches).
content_hash: afde859b97ca114c587756d63bd8a7005c92729e49dad6d950e2ac7eee83446f
example:
  ```python
  evens, odds = partition([1, 2, 3, 4, 5], lambda n: n % 2 == 0)
  # evens = [2, 4], odds = [1, 3, 5]
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista a dividir"},{"name":"pred","desc":"Función predicado que retorna True/False"}],"output":"Tupla (elementos_que_cumplen, elementos_que_no_cumplen)"}


id: pdf_to_markdown_py_core
name: pdf_to_markdown
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def pdf_to_markdown(pdf_path: str, heading_detection: str = 'auto') -> tuple[str, dict]
tags: ["pdf","markdown","conversion","headings","tables","pdfplumber","parsing"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["pdfplumber","os","sys"]
tests: []
file_path: python/functions/core/pdf_to_markdown.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 440e3d88d63ac293d2815e1a880ce883a697d7ade029a6702c75c23142b003d5
description:
  Convierte un PDF a markdown. Extrae texto, tablas e inyecta headings detectados desde bookmarks o analisis de fuentes. Retorna (markdown_content, metadata_dict).

uses_functions:
  ["extract_pdf_bookmarks_py_core","detect_headings_by_font_py_core","format_table_to_markdown_py_core"]

example:
  ```python
  from pdf_to_markdown import pdf_to_markdown

  md, meta = pdf_to_markdown("report.pdf")
  print(f"Pages: {meta['pages']}, Headings: {meta['headings_found']}, Tables: {meta['tables_found']}")
  print(md[:500])

  # Con estrategia especifica
  md, meta = pdf_to_markdown("report.pdf", heading_detection="font")
  print(f"Strategy used: {meta['heading_strategy']}")
  ```

notes:
  Estrategias de heading_detection:
  - `"bookmarks"`: extrae outlines/bookmarks del PDF via pdfplumber. Rapido y preciso cuando el PDF tiene bookmarks bien formados.
  - `"font"`: analiza distribucion de font sizes para detectar headings. Util para PDFs sin bookmarks.
  - `"auto"`: intenta bookmarks primero; si no hay, usa analisis de font. Estrategia por defecto.

  El metadata_dict retornado contiene: `pages` (int), `heading_strategy` (str: "bookmarks"|"font"|"none"), `headings_found` (int), `tables_found` (int).

  Tablas se extraen via `page.extract_tables()` de pdfplumber y se formatean como tablas markdown. Requiere `pip install pdfplumber`.

code:
  """Convert a PDF file to markdown, with heading detection and table extraction."""

  import os
  import sys

  import pdfplumber

  # Allow import from same directory when run directly
  _DIR = os.path.dirname(os.path.abspath(__file__))
  if _DIR not in sys.path:
      sys.path.insert(0, _DIR)

  from extract_pdf_bookmarks import extract_pdf_bookmarks
  from detect_headings_by_font import detect_headings_by_font
  from format_table_to_markdown import format_table_to_markdown


  def pdf_to_markdown(
      pdf_path: str,
      heading_detection: str = "auto",
  ) -> tuple[str, dict]:
      """Convert a PDF file to markdown.

      Extracts text, tables, and injects detected headings from bookmarks or
      font size analysis.

      Args:
          pdf_path: Path to the PDF file.
          heading_detection: Strategy for detecting headings:
              - "bookmarks": use PDF outline/bookmarks only.
              - "font": analyze font size distribution.
              - "auto": try bookmarks first, fall back to font analysis.

      Returns:
          tuple[str, dict]: (markdown_content, metadata_dict).
              metadata_dict contains: {"pages": int, "heading_strategy": str,
              "headings_found": int, "tables_found": int}

      Raises:
          FileNotFoundError: If pdf_path does not exist.
          Exception: If the PDF cannot be opened or is corrupted.
      """
      if not os.path.exists(pdf_path):
          raise FileNotFoundError(f"PDF not found: {pdf_path}")

      with pdfplumber.open(pdf_path) as pdf:
          total_pages = len(pdf.pages)

          # Step 1: Detect headings
          headings: list[dict] = []
          strategy_used = heading_detection

          if heading_detection in ("bookmarks", "auto"):
              headings = extract_pdf_bookmarks(pdf)
              if headings:
                  strategy_used = "bookmarks"
              elif heading_detection == "auto":
                  headings = detect_headings_by_font(pdf)
                  strategy_used = "font" if headings else "none"
          elif heading_detection == "font":
              headings = detect_headings_by_font(pdf)
              strategy_used = "font"

          # Step 2: Group headings by page number
          headings_by_page: dict[int, list[dict]] = {}
          for h in headings:
              pn = h.get("page_num")
              if pn is not None:
                  headings_by_page.setdefault(pn, []).append(h)

          # Step 3: Build markdown page by page
          sections: list[str] = []
          tables_found = 0

          for page_idx, page in enumerate(pdf.pages):
              page_num = page_idx + 1
              page_parts: list[str] = []

              # Inject headings for this page
              for h in headings_by_page.get(page_num, []):
                  level = h["level"]
                  title = h["title"]
                  page_parts.append(f"{'#' * level} {title}")

              # Extract page text
              text = page.extract_text() or ""
              if text.strip():
                  page_parts.append(text.strip())

              # Extract tables
              try:
                  tables = page.extract_tables()
              except Exception:
                  tables = []

              for table in tables:
                  if not table:
                      continue
                  # Convert all cells to strings, treating None as empty
                  str_table = [
                      [str(cell) if cell is not None else "" for cell in row]
                      for row in table
                  ]
                  md_table = format_table_to_markdown(str_table)
                  if md_table:
                      page_parts.append(md_table)
                      tables_found += 1

              if page_parts:
                  sections.append("\n\n".join(page_parts))

          markdown_content = "\n\n".join(sections)

          metadata = {
              "pages": total_pages,
              "heading_strategy": strategy_used,
              "headings_found": len(headings),
              "tables_found": tables_found,
          }

          return markdown_content, metadata

params_schema:
  {"params":[{"name":"pdf_path","desc":"Ruta al archivo PDF a convertir"},{"name":"heading_detection","desc":"Estrategia de detección de headings: 'auto', 'bookmarks', 'font'"}],"output":"Tupla (markdown_content, metadata_dict) con el PDF convertido a markdown y metadatos"}


id: pipe_py_core
name: pipe
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def pipe(value, *fns)
description: Pasa un valor a traves de una secuencia de funciones de izquierda a derecha.
tags: ["functional","pipe","composition","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a3950c6e7d628ef3dd17ec6ef31b5a87f701260a9ddf46e9f8caa4a6a6c0000b
example:
  ```python
  result = pipe(
      [1, 2, 3, 4, 5],
      lambda xs: filter_list(xs, lambda n: n > 2),
      lambda xs: map_list(xs, lambda n: n * 10),
  )
  # [30, 40, 50]
  ```

notes:
  Funcion pura. Ejecuta las funciones en orden de izquierda a derecha: f1(value), luego f2(result), etc.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"value","desc":"Valor inicial a pasar por las funciones"},{"name":"fns","desc":"Funciones a aplicar secuencialmente de izquierda a derecha"}],"output":"Resultado final después de aplicar todas las funciones en orden"}


id: preprocess_text_py_core
name: preprocess_text
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def preprocess_text(text: str) -> str
description: Normaliza whitespace y newlines de un texto crudo. Util como paso previo a chunking o indexing.
tags: ["text","preprocessing","normalization","whitespace","newlines"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/preprocess_text_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1ea3ef46077bb8518bacfe823abf1c5669466407f3348258d13ecb8fcfeff373
example:
  ```python
  raw = "  hello  \r\n  world  \r\n\n\n\n  end  "
  result = preprocess_text(raw)
  # "hello\nworld\n\nend"
  ```

tests:
  ["texto con saltos de linea Windows CRLF","texto con multiples newlines consecutivos","texto con espacios leading y trailing en cada linea"]

notes:
  Funcion pura. Aplica tres pasos en orden: normalizar line endings (CRLF y CR a LF), colapsar 3+ newlines consecutivos a 2, strip de cada linea. El strip global final elimina whitespace al inicio y al final del texto completo.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"text","desc":"Texto crudo a normalizar"}],"output":"Texto con whitespace normalizado y newlines ajustados"}


id: react_loop_py_core
name: react_loop
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["llm","agent","react","reasoning","tools","loop"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["json","re","typing"]
test_file_path: python/functions/core/react_loop_test.py
file_path: python/functions/core/react_loop.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2503ed1aeec783924cb5e7d9bd1279013f6c1ff7afe822b6b1578ca6fb8b4e2
signature:
  def react_loop(llm_chat: Callable[[list[dict]], str], tools: dict[str, Callable[..., str]], system_prompt: str, user_prompt: str, max_iterations: int = 5, on_thought: Callable[[str], None] | None = None, on_action: Callable[[str, dict], None] | None = None, on_observation: Callable[[str], None] | None = None) -> str

description:
  Implementa el patron ReACT (Reasoning + Acting) para agentes LLM. El agente razona, decide usar herramientas, observa resultados, y repite hasta producir una respuesta final.

example:
  ```python
  import openai

  client = openai.OpenAI()

  def llm_chat(messages: list[dict]) -> str:
      resp = client.chat.completions.create(model="gpt-4o", messages=messages)
      return resp.choices[0].message.content

  def search(query: str) -> str:
      """Search the knowledge base."""
      return f"Results for: {query}"

  def analyze(entity_id: str) -> str:
      """Deep dive into an entity."""
      return f"Analysis of entity {entity_id}"

  answer = react_loop(
      llm_chat=llm_chat,
      tools={"search": search, "analyze": analyze},
      system_prompt="You are a research assistant.",
      user_prompt="What is quantum entanglement?",
      max_iterations=5,
      on_thought=lambda t: print(f"[thought] {t}"),
      on_action=lambda a, i: print(f"[action] {a}({i})"),
      on_observation=lambda o: print(f"[obs] {o}"),
  )
  print(answer)
  ```

tests:
  ["agente que usa 1 tool y da respuesta final","agente que usa multiples tools iterativamente","agente que da respuesta directa sin tools","max iterations alcanzado fallback","tool que falla error handling"]

notes:
  El formato de parseo sigue el patron ReACT clasico (Thought/Action/Action Input/Observation/Final Answer), compatible con LangChain y agentes basados en este patron.

  Los callbacks (on_thought, on_action, on_observation) permiten logging, UI y tracing sin acoplar al agente.

  La funcion es completamente generica: funciona con cualquier LLM (OpenAI, Anthropic, litellm) y cualquier conjunto de tools. El tool description se extrae del primer renglon del docstring de cada callable.

  Si max_iterations se agota sin Final Answer, retorna el ultimo response crudo del LLM como fallback.

  Los errores de tools se capturan y se entregan como Observation al LLM para que pueda recuperarse. Los errores del LLM propio se propagan hacia arriba.

code:
  """ReACT (Reasoning + Acting) agent loop for LLM-based agents."""

  import json
  import re
  from typing import Callable


  def react_loop(
      llm_chat: Callable[[list[dict]], str],
      tools: dict[str, Callable[..., str]],
      system_prompt: str,
      user_prompt: str,
      max_iterations: int = 5,
      on_thought: Callable[[str], None] | None = None,
      on_action: Callable[[str, dict], None] | None = None,
      on_observation: Callable[[str], None] | None = None,
  ) -> str:
      """Execute a ReACT (Reasoning + Acting) agent loop.

      The agent reasons, selects tools, observes results, and repeats until
      producing a Final Answer or exhausting max_iterations.

      Args:
          llm_chat: Callable that takes a list of messages and returns a string response.
          tools: Mapping of tool name to callable. Each tool returns a string.
          system_prompt: System instruction for the agent.
          user_prompt: Initial user request.
          max_iterations: Maximum reasoning iterations before fallback.
          on_thought: Optional callback invoked with each Thought string.
          on_action: Optional callback invoked with (action_name, action_input_dict).
          on_observation: Optional callback invoked with each Observation string.

      Returns:
          The Final Answer string, or the last raw LLM response as fallback.
      """
      tools_description = _format_tools(tools)

      augmented_system = (
          f"{system_prompt}\n\n"
          "You have access to the following tools:\n"
          f"{tools_description}\n\n"
          "Use this format:\n"
          "Thought: <your reasoning>\n"
          "Action: <tool name>\n"
          "Action Input: <JSON object with tool arguments>\n"
          "Observation: <tool result will appear here>\n"
          "... (repeat as needed)\n"
          "Final Answer: <your final response to the user>"
      )

      messages: list[dict] = [
          {"role": "system", "content": augmented_system},
          {"role": "user", "content": user_prompt},
      ]

      last_response = ""

      for _ in range(max_iterations):
          response = llm_chat(messages)
          last_response = response

          final_answer = _parse_final_answer(response)
          if final_answer is not None:
              return final_answer

          thought = _parse_section(response, "Thought")
          action = _parse_section(response, "Action")
          action_input_raw = _parse_section(response, "Action Input")

          if thought and on_thought:
              on_thought(thought)

          if not action:
              # No action and no final answer — return as-is
              return response

          action_input: dict = {}
          if action_input_raw:
              try:
                  action_input = json.loads(action_input_raw)
              except json.JSONDecodeError:
                  action_input = {"input": action_input_raw}

          if on_action:
              on_action(action, action_input)

          tool_fn = tools.get(action)
          if tool_fn is None:
              observation = f"Error: tool '{action}' not found. Available tools: {list(tools.keys())}"
          else:
              try:
                  observation = tool_fn(**action_input)
              except Exception as exc:
                  observation = f"Error executing tool '{action}': {exc}"

          if on_observation:
              on_observation(observation)

          # Append assistant turn + observation to continue the conversation
          messages.append({"role": "assistant", "content": response})
          messages.append({"role": "user", "content": f"Observation: {observation}"})

      return last_response


  # ---------------------------------------------------------------------------
  # Private helpers
  # ---------------------------------------------------------------------------

  def _format_tools(tools: dict[str, Callable[..., str]]) -> str:
      """Build a human-readable tool list from the tools dict."""
      lines = []
      for name, fn in tools.items():
          doc = (fn.__doc__ or "").strip().split("\n")[0]
          lines.append(f"- {name}: {doc}" if doc else f"- {name}")
      return "\n".join(lines)


  def _parse_section(text: str, label: str) -> str | None:
      """Extract the value after a 'Label:' marker, stopping at the next label."""
      pattern = rf"^{label}:\s*(.+?)(?=\n(?:Thought|Action Input|Action|Observation|Final Answer):|$)"
      match = re.search(pattern, text, re.IGNORECASE | re.MULTILINE | re.DOTALL)
      if match:
          return match.group(1).strip()
      return None


  def _parse_final_answer(text: str) -> str | None:
      """Extract Final Answer from agent response."""
      match = re.search(r"Final Answer:\s*(.+)", text, re.IGNORECASE | re.DOTALL)
      if match:
          return match.group(1).strip()
      return None

params_schema:
  {"params":[{"name":"llm_chat","desc":"Función que envía mensajes al LLM y retorna la respuesta"},{"name":"tools","desc":"Diccionario de funciones disponibles como tools {nombre: callable}"},{"name":"system_prompt","desc":"Instrucciones del sistema para el agente"},{"name":"user_prompt","desc":"Pregunta/tarea inicial del usuario"},{"name":"max_iterations","desc":"Máximo número de iteraciones antes de fallback"},{"name":"on_thought","desc":"Callback opcional para logging de pensamientos"},{"name":"on_action","desc":"Callback opcional para logging de acciones"},{"name":"on_observation","desc":"Callback opcional para logging de observaciones"}],"output":"String con la respuesta final del agente"}


id: reduce_list_py_core
name: reduce_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def reduce_list(xs: list, initial, fn: callable)
description: Reduce una lista con un acumulador y una funcion binaria fn(acc, x).
tags: ["list","functional","reduce","fold","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  total = reduce_list([1, 2, 3, 4], 0, lambda acc, x: acc + x)
  # 10
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 93e872393ca31a41f7deb8f2296e7eb221f5125464224d53b5b185dc581b6d28
notes:
  Funcion pura. Usa functools.reduce internamente. El valor inicial es obligatorio para evitar errores con listas vacias.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista a reducir"},{"name":"initial","desc":"Valor inicial del acumulador"},{"name":"fn","desc":"Función binaria fn(acumulador, elemento)"}],"output":"Valor final del acumulador después de procesar todos los elementos"}


id: remove_tree_fields_py_core
name: remove_tree_fields
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def remove_tree_fields(data: Any, fields: list[str] = None) -> Any
description: Elimina campos especificados recursivamente de un arbol (dict/list). Por defecto elimina 'text'.
tags: ["tree","filter","fields","recursive"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Crea nueva estructura sin los campos indicados. No muta el original.
content_hash: 21e57aa89cb454b7926d07ad124eb4a0a37edf7d238df6a4285407565d159bb4
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  data = {"title": "A", "text": "largo...", "nodes": [{"title": "B", "text": "otro..."}]}
  remove_tree_fields(data, ["text"])
  # {"title": "A", "nodes": [{"title": "B"}]}
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"data","desc":"Estructura árbol (dict/list/valor) a procesar"},{"name":"fields","desc":"Lista de nombres de campo a eliminar recursivamente (default: ['text'])"}],"output":"Copia de la estructura sin los campos especificados"}


id: render_template_py_core
name: render_template
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: render_template(template: str, context: dict, missing: str = '') -> str
tags: ["template","render","html","string","format","jinja"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/render_template_test.py
file_path: python/functions/core/render_template.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 06947a4d35706ad4b354da5321505e33bbfaaa05cf26e4fa71a5f0d1d6f1ce79
description:
  Motor de templates minimalista sin dependencias. Soporta {{var}}, {{{raw}}}, {{obj.field}} dot-path, {% for x in list %}...{% endfor %}, {% if cond %}...{% endif %}, {% if not cond %}...{% endif %}. HTML-escaped por defecto.

example:
  ```python
  # Sustitucion simple
  result = render_template("Hola {{nombre}}!", {"nombre": "Ana"})
  # "Hola Ana!"

  # Dot-path
  ctx = {"user": {"name": "Bob", "city": "Madrid"}}
  result = render_template("{{user.name}} en {{user.city}}", ctx)
  # "Bob en Madrid"

  # For loop
  ctx = {"items": ["a", "b", "c"]}
  result = render_template("{% for x in items %}-{{x}}{% endfor %}", ctx)
  # "-a-b-c"

  # If condicional
  result = render_template("{% if show %}visible{% endif %}", {"show": True})
  # "visible"

  # Sin escape (raw)
  result = render_template("{{{html}}}", {"html": "<b>bold</b>"})
  # "<b>bold</b>"

  # Variable ausente con valor configurable
  result = render_template("{{falta}}", {}, missing="N/A")
  # "N/A"
  ```

tests:
  ["sustitucion simple","dot path en nested dicts","for loop con lista de strings","for loop con lista de dicts","if endif condicional","variable missing retorna string vacio","variable missing configurable","html escaping por defecto","triple braces sin escape"]

notes:
  No pretende reemplazar Jinja2. Cubre el 80% de casos simples sin dependencias.
  HTML escape por defecto protege contra XSS al generar HTML desde datos de usuario.
  Usa {{{triple}}} cuando el valor ya es HTML confiable.
  Los bloques for/if pueden anidarse pero el parsing es iterativo, no recursivo profundo.

code:
  """Motor de templates minimalista sin dependencias externas."""

  import re


  _HTML_ESCAPES = {
      "&": "&amp;",
      "<": "&lt;",
      ">": "&gt;",
      '"': "&quot;",
      "'": "&#x27;",
  }


  def _html_escape(value: str) -> str:
      for ch, entity in _HTML_ESCAPES.items():
          value = value.replace(ch, entity)
      return value


  def _resolve_path(context: dict, path: str, missing: str) -> str:
      """Resuelve un dot-path en un dict anidado."""
      parts = path.strip().split(".")
      current = context
      for part in parts:
          if isinstance(current, dict) and part in current:
              current = current[part]
          else:
              return missing
      return str(current) if current is not None else ""


  def _process_block(template: str, context: dict, missing: str) -> str:
      """Procesa un bloque de template (puede contener for/if anidados)."""
      result = template

      # Procesar bloques for: {% for x in items %}...{% endfor %}
      for_pattern = re.compile(
          r"\{%\s*for\s+(\w+)\s+in\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endfor\s*%\}",
          re.DOTALL,
      )
      while for_pattern.search(result):
          def replace_for(m: re.Match) -> str:
              var_name = m.group(1)
              list_path = m.group(2)
              body = m.group(3)
              items = _resolve_raw(context, list_path)
              if not isinstance(items, list):
                  return ""
              parts = []
              for item in items:
                  loop_context = dict(context)
                  loop_context[var_name] = item
                  parts.append(_process_block(body, loop_context, missing))
              return "".join(parts)

          result = for_pattern.sub(replace_for, result)

      # Procesar bloques if not: {% if not cond %}...{% endif %}
      if_not_pattern = re.compile(
          r"\{%\s*if\s+not\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endif\s*%\}",
          re.DOTALL,
      )
      while if_not_pattern.search(result):
          def replace_if_not(m: re.Match) -> str:
              cond_path = m.group(1)
              body = m.group(2)
              value = _resolve_raw(context, cond_path)
              if not value:
                  return _process_block(body, context, missing)
              return ""

          result = if_not_pattern.sub(replace_if_not, result)

      # Procesar bloques if: {% if cond %}...{% endif %}
      if_pattern = re.compile(
          r"\{%\s*if\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endif\s*%\}",
          re.DOTALL,
      )
      while if_pattern.search(result):
          def replace_if(m: re.Match) -> str:
              cond_path = m.group(1)
              body = m.group(2)
              value = _resolve_raw(context, cond_path)
              if value:
                  return _process_block(body, context, missing)
              return ""

          result = if_pattern.sub(replace_if, result)

      # Sustitucion triple braces (sin escape): {{{var}}}
      raw_pattern = re.compile(r"\{\{\{(\s*[\w.]+\s*)\}\}\}")
      result = raw_pattern.sub(
          lambda m: _resolve_path(context, m.group(1), missing), result
      )

      # Sustitucion doble braces (HTML-escaped): {{var}}
      escaped_pattern = re.compile(r"\{\{(\s*[\w.]+\s*)\}\}")
      result = escaped_pattern.sub(
          lambda m: _html_escape(_resolve_path(context, m.group(1), missing)), result
      )

      return result


  def _resolve_raw(context: dict, path: str):
      """Resuelve un dot-path y retorna el valor raw (no convertido a string)."""
      parts = path.strip().split(".")
      current = context
      for part in parts:
          if isinstance(current, dict) and part in current:
              current = current[part]
          else:
              return None
      return current


  def render_template(template: str, context: dict, missing: str = "") -> str:
      """Motor de templates minimalista sin dependencias.

      Soporta sustitucion de variables, dot-path traversal, bucles for
      e condicionales if/if not. No pretende reemplazar Jinja2 — cubre
      el 80% de casos simples.

      Sintaxis:
          {{var}}             — sustitucion con HTML escape
          {{{var}}}           — sustitucion sin escape (raw)
          {{obj.key.subkey}}  — dot-path traversal en dicts anidados
          {% for x in items %}...{% endfor %} — iteracion sobre lista
          {% if cond %}...{% endif %}          — condicional truthy
          {% if not cond %}...{% endif %}      — negacion

      Args:
          template: String con la plantilla.
          context: Dict con los valores disponibles para sustitucion.
          missing: Valor por defecto para variables no encontradas.
                   Por defecto string vacio.

      Returns:
          String con todos los placeholders procesados.
      """
      return _process_block(template, context, missing)

params_schema:
  {"params":[{"name":"template","desc":"String con template usando {{var}}, {{{raw}}}, {% for %}, {% if %}"},{"name":"context","desc":"Diccionario con valores para las variables"},{"name":"missing","desc":"Valor a usar cuando falta una variable (default: '')"}],"output":"String renderizado con variables sustituidas y lógica aplicada"}


id: retry_async_py_core
name: retry_async
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","async","resilience","api","asyncio"]
uses_functions: ["classify_api_error_py_core","compute_backoff_delay_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["asyncio","typing"]
tests: []
file_path: python/functions/core/retry_async.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e2232c77a0f2085d0a151f77b95a39da971e6edb83748a857841ac18fba56516
signature:
  async def retry_async(func: Callable[[], Awaitable[T]], max_retries: int, base_delay: float = 0.5, max_delay: float = 8.0, jitter: bool = True, is_retryable: Callable[[Exception], bool] | None = None) -> T

description:
  Version async de retry_sync. Reintenta una corrutina en errores transitorios con exponential backoff. Usa asyncio.sleep en vez de time.sleep.

example:
  ```python
  import httpx

  async def fetch():
      async with httpx.AsyncClient() as client:
          return await client.get("https://api.example.com/data")

  # Reintentar hasta 3 veces con backoff
  response = await retry_async(func=fetch, max_retries=3)

  # Con predicado personalizado
  result = await retry_async(
      func=my_async_db_query,
      max_retries=5,
      is_retryable=lambda e: "deadlock" in str(e),
  )
  ```

notes:
  Version async de retry_sync: usa `asyncio.sleep` en lugar de `time.sleep`, permitiendo que el event loop atienda otras tareas durante la espera. El comportamiento de clasificacion y backoff es identico a retry_sync.

code:
  """Retry an async callable with exponential backoff."""

  import asyncio
  from typing import Awaitable, Callable, TypeVar

  from classify_api_error import classify_api_error
  from compute_backoff_delay import compute_backoff_delay

  T = TypeVar("T")


  async def retry_async(
      func: Callable[[], Awaitable[T]],
      max_retries: int,
      base_delay: float = 0.5,
      max_delay: float = 8.0,
      jitter: bool = True,
      is_retryable: Callable[[Exception], bool] | None = None,
  ) -> T:
      """Retry an async function on transient errors with exponential backoff.

      Args:
          func: Zero-argument async callable to execute and retry.
          max_retries: Maximum number of retry attempts after the first failure.
          base_delay: Base delay in seconds for backoff calculation.
          max_delay: Maximum delay cap in seconds.
          jitter: If True, adds random jitter to backoff delay.
          is_retryable: Optional predicate; if None, uses classify_api_error to
                        classify transient errors as retryable.

      Returns:
          The return value of func on success.

      Raises:
          Exception: The last exception raised if all attempts are exhausted or
                     if the error is not retryable.
      """
      if is_retryable is None:
          is_retryable = lambda e: classify_api_error(e) == "transient"

      last_exc: Exception | None = None
      for attempt in range(max_retries + 1):
          try:
              return await func()
          except Exception as exc:
              last_exc = exc
              if not is_retryable(exc) or attempt >= max_retries:
                  raise
              delay = compute_backoff_delay(attempt, base_delay, max_delay, jitter)
              await asyncio.sleep(delay)

      raise last_exc  # unreachable but satisfies type checkers

params_schema:
  {"params":[{"name":"func","desc":"Corrutina a reintentar"},{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"base_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"is_retryable","desc":"Predicado opcional para decidir si reintentar una excepción"}],"output":"Resultado de ejecutar la función si tiene éxito"}


id: retry_sync_py_core
name: retry_sync
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","sync","resilience","api"]
uses_functions: ["classify_api_error_py_core","compute_backoff_delay_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["time","typing"]
tests: []
file_path: python/functions/core/retry_sync.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 70ff8396d5324c071f29ee769f47d444f351f55a89fecbbc3b79848a66b6ffff
signature:
  def retry_sync(func: Callable[[], T], max_retries: int, base_delay: float = 0.5, max_delay: float = 8.0, jitter: bool = True, is_retryable: Callable[[Exception], bool] | None = None) -> T

description:
  Reintenta una funcion sincrona en errores transitorios con exponential backoff. Por defecto usa classify_api_error para decidir si reintentar. Hace time.sleep entre intentos.

example:
  ```python
  import httpx

  # Reintentar una llamada HTTP transitoria hasta 3 veces
  response = retry_sync(
      func=lambda: httpx.get("https://api.example.com/data"),
      max_retries=3,
  )

  # Con predicado personalizado
  retry_sync(
      func=my_db_query,
      max_retries=5,
      is_retryable=lambda e: "deadlock" in str(e),
  )
  ```

notes:
  Impure: hace `time.sleep` entre reintentos. Usa `classify_api_error` como clasificador por defecto — solo reintenta errores marcados como "transient". Si `is_retryable` devuelve False en la primera excepcion, re-raise inmediato sin esperar. El parametro `max_retries` es el numero de reintentos despues del primer intento (total de llamadas = max_retries + 1).

code:
  """Retry a synchronous callable with exponential backoff."""

  import time
  from typing import Callable, TypeVar

  from classify_api_error import classify_api_error
  from compute_backoff_delay import compute_backoff_delay

  T = TypeVar("T")


  def retry_sync(
      func: Callable[[], T],
      max_retries: int,
      base_delay: float = 0.5,
      max_delay: float = 8.0,
      jitter: bool = True,
      is_retryable: Callable[[Exception], bool] | None = None,
  ) -> T:
      """Retry a synchronous function on transient errors with exponential backoff.

      Args:
          func: Zero-argument callable to execute and retry.
          max_retries: Maximum number of retry attempts after the first failure.
          base_delay: Base delay in seconds for backoff calculation.
          max_delay: Maximum delay cap in seconds.
          jitter: If True, adds random jitter to backoff delay.
          is_retryable: Optional predicate; if None, uses classify_api_error to
                        classify transient errors as retryable.

      Returns:
          The return value of func on success.

      Raises:
          Exception: The last exception raised if all attempts are exhausted or
                     if the error is not retryable.
      """
      if is_retryable is None:
          is_retryable = lambda e: classify_api_error(e) == "transient"

      last_exc: Exception | None = None
      for attempt in range(max_retries + 1):
          try:
              return func()
          except Exception as exc:
              last_exc = exc
              if not is_retryable(exc) or attempt >= max_retries:
                  raise
              delay = compute_backoff_delay(attempt, base_delay, max_delay, jitter)
              time.sleep(delay)

      raise last_exc  # unreachable but satisfies type checkers

params_schema:
  {"params":[{"name":"func","desc":"Función a reintentar"},{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"base_delay","desc":"Delay inicial en segundos entre reintentos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"is_retryable","desc":"Predicado opcional para decidir si reintentar"}],"output":"Resultado de ejecutar la función si tiene éxito"}


id: retry_with_backoff_py_core
name: retry_with_backoff
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","decorator","sync","resilience","jitter"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["functools","time","random","asyncio","collections.abc","typing"]
test_file_path: python/functions/core/retry_with_backoff_test.py
file_path: python/functions/core/retry_with_backoff.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2ec2aa207e45f592ea1caa92f5c03164b63a405400d3a33256e42e65bec30bb
signature:
  def retry_with_backoff(max_retries: int = 3, initial_delay: float = 1.0, max_delay: float = 30.0, backoff_factor: float = 2.0, jitter: bool = True, exceptions: tuple[type[Exception], ...] = (Exception,), on_retry: Callable[[Exception, int], None] | None = None) -> Callable

description:
  Decorador que reintenta una funcion sincrona con exponential backoff cuando lanza excepciones del tipo especificado. Soporta jitter, callback on_retry y filtrado por tipo de excepcion.

example:
  ```python
  from retry_with_backoff import retry_with_backoff

  @retry_with_backoff(max_retries=3, exceptions=(ConnectionError, TimeoutError))
  def call_api():
      response = requests.get("https://api.example.com/data")
      response.raise_for_status()
      return response.json()

  # Con callback para logging
  def log_retry(exc: Exception, attempt: int) -> None:
      print(f"Attempt {attempt} failed: {exc}. Retrying...")

  @retry_with_backoff(max_retries=5, on_retry=log_retry)
  def fetch_resource():
      ...
  ```

tests:
  ["funcion que falla dos veces y luego exito","funcion que siempre falla agota retries y lanza","on_retry callback se llama","jitter produce delays variables","solo reintenta excepciones especificadas"]

notes:
  Impure: hace `time.sleep` entre reintentos. El delay entre intentos se calcula como `min(delay, max_delay)`, opcionalmente multiplicado por un factor aleatorio en `[0.5, 1.0]` si `jitter=True`. Tras cada intento fallido, `delay *= backoff_factor`. El parametro `max_retries` es el numero de reintentos despues del primer intento (total de llamadas = max_retries + 1). Solo reintenta excepciones que sean instancias de los tipos en `exceptions`. Implementado en el mismo archivo que `retry_with_backoff_async`.

code:
  """Retry decorators with exponential backoff for sync and async functions."""

  import asyncio
  import functools
  import random
  import time
  from collections.abc import Callable
  from typing import Any


  def retry_with_backoff(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries a synchronous function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      time.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator


  def retry_with_backoff_async(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries an async function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target async function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          async def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return await func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      await asyncio.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator

params_schema:
  {"params":[{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"initial_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"backoff_factor","desc":"Factor multiplicativo para el delay exponencial"},{"name":"jitter","desc":"Si True, añade variabilidad aleatoria al delay"},{"name":"exceptions","desc":"Tupla de tipos de excepción a reintentar"},{"name":"on_retry","desc":"Callback opcional llamado cuando se reintenta"}],"output":"Decorador que envuelve la función con lógica de retry"}


id: retry_with_backoff_async_py_core
name: retry_with_backoff_async
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","decorator","async","asyncio","resilience","jitter"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["functools","asyncio","random","collections.abc","typing"]
test_file_path: python/functions/core/retry_with_backoff_test.py
file_path: python/functions/core/retry_with_backoff.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 182bb56a243c9c38de789be672fbdf28de81fc6fd00533e7855bd5aaa18aa672
signature:
  def retry_with_backoff_async(max_retries: int = 3, initial_delay: float = 1.0, max_delay: float = 30.0, backoff_factor: float = 2.0, jitter: bool = True, exceptions: tuple[type[Exception], ...] = (Exception,), on_retry: Callable[[Exception, int], None] | None = None) -> Callable

description:
  Decorador que reintenta una funcion async con exponential backoff cuando lanza excepciones del tipo especificado. Usa asyncio.sleep en vez de time.sleep.

example:
  ```python
  from retry_with_backoff import retry_with_backoff_async

  @retry_with_backoff_async(max_retries=3, exceptions=(ConnectionError, TimeoutError))
  async def fetch_data():
      async with httpx.AsyncClient() as client:
          response = await client.get("https://api.example.com/data")
          response.raise_for_status()
          return response.json()

  # Con callback para logging
  def log_retry(exc: Exception, attempt: int) -> None:
      print(f"Async attempt {attempt} failed: {exc}. Retrying...")

  @retry_with_backoff_async(max_retries=5, on_retry=log_retry)
  async def call_service():
      ...
  ```

tests:
  ["async funcion que falla dos veces y luego exito","async funcion que siempre falla agota retries y lanza","async on_retry callback se llama","async jitter produce delays variables","async solo reintenta excepciones especificadas"]

notes:
  Impure: usa `asyncio.sleep` entre reintentos — no bloquea el event loop. Tiene la misma logica de backoff que `retry_with_backoff` pero para coroutines. El wrapper generado es una coroutine (`async def`), por lo que es compatible con `await` y `asyncio.gather`. Definido en el mismo archivo que `retry_with_backoff`.

code:
  """Retry decorators with exponential backoff for sync and async functions."""

  import asyncio
  import functools
  import random
  import time
  from collections.abc import Callable
  from typing import Any


  def retry_with_backoff(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries a synchronous function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      time.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator


  def retry_with_backoff_async(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries an async function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target async function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          async def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return await func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      await asyncio.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator

params_schema:
  {"params":[{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"initial_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"backoff_factor","desc":"Factor multiplicativo para exponential backoff"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"exceptions","desc":"Tupla de tipos de excepción a reintentar"},{"name":"on_retry","desc":"Callback opcional para logging de reintentos"}],"output":"Decorador que envuelve función async con lógica de retry"}


id: sanitize_for_path_py_core
name: sanitize_for_path
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def sanitize_for_path(text: str, max_length: int = 50) -> str
tags: ["path","sanitize","string","filesystem","slug"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","hashlib"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c7230e66c9e245e8c30e4be0c11c632924d635e44f28b7cf1e3b055750bfc5b1
description:
  Convierte texto a nombre seguro para uso en paths. Remueve caracteres especiales, reemplaza espacios con _, trunca con hash suffix si excede max_length.

example:
  ```python
  sanitize_for_path("Hello World!")       # "Hello_World"
  sanitize_for_path("中文标题")             # "中文标题"
  sanitize_for_path("a" * 100)           # "aaa...aaa_<sha256[:8]>" (max 50 chars)
  sanitize_for_path("")                  # "section"
  sanitize_for_path("price: $100 @2024") # "price__100_2024"
  ```

tests:
  ["texto normal produce slug con underscores","caracteres especiales son removidos","texto muy largo trunca con hash suffix","texto vacio retorna section","texto CJK es preservado"]

notes:
  Funcion pura. Caracteres permitidos: `\w` (letras, digitos, _), rangos CJK `[\u4e00-\u9fff\u3040-\u30ff\uac00-\ud7af]`, espacios (convertidos a `_`) y guiones. El sufijo hash es `_` + primeros 8 caracteres del sha256 del texto original, lo que hace la truncacion determinista y evita colisiones entre textos largos similares. Si el texto es todo caracteres especiales y queda vacio tras limpiar, retorna `"section"` como fallback seguro.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"text","desc":"Texto a convertir en nombre seguro para filesystem"},{"name":"max_length","desc":"Longitud máxima permitida del resultado"}],"output":"String sanitizado apto para usar en paths y nombres de archivos"}


id: smart_split_content_py_core
name: smart_split_content
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def smart_split_content(content: str, max_tokens: int = 1024, max_chars: int = 8000) -> list[str]
tags: ["chunking","splitting","tokens","text","markdown","llm"]
uses_functions: ["estimate_token_count_py_core"]
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e595ce1ddc0fbd0d2ccd1820f6ba05c174fbbb0d73a74706ccbed7da90a6c5ce
description:
  Divide contenido grande en partes respetando limites de tokens y caracteres. Divide por parrafos (doble newline). Si un parrafo individual excede el limite, lo corta por caracteres.

example:
  ```python
  # Short content — stays as one chunk
  parts = smart_split_content("Hello.\n\nWorld.", max_tokens=1024, max_chars=8000)
  # ["Hello.\n\nWorld."]

  # Long content — split at paragraph boundaries
  long_md = "\n\n".join(["paragraph text"] * 300)
  parts = smart_split_content(long_md, max_tokens=100, max_chars=8000)
  # [<chunk1>, <chunk2>, ...]
  ```

tests:
  ["contenido corto produce una sola parte","contenido largo divide en multiples partes","parrafo gigante que requiere forzar corte"]

notes:
  Funcion pura. Compone `estimate_token_count` para la estimacion de tokens. La estrategia de split es conservadora: acumula parrafos mientras quepan bajo AMBOS limites (tokens Y caracteres). Si un parrafo individual excede cualquier limite se fuerza corte byte a byte cada `max_chars` caracteres. Util para partir documentos grandes antes de enviarlos a LLMs con ventana de contexto limitada.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"Contenido grande a dividir"},{"name":"max_tokens","desc":"Límite máximo de tokens por chunk"},{"name":"max_chars","desc":"Límite máximo de caracteres por chunk"}],"output":"Lista de strings donde cada string es un chunk que respeta ambos límites"}


id: split_text_into_chunks_py_core
name: split_text_into_chunks
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def split_text_into_chunks(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]
tags: ["text","chunking","nlp","split","overlap","sentence","boundary","rag"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/split_text_into_chunks_test.py
file_path: python/functions/core/split_text_into_chunks.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2cdc661bae12ca7d31d66a992efbfedb9e8dd14fc7c986c1f73d5911619f7d5
description:
  Divide texto en chunks de tamaño fijo con overlap, intentando cortar en límites de oración para no romper frases a mitad. Soporta separadores CJK (。！？) y occidentales (. ! ?).

example:
  ```python
  text = "First sentence is here. Second sentence follows. Third ends it."
  chunks = split_text_into_chunks(text, chunk_size=40, overlap=10)
  # ['First sentence is here.', 'is here. Second sentence follows.', ...]

  # Texto vacío
  split_text_into_chunks("")  # []

  # Texto corto (cabe en un chunk)
  split_text_into_chunks("Hello world.", chunk_size=500)  # ['Hello world.']
  ```

tests:
  ["texto corto cabe en 1 chunk","texto largo con oraciones corta en punto","texto cjk con separador ideografico","texto sin separadores corta en chunk size exacto","overlap funciona","texto vacio retorna lista vacia"]

notes:
  Algoritmo de sliding window con detección de límites de oración. Los separadores se evalúan en orden de prioridad:
  `。`, `！`, `？`, `.\n`, `!\n`, `?\n`, `\n\n`, `. `, `! `, `? `

  Un separador solo se acepta si el corte resultante está después del 30% del chunk (evita chunks muy cortos). Si no se encuentra ningún separador válido, el texto se corta en `chunk_size` exacto.

  El siguiente chunk empieza en `end - overlap`, garantizando solapamiento entre chunks consecutivos. Esto es útil para RAG (Retrieval-Augmented Generation) donde el contexto de límite de chunk puede ser crítico.

  Diferencia con `page_list_to_groups_py_core`: ese agrupa páginas por tokens con overlap configurable entre grupos de páginas. Este divide texto plano por caracteres con detección de oraciones.

code:
  """Split text into overlapping chunks with sentence-boundary awareness."""


  def split_text_into_chunks(
      text: str, chunk_size: int = 500, overlap: int = 50
  ) -> list[str]:
      """Divide texto en chunks de tamaño fijo con overlap, cortando en límites de oración.

      Args:
          text: Texto a dividir.
          chunk_size: Tamaño máximo de cada chunk en caracteres.
          overlap: Número de caracteres de solapamiento entre chunks consecutivos.

      Returns:
          Lista de chunks. Vacía si el texto es vacío.
      """
      if not text:
          return []

      if len(text) <= chunk_size:
          stripped = text.strip()
          return [stripped] if stripped else []

      # Separadores en orden de prioridad (más específicos primero)
      separators = ["。", "！", "？", ".\n", "!\n", "?\n", "\n\n", ". ", "! ", "? "]

      chunks: list[str] = []
      start = 0
      text_len = len(text)

      while start < text_len:
          end = start + chunk_size

          if end < text_len:
              # Buscar el último separador de oración dentro de text[start:end]
              # Solo aceptar si está después del 30% del chunk
              min_pos = start + int(chunk_size * 0.30)
              best_end = None

              for sep in separators:
                  sep_len = len(sep)
                  # Buscar la última ocurrencia del separador en text[start:end]
                  search_region = text[start:end]
                  pos = search_region.rfind(sep)
                  if pos == -1:
                      continue
                  abs_pos = start + pos + sep_len
                  if abs_pos > min_pos:
                      # Usar este separador solo si produce un corte más tarde que el mínimo
                      # y más temprano que chunk_size (ya garantizado por rfind en [start:end])
                      if best_end is None or abs_pos > best_end:
                          best_end = abs_pos

              if best_end is not None:
                  end = best_end

          chunk = text[start:end].strip()
          if chunk:
              chunks.append(chunk)

          start = end - overlap
          # Protección contra bucle infinito si overlap >= chunk_size o end no avanza
          if start >= end:
              start = end

      return chunks

params_schema:
  {"params":[{"name":"text","desc":"Texto a dividir en chunks"},{"name":"chunk_size","desc":"Tamaño objetivo de cada chunk en caracteres"},{"name":"overlap","desc":"Número de caracteres a solapar entre chunks consecutivos"}],"output":"Lista de strings donde cada uno es un chunk con solapamiento"}


id: strip_markdown_codeblock_py_core
name: strip_markdown_codeblock
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def strip_markdown_codeblock(text: str) -> str
tags: ["llm","text","cleaning","markdown","codeblock","json"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/strip_markdown_codeblock_test.py
file_path: python/functions/core/strip_markdown_codeblock.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 36e0897ec48d8417649bbdb885191882e7952c1522b04c2d79847f804266c23d
description:
  Remueve wrapping de code blocks markdown que modelos a veces anaden al responder JSON. Ej: ```json\n{...}\n``` → {...}

example:
  ```python
  wrapped = '```json\n{"key": "value"}\n```'
  result = strip_markdown_codeblock(wrapped)
  # '{"key": "value"}'

  plain = '{"key": "value"}'
  result = strip_markdown_codeblock(plain)
  # '{"key": "value"}'
  ```

tests:
  ["JSON sin codeblock no cambia","JSON con wrapper ```json es desenvuelto","JSON con wrapper ``` solo es desenvuelto","texto que no es JSON pasa sin cambio"]

notes:
  Funcion pura. Aplica cuatro pasos: strip inicial, remover prefijo ` ``` ` o ` ```json ` (case-insensitive), remover sufijo ` ``` `, strip final. Maneja tanto ` ```json ` como ` ``` ` generico. El flag IGNORECASE permite ` ```JSON ` o ` ```Json `.

code:
  """Remove markdown code block wrapping from LLM responses."""

  import re


  def strip_markdown_codeblock(text: str) -> str:
      """Remove markdown code block wrapping from LLM responses.

      LLMs sometimes wrap JSON (or other content) in markdown code fences like:
          ```json
          {"key": "value"}
          ```
      This function unwraps the content so it can be parsed directly.

      Args:
          text: Text that may be wrapped in a markdown code block.

      Returns:
          Unwrapped text with leading/trailing whitespace stripped.
      """
      text = text.strip()
      text = re.sub(r'^```(?:json)?\s*\n?', '', text, flags=re.IGNORECASE)
      text = re.sub(r'\n?```\s*$', '', text)
      return text.strip()

params_schema:
  {"params":[{"name":"text","desc":"String posiblemente envuelto en markdown code block"}],"output":"String con los delimitadores ```json/``` removidos si estaban presentes"}


id: strip_think_tags_py_core
name: strip_think_tags
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def strip_think_tags(text: str) -> str
tags: ["llm","text","cleaning","think","chain-of-thought","deepseek","minimax"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/strip_think_tags_test.py
file_path: python/functions/core/strip_think_tags.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 345039965dc09cbe89872386bb99e4069c961b695a0cf236f1d2abf483a0423a
description:
  Remueve tags <think>...</think> que algunos modelos (MiniMax, DeepSeek) incluyen en sus respuestas como chain of thought interno.

example:
  ```python
  raw = "<think>Let me reason step by step...</think>La respuesta es 42."
  result = strip_think_tags(raw)
  # "La respuesta es 42."

  multiline = "<think>\nPrimero analizo...\nLuego concluyo...\n</think>Respuesta final."
  result = strip_think_tags(multiline)
  # "Respuesta final."
  ```

tests:
  ["texto sin tags no cambia","texto con think tags los remueve","tags multilinea son removidos","multiples tags son removidos"]

notes:
  Funcion pura. Usa `re.sub` con el flag `[\s\S]*?` (lazy) para manejar contenido multilinea sin capturar de mas cuando hay multiples tags. El `.strip()` final elimina whitespace residual alrededor del texto util.

code:
  """Remove <think>...</think> tags from LLM responses."""

  import re


  def strip_think_tags(text: str) -> str:
      """Remove <think>...</think> tags from LLM responses.

      Some models (MiniMax, DeepSeek) include chain-of-thought blocks wrapped in
      <think> tags as part of their output. This function strips those blocks so
      that downstream code only sees the final answer.

      Args:
          text: Raw LLM response that may contain <think>...</think> blocks.

      Returns:
          Text with all <think>...</think> blocks removed and outer whitespace
          stripped.
      """
      return re.sub(r'<think>[\s\S]*?</think>', '', text).strip()

params_schema:
  {"params":[{"name":"text","desc":"String posiblemente con tags \u003cthink\u003e...\u003c/think\u003e"}],"output":"String con los think tags removidos"}


id: t_py_core
name: t
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def t(key: str, locale: str | None = None, **kwargs) -> str
tags: ["i18n","translation","locale","dot-path","interpolation"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["threading"]
test_file_path: python/functions/core/t_test.py
file_path: python/functions/core/t.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 544444150baa4e28a266e79675567b1e4cd51b1efa4f1f9369689ca7051b6f06
description:
  Traduce una clave con dot-path notation al idioma activo. Prioridad: parametro locale > thread-local > default. Soporta interpolacion {variable}. Fallback al locale default si la clave no existe; si tampoco existe, retorna la key.

example:
  ```python
  from t import t, _set_translations

  translations = {
      "en": {
          "report": {
              "sectionStart": "Generating section: {title}",
              "done": "Done"
          }
      },
      "es": {
          "report": {
              "done": "Listo"
          }
      }
  }
  _set_translations(translations, default_locale="en")

  t("report.sectionStart", locale="en", title="Introduction")
  # → "Generating section: Introduction"

  t("report.done", locale="es")
  # → "Listo"

  t("report.sectionStart", locale="es", title="Intro")
  # → "Generating section: Intro"  (fallback a en)

  t("nonexistent.key", locale="en")
  # → "nonexistent.key"
  ```

tests:
  ["key existente retorna traduccion","key inexistente retorna la key","interpolacion de variables","dot-path profundo","fallback a locale default"]

notes:
  Lee estado global (modulo-level `_translations` y `_locale_local` thread-local), por eso es impura. Configurar con `_set_translations()` al inicio de la aplicacion y `_set_locale()` por thread/request. La interpolacion usa `str.format(**kwargs)` — si hay un placeholder faltante, retorna el string sin interpolar para no romper el flujo. Inspirada conceptualmente en el modulo `locale.py` de MiroFish (AGPL-3.0); reimplementada desde cero.

code:
  """Traduccion de claves con dot-path notation e interpolacion de variables."""

  import threading

  _locale_local = threading.local()
  _translations: dict = {}
  _default_locale: str = "en"


  def _set_translations(translations: dict, default_locale: str = "en") -> None:
      """Configura el diccionario global de traducciones y el locale default.

      Llamar desde load_translations o al inicio de la aplicacion.
      """
      global _translations, _default_locale
      _translations = translations
      _default_locale = default_locale


  def _set_locale(locale: str) -> None:
      """Establece el locale para el thread actual."""
      _locale_local.locale = locale


  def _get_locale() -> str:
      """Retorna el locale del thread actual, o el default si no esta configurado."""
      return getattr(_locale_local, "locale", _default_locale)


  def _resolve_key(translations: dict, locale: str, key: str) -> str | None:
      """Navega dot-path en el diccionario de traducciones para un locale.

      Returns:
          El string de traduccion si existe, None si no.
      """
      locale_dict = translations.get(locale)
      if not locale_dict:
          return None

      parts = key.split(".")
      node = locale_dict
      for part in parts:
          if not isinstance(node, dict):
              return None
          node = node.get(part)
          if node is None:
              return None

      return node if isinstance(node, str) else None


  def t(key: str, locale: str | None = None, **kwargs) -> str:
      """Traduce una clave con dot-path notation al idioma actual.

      Determina el locale en orden de prioridad: parametro > thread-local > default.
      Soporta interpolacion de variables con {nombre} en el valor traducido.
      Si la clave no existe en el locale solicitado, intenta el locale default.
      Si tampoco existe, retorna la clave tal cual.

      Args:
          key: Clave de traduccion en dot-path notation (ej: "report.taskStarted").
          locale: Locale a usar. Si es None usa el locale del thread actual.
          **kwargs: Variables para interpolar en el string traducido.

      Returns:
          String traducido con variables interpoladas, o la key si no se encontro.

      Example:
          >>> # translations = {"en": {"report": {"sectionStart": "Section: {title}"}}}
          >>> t("report.sectionStart", locale="en", title="Introduction")
          'Section: Introduction'
          >>> t("nonexistent.key", locale="en")
          'nonexistent.key'
      """
      resolved_locale = locale if locale is not None else _get_locale()

      value = _resolve_key(_translations, resolved_locale, key)

      if value is None and resolved_locale != _default_locale:
          value = _resolve_key(_translations, _default_locale, key)

      if value is None:
          return key

      if kwargs:
          try:
              value = value.format(**kwargs)
          except (KeyError, ValueError):
              pass

      return value

params_schema:
  {"params":[{"name":"key","desc":"Clave de traducción con dot-path notation (ej: 'report.done')"},{"name":"locale","desc":"Locale a usar (default: usa locale thread-local)"},{"name":"kwargs","desc":"Variables para interpolación en el template"}],"output":"String traducido e interpolado, o la key si no se encuentra traducción"}


id: take_py_core
name: take
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def take(xs: list, n: int) -> list
description: Toma los primeros n elementos de una lista.
tags: ["list","functional","take","slice","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = take([1, 2, 3, 4, 5], 3)
  # [1, 2, 3]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Si n > len(xs), retorna toda la lista. No muta la original.
content_hash: 069d9e069f12f7eba8edacdf5f5537834fe9981ed4cf5123ad38906a97c77bac
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista de la que tomar elementos"},{"name":"n","desc":"Número de elementos a tomar desde el inicio"}],"output":"Lista con los primeros n elementos"}


id: task_manager_py_core
name: task_manager
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["task","manager","singleton","thread-safe","background","async","polling","progress"]
uses_functions: []
uses_types: ["task_py_core","task_status_py_core"]
returns: []
error_type: error_go_core
imports: ["threading","uuid","datetime"]
tests: []
file_path: python/functions/core/task_manager.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a1a232a29013277dc2eec254bdb829488642ddd69c1506907c65d4d83eaf4ac8
params_schema: {"output":"Instancia singleton thread-safe para gestionar tareas de larga duración"}
signature:
  class TaskManager: create_task(task_type: str, metadata: dict | None = None) -> str; get_task(task_id: str) -> Task | None; update_task(task_id: str, ...) -> None; complete_task(task_id: str, result: dict) -> None; fail_task(task_id: str, error: str) -> None; list_tasks(task_type: str | None = None) -> list[dict]; cleanup_old_tasks(max_age_hours: int = 24) -> None

description:
  Gestor de tareas background en memoria, thread-safe, con patron singleton. Permite crear, actualizar, completar y limpiar tareas de larga duracion. Patron tipico: hilo background actualiza progreso via update_task(), frontend hace polling via get_task().

example:
  ```python
  manager = TaskManager()

  # Crear tarea
  task_id = manager.create_task("graph_build", metadata={"nodes": 500})

  # Hilo background actualiza progreso
  manager.update_task(task_id, status=TaskStatus.PROCESSING, progress=25, message="Cargando datos...")
  manager.update_task(task_id, progress=75, message="Construyendo grafo...")
  manager.complete_task(task_id, result={"graph_id": "abc-123"})

  # Frontend hace polling
  task = manager.get_task(task_id)
  print(task.to_dict())
  # {"task_id": "...", "status": "completed", "progress": 100, ...}

  # Listar todas las tareas de un tipo
  tasks = manager.list_tasks(task_type="graph_build")

  # Limpiar tareas antiguas (> 24h)
  manager.cleanup_old_tasks(max_age_hours=24)
  ```

notes:
  Singleton con double-checked locking: la primera instancia se crea una sola vez aunque multiples hilos llamen a `TaskManager()` simultaneamente.

  El lock `_tasks_lock` protege el diccionario interno `_tasks` en todas las operaciones de lectura y escritura.

  `update_task()` solo actualiza los campos que recibe como no-None, lo que permite actualizaciones parciales sin sobreescribir el estado existente.

  `cleanup_old_tasks()` solo elimina tareas en estado terminal (COMPLETED o FAILED). Las tareas PENDING o PROCESSING no se eliminan independientemente de su edad.

  El `error_type` referencia `error_go_core` como convencion del registry — los errores reales en Python se lanzan como excepciones, no se retornan.

code:
  """TaskManager — gestor de tareas background thread-safe con patron singleton."""

  import threading
  import uuid
  from datetime import datetime, timezone

  from task_status import TaskStatus
  from task import Task


  class TaskManager:
      """Gestor de tareas en memoria, thread-safe, con patron singleton.

      Uso tipico:
          manager = TaskManager()
          task_id = manager.create_task("graph_build", metadata={"nodes": 500})
          # en un hilo background:
          manager.update_task(task_id, progress=50, message="Procesando nodos...")
          manager.complete_task(task_id, result={"graph_id": "abc"})
          # desde el frontend (polling):
          task = manager.get_task(task_id)
      """

      _instance: "TaskManager | None" = None
      _lock: threading.Lock = threading.Lock()

      def __new__(cls) -> "TaskManager":
          if cls._instance is None:
              with cls._lock:
                  if cls._instance is None:
                      instance = super().__new__(cls)
                      instance._tasks: dict[str, Task] = {}
                      instance._tasks_lock = threading.Lock()
                      cls._instance = instance
          return cls._instance

      # ------------------------------------------------------------------
      # Operaciones publicas
      # ------------------------------------------------------------------

      def create_task(self, task_type: str, metadata: dict | None = None) -> str:
          """Crea una nueva tarea en estado PENDING y retorna su task_id.

          Args:
              task_type: Identificador libre del tipo de tarea ("graph_build", etc.).
              metadata:  Datos adicionales asociados a la tarea (parametros de entrada).

          Returns:
              task_id — UUID string de la tarea creada.
          """
          task_id = str(uuid.uuid4())
          now = datetime.now(timezone.utc)
          task = Task(
              task_id=task_id,
              task_type=task_type,
              status=TaskStatus.PENDING,
              created_at=now,
              updated_at=now,
              metadata=metadata or {},
          )
          with self._tasks_lock:
              self._tasks[task_id] = task
          return task_id

      def get_task(self, task_id: str) -> Task | None:
          """Retorna la tarea con el id dado, o None si no existe.

          Args:
              task_id: UUID de la tarea.

          Returns:
              Task o None.
          """
          with self._tasks_lock:
              return self._tasks.get(task_id)

      def update_task(
          self,
          task_id: str,
          status: TaskStatus | None = None,
          progress: int | None = None,
          message: str | None = None,
          result: dict | None = None,
          error: str | None = None,
          progress_detail: dict | None = None,
      ) -> None:
          """Actualiza campos de una tarea existente.

          Solo actualiza los campos que no sean None. Actualiza siempre updated_at.

          Args:
              task_id:         UUID de la tarea a actualizar.
              status:          Nuevo estado (TaskStatus), opcional.
              progress:        Nuevo porcentaje 0-100, opcional.
              message:         Nuevo mensaje de estado, opcional.
              result:          Resultado de la tarea, opcional.
              error:           Mensaje de error, opcional.
              progress_detail: Detalles granulares de progreso, opcional.
          """
          with self._tasks_lock:
              task = self._tasks.get(task_id)
              if task is None:
                  return
              if status is not None:
                  task.status = status
              if progress is not None:
                  task.progress = progress
              if message is not None:
                  task.message = message
              if result is not None:
                  task.result = result
              if error is not None:
                  task.error = error
              if progress_detail is not None:
                  task.progress_detail = progress_detail
              task.updated_at = datetime.now(timezone.utc)

      def complete_task(self, task_id: str, result: dict) -> None:
          """Marca la tarea como COMPLETED con el resultado dado.

          Args:
              task_id: UUID de la tarea.
              result:  Resultado serializable de la tarea completada.
          """
          self.update_task(
              task_id,
              status=TaskStatus.COMPLETED,
              progress=100,
              result=result,
          )

      def fail_task(self, task_id: str, error: str) -> None:
          """Marca la tarea como FAILED con el mensaje de error dado.

          Args:
              task_id: UUID de la tarea.
              error:   Descripcion del error.
          """
          self.update_task(
              task_id,
              status=TaskStatus.FAILED,
              error=error,
          )

      def list_tasks(self, task_type: str | None = None) -> list[dict]:
          """Lista todas las tareas, opcionalmente filtradas por tipo.

          Args:
              task_type: Si se especifica, solo retorna tareas de ese tipo.

          Returns:
              Lista de dicts serializados (via Task.to_dict()).
          """
          with self._tasks_lock:
              tasks = list(self._tasks.values())
          if task_type is not None:
              tasks = [t for t in tasks if t.task_type == task_type]
          return [t.to_dict() for t in tasks]

      def cleanup_old_tasks(self, max_age_hours: int = 24) -> None:
          """Elimina tareas terminadas (COMPLETED o FAILED) con mas de max_age_hours horas.

          Args:
              max_age_hours: Edad maxima en horas para tareas terminadas. Default: 24.
          """
          now = datetime.now(timezone.utc)
          terminal = {TaskStatus.COMPLETED, TaskStatus.FAILED}
          with self._tasks_lock:
              to_delete = [
                  tid
                  for tid, task in self._tasks.items()
                  if task.status in terminal
                  and (now - task.updated_at).total_seconds() > max_age_hours * 3600
              ]
              for tid in to_delete:
                  del self._tasks[tid]


id: to_csv_py_core
name: to_csv
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["csv","serialization","export","tabular","format"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/to_csv_test.py
file_path: python/functions/core/to_csv.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a47505629d2e1127723fc259adb900d528901626f23b34d746bae86a999e35a7
signature:
  to_csv(rows: list[dict], columns: list[str] | None = None, delimiter: str = ',', include_header: bool = True) -> str

description:
  Serializa datos tabulares a CSV. Si columns es None, usa las keys de la primera fila. Escapa campos con comillas, newlines y delimiters correctamente (RFC 4180). Sin dependencias externas.

example:
  ```python
  rows = [{"nombre": "Ana", "edad": 30}, {"nombre": "Bob", "edad": 25}]
  csv = to_csv(rows)
  # "nombre,edad\r\nAna,30\r\nBob,25"

  # Con columnas especificas
  csv = to_csv(rows, columns=["edad", "nombre"])
  # "edad,nombre\r\n30,Ana\r\n25,Bob"

  # Sin header
  csv = to_csv(rows, include_header=False)
  # "Ana,30\r\nBob,25"
  ```

tests:
  ["lista simple csv con header","campos con comas comillas newlines","columns explicitas reordena y filtra","include header false","lista vacia retorna string vacio","valores none campo vacio"]

notes:
  Implementacion manual sin el modulo csv de stdlib para control total del escaping.
  Cumple RFC 4180: escapa con comillas dobles cualquier campo que contenga el delimiter, comillas o saltos de linea. Los valores None se serializan como campo vacio.

code:
  """Serializa datos tabulares a formato CSV (RFC 4180)."""


  def _escape_field(value: str, delimiter: str) -> str:
      """Escapa un campo segun RFC 4180 si contiene delimiter, comillas o newlines."""
      if any(c in value for c in (delimiter, '"', "\n", "\r")):
          return '"' + value.replace('"', '""') + '"'
      return value


  def to_csv(
      rows: list[dict],
      columns: list[str] | None = None,
      delimiter: str = ",",
      include_header: bool = True,
  ) -> str:
      """Serializa datos tabulares a CSV.

      Implementacion manual sin csv module para control total del escaping.
      Cumple RFC 4180: escapa campos con comillas, newlines y delimiters.

      Args:
          rows: Lista de dicts con los datos. Todas las filas deben tener
                las mismas keys (o un subconjunto de columns).
          columns: Columnas a incluir y su orden. Si es None, usa las keys
                   de la primera fila en orden de insercion.
          delimiter: Separador de campos. Por defecto coma.
          include_header: Si True, incluye la fila de encabezados.

      Returns:
          String CSV completo. String vacio si rows es vacio.
      """
      if not rows:
          return ""

      cols = columns if columns is not None else list(rows[0].keys())

      output_lines: list[str] = []

      if include_header:
          output_lines.append(delimiter.join(_escape_field(c, delimiter) for c in cols))

      for row in rows:
          fields = []
          for col in cols:
              raw = row.get(col)
              if raw is None:
                  fields.append("")
              else:
                  fields.append(_escape_field(str(raw), delimiter))
          output_lines.append(delimiter.join(fields))

      return "\r\n".join(output_lines)

params_schema:
  {"params":[{"name":"rows","desc":"Lista de diccionarios a serializar como CSV"},{"name":"columns","desc":"Lista opcional de nombres de columnas (orden y filtrado)"},{"name":"delimiter","desc":"Carácter delimitador para campos"},{"name":"include_header","desc":"Si True, incluye fila de encabezados"}],"output":"String con datos en formato CSV válido (RFC 4180)"}


id: to_jsonl_py_core
name: to_jsonl
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: to_jsonl(rows: list[dict]) -> str
tags: ["jsonl","json","serialization","export","streaming","format"]
uses_functions: []
uses_types: []
returns: []
imports: ["json"]
test_file_path: python/functions/core/to_jsonl_test.py
file_path: python/functions/core/to_jsonl.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 5ae54c6aabf7efa66f91b9ccb4a9860d84eec8a41affee153dd08e7cdbc9f4fe
description:
  Serializa a JSON Lines (newline-delimited JSON). Cada dict se serializa como una linea JSON independiente. Util para streaming, logging estructurado y formatos de intercambio.

example:
  ```python
  rows = [{"id": 1, "name": "Ana"}, {"id": 2, "name": "Bob"}]
  jsonl = to_jsonl(rows)
  # '{"id": 1, "name": "Ana"}\n{"id": 2, "name": "Bob"}'

  # Unicode preservado
  rows = [{"emoji": "🔥", "val": None}]
  jsonl = to_jsonl(rows)
  # '{"emoji": "🔥", "val": null}'
  ```

tests:
  ["lista de dicts jsonl","valores unicode none nested","lista vacia retorna string vacio","cada linea es json parseable"]

notes:
  Usa json.dumps con ensure_ascii=False para preservar Unicode. No incluye newline al final.
  Cada linea es un JSON valido independiente, parseable con json.loads.

code:
  """Serializa lista de dicts a formato JSON Lines (JSONL)."""

  import json


  def to_jsonl(rows: list[dict]) -> str:
      """Serializa a JSON Lines (newline-delimited JSON).

      Cada dict se serializa como una linea JSON independiente separada por \\n.
      Util para streaming, logging estructurado y formatos de intercambio de datos.

      Args:
          rows: Lista de dicts a serializar. Cada dict puede contener valores
                anidados, Unicode, None, numeros, etc.

      Returns:
          String con un JSON por linea. String vacio si rows es vacio.
          No incluye newline al final.
      """
      if not rows:
          return ""

      return "\n".join(json.dumps(row, ensure_ascii=False) for row in rows)

params_schema:
  {"params":[{"name":"rows","desc":"Lista de diccionarios a serializar"}],"output":"String en formato JSON Lines (newline-delimited JSON)"}


id: to_pascal_case_py_core
name: to_pascal_case
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def to_pascal_case(name: str) -> str
tags: ["string","naming","pascal-case","case-conversion","formatting","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/to_pascal_case_test.py
file_path: python/functions/core/to_pascal_case.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 6375eadc015917983a4d59867adae7824e264b517cc3726251561224a33ba69b
description:
  Convierte cualquier formato de nombre (snake_case, camelCase, kebab-case, mixto) a PascalCase. Retorna "Unknown" para entrada vacia.

example:
  ```python
  to_pascal_case("works_for")        # "WorksFor"
  to_pascal_case("camelCaseExample") # "CamelCaseExample"
  to_pascal_case("SCREAMING_SNAKE")  # "ScreamingSnake"
  to_pascal_case("kebab-case")       # "KebabCase"
  to_pascal_case("")                 # "Unknown"
  ```

tests:
  ["snake_case","camelCase","UPPER_SNAKE","kebab-case","pascal_case_idempotente","string_vacio","con_numeros"]

notes:
  Funcion pura, sin dependencias externas (solo `re` de stdlib).

  Algoritmo en dos pasos:
  1. Split por caracteres no alfanumericos: `re.split(r'[^a-zA-Z0-9]+', name)`
  2. Split adicional por boundary camelCase en cada parte: `re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_')`

  Cada word resultante se capitaliza (primera letra upper, resto lower) y se concatenan. Si no hay words validas, retorna `"Unknown"`.

code:
  """Convert any name format to PascalCase."""

  import re


  def to_pascal_case(name: str) -> str:
      """Convert any name format to PascalCase.

      Handles snake_case, camelCase, kebab-case, SCREAMING_SNAKE, and mixed formats.
      Splits on non-alphanumeric characters and camelCase boundaries, then capitalizes
      each word and joins them.

      Args:
          name: Input name in any format.

      Returns:
          str: Name converted to PascalCase. Returns "Unknown" for empty input.
      """
      parts = re.split(r'[^a-zA-Z0-9]+', name)
      words = []
      for part in parts:
          if not part:
              continue
          sub_parts = re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_')
          for w in sub_parts:
              if w:
                  words.append(w[0].upper() + w[1:].lower() if len(w) > 1 else w.upper())
      return ''.join(words) if words else 'Unknown'

params_schema:
  {"params":[{"name":"name","desc":"String en cualquier formato de naming (snake_case, camelCase, kebab-case, etc.)"}],"output":"String convertido a PascalCase, o 'Unknown' si está vacío"}


id: tree_to_flat_list_py_core
name: tree_to_flat_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def tree_to_flat_list(structure: Any) -> list[dict]
description: Convierte arbol jerarquico a lista plana en orden DFS. Mantiene nodos internos con sus campos.
tags: ["tree","flatten","dfs","hierarchy"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 8320935b3492bca41e64acbb6de1dc68d9f5ea55058fd5186b577323cd197a18
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  tree = {"title": "Root", "nodes": [{"title": "Child", "nodes": []}]}
  tree_to_flat_list(tree)
  # [{"title": "Root", "nodes": [...]}, {"title": "Child", "nodes": []}]
  ```

notes:
  Funcion pura. A diferencia de flatten_tree, mantiene los nodos internos con su campo 'nodes' intacto (referencias, no copias).

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"structure","desc":"Árbol jerárquico con estructura anidada"}],"output":"Lista plana de nodos en orden DFS (depth-first search)"}


id: unique_py_core
name: unique
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def unique(xs: list) -> list
description: Elimina duplicados de una lista preservando el orden de aparicion.
tags: ["list","functional","unique","deduplicate","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = unique([1, 2, 2, 3, 1, 4])
  # [1, 2, 3, 4]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Preserva el orden de la primera aparicion. Requiere elementos hashables.
content_hash: 6ec7be71df965029b4a4916c98998085b760fbd8ece0ba5e92e8bd3e9d4c2d4e
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista potencialmente con elementos duplicados"}],"output":"Lista sin duplicados preservando orden de primera aparición"}


id: validate_git_ssh_uri_py_core
name: validate_git_ssh_uri
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def validate_git_ssh_uri(url: str) -> None
description: Valida el formato de una URI SSH de git (git@host:path). Lanza ValueError si la URI es invalida.
tags: ["git","ssh","validation","uri"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["URI valida","URI sin git@","URI sin colon","URI con path vacio"]
test_file_path: python/functions/core/parse_git_url_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f99b598639bcd61c5cba94d89b1ac3b8a4b8d8a0306afec89c6652cc88e09c00
example:
  ```python
  validate_git_ssh_uri("git@github.com:org/repo.git")
  # None (sin excepcion)

  validate_git_ssh_uri("https://github.com/org/repo")
  # ValueError: git SSH URI must start with 'git@', got: 'https://github.com/org/repo'

  validate_git_ssh_uri("git@github.com/org/repo")
  # ValueError: git SSH URI must contain ':'

  validate_git_ssh_uri("git@github.com:")
  # ValueError: git SSH URI must have a non-empty path after ':'
  ```

notes:
  Funcion pura. Valida solo la estructura sintatica del formato SSH de git: prefijo `git@`, separador `:`, y path no vacio. No verifica que el host sea alcanzable ni que el repo exista. Usar antes de `subprocess.run(["git", "clone", uri])` para dar mensajes de error claros al usuario.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"url","desc":"String con URI SSH de git a validar (formato git@host:path)"}],"output":"None si la URI es válida, lanza ValueError si es inválida"}


id: validate_json_schema_py_core
name: validate_json_schema
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def validate_json_schema(data: dict, schema: dict) -> tuple[bool, list[str]]
tags: ["validation","schema","json-schema","pure","core"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/validate_json_schema_test.py
file_path: python/functions/core/validate_json_schema.py
created_at: 2026-04-05T11:22:55Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e635587635f73b2cbd5f9699243f9531a591e80ee644a9e3e75ceda40657f6df
description:
  Valida un dict contra un subset practico de JSON Schema (draft 2020-12) sin dependencias externas. Soporta type, required, properties, items, minItems/maxItems, minimum/maximum, exclusiveMinimum/exclusiveMaximum, minLength/maxLength, pattern, enum. Acumula todos los errores con paths descriptivos.

example:
  ```python
  schema = {
      "type": "object",
      "required": ["name", "age"],
      "properties": {
          "name": {"type": "string"},
          "age": {"type": "integer", "minimum": 0},
          "email": {"type": "string", "pattern": r"^[^@]+@[^@]+$"},
      },
  }

  valid, errors = validate_json_schema({"name": "Alice", "age": 30}, schema)
  # valid = True, errors = []

  valid, errors = validate_json_schema({"name": 123, "age": -1}, schema)
  # valid = False
  # errors = [
  #   "$.name: expected string, got integer",
  #   "$.age: -1 < minimum 0",
  #   "$.email: required field missing or empty",  (si email en required)
  # ]
  ```

tests:
  ["schema simple con types y required","nested object validation","array con items schema","pattern validation en strings","numeric ranges","multiples errores acumulados con paths correctos","schema vacio acepta todo","data none contra required field"]

notes:
  Funcion pura. Solo usa `re` de la stdlib. No cubre el spec completo de JSON Schema — esta disenada para validacion practica de metadata de entities y resultados de queries. El path usa notacion `$.field[i]` para facilitar debugging. Schema vacio acepta cualquier valor (comportamiento estandar JSON Schema).

code:
  """Validacion de dicts contra un subset de JSON Schema (draft 2020-12) sin dependencias externas."""

  import re


  def validate_json_schema(data: dict, schema: dict) -> tuple[bool, list[str]]:
      """Valida un dict contra un schema JSON Schema (draft 2020-12).

      Soporta: type, required, properties, items, minItems/maxItems,
      minimum/maximum, exclusiveMinimum/exclusiveMaximum,
      minLength/maxLength, pattern, enum.

      Args:
          data: Datos a validar.
          schema: Schema JSON Schema (draft 2020-12) como dict.

      Returns:
          (valid, errors) — True y lista vacia si cumple, False y lista de
          errores descriptivos si no. Cada error incluye el path (ej: "$.address.zip").
      """
      errors: list[str] = []
      _validate(data, schema, "$", errors)
      return (len(errors) == 0, errors)


  _TYPE_CHECKS = {
      "string": lambda v: isinstance(v, str),
      "number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
      "integer": lambda v: isinstance(v, int) and not isinstance(v, bool),
      "boolean": lambda v: isinstance(v, bool),
      "array": lambda v: isinstance(v, list),
      "object": lambda v: isinstance(v, dict),
      "null": lambda v: v is None,
  }

  _TYPE_NAMES = {
      str: "string",
      int: "integer",
      float: "number",
      bool: "boolean",
      list: "array",
      dict: "object",
      type(None): "null",
  }


  def _type_name(v: object) -> str:
      return _TYPE_NAMES.get(type(v), type(v).__name__)


  def _validate(data: object, schema: dict, path: str, errors: list[str]) -> None:
      if not schema:
          return

      # --- type ---
      expected_type = schema.get("type")
      if expected_type is not None:
          checker = _TYPE_CHECKS.get(expected_type)
          if checker and not checker(data):
              errors.append(
                  f"{path}: expected {expected_type}, got {_type_name(data)}"
              )
              # no seguir validando si el tipo base ya fallo
              return

      # --- enum ---
      if "enum" in schema:
          if data not in schema["enum"]:
              errors.append(
                  f"{path}: value {data!r} not in enum {schema['enum']}"
              )

      # --- object ---
      if isinstance(data, dict):
          required = schema.get("required", [])
          for field in required:
              if field not in data or data[field] is None or data[field] == "":
                  errors.append(f"{path}.{field}: required field missing or empty")

          properties = schema.get("properties", {})
          for field, sub_schema in properties.items():
              if field in data:
                  _validate(data[field], sub_schema, f"{path}.{field}", errors)

      # --- array ---
      elif isinstance(data, list):
          min_items = schema.get("minItems")
          max_items = schema.get("maxItems")
          if min_items is not None and len(data) < min_items:
              errors.append(
                  f"{path}: array length {len(data)} < minItems {min_items}"
              )
          if max_items is not None and len(data) > max_items:
              errors.append(
                  f"{path}: array length {len(data)} > maxItems {max_items}"
              )

          items_schema = schema.get("items")
          if items_schema:
              for i, item in enumerate(data):
                  _validate(item, items_schema, f"{path}[{i}]", errors)

      # --- string ---
      elif isinstance(data, str):
          min_len = schema.get("minLength")
          max_len = schema.get("maxLength")
          if min_len is not None and len(data) < min_len:
              errors.append(
                  f"{path}: string length {len(data)} < minLength {min_len}"
              )
          if max_len is not None and len(data) > max_len:
              errors.append(
                  f"{path}: string length {len(data)} > maxLength {max_len}"
              )
          pattern = schema.get("pattern")
          if pattern is not None and not re.search(pattern, data):
              errors.append(
                  f"{path}: value {data!r} does not match pattern {pattern!r}"
              )

      # --- number / integer ---
      elif isinstance(data, (int, float)) and not isinstance(data, bool):
          minimum = schema.get("minimum")
          maximum = schema.get("maximum")
          ex_min = schema.get("exclusiveMinimum")
          ex_max = schema.get("exclusiveMaximum")
          if minimum is not None and data < minimum:
              errors.append(f"{path}: {data} < minimum {minimum}")
          if maximum is not None and data > maximum:
              errors.append(f"{path}: {data} > maximum {maximum}")
          if ex_min is not None and data <= ex_min:
              errors.append(f"{path}: {data} <= exclusiveMinimum {ex_min}")
          if ex_max is not None and data >= ex_max:
              errors.append(f"{path}: {data} >= exclusiveMaximum {ex_max}")

params_schema:
  {"params":[{"name":"data","desc":"Diccionario a validar contra el schema"},{"name":"schema","desc":"Esquema JSON Schema (subset practico)"}],"output":"Tupla (válido: bool, errores: list[str]) con lista de errores si hay"}


id: write_node_ids_py_core
name: write_node_ids
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def write_node_ids(data: Any, node_id: int = 0) -> int
tags: ["tree","id","hierarchy","mutation"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 388c388bd905a288d5e642563af3c2ac374e227043df2c9a4699888ce7cde9fc
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
description:
  Asigna IDs secuenciales zero-padded (0001, 0002...) a todos los nodos de un arbol. Retorna siguiente contador.

example:
  ```python
  tree = [{"title": "A", "nodes": [{"title": "B", "nodes": []}]}]
  next_id = write_node_ids(tree)
  # tree[0]["node_id"] == "0000", tree[0]["nodes"][0]["node_id"] == "0001"
  # next_id == 2
  ```

notes:
  Funcion pura en el sentido de no tener I/O, pero muta el arbol de entrada in-place. IDs son strings de 4 digitos.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"data","desc":"Estructura árbol a la que añadir IDs"},{"name":"node_id","desc":"Contador inicial para los IDs (default: 0)"}],"output":"Próximo contador a usar después de asignar IDs"}


id: zip_with_py_core
name: zip_with
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def zip_with(xs: list, ys: list, fn: callable) -> list
description: Combina dos listas elemento a elemento con una funcion. Se detiene en la mas corta.
tags: ["list","functional","zip","combine","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = zip_with([1, 2, 3], [10, 20, 30], lambda a, b: a + b)
  # [11, 22, 33]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Se detiene en la lista mas corta, como zip() de Python.
content_hash: de36b74fef535c15ad34e9feacd033853f630e46bcb33ec03ff38b9d384fce2b
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Primera lista a combinar"},{"name":"ys","desc":"Segunda lista a combinar"},{"name":"fn","desc":"Función que combina elementos: fn(x, y)"}],"output":"Lista de resultados combinados, longitud igual a la lista más corta"}


id: detect_sql_injection_py_cybersecurity
name: detect_sql_injection
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def detect_sql_injection(input_str: str) -> tuple
tags: ["sqli","injection","detection","security","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 559cca8d964dbd534a83887182b5042d3bee818815bfc6b7f139131de4177e49
description:
  Detecta patrones de SQL injection en un string. Retorna (is_threat, pattern) con el nombre del patron detectado.

example:
  ```python
  detect_sql_injection("' OR '1'='1")
  # (True, "string_tautology")

  detect_sql_injection("; DROP TABLE users")
  # (True, "stacked_query")

  detect_sql_injection("hello world")
  # (False, "")
  ```

notes:
  Detecta 10 patrones: sql_keyword, tautology, stacked_query, comment_injection, string_tautology, union_select, hex_literal, char_function, concat_function, time_based. No reemplaza un WAF pero es util para logging y alertas tempranas.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"input_str","desc":"string a analizar en busca de patrones de SQL injection"}],"output":"tupla (is_threat, pattern) indicando si se detectó amenaza y el nombre del patrón"}


id: entropy_shannon_py_cybersecurity
name: entropy_shannon
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def entropy_shannon(data: bytes) -> float
tags: ["entropy","shannon","analysis","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math","collections"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f74d9a99d61bd29572f136d680d8971d4e964779c382c7174899957fd1f381df
description:
  Calcula la entropia de Shannon de datos binarios (0-8 bits por byte). Util para detectar datos cifrados o comprimidos.

example:
  ```python
  # Datos aleatorios (alta entropia)
  entropy_shannon(bytes(range(256)))
  # ~8.0

  # Datos repetitivos (baja entropia)
  entropy_shannon(b"aaaaaaaaaa")
  # 0.0

  # Texto normal
  entropy_shannon(b"hello world")
  # ~2.84
  ```

notes:
  Entropia alta (>7.5) sugiere datos cifrados o comprimidos. Entropia baja (<3) sugiere datos estructurados o repetitivos. Retorna 0.0 para datos vacios.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"data","desc":"bytes cuya entropia de Shannon se desea calcular"}],"output":"valor float de entropia entre 0 y 8 bits por byte"}


id: envelope_decrypt_py_cybersecurity
name: envelope_decrypt
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: impure
signature: def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes
uses_functions: ["envelope_encrypt_py_cybersecurity"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["cryptography","struct"]
test_file_path: python/functions/cybersecurity/envelope_encrypt_test.py
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: d7dc98b639867b817cdbac801b6a3b3dd601652fcec298478c9552f5d64301df
description:
  Descifra datos cifrados con envelope_encrypt. Si los datos no comienzan con el magic b'OVE1', los retorna sin modificar (passthrough). Soporta archivos que pueden o no estar cifrados sin necesidad de chequeo previo.

tags:
  ["decryption","aes","gcm","envelope-encryption","dek","kek","cryptography","cybersecurity","passthrough"]

example:
  ```python
  import secrets
  from cybersecurity import envelope_encrypt, envelope_decrypt

  master_key = secrets.token_bytes(32)

  # Caso 1: descifrar datos cifrados
  ciphertext = envelope_encrypt(b"datos secretos", master_key)
  plaintext = envelope_decrypt(ciphertext, master_key)
  # plaintext == b"datos secretos"

  # Caso 2: passthrough — datos no cifrados
  raw = b"archivo en plano"
  result = envelope_decrypt(raw, master_key)
  # result == b"archivo en plano"  (sin modificar)

  # Caso 3: key incorrecta — lanza InvalidTag
  wrong_key = secrets.token_bytes(32)
  # envelope_decrypt(ciphertext, wrong_key)  → cryptography.exceptions.InvalidTag
  ```

tests:
  ["decrypt de datos cifrados","decrypt de datos no cifrados passthrough","key incorrecta","envelope truncado","magic invalido"]

notes:
  Implementacion original inspirada en OpenViking `openviking/crypto/encryptor.py` (AGPL-3.0). Reimplementada desde cero.

  - **Passthrough**: si `ciphertext` no empieza con `b"OVE1"`, se retorna sin modificar. Permite usar la funcion indistintamente en archivos cifrados y no cifrados.
  - **Autenticacion GCM**: si la master_key es incorrecta o los datos fueron manipulados, `cryptography.exceptions.InvalidTag` es lanzado por la capa GCM — nunca se retorna texto corrupto.
  - **ValueError**: lanzado si el envelope tiene magic correcto pero estructura invalida (truncado o version no soportada).
  - `master_key` debe ser de exactamente 32 bytes para AES-256.
  - Requiere `cryptography` instalado: `uv add cryptography`.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"ciphertext","desc":"datos cifrados con envelope_encrypt o datos sin cifrar (passthrough si no comienzan con magic OVE1)"},{"name":"master_key","desc":"clave maestra KEK de 32 bytes para descifrar la file key"}],"output":"bytes desencriptados, o bytes sin modificar si el magic no es OVE1"}


id: envelope_encrypt_py_cybersecurity
name: envelope_encrypt
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: impure
signature: def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes
tags: ["encryption","aes","gcm","envelope-encryption","dek","kek","cryptography","cybersecurity"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["cryptography","secrets","struct"]
test_file_path: python/functions/cybersecurity/envelope_encrypt_test.py
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 7e61accc062c67b606ea417b018a6c2d43a70e6c8c131d3504526469f60e84a3
description:
  Cifra datos usando patron Envelope Encryption con AES-256-GCM. Genera una file key aleatoria (DEK), cifra los datos con ella, luego cifra la file key con la master_key (KEK). Retorna un envelope binario con magic b'OVE1'.

example:
  ```python
  import secrets
  from cybersecurity import envelope_encrypt, envelope_decrypt

  master_key = secrets.token_bytes(32)  # 256-bit KEK
  plaintext = b"datos confidenciales"

  ciphertext = envelope_encrypt(plaintext, master_key)
  # ciphertext[:4] == b"OVE1"

  recovered = envelope_decrypt(ciphertext, master_key)
  # recovered == plaintext
  ```

tests:
  ["encrypt → decrypt roundtrip","datos vacios","datos grandes","ciphertext tiene magic correcto","ciphertext es distinto cada vez"]

notes:
  Implementacion original inspirada en OpenViking `openviking/crypto/encryptor.py` (AGPL-3.0). Reimplementada desde cero.

  - La file key (DEK) es de 32 bytes generados con `secrets.token_bytes` (CSPRNG).
  - Tanto el cifrado de datos como el de la file key usan AES-256-GCM con IVs de 12 bytes.
  - El GCM auth tag (16 bytes) garantiza autenticidad e integridad.
  - `master_key` debe ser de exactamente 32 bytes para AES-256.
  - Requiere `cryptography` instalado: `uv add cryptography`.

documentation:
  ## Formato del envelope

  ```
  Magic    (4B): b"OVE1"       identificador de formato
  Version  (1B): 0x01          version del protocolo
  Reserved (1B): 0x00          reservado para uso futuro
  EFK_len  (2B): big-endian    longitud de encrypted_file_key
  KIV_len  (2B): big-endian    longitud de key_iv
  DIV_len  (2B): big-endian    longitud de data_iv
  --- header: 12 bytes total ---
  Encrypted File Key (variable, incluye GCM auth tag de 16B)
  Key IV             (12B)
  Data IV            (12B)
  Encrypted Content  (variable, incluye GCM auth tag de 16B)
  ```

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"plaintext","desc":"datos a cifrar con AES-256-GCM"},{"name":"master_key","desc":"clave maestra KEK de 32 bytes para cifrar la file key"}],"output":"envelope binario con magic b'OVE1' conteniendo datos cifrados"}


id: extract_urls_py_cybersecurity
name: extract_urls
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def extract_urls(text: str) -> list
description: Extrae todas las URLs (http/https) de un texto. Util para analisis de IoCs y threat intelligence.
tags: ["url","extract","parsing","ioc","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f67c7bf3f5e420dfe573229d1f7071387209520ca94456c451c75651c7444939
example:
  ```python
  extract_urls("Visit https://example.com and http://test.org/path?q=1")
  # ["https://example.com", "http://test.org/path?q=1"]

  extract_urls("no urls here")
  # []
  ```

notes:
  Usa regex para extraer URLs con esquema http/https. No valida que las URLs sean alcanzables. Util para extraer indicadores de compromiso (IoCs) de logs, emails o reportes de threat intelligence.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"text","desc":"string de texto del cual extraer URLs HTTP/HTTPS"}],"output":"lista de strings con todas las URLs encontradas"}


id: hash_md5_py_cybersecurity
name: hash_md5
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def hash_md5(data: bytes) -> str
description: Calcula el hash MD5 de datos binarios. Retorna hex digest.
tags: ["hash","md5","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["hashlib"]
example:
  ```python
  h = hash_md5(b"hello")
  # "5d41402abc4b2a76b9719d911017c592"
  ```

tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 63462af1382995eef2107d0865249b487cda23e7d6c51de3e2c125d5c8927187
notes:
  Usa hashlib de stdlib. MD5 no es seguro para propositos criptograficos pero es util para checksums, fingerprinting de archivos e identificacion rapida de IoCs.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"data","desc":"bytes a ser hasheado"}],"output":"string hexadecimal del hash MD5 (32 caracteres)"}


id: hash_sha256_py_cybersecurity
name: hash_sha256
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def hash_sha256(data: bytes) -> str
description: Calcula el hash SHA-256 de datos binarios. Retorna hex digest.
tags: ["hash","sha256","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["hashlib"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ceeb9f43a5181fcd5d6e2f6d2ba095589625439bdb91f949638bbf29cb56009b
example:
  ```python
  h = hash_sha256(b"hello")
  # "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
  ```

notes:
  Usa hashlib de stdlib. Funcion pura sin side effects. SHA-256 produce un digest de 64 caracteres hexadecimales (256 bits).

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"data","desc":"bytes a ser hasheado"}],"output":"string hexadecimal del hash SHA-256 (64 caracteres)"}


id: is_base64_py_cybersecurity
name: is_base64
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def is_base64(s: str) -> bool
tags: ["base64","validation","encoding","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","base64"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ef44b434fe7e4394057ada22677d572da9ff67134609776c4eab600d1136ad61
description:
  Verifica si un string es base64 valido. Acepta base64 estandar y URL-safe. Requiere minimo 4 caracteres.

example:
  ```python
  is_base64("aGVsbG8=")
  # True

  is_base64("not!valid")
  # False

  is_base64("ab")
  # False (menos de 4 caracteres)
  ```

notes:
  Verifica tanto el formato (regex) como que el decode sea exitoso. Util para detectar datos codificados en payloads sospechosos, headers HTTP o parametros de URL.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"s","desc":"string a validar como base64"}],"output":"booleano indicando si el string es base64 valido (minimo 4 caracteres)"}


id: is_hex_py_cybersecurity
name: is_hex
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def is_hex(s: str) -> bool
tags: ["hex","validation","encoding","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 046c08bf7df38b84716681c2c5fe9c33591575c6923dc1e64e9df29b1350cd6a
description:
  Verifica si un string es hexadecimal valido. Acepta con o sin prefijo 0x. Requiere minimo 2 caracteres.

example:
  ```python
  is_hex("4a6f686e")
  # True

  is_hex("0x4a6f686e")
  # True

  is_hex("xyz")
  # False

  is_hex("a")
  # False (menos de 2 caracteres)
  ```

notes:
  Util para validar hashes, direcciones de memoria, shellcode y otros datos hexadecimales en contexto de seguridad.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"s","desc":"string a validar como hexadecimal"}],"output":"booleano indicando si el string es hexadecimal valido con o sin prefijo 0x (minimo 2 caracteres)"}


id: jaccard_similarity_py_cybersecurity
name: jaccard_similarity
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def jaccard_similarity(a: list, b: list) -> float
tags: ["jaccard","similarity","comparison","sets","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 3fbb86722a904baaea67e84a58e079b1974569b05c334ed67f872d34e38329c8
description:
  Calcula el coeficiente de similitud de Jaccard entre dos listas. J(A,B) = |A interseccion B| / |A union B|.

example:
  ```python
  jaccard_similarity(["a", "b", "c"], ["b", "c", "d"])
  # 0.5

  jaccard_similarity(["a", "b"], ["a", "b"])
  # 1.0

  jaccard_similarity([], [])
  # 0.0
  ```

notes:
  Convierte las listas a sets internamente. Retorna 0.0 si ambas listas son vacias. Util para comparar conjuntos de tokens, features de malware, IoCs compartidos entre muestras, o tags de vulnerabilidades.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"a","desc":"primera lista para comparacion"},{"name":"b","desc":"segunda lista para comparacion"}],"output":"coeficiente float de similitud de Jaccard entre 0.0 y 1.0"}


id: levenshtein_distance_py_cybersecurity
name: levenshtein_distance
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def levenshtein_distance(a: str, b: str) -> int
tags: ["levenshtein","distance","fuzzy","typosquatting","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 2e6ecac9eba12e49b1b70ffbd1f30d865f7119e1a5769764cf3b559881d73226
description:
  Calcula la distancia de Levenshtein (edit distance) entre dos strings. Util para deteccion de typosquatting en dominios.

example:
  ```python
  levenshtein_distance("google.com", "gooogle.com")
  # 1

  levenshtein_distance("paypal.com", "paypa1.com")
  # 1

  levenshtein_distance("abc", "abc")
  # 0
  ```

notes:
  Implementacion O(n*m) con optimizacion de espacio (dos filas). Sin dependencias externas. Util para detectar dominios de typosquatting comparando contra dominios legitimos conocidos.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"a","desc":"primer string para comparacion de distancia de edicion"},{"name":"b","desc":"segundo string para comparacion de distancia de edicion"}],"output":"entero con la distancia de Levenshtein (minimo numero de ediciones)"}


id: normalize_url_py_cybersecurity
name: normalize_url
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def normalize_url(raw_url: str) -> str
tags: ["url","normalize","ioc","dedup","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["urllib.parse"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 99a36ba484bcf6398fc8f6e800cf79d6944cac72f186181d8fb1acf55e990ece
description:
  Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros. Util para deduplicacion de IoCs.

example:
  ```python
  normalize_url("HTTPS://Example.COM/path?b=2&a=1#frag")
  # "https://example.com/path?a=1&b=2"

  normalize_url("http://test.org/path/")
  # "http://test.org/path"
  ```

notes:
  Operaciones de normalizacion: lowercase de scheme y host, eliminacion de trailing slash (excepto root), ordenamiento alfabetico de query parameters, eliminacion de fragmentos. Usa urllib.parse de stdlib.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"raw_url","desc":"URL en formato string a normalizar"}],"output":"URL normalizada con host en minusculas, fragmentos eliminados y parametros ordenados"}


id: aggregate_by_group_py_datascience
name: aggregate_by_group
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def aggregate_by_group(rows: list[dict], group_by: list[str], aggs: dict[str, str]) -> list[dict]
tags: ["datascience","tabular","groupby","aggregate","transform","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["collections"]
test_file_path: python/functions/datascience/aggregate_by_group_test.py
file_path: python/functions/datascience/aggregate_by_group.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ef3fece0affeb2b2c57afdb5771f4cb130e89da3f0306a1e2297ea0f94a35376
description:
  GROUP BY + agregaciones sobre datos tabulares. aggs es un dict de columna a funcion (sum, mean, count, min, max, first, last, collect). collect acumula valores en lista. None se ignora en agregaciones numericas.

example:
  ```python
  rows = [
      {"dept": "eng", "salary": 100},
      {"dept": "eng", "salary": 120},
      {"dept": "sales", "salary": 80},
  ]
  aggregate_by_group(rows, group_by=["dept"], aggs={"salary": "mean"})
  # [{"dept": "eng", "salary": 110.0}, {"dept": "sales", "salary": 80.0}]
  ```

tests:
  ["Group by una columna con sum","Group by multiples columnas","Agregacion mean count min max","collect acumula en lista","Grupo con una sola fila","Campo con None se ignora en agregaciones numericas"]

notes:
  Funcion pura sin dependencias externas (solo collections.defaultdict de stdlib).
  Preserva el orden de primera aparicion de cada grupo.
  La funcion 'collect' no filtra None — acumula todos los valores incluyendo None.

code:
  """GROUP BY + agregaciones sobre datos tabulares list[dict]."""

  from collections import defaultdict


  def aggregate_by_group(
      rows: list[dict],
      group_by: list[str],
      aggs: dict[str, str],
  ) -> list[dict]:
      """Agrupa filas por una o varias columnas y aplica agregaciones.

      Equivalente a SQL GROUP BY con funciones de agregacion.
      La funcion 'collect' acumula todos los valores en una lista.
      Los valores None se ignoran en agregaciones numericas (sum, mean, min, max).

      Args:
          rows: Lista de dicts con los datos.
          group_by: Lista de columnas por las que agrupar.
          aggs: Dict de {columna: funcion}. Funciones: sum, mean, count,
                min, max, first, last, collect.

      Returns:
          Lista de dicts con las columnas de group_by mas los campos agregados.
          El orden de las filas sigue el orden de primera aparicion del grupo.
      """
      # Mantener orden de grupos con lista de claves
      group_keys: list[tuple] = []
      seen_groups: set[tuple] = set()
      buckets: dict[tuple, dict[str, list]] = defaultdict(lambda: defaultdict(list))

      for row in rows:
          gk = tuple(row.get(col) for col in group_by)
          if gk not in seen_groups:
              seen_groups.add(gk)
              group_keys.append(gk)
          for col in aggs:
              val = row.get(col)
              buckets[gk][col].append(val)

      def _aggregate(vals: list, func: str):
          if func == "collect":
              return vals
          if func == "count":
              return len(vals)
          if func == "first":
              return vals[0] if vals else None
          if func == "last":
              return vals[-1] if vals else None
          # Para sum, mean, min, max: ignorar None
          numeric = [v for v in vals if v is not None]
          if not numeric:
              return None
          if func == "sum":
              return sum(numeric)
          if func == "mean":
              return sum(numeric) / len(numeric)
          if func == "min":
              return min(numeric)
          if func == "max":
              return max(numeric)
          raise ValueError(f"Funcion de agregacion no soportada: {func}")

      result = []
      for gk in group_keys:
          record: dict = dict(zip(group_by, gk))
          for col, func in aggs.items():
              record[col] = _aggregate(buckets[gk][col], func)
          result.append(record)

      return result

params_schema:
  {"params":[{"name":"rows","desc":"lista de dicts donde cada dict representa una fila (ej: [{'dept': 'eng', 'salary': 100}, ...])"},{"name":"group_by","desc":"lista de nombres de columnas para agrupar (ej: ['dept']). Pueden ser multiples columnas."},{"name":"aggs","desc":"dict de columna -\u003e funcion de agregacion (ej: {'salary': 'mean'}). Soporta: sum, mean, count, min, max, first, last, collect"}],"output":"lista de dicts donde cada dict es un grupo con sus resultados de agregacion"}


id: autocorrelation_py_datascience
name: autocorrelation
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def autocorrelation(data: list, lag: int) -> float
description: Calcula la autocorrelacion de una serie temporal para un lag dado.
tags: ["statistics","timeseries","correlation","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  autocorrelation([1, 2, 3, 4, 5, 4, 3, 2, 1], 1)
  # ~0.489
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Autocorrelacion normalizada por la varianza. Retorna 0.0 si lag es invalido o la varianza es cero.
content_hash: b124fbfed9d8738af39c456cd34ac2675763020628dea39499b56fee49a4d4e5
code:
  """Pure datascience utilities — statistics and numerical functions.

  Uses only math stdlib. No external dependencies.
  """

  import math


  def pearson(xs: list, ys: list) -> float:
      """Pearson correlation coefficient between two lists of floats."""
      n = len(xs)
      if n != len(ys) or n == 0:
          return 0.0
      mean_x = sum(xs) / n
      mean_y = sum(ys) / n
      num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
      den_x = math.sqrt(sum((x - mean_x) ** 2 for x in xs))
      den_y = math.sqrt(sum((y - mean_y) ** 2 for y in ys))
      if den_x == 0.0 or den_y == 0.0:
          return 0.0
      return num / (den_x * den_y)


  def standardize(data: list) -> list:
      """Z-score standardization (mean=0, std=1)."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [0.0] * n
      return [(x - mean) / std for x in data]


  def min_max_scale(data: list) -> list:
      """Scale values to [0, 1] range."""
      if not data:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          return [0.0] * len(data)
      return [(x - lo) / (hi - lo) for x in data]


  def clip(data: list, lo: float, hi: float) -> list:
      """Clip values to [lo, hi]."""
      return [max(lo, min(hi, x)) for x in data]


  def detect_outliers(data: list, threshold: float) -> list:
      """Returns list of bools, True where |z-score| > threshold."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [False] * n
      return [abs((x - mean) / std) > threshold for x in data]


  def impute(data: list) -> list:
      """Replace None/NaN with mean of non-null values."""
      valid = [x for x in data if x is not None and not (isinstance(x, float) and math.isnan(x))]
      if not valid:
          return [0.0] * len(data)
      mean = sum(valid) / len(valid)
      return [
          mean if (x is None or (isinstance(x, float) and math.isnan(x))) else x
          for x in data
      ]


  def histogram(data: list, buckets: int) -> list:
      """Returns list of counts per bucket."""
      if not data or buckets <= 0:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          counts = [0] * buckets
          counts[0] = len(data)
          return counts
      width = (hi - lo) / buckets
      counts = [0] * buckets
      for x in data:
          idx = int((x - lo) / width)
          if idx >= buckets:
              idx = buckets - 1
          counts[idx] += 1
      return counts


  def rolling_window(xs: list, size: int) -> list:
      """Returns list of sublists (sliding windows of given size)."""
      if size <= 0 or size > len(xs):
          return []
      return [xs[i : i + size] for i in range(len(xs) - size + 1)]


  def autocorrelation(data: list, lag: int) -> float:
      """Autocorrelation at given lag."""
      n = len(data)
      if lag < 0 or lag >= n or n == 0:
          return 0.0
      mean = sum(data) / n
      var = sum((x - mean) ** 2 for x in data) / n
      if var == 0.0:
          return 0.0
      cov = sum((data[i] - mean) * (data[i + lag] - mean) for i in range(n - lag)) / n
      return cov / var


  def linspace(start: float, stop: float, num: int) -> list:
      """Generate evenly spaced values from start to stop (inclusive)."""
      if num <= 0:
          return []
      if num == 1:
          return [start]
      step = (stop - start) / (num - 1)
      return [start + i * step for i in range(num)]


  def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict:
      """Estima parámetros de un proceso Hawkes desde autocorrelación de arrivals.

      Ajusta exponencial a*exp(-b*lag) sobre la ACF.
      Retorna dict con alpha, beta, branching_ratio, acf.
      """
      import numpy as np
      from scipy.optimize import curve_fit

      arr = np.array(arrivals, dtype=float)
      mean_a = np.mean(arr)
      var_a = np.var(arr)
      if var_a == 0:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': [1.0]}

      acf = [1.0] + [
          float(np.mean((arr[lag:] - mean_a) * (arr[:-lag] - mean_a)) / var_a)
          for lag in range(1, max_lag)
      ]

      lags = np.arange(1, max_lag)
      acf_vals = np.array(acf[1:])

      if acf_vals[0] <= 0.01:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': acf}

      exp_decay = lambda x, a, b: a * np.exp(-b * x)
      try:
          popt, _ = curve_fit(exp_decay, lags, acf_vals, p0=[0.5, 0.5], maxfev=5000)
          alpha_est, beta_est = abs(popt[0]), abs(popt[1])
      except RuntimeError:
          alpha_est, beta_est = 0.0, 1.0

      branching = alpha_est / beta_est if beta_est > 0 else 0.0
      return {
          'alpha': round(alpha_est, 4),
          'beta': round(beta_est, 4),
          'branching_ratio': round(branching, 4),
          'acf': acf,
      }


  def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict:
      """Estima el exponente alpha de una distribución Pareto via MLE.

      α = n / Σ ln(xi / x_min) donde x_min es el percentil indicado.
      Alpha bajo = cola más pesada = más valores extremos.
      """
      import numpy as np

      arr = np.array([v for v in values if v > 0], dtype=float)
      if len(arr) < 10:
          return {'alpha': 0.0, 'x_min': 0.0, 'n_tail': 0}

      x_min = float(np.percentile(arr, x_min_percentile))
      tail = arr[arr >= x_min]

      if len(tail) < 2 or x_min <= 0:
          return {'alpha': 0.0, 'x_min': x_min, 'n_tail': len(tail)}

      alpha = float(len(tail) / np.sum(np.log(tail / x_min)))

      return {
          'alpha': round(alpha, 4),
          'x_min': round(x_min, 6),
          'n_tail': len(tail),
      }

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos de una serie temporal (ej: precios diarios, cantidades de eventos)"},{"name":"lag","desc":"numero de periodos para desplazar (ej: 1 para autocorrelacion con el valor anterior). Debe ser positivo."}],"output":"coeficiente de autocorrelacion normalizado en rango [-1, 1]. 1.0=correlacion perfecta, 0.0=sin correlacion"}


id: build_entity_schema_prompt_py_datascience
name: build_entity_schema_prompt
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def build_entity_schema_prompt(entity_presets: list[dict]) -> str
tags: ["prompt","llm","entity","schema","osint","graph","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["lista con varios presets","lista vacia retorna string vacio","preset sin metadata_fields"]
test_file_path: python/functions/datascience/build_entity_schema_prompt_test.py
file_path: python/functions/datascience/build_entity_schema_prompt.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 7b90e3fdfb41131299a13e16ffb96aff05cb84fa0ae3d3cd76b9f32d67e4b943
description:
  Genera la seccion del system prompt que describe los entity types disponibles para extraccion. Formatea los presets del registry en texto legible para el LLM.

example:
  ```python
  from build_entity_schema_prompt import build_entity_schema_prompt

  presets = [
      {
          "type_ref": "osint_person_go_cybersecurity",
          "label": "Person",
          "metadata_fields": ["full_name", "alias", "nationality", "dob", "risk_score"],
      },
      {
          "type_ref": "osint_organization_go_cybersecurity",
          "label": "Organization",
          "metadata_fields": ["legal_name", "country", "sector", "founded", "risk_score"],
      },
  ]

  prompt = build_entity_schema_prompt(presets)
  # Entity types available for extraction:
  #
  # 1. Person (type_ref: osint_person_go_cybersecurity)
  #    Attributes: full_name, alias, nationality, dob, risk_score
  #
  # 2. Organization (type_ref: osint_organization_go_cybersecurity)
  #    Attributes: legal_name, country, sector, founded, risk_score
  ```

notes:
  Funcion pura. No requiere dependencias externas.

  El formato de salida es deliberadamente sencillo para maximizar la comprension por el LLM: numero de orden, label humano, type_ref del registry y lista de atributos en una sola linea.

  Si un preset no tiene `metadata_fields` (o tiene lista vacia), se omite la linea de atributos.

  Pensada para componer con `build_relation_schema_prompt` al construir el system prompt completo de extraccion de grafos OSINT.

code:
  """Genera la seccion del system prompt que describe los entity types disponibles para extraccion."""


  def build_entity_schema_prompt(entity_presets: list[dict]) -> str:
      """Genera texto legible para el LLM describiendo los entity types disponibles.

      Formatea los presets del registry en una seccion del system prompt que indica
      al LLM que tipos de entidades puede extraer y que atributos tiene cada uno.

      Args:
          entity_presets: Lista de presets con campos 'label', 'type_ref' y
                          opcionalmente 'metadata_fields'. Ejemplo:
                          [{"type_ref": "osint_person_go_cybersecurity",
                            "label": "Person",
                            "metadata_fields": ["full_name", "alias"]}]

      Returns:
          String formateado con la seccion del prompt. Retorna string vacio si
          la lista de presets esta vacia.
      """
      if not entity_presets:
          return ""

      lines = ["Entity types available for extraction:", ""]

      for i, preset in enumerate(entity_presets, start=1):
          label = preset.get("label", "Unknown")
          type_ref = preset.get("type_ref", "")
          metadata_fields = preset.get("metadata_fields", [])

          lines.append(f"{i}. {label} (type_ref: {type_ref})")

          if metadata_fields:
              attrs = ", ".join(metadata_fields)
              lines.append(f"   Attributes: {attrs}")

          lines.append("")

      # Remove trailing blank line
      if lines and lines[-1] == "":
          lines.pop()

      return "\n".join(lines)

params_schema:
  {"params":[{"name":"entity_presets","desc":"lista de dicts describiendo tipos de entidad (ej: [{'type_ref': 'osint_person_go_cybersecurity', 'label': 'Person', 'metadata_fields': ['name', 'alias']}])"}],"output":"string con seccion del system prompt describiendo los entity types en formato legible para LLM"}


id: build_relation_schema_prompt_py_datascience
name: build_relation_schema_prompt
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def build_relation_schema_prompt(relation_types: list[str]) -> str
tags: ["prompt","llm","relation","schema","osint","graph","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["lista con varios tipos","lista vacia retorna string vacio","un solo tipo"]
test_file_path: python/functions/datascience/build_relation_schema_prompt_test.py
file_path: python/functions/datascience/build_relation_schema_prompt.py
created_at: 2026-04-05T11:23:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 925a17dd2d241cf132753d0cc8e84278049fbc836e796d8bcb32b6fe829505fa
description:
  Genera la seccion del system prompt con los tipos de relacion permitidos para extraccion. Formatea la lista de tipos en texto legible para el LLM.

example:
  ```python
  from build_relation_schema_prompt import build_relation_schema_prompt

  types = ["funds", "employs", "communicates_with", "owns"]
  prompt = build_relation_schema_prompt(types)
  # Allowed relation types:
  # funds, employs, communicates_with, owns
  ```

notes:
  Funcion pura. No requiere dependencias externas.

  La salida es una sola linea con todos los tipos separados por coma, precedida por el encabezado. El formato es minimal para no consumir tokens innecesarios del contexto del LLM.

  Pensada para componer con `build_entity_schema_prompt` al construir el system prompt completo de extraccion de grafos OSINT.

code:
  """Genera la seccion del system prompt con los tipos de relacion permitidos."""


  def build_relation_schema_prompt(relation_types: list[str]) -> str:
      """Genera texto legible para el LLM describiendo los tipos de relacion permitidos.

      Formatea la lista de tipos de relacion en una seccion del system prompt que
      indica al LLM que relaciones puede extraer entre entidades.

      Args:
          relation_types: Lista de strings con los tipos de relacion permitidos.
                          Ejemplo: ["funds", "employs", "communicates_with"]

      Returns:
          String formateado con la seccion del prompt. Retorna string vacio si
          la lista esta vacia.
      """
      if not relation_types:
          return ""

      joined = ", ".join(relation_types)
      return f"Allowed relation types:\n{joined}"

params_schema:
  {"params":[{"name":"relation_types","desc":"lista de nombres de tipos de relacion permitidos (ej: ['funds', 'employs', 'owns', 'communicates_with'])"}],"output":"string con una linea describiendo los tipos de relacion permitidos en formato legible para LLM"}


id: clip_py_datascience
name: clip
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def clip(data: list, lo: float, hi: float) -> list
description: Recorta los valores de la lista al rango [lo, hi].
tags: ["clipping","bounds","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  clip([1, 5, 10, -3], 0, 7)
  # [1, 5, 7, 0]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura sin dependencias. Equivalente a numpy.clip pero sin numpy.
content_hash: 395e574ff4c402047c4a1dec4d0a82c5a6ee5f9f1629cb5f08b6ac691e8f5ffe
code:
  """Pure datascience utilities — statistics and numerical functions.

  Uses only math stdlib. No external dependencies.
  """

  import math


  def pearson(xs: list, ys: list) -> float:
      """Pearson correlation coefficient between two lists of floats."""
      n = len(xs)
      if n != len(ys) or n == 0:
          return 0.0
      mean_x = sum(xs) / n
      mean_y = sum(ys) / n
      num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
      den_x = math.sqrt(sum((x - mean_x) ** 2 for x in xs))
      den_y = math.sqrt(sum((y - mean_y) ** 2 for y in ys))
      if den_x == 0.0 or den_y == 0.0:
          return 0.0
      return num / (den_x * den_y)


  def standardize(data: list) -> list:
      """Z-score standardization (mean=0, std=1)."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [0.0] * n
      return [(x - mean) / std for x in data]


  def min_max_scale(data: list) -> list:
      """Scale values to [0, 1] range."""
      if not data:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          return [0.0] * len(data)
      return [(x - lo) / (hi - lo) for x in data]


  def clip(data: list, lo: float, hi: float) -> list:
      """Clip values to [lo, hi]."""
      return [max(lo, min(hi, x)) for x in data]


  def detect_outliers(data: list, threshold: float) -> list:
      """Returns list of bools, True where |z-score| > threshold."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [False] * n
      return [abs((x - mean) / std) > threshold for x in data]


  def impute(data: list) -> list:
      """Replace None/NaN with mean of non-null values."""
      valid = [x for x in data if x is not None and not (isinstance(x, float) and math.isnan(x))]
      if not valid:
          return [0.0] * len(data)
      mean = sum(valid) / len(valid)
      return [
          mean if (x is None or (isinstance(x, float) and math.isnan(x))) else x
          for x in data
      ]


  def histogram(data: list, buckets: int) -> list:
      """Returns list of counts per bucket."""
      if not data or buckets <= 0:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          counts = [0] * buckets
          counts[0] = len(data)
          return counts
      width = (hi - lo) / buckets
      counts = [0] * buckets
      for x in data:
          idx = int((x - lo) / width)
          if idx >= buckets:
              idx = buckets - 1
          counts[idx] += 1
      return counts


  def rolling_window(xs: list, size: int) -> list:
      """Returns list of sublists (sliding windows of given size)."""
      if size <= 0 or size > len(xs):
          return []
      return [xs[i : i + size] for i in range(len(xs) - size + 1)]


  def autocorrelation(data: list, lag: int) -> float:
      """Autocorrelation at given lag."""
      n = len(data)
      if lag < 0 or lag >= n or n == 0:
          return 0.0
      mean = sum(data) / n
      var = sum((x - mean) ** 2 for x in data) / n
      if var == 0.0:
          return 0.0
      cov = sum((data[i] - mean) * (data[i + lag] - mean) for i in range(n - lag)) / n
      return cov / var


  def linspace(start: float, stop: float, num: int) -> list:
      """Generate evenly spaced values from start to stop (inclusive)."""
      if num <= 0:
          return []
      if num == 1:
          return [start]
      step = (stop - start) / (num - 1)
      return [start + i * step for i in range(num)]


  def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict:
      """Estima parámetros de un proceso Hawkes desde autocorrelación de arrivals.

      Ajusta exponencial a*exp(-b*lag) sobre la ACF.
      Retorna dict con alpha, beta, branching_ratio, acf.
      """
      import numpy as np
      from scipy.optimize import curve_fit

      arr = np.array(arrivals, dtype=float)
      mean_a = np.mean(arr)
      var_a = np.var(arr)
      if var_a == 0:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': [1.0]}

      acf = [1.0] + [
          float(np.mean((arr[lag:] - mean_a) * (arr[:-lag] - mean_a)) / var_a)
          for lag in range(1, max_lag)
      ]

      lags = np.arange(1, max_lag)
      acf_vals = np.array(acf[1:])

      if acf_vals[0] <= 0.01:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': acf}

      exp_decay = lambda x, a, b: a * np.exp(-b * x)
      try:
          popt, _ = curve_fit(exp_decay, lags, acf_vals, p0=[0.5, 0.5], maxfev=5000)
          alpha_est, beta_est = abs(popt[0]), abs(popt[1])
      except RuntimeError:
          alpha_est, beta_est = 0.0, 1.0

      branching = alpha_est / beta_est if beta_est > 0 else 0.0
      return {
          'alpha': round(alpha_est, 4),
          'beta': round(beta_est, 4),
          'branching_ratio': round(branching, 4),
          'acf': acf,
      }


  def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict:
      """Estima el exponente alpha de una distribución Pareto via MLE.

      α = n / Σ ln(xi / x_min) donde x_min es el percentil indicado.
      Alpha bajo = cola más pesada = más valores extremos.
      """
      import numpy as np

      arr = np.array([v for v in values if v > 0], dtype=float)
      if len(arr) < 10:
          return {'alpha': 0.0, 'x_min': 0.0, 'n_tail': 0}

      x_min = float(np.percentile(arr, x_min_percentile))
      tail = arr[arr >= x_min]

      if len(tail) < 2 or x_min <= 0:
          return {'alpha': 0.0, 'x_min': x_min, 'n_tail': len(tail)}

      alpha = float(len(tail) / np.sum(np.log(tail / x_min)))

      return {
          'alpha': round(alpha, 4),
          'x_min': round(x_min, 6),
          'n_tail': len(tail),
      }

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos a recortar"},{"name":"lo","desc":"limite inferior (minimo permitido). Valores menores seran reemplazados por lo."},{"name":"hi","desc":"limite superior (maximo permitido). Valores mayores seran reemplazados por hi."}],"output":"lista de valores con todos los elementos recortados al rango [lo, hi]"}


id: deduplicate_entities_py_datascience
name: deduplicate_entities
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
uses_functions: ["normalize_entity_name_py_core","merge_entity_attributes_py_core"]
uses_types: ["entity_candidate_py_datascience","deduplication_result_py_datascience"]
returns: ["deduplication_result_py_datascience"]
imports: ["uuid"]
test_file_path: python/functions/datascience/deduplicate_entities_test.py
file_path: python/functions/datascience/deduplicate_entities.py
created_at: 2026-04-05T11:24:52Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: dbbaff336f28c177533e8da1fe7ddc34331e6524303840a4e1fa5de8653b0119
signature:
  def deduplicate_entities(candidates: list[EntityCandidate], name_threshold: float = 0.85, same_type_only: bool = True) -> DeduplicationResult

description:
  Agrupa entidades candidatas que refieren a la misma entidad real usando fuzzy matching de nombres (Levenshtein + Jaccard) y Union-Find para clusters transitivos. Retorna entidades mergeadas con mapas de resolucion de IDs y log de merges.

tags:
  ["deduplication","entity","fuzzy","levenshtein","jaccard","union-find","knowledge-graph","nlp","fuzzygraph","datascience"]

example:
  ```python
  from python.types.datascience.entity_candidate import EntityCandidate
  from python.functions.datascience.deduplicate_entities import deduplicate_entities

  candidates = [
      EntityCandidate(name="John Smith", type_ref="person", confidence=0.9),
      EntityCandidate(name="Smith, John", type_ref="person", confidence=0.85),
      EntityCandidate(name="Google", type_ref="organization", confidence=0.95),
      EntityCandidate(name="Google LLC", type_ref="organization", confidence=0.88),
  ]

  result = deduplicate_entities(candidates, name_threshold=0.85, same_type_only=True)
  # result.total_before = 4
  # result.total_after = 2
  # result.merge_log = [
  #   {"canonical": "John Smith", "merged": ["Smith, John"], "score": 0.91, "reason": "fuzzy_name"},
  #   {"canonical": "Google", "merged": ["Google LLC"], "score": 0.89, "reason": "fuzzy_name"},
  # ]
  ```

tests:
  ["John Smith y Smith, John se mergean","Google y Google LLC se mergean","192.168.1.1 y 192.168.1.1 se mergean por matching exacto","John Smith (person) y John Smith (organization) NO se mergean","Clusters transitivos: A~B, B~C -\u003e {A, B, C} en un solo cluster","Entidades sin duplicados pasan sin modificacion","Confidence toma el max del cluster; atributos se fusionan","Lista vacia retorna resultado vacio","name_to_id contiene todos los nombres originales del cluster"]

notes:
  Funcion pura. Implementa Levenshtein y Jaccard internamente para evitar dependencias externas a este modulo. Las funciones del registry `levenshtein_distance_py_cybersecurity` y `jaccard_similarity_py_cybersecurity` son equivalentes pero requieren imports adicionales — la implementacion inline mantiene la funcion sin dependencias de stdlib.

  El `name_to_id` del resultado es el mapa de resolucion principal para la fase de deduplicacion de relaciones: permite resolver cualquier variante de nombre de una entidad a su ID canonico.

documentation:
  ## Algoritmo

  1. **Normalizar nombres** usando `normalize_entity_name()` sobre cada candidato segun su `type_ref`
  2. **Comparacion pairwise** dentro del mismo tipo (si `same_type_only=True`):
     - Para tipos tecnicos (ip, email, domain, crypto_wallet, phone): matching exacto normalizado
     - Para el resto: `score = max(levenshtein_sim, jaccard_sim)` + bonus por contencion (+0.3) y acronimos (+0.3)
  3. **Union-Find** para clusters transitivos: si A~B y B~C, entonces {A, B, C} forman un cluster
  4. **Merge por cluster:**
     - Nombre canonico: candidato con mayor `confidence`
     - Atributos: `merge_entity_attributes()` sobre todos los candidatos del cluster
     - Confidence: `max` del cluster
     - Source chunks: union de todos los candidatos
     - `merged_from`: union de todos los nombres originales

  ## Heuristicas de similitud de nombres

  | Heuristica | Efecto |
  |---|---|
  | Levenshtein | `1 - (edit_distance / max_len)` |
  | Jaccard sobre tokens | `\|A ∩ B\| / \|A ∪ B\|` |
  | Score base | `max(lev_sim, jaccard_sim)` |
  | Contencion (a in b o b in a) | `+0.3` hasta max 1.0 |
  | Acronimo ("FBI" ~ "Federal Bureau of Investigation") | `+0.3` hasta max 1.0 |
  | Tipos exactos (ip/email/domain) | solo matching exacto, ignora umbral |

  ## Complejidad

  - Pairwise: O(N^2) — aceptable para <1000 entidades (tipico por documento)
  - Union-Find con path compression: O(α(N)) amortizado por operacion
  - Para escalar a >1000: pre-filtrar por primera letra o n-gram index antes de comparar

code:
  """Deduplica entidades candidatas usando fuzzy matching de nombres."""

  from __future__ import annotations

  import sys
  import os
  import uuid

  sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", ".."))

  from python.types.datascience.entity_candidate import EntityCandidate
  from python.types.datascience.deduplication_result import DeduplicationResult
  from python.functions.core.normalize_entity_name import normalize_entity_name
  from python.functions.core.merge_entity_attributes import merge_entity_attributes


  # ── Similitud helpers ──────────────────────────────────────────────────────────

  def _levenshtein(a: str, b: str) -> int:
      """Distancia de edicion Levenshtein entre dos strings."""
      if a == b:
          return 0
      if not a:
          return len(b)
      if not b:
          return len(a)
      prev = list(range(len(b) + 1))
      for i, ca in enumerate(a, 1):
          curr = [i]
          for j, cb in enumerate(b, 1):
              cost = 0 if ca == cb else 1
              curr.append(min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + cost))
          prev = curr
      return prev[-1]


  def _jaccard(tokens_a: list[str], tokens_b: list[str]) -> float:
      """Similitud de Jaccard entre dos conjuntos de tokens."""
      set_a = set(tokens_a)
      set_b = set(tokens_b)
      if not set_a and not set_b:
          return 1.0
      inter = len(set_a & set_b)
      union = len(set_a | set_b)
      return inter / union if union else 0.0


  def _name_similarity(a: str, b: str) -> float:
      """Score de similitud entre dos nombres normalizados.

      Combina similitud de Levenshtein y Jaccard sobre tokens.
      Aplica bonus de contencion (+0.3) y deteccion de acronimos.
      """
      if a == b:
          return 1.0

      # Similitud Levenshtein
      max_len = max(len(a), len(b))
      lev_sim = 1.0 - (_levenshtein(a, b) / max_len) if max_len else 1.0

      # Similitud Jaccard sobre tokens
      tokens_a = a.split()
      tokens_b = b.split()
      jac_sim = _jaccard(tokens_a, tokens_b)

      score = max(lev_sim, jac_sim)

      # Bonus de contencion: un nombre contiene al otro
      if a in b or b in a:
          score = min(1.0, score + 0.3)

      # Deteccion de acronimo: "FBI" ~ "Federal Bureau of Investigation"
      if _is_acronym_of(a, tokens_b) or _is_acronym_of(b, tokens_a):
          score = min(1.0, score + 0.3)

      return score


  def _is_acronym_of(candidate: str, tokens: list[str]) -> bool:
      """Comprueba si candidate es un acronimo formado por las iniciales de tokens."""
      if not candidate or not tokens:
          return False
      initials = "".join(t[0] for t in tokens if t).upper()
      return candidate.upper() == initials


  _EXACT_TYPES = {"ip", "email", "domain", "crypto_wallet", "phone"}


  def _is_exact_type(entity_type: str) -> bool:
      """Tipos tecnicos donde solo se acepta matching exacto."""
      return entity_type.lower() in _EXACT_TYPES


  # ── Union-Find ─────────────────────────────────────────────────────────────────

  class _UnionFind:
      def __init__(self, n: int) -> None:
          self._parent = list(range(n))
          self._rank = [0] * n

      def find(self, x: int) -> int:
          while self._parent[x] != x:
              self._parent[x] = self._parent[self._parent[x]]
              x = self._parent[x]
          return x

      def union(self, x: int, y: int) -> None:
          rx, ry = self.find(x), self.find(y)
          if rx == ry:
              return
          if self._rank[rx] < self._rank[ry]:
              rx, ry = ry, rx
          self._parent[ry] = rx
          if self._rank[rx] == self._rank[ry]:
              self._rank[rx] += 1


  # ── Implementacion principal ────────────────────────────────────────────────────

  def deduplicate_entities(
      candidates: list[EntityCandidate],
      name_threshold: float = 0.85,
      same_type_only: bool = True,
  ) -> DeduplicationResult:
      """Agrupa entidades candidatas que refieren a la misma entidad real.

      Usa fuzzy matching de nombres (Levenshtein + Jaccard) y Union-Find para
      detectar clusters transitivos. Por cada cluster genera una entidad canonica
      mergeando atributos de todos sus miembros.

      Para tipos tecnicos (ip, email, domain, crypto_wallet, phone) solo se
      acepta matching exacto normalizado, ignorando el umbral de nombre.

      Args:
          candidates: lista de EntityCandidate a deduplicar.
          name_threshold: score minimo para considerar dos nombres iguales (0-1).
          same_type_only: si True, solo compara entidades del mismo type_ref.

      Returns:
          DeduplicationResult con entidades deduplicadas, mapas de resolucion
          e historial de merges.
      """
      if not candidates:
          return DeduplicationResult(
              entities=[],
              entity_id_map={},
              name_to_id={},
              merge_log=[],
              total_before=0,
              total_after=0,
          )

      n = len(candidates)

      # Paso 1: normalizar nombres
      normalized: list[str] = []
      for c in candidates:
          norm = normalize_entity_name(c.name, c.type_ref)
          normalized.append(norm)

      # Paso 2: Union-Find sobre todos los indices
      uf = _UnionFind(n)

      # Paso 3: comparacion pairwise (con agrupacion por tipo si same_type_only)
      merge_pairs: list[tuple[int, int, float]] = []

      for i in range(n):
          for j in range(i + 1, n):
              if same_type_only and candidates[i].type_ref != candidates[j].type_ref:
                  continue

              ni, nj = normalized[i], normalized[j]
              et = candidates[i].type_ref.lower()

              if _is_exact_type(et):
                  if ni == nj:
                      uf.union(i, j)
                      merge_pairs.append((i, j, 1.0))
                  continue

              score = _name_similarity(ni, nj)
              if score >= name_threshold:
                  uf.union(i, j)
                  merge_pairs.append((i, j, score))

      # Paso 4: agrupar indices por raiz del Union-Find
      clusters: dict[int, list[int]] = {}
      for i in range(n):
          root = uf.find(i)
          clusters.setdefault(root, []).append(i)

      # Paso 5: merge por cluster
      merged_entities: list[EntityCandidate] = []
      entity_id_map: dict[str, str] = {}
      name_to_id: dict[str, str] = {}
      merge_log: list[dict] = []

      # Pares mergeados para construir el log
      merged_pairs_by_root: dict[int, list[tuple[int, int, float]]] = {}
      for i, j, score in merge_pairs:
          root = uf.find(i)
          merged_pairs_by_root.setdefault(root, []).append((i, j, score))

      for root, indices in clusters.items():
          cluster_candidates = [candidates[idx] for idx in indices]

          if len(cluster_candidates) == 1:
              c = cluster_candidates[0]
              canonical_name = c.name
              canonical_norm = normalized[indices[0]]
              merged_attrs = c.attributes
              merged_confidence = c.confidence
              merged_chunks = list(c.source_chunk_indices)
              merged_from = list(c.merged_from) if c.merged_from else [c.name]
          else:
              # Candidato con mayor confidence es el canonico
              best = max(cluster_candidates, key=lambda c: c.confidence)
              canonical_name = best.name
              canonical_norm = normalize_entity_name(best.name, best.type_ref)

              merged_attrs = merge_entity_attributes(
                  [c.attributes for c in cluster_candidates]
              )
              merged_confidence = max(c.confidence for c in cluster_candidates)

              merged_chunks: list[int] = []
              seen_chunks: set[int] = set()
              for c in cluster_candidates:
                  for idx in c.source_chunk_indices:
                      if idx not in seen_chunks:
                          merged_chunks.append(idx)
                          seen_chunks.add(idx)

              merged_from: list[str] = []
              seen_names: set[str] = set()
              for c in cluster_candidates:
                  names_to_add = c.merged_from if c.merged_from else [c.name]
                  for nm in names_to_add:
                      if nm not in seen_names:
                          merged_from.append(nm)
                          seen_names.add(nm)

              # Log de merge
              other_names = [c.name for c in cluster_candidates if c is not best]
              pairs = merged_pairs_by_root.get(root, [])
              max_score = max((s for _, _, s in pairs), default=1.0)
              merge_log.append(
                  {
                      "canonical": canonical_name,
                      "merged": other_names,
                      "score": round(max_score, 4),
                      "reason": "fuzzy_name",
                  }
              )

          ent_id = str(uuid.uuid4())
          entity = EntityCandidate(
              name=canonical_name,
              name_normalized=canonical_norm,
              type_ref=cluster_candidates[0].type_ref,
              type_label=cluster_candidates[0].type_label,
              attributes=merged_attrs,
              confidence=merged_confidence,
              source_chunk_indices=merged_chunks,
              merged_from=merged_from,
          )
          merged_entities.append(entity)

          # Poblar mapas de resolucion
          entity_id_map[canonical_norm] = ent_id
          for orig_name in merged_from:
              name_to_id[orig_name] = ent_id
          name_to_id[canonical_norm] = ent_id

      return DeduplicationResult(
          entities=merged_entities,
          entity_id_map=entity_id_map,
          name_to_id=name_to_id,
          merge_log=merge_log,
          total_before=n,
          total_after=len(merged_entities),
      )

params_schema:
  {"params":[{"name":"candidates","desc":"lista de EntityCandidate a deduplicar. Cada uno tiene name, type_ref, confidence."},{"name":"name_threshold","desc":"umbral de similitud para fuzzy matching de nombres en rango [0, 1] (tipico: 0.85). Mayor = mas estricto."},{"name":"same_type_only","desc":"si True, solo mergea candidatos con el mismo type_ref. Si False, mergea por similitud ignoring tipo."}],"output":"DeduplicationResult con entidades mergeadas, mapas de resolucion de IDs (name_to_id) y log de merges realizados"}


id: deduplicate_relations_py_datascience
name: deduplicate_relations
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["datascience","extraction","knowledge-graph","nlp","deduplication","fuzzy-match","fuzzygraph"]
uses_functions: ["levenshtein_distance_py_cybersecurity"]
uses_types: ["relation_candidate_py_datascience"]
returns: []
imports: []
test_file_path: python/functions/datascience/deduplicate_relations_test.py
file_path: python/functions/datascience/deduplicate_relations.py
created_at: 2026-04-05T11:24:52Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: fbd08ccf219db9d19d0960a9925fef047b2b5ee4ea669321e76b2ff8dc3eefa1
signature:
  def deduplicate_relations(relations: list[RelationCandidate], entity_id_map: dict[str, str]) -> list[RelationCandidate]

description:
  Deduplica relaciones candidatas resolviendo from_name/to_name a entity IDs finales via entity_id_map. Descarta self-loops y relaciones sin match. Mergea duplicados (mismo from_id, to_id, relation_type) concatenando descripciones unicas y tomando max confidence.

example:
  ```python
  from python.types.datascience.relation_candidate import RelationCandidate
  from python.functions.datascience.deduplicate_relations import deduplicate_relations

  # entity_id_map producido por deduplicate_entities
  entity_id_map = {
      "john smith": "entity_001",
      "smith, john": "entity_001",  # alias mergeado
      "acme corp": "entity_002",
  }

  relations = [
      RelationCandidate(from_name="John Smith", to_name="Acme Corp",
                        relation_type="works_at", description="John es CEO",
                        confidence=0.9, source_chunk_index=0),
      RelationCandidate(from_name="Smith, John", to_name="Acme Corp",
                        relation_type="works_at", description="CEO de Acme",
                        confidence=0.7, source_chunk_index=2),
  ]

  result = deduplicate_relations(relations, entity_id_map)
  # → 1 RelationCandidate con from_id="entity_001", to_id="entity_002",
  #   confidence=0.9, description="John es CEO; CEO de Acme"
  ```

tests:
  ["dos relaciones identicas se colapsan en una","relacion con nombre mergeado se resuelve al id correcto","self loop se descarta","nombre no mapeado sin fuzzy match se descarta","relaciones distintas se mantienen","merge descripcion concatena unicas","lista vacia retorna lista vacia","fuzzy match resuelve nombre cercano"]

notes:
  La funcion es pura: no hace I/O, no tiene efectos secundarios. El logging es
  de nivel DEBUG/WARNING — en produccion configurar el logger de la aplicacion.

  **Resolucion de nombres:**
  - Lookup exacto primero (lowercase strip del nombre contra las claves del mapa).
  - Si no hay match exacto, fuzzy match con Levenshtein (threshold=3 ediciones).
  - Si sigue sin match, la relacion se descarta con `logger.warning`.

  **Self-loops:** relaciones donde `from_id == to_id` siempre se descartan.

  **Merge:** cuando varias relaciones comparten `(from_id, to_id, relation_type)`:
  - `confidence`: max del grupo.
  - `description`: union de descripciones unicas (no duplicadas), separadas por `'; '`.
  - `from_name` / `to_name` / `source_chunk_index`: del primer candidato del grupo.

  **Integracion con fuzzygraph:**
  Esta funcion es el paso 4 del pipeline de extraccion. Recibe el output de
  `extract_relations_llm` (relaciones crudas con nombres de texto) y el
  `entity_id_map` producido por `deduplicate_entities`. Produce la lista final
  de relaciones para `ExtractionResult`.

code:
  """Deduplica RelationCandidate resolviendo nombres a IDs y colapsando duplicados."""

  import logging
  import os
  import sys

  logger = logging.getLogger(__name__)

  # --- Importar levenshtein_distance desde cybersecurity ---
  # Soporta dos contextos:
  #   1. Ejecutado desde python/functions/datascience/ (pytest local)
  #   2. Ejecutado desde la raiz del registry (fn run)
  def _levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de edicion de Levenshtein entre dos strings."""
      if len(a) < len(b):
          return _levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)
      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(curr_row[j] + 1, prev_row[j + 1] + 1, prev_row[j] + cost)
              )
          prev_row = curr_row
      return prev_row[-1]


  try:
      _here = os.path.dirname(os.path.abspath(__file__))
      _cyber_path = os.path.join(_here, "..", "cybersecurity")
      if _cyber_path not in sys.path:
          sys.path.insert(0, _cyber_path)
      from cybersecurity import levenshtein_distance as _lev
  except ImportError:
      _lev = None  # type: ignore

  levenshtein_distance = _lev if _lev is not None else _levenshtein_distance


  def _fuzzy_resolve(name: str, entity_id_map: dict[str, str], threshold: int = 3) -> str:
      """Intenta resolver un nombre contra las claves del mapa por fuzzy match.

      Recorre todas las claves de entity_id_map y busca la mas cercana segun
      distancia de Levenshtein. Retorna el entity_id si la distancia es <=
      threshold, o '' si no hay match aceptable.

      Args:
          name: nombre a resolver (ya en lowercase strip).
          entity_id_map: mapa nombre_normalizado -> entity_id.
          threshold: distancia maxima de edicion para considerar match (default 3).

      Returns:
          entity_id del mejor match o '' si no hay match.
      """
      best_id = ""
      best_dist = threshold + 1
      for key, entity_id in entity_id_map.items():
          dist = levenshtein_distance(name, key)
          if dist < best_dist:
              best_dist = dist
              best_id = entity_id
      return best_id if best_dist <= threshold else ""


  def deduplicate_relations(
      relations: list,
      entity_id_map: dict[str, str],
  ) -> list:
      """Deduplica relaciones candidatas resolviendo nombres a IDs de entidad finales.

      Algoritmo:
      1. Para cada RelationCandidate, intentar resolver from_name y to_name al
         entity_id via entity_id_map (lookup exacto primero, ignorando mayusculas).
         Si no hay match exacto, intentar fuzzy match con levenshtein_distance.
         Si sigue sin match, descartar la relacion con warning.
      2. Descartar self-loops (from_id == to_id).
      3. Deduplicar por (from_id, to_id, relation_type):
         - description: concatenar descripciones unicas separadas por '; '
         - confidence: max del grupo
      4. Retornar lista limpia de RelationCandidate con from_id y to_id resueltos.

      Args:
          relations: lista de RelationCandidate con from_name/to_name originales.
          entity_id_map: mapa nombre_normalizado -> entity_id (output de
              deduplicate_entities). Permite resolver nombres que fueron mergeados.

      Returns:
          Lista deduplicada de RelationCandidate con from_id y to_id resueltos.
      """
      # Importar tipo — funciona tanto desde datascience/ como desde raiz del registry
      try:
          _types_path = os.path.join(
              os.path.dirname(os.path.abspath(__file__)),
              "..", "..", "..", "python", "types", "datascience",
          )
          if _types_path not in sys.path:
              sys.path.insert(0, _types_path)
          from relation_candidate import RelationCandidate
      except ImportError:
          from python.types.datascience.relation_candidate import RelationCandidate  # type: ignore

      resolved: list = []

      for rel in relations:
          # --- Resolver from_name ---
          from_key = rel.from_name.lower().strip()
          from_id = entity_id_map.get(from_key, "")
          if not from_id:
              from_id = _fuzzy_resolve(from_key, entity_id_map)
              if not from_id:
                  logger.warning(
                      "deduplicate_relations: no se pudo resolver from_name=%r — descartando",
                      rel.from_name,
                  )
                  continue

          # --- Resolver to_name ---
          to_key = rel.to_name.lower().strip()
          to_id = entity_id_map.get(to_key, "")
          if not to_id:
              to_id = _fuzzy_resolve(to_key, entity_id_map)
              if not to_id:
                  logger.warning(
                      "deduplicate_relations: no se pudo resolver to_name=%r — descartando",
                      rel.to_name,
                  )
                  continue

          # --- Descartar self-loops ---
          if from_id == to_id:
              logger.debug(
                  "deduplicate_relations: self-loop descartado (from=%r, to=%r, type=%r)",
                  rel.from_name,
                  rel.to_name,
                  rel.relation_type,
              )
              continue

          resolved.append(
              RelationCandidate(
                  from_name=rel.from_name,
                  to_name=rel.to_name,
                  from_id=from_id,
                  to_id=to_id,
                  relation_type=rel.relation_type,
                  description=rel.description,
                  confidence=rel.confidence,
                  source_chunk_index=rel.source_chunk_index,
              )
          )

      # --- Deduplicar por (from_id, to_id, relation_type) ---
      groups: dict[tuple, list] = {}
      for rel in resolved:
          key = (rel.from_id, rel.to_id, rel.relation_type)
          groups.setdefault(key, []).append(rel)

      result: list = []
      for (from_id, to_id, rel_type), group in groups.items():
          if len(group) == 1:
              result.append(group[0])
              continue

          # Mergear: max confidence + union de descripciones unicas
          best_confidence = max(r.confidence for r in group)
          seen_desc: set[str] = set()
          descriptions: list[str] = []
          for r in group:
              if r.description and r.description not in seen_desc:
                  descriptions.append(r.description)
                  seen_desc.add(r.description)

          result.append(
              RelationCandidate(
                  from_name=group[0].from_name,
                  to_name=group[0].to_name,
                  from_id=from_id,
                  to_id=to_id,
                  relation_type=rel_type,
                  description="; ".join(descriptions),
                  confidence=best_confidence,
                  source_chunk_index=group[0].source_chunk_index,
              )
          )

      return result

params_schema:
  {"params":[{"name":"relations","desc":"lista de RelationCandidate con from_name, to_name, relation_type, description, confidence"},{"name":"entity_id_map","desc":"dict producido por deduplicate_entities mapando nombres mergeados a IDs canonicos (ej: {'john smith': 'entity_001'})"}],"output":"lista de RelationCandidate deduplicadas, con from_id y to_id resueltos al entity_id_map, sin self-loops"}


id: detect_drift_py_datascience
name: detect_drift
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["drift","statistics","z-score","monitoring","executions","operations","datascience"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
test_file_path: python/functions/datascience/detect_drift_test.py
file_path: python/functions/datascience/detect_drift.py
created_at: 2026-04-05T11:25:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a7a17e4855a17c9dd5b3d7a709ceede9d8bd314f0f733d7b44b8673bd09d78af
signature:
  def detect_drift(history: list[dict], current: dict, fields: list[str], threshold: float = 2.0) -> list[dict]

description:
  Detecta drift estadistico comparando metricas de la ejecucion actual contra el historial usando z-score. Si |z| > threshold, el campo ha drifteado. Util para monitorizar executions en operations.db.

example:
  ```python
  history = [
      {"records_out": 100, "duration_ms": 500},
      {"records_out": 105, "duration_ms": 480},
      {"records_out": 98,  "duration_ms": 510},
  ]
  current = {"records_out": 50, "duration_ms": 2000}

  results = detect_drift(history, current, ["records_out", "duration_ms"])
  # [
  #   {"field": "records_out", "current": 50, "mean": 101.0, "std": 3.6, "z_score": -14.2, "drifted": True},
  #   {"field": "duration_ms", "current": 2000, "mean": 496.7, "std": 15.3, "z_score": 98.3, "drifted": True},
  # ]
  ```

tests:
  ["campo con drift claro (z \u003e threshold)","campo estable (z \u003c threshold)","historial con un solo punto → std=0, no puede calcular → drifted=False con nota","historial vacio → todos drifted=False","threshold custom"]

notes:
  Funcion pura. Solo stdlib (`math`).

  El z-score usa desviacion estandar poblacional (dividir por N, no N-1) para ser consistente con historial de cualquier tamanio.

  Casos especiales:
  - **Historial vacio**: z_score=0.0, drifted=False para todos los campos.
  - **Un solo punto en historial**: std=0.0, z_score=0.0, drifted=False. No hay suficiente historia para calcular variabilidad.
  - **Std=0 con N>=2**: todos los valores historicos identicos. z_score=0.0, drifted=False (cualquier desviacion seria tecnicamente infinita, pero se asume que el sistema es muy estable).

  Pensado para el paso ANALIZAR del bucle reactivo: comparar `metrics` de la ejecucion actual con executions historicas de `operations.db`.

code:
  """detect_drift — detecta drift estadistico por z-score comparando metricas contra historial."""

  import math


  def detect_drift(
      history: list[dict],
      current: dict,
      fields: list[str],
      threshold: float = 2.0,
  ) -> list[dict]:
      """Detecta drift estadistico comparando metricas actuales contra el historial.

      Usa z-score: si |z| > threshold, el campo ha drifteado. Pensado para
      comparar metrics de executions sucesivas en operations.db.

      Args:
          history: Lista de dicts con metricas historicas. Cada dict puede
                   contener cualquier combinacion de los campos indicados.
          current: Dict con las metricas de la ejecucion actual.
          fields: Lista de campos numericos a analizar.
          threshold: Umbral de z-score para considerar drift. Default 2.0.

      Returns:
          Lista de dicts con: field, current, mean, std, z_score, drifted.
          Si el historial tiene 0 o 1 punto, z_score=0.0 y drifted=False
          porque no hay suficiente informacion estadistica.
      """
      results = []

      for field in fields:
          values = [
              float(h[field])
              for h in history
              if field in h and h[field] is not None
          ]

          current_val = float(current.get(field, 0))

          if len(values) == 0:
              results.append({
                  "field": field,
                  "current": current_val,
                  "mean": 0.0,
                  "std": 0.0,
                  "z_score": 0.0,
                  "drifted": False,
              })
              continue

          n = len(values)
          mean = sum(values) / n

          if n < 2:
              # Un solo punto: no hay std, no podemos calcular z-score
              results.append({
                  "field": field,
                  "current": current_val,
                  "mean": mean,
                  "std": 0.0,
                  "z_score": 0.0,
                  "drifted": False,
              })
              continue

          variance = sum((v - mean) ** 2 for v in values) / n
          std = math.sqrt(variance)

          if std == 0.0:
              # Todos los valores identicos: z_score indeterminado, no drift
              z_score = 0.0
              drifted = False
          else:
              z_score = (current_val - mean) / std
              drifted = abs(z_score) > threshold

          results.append({
              "field": field,
              "current": current_val,
              "mean": mean,
              "std": std,
              "z_score": z_score,
              "drifted": drifted,
          })

      return results

params_schema:
  {"params":[{"name":"history","desc":"lista de dicts con metricas historicas (ej: [{'records_out': 100, 'duration_ms': 500}, ...])"},{"name":"current","desc":"dict con metricas actuales a comparar (ej: {'records_out': 50, 'duration_ms': 2000})"},{"name":"fields","desc":"lista de nombres de campos a monitorizar en current vs history (ej: ['records_out', 'duration_ms'])"},{"name":"threshold","desc":"umbral de z-score para declarar drift (tipico: 2.0-3.0). Mayor = menos sensible."}],"output":"lista de dicts con {field, current, mean, std, z_score, drifted} para cada campo monitoreado"}


id: detect_outliers_py_datascience
name: detect_outliers
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def detect_outliers(data: list, threshold: float) -> list
description: Detecta outliers por z-score. Retorna lista de bools, True donde |z-score| > threshold.
tags: ["statistics","outliers","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Usa z-score poblacional. Threshold tipico: 2.0 o 3.0. Si la desviacion es cero, no hay outliers.
content_hash: 2f20cfd448f740d6807497c607cb8b49e77a6ceb377fbff267f4bdcd2ef70ca1
example:
  ```python
  detect_outliers([1, 2, 3, 100, 2, 3], 2.0)
  # [False, False, False, True, False, False]
  ```

code:
  """Pure datascience utilities — statistics and numerical functions.

  Uses only math stdlib. No external dependencies.
  """

  import math


  def pearson(xs: list, ys: list) -> float:
      """Pearson correlation coefficient between two lists of floats."""
      n = len(xs)
      if n != len(ys) or n == 0:
          return 0.0
      mean_x = sum(xs) / n
      mean_y = sum(ys) / n
      num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
      den_x = math.sqrt(sum((x - mean_x) ** 2 for x in xs))
      den_y = math.sqrt(sum((y - mean_y) ** 2 for y in ys))
      if den_x == 0.0 or den_y == 0.0:
          return 0.0
      return num / (den_x * den_y)


  def standardize(data: list) -> list:
      """Z-score standardization (mean=0, std=1)."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [0.0] * n
      return [(x - mean) / std for x in data]


  def min_max_scale(data: list) -> list:
      """Scale values to [0, 1] range."""
      if not data:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          return [0.0] * len(data)
      return [(x - lo) / (hi - lo) for x in data]


  def clip(data: list, lo: float, hi: float) -> list:
      """Clip values to [lo, hi]."""
      return [max(lo, min(hi, x)) for x in data]


  def detect_outliers(data: list, threshold: float) -> list:
      """Returns list of bools, True where |z-score| > threshold."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [False] * n
      return [abs((x - mean) / std) > threshold for x in data]


  def impute(data: list) -> list:
      """Replace None/NaN with mean of non-null values."""
      valid = [x for x in data if x is not None and not (isinstance(x, float) and math.isnan(x))]
      if not valid:
          return [0.0] * len(data)
      mean = sum(valid) / len(valid)
      return [
          mean if (x is None or (isinstance(x, float) and math.isnan(x))) else x
          for x in data
      ]


  def histogram(data: list, buckets: int) -> list:
      """Returns list of counts per bucket."""
      if not data or buckets <= 0:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          counts = [0] * buckets
          counts[0] = len(data)
          return counts
      width = (hi - lo) / buckets
      counts = [0] * buckets
      for x in data:
          idx = int((x - lo) / width)
          if idx >= buckets:
              idx = buckets - 1
          counts[idx] += 1
      return counts


  def rolling_window(xs: list, size: int) -> list:
      """Returns list of sublists (sliding windows of given size)."""
      if size <= 0 or size > len(xs):
          return []
      return [xs[i : i + size] for i in range(len(xs) - size + 1)]


  def autocorrelation(data: list, lag: int) -> float:
      """Autocorrelation at given lag."""
      n = len(data)
      if lag < 0 or lag >= n or n == 0:
          return 0.0
      mean = sum(data) / n
      var = sum((x - mean) ** 2 for x in data) / n
      if var == 0.0:
          return 0.0
      cov = sum((data[i] - mean) * (data[i + lag] - mean) for i in range(n - lag)) / n
      return cov / var


  def linspace(start: float, stop: float, num: int) -> list:
      """Generate evenly spaced values from start to stop (inclusive)."""
      if num <= 0:
          return []
      if num == 1:
          return [start]
      step = (stop - start) / (num - 1)
      return [start + i * step for i in range(num)]


  def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict:
      """Estima parámetros de un proceso Hawkes desde autocorrelación de arrivals.

      Ajusta exponencial a*exp(-b*lag) sobre la ACF.
      Retorna dict con alpha, beta, branching_ratio, acf.
      """
      import numpy as np
      from scipy.optimize import curve_fit

      arr = np.array(arrivals, dtype=float)
      mean_a = np.mean(arr)
      var_a = np.var(arr)
      if var_a == 0:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': [1.0]}

      acf = [1.0] + [
          float(np.mean((arr[lag:] - mean_a) * (arr[:-lag] - mean_a)) / var_a)
          for lag in range(1, max_lag)
      ]

      lags = np.arange(1, max_lag)
      acf_vals = np.array(acf[1:])

      if acf_vals[0] <= 0.01:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': acf}

      exp_decay = lambda x, a, b: a * np.exp(-b * x)
      try:
          popt, _ = curve_fit(exp_decay, lags, acf_vals, p0=[0.5, 0.5], maxfev=5000)
          alpha_est, beta_est = abs(popt[0]), abs(popt[1])
      except RuntimeError:
          alpha_est, beta_est = 0.0, 1.0

      branching = alpha_est / beta_est if beta_est > 0 else 0.0
      return {
          'alpha': round(alpha_est, 4),
          'beta': round(beta_est, 4),
          'branching_ratio': round(branching, 4),
          'acf': acf,
      }


  def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict:
      """Estima el exponente alpha de una distribución Pareto via MLE.

      α = n / Σ ln(xi / x_min) donde x_min es el percentil indicado.
      Alpha bajo = cola más pesada = más valores extremos.
      """
      import numpy as np

      arr = np.array([v for v in values if v > 0], dtype=float)
      if len(arr) < 10:
          return {'alpha': 0.0, 'x_min': 0.0, 'n_tail': 0}

      x_min = float(np.percentile(arr, x_min_percentile))
      tail = arr[arr >= x_min]

      if len(tail) < 2 or x_min <= 0:
          return {'alpha': 0.0, 'x_min': x_min, 'n_tail': len(tail)}

      alpha = float(len(tail) / np.sum(np.log(tail / x_min)))

      return {
          'alpha': round(alpha, 4),
          'x_min': round(x_min, 6),
          'n_tail': len(tail),
      }

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos para detectar outliers"},{"name":"threshold","desc":"umbral de z-score absoluto (tipico: 2.0 para 95% confianza, 3.0 para 99%). Mayor = menos sensible."}],"output":"lista de booleanos paralela a data, True donde |z-score| \u003e threshold"}


id: diff_entities_py_datascience
name: diff_entities
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["diff","entities","snapshot","operations","comparison","datascience"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/datascience/diff_entities_test.py
file_path: python/functions/datascience/diff_entities.py
created_at: 2026-04-05T11:24:00Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e1beba8059ef4b69f0684a836fff11906d2fc86f8d13e479fdd50c4cbf3f0617
signature:
  def diff_entities(before: list[dict], after: list[dict], key: str = 'id', ignore_fields: list[str] | None = None, compare_fields: list[str] | None = None) -> dict

description:
  Compara dos snapshots de entities y devuelve diferencias campo a campo. Detecta añadidas, eliminadas, modificadas e inalteradas. Ignora created_at y updated_at por defecto.

example:
  ```python
  before = [
      {"id": "1", "name": "Alice", "status": "active", "updated_at": "2024-01-01"},
      {"id": "2", "name": "Bob", "status": "active", "updated_at": "2024-01-01"},
  ]
  after = [
      {"id": "1", "name": "Alice", "status": "inactive", "updated_at": "2024-01-02"},
      {"id": "3", "name": "Carol", "status": "active", "updated_at": "2024-01-02"},
  ]

  result = diff_entities(before, after)
  # result["added"]    -> [{"id": "3", "name": "Carol", ...}]
  # result["removed"]  -> [{"id": "2", "name": "Bob", ...}]
  # result["modified"] -> [{"key": "1", "changes": {"status": {"old": "active", "new": "inactive"}}}]
  # result["unchanged"] -> 0
  # result["summary"]  -> "1 added, 1 removed, 1 modified, 0 unchanged"
  ```

tests:
  ["entity añadida","entity eliminada","entity modificada con detalle de campos","entities identicas → unchanged","ignore_fields funciona","compare_fields filtra correctamente","lista vacia vs lista con datos"]

notes:
  Funcion pura. No hace I/O — toma listas de dicts ya cargadas en memoria.

  El campo `key` debe existir en todas las entities; las que no lo tengan se ignoran silenciosamente.

  Si `compare_fields` se da, tiene prioridad sobre `ignore_fields`. Esto permite comparar solo un subconjunto especifico de campos sin preocuparse por los campos temporales.

  El orden de `added` y `removed` no esta garantizado (depende del orden de iteracion de sets).

code:
  """diff_entities — compara dos snapshots de entities detectando cambios campo a campo."""


  def diff_entities(
      before: list[dict],
      after: list[dict],
      key: str = "id",
      ignore_fields: list[str] | None = None,
      compare_fields: list[str] | None = None,
  ) -> dict:
      """Compara dos snapshots de entities y devuelve diferencias campo a campo.

      Detecta entities añadidas, eliminadas, modificadas e inalteradas.
      Ignora campos de metadata temporal por defecto (created_at, updated_at).

      Args:
          before: Lista de entities del snapshot anterior.
          after: Lista de entities del snapshot posterior.
          key: Campo que identifica unicamente cada entity. Default "id".
          ignore_fields: Campos a excluir de la comparacion.
                         Default ["created_at", "updated_at"].
          compare_fields: Si se da, solo compara estos campos (tiene prioridad
                          sobre ignore_fields).

      Returns:
          Dict con keys: added, removed, modified, unchanged, summary.
          modified contiene lista de {"key": str, "changes": {"field": {"old": ..., "new": ...}}}.
      """
      if ignore_fields is None:
          ignore_fields = ["created_at", "updated_at"]

      before_map = {str(e[key]): e for e in before if key in e}
      after_map = {str(e[key]): e for e in after if key in e}

      before_keys = set(before_map.keys())
      after_keys = set(after_map.keys())

      added = [after_map[k] for k in after_keys - before_keys]
      removed = [before_map[k] for k in before_keys - after_keys]

      modified = []
      unchanged = 0

      for k in before_keys & after_keys:
          b = before_map[k]
          a = after_map[k]

          if compare_fields is not None:
              fields_to_check = compare_fields
          else:
              all_fields = set(b.keys()) | set(a.keys())
              fields_to_check = [f for f in all_fields if f not in ignore_fields and f != key]

          changes = {}
          for field in fields_to_check:
              old_val = b.get(field)
              new_val = a.get(field)
              if old_val != new_val:
                  changes[field] = {"old": old_val, "new": new_val}

          if changes:
              modified.append({"key": k, "changes": changes})
          else:
              unchanged += 1

      n_added = len(added)
      n_removed = len(removed)
      n_modified = len(modified)
      summary = f"{n_added} added, {n_removed} removed, {n_modified} modified, {unchanged} unchanged"

      return {
          "added": added,
          "removed": removed,
          "modified": modified,
          "unchanged": unchanged,
          "summary": summary,
      }

params_schema:
  {"params":[{"name":"before","desc":"lista de dicts con entities antes de cambios (ej: [{'id': '1', 'name': 'Alice', 'status': 'active'}, ...])"},{"name":"after","desc":"lista de dicts con entities despues de cambios, misma estructura que before"},{"name":"key","desc":"nombre del campo que identifica cada entity (tipico: 'id'). Debe existir en todas las entities."},{"name":"ignore_fields","desc":"lista opcional de campos a ignorar en la comparacion (ej: ['created_at', 'updated_at'])"},{"name":"compare_fields","desc":"lista opcional de campos SOLO a comparar (si se da, tiene prioridad sobre ignore_fields)"}],"output":"dict con {added, removed, modified, unchanged, summary} describiendo los cambios campo a campo"}


id: diff_relations_py_datascience
name: diff_relations
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["diff","relations","graph","snapshot","operations","comparison","datascience"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/datascience/diff_relations_test.py
file_path: python/functions/datascience/diff_relations.py
created_at: 2026-04-05T11:24:27Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: cd900987cdb155ecc8af4a2af7e608218e71f79a5dc59165d50bb5f99fa40cb5
signature:
  def diff_relations(before: list[dict], after: list[dict], key: tuple[str, str, str] = ('source_id', 'target_id', 'relation_type'), ignore_fields: list[str] | None = None, compare_fields: list[str] | None = None) -> dict

description:
  Compara relaciones entre dos snapshots usando key compuesta (source_id, target_id, relation_type). Detecta relaciones añadidas, eliminadas y modificadas con detalle campo a campo.

example:
  ```python
  before = [
      {"source_id": "A", "target_id": "B", "relation_type": "knows", "weight": 1.0},
      {"source_id": "B", "target_id": "C", "relation_type": "owns", "weight": 0.5},
  ]
  after = [
      {"source_id": "A", "target_id": "B", "relation_type": "knows", "weight": 2.0},
      {"source_id": "C", "target_id": "D", "relation_type": "knows", "weight": 1.0},
  ]

  result = diff_relations(before, after)
  # result["added"]    -> [{"source_id": "C", "target_id": "D", ...}]
  # result["removed"]  -> [{"source_id": "B", "target_id": "C", ...}]
  # result["modified"] -> [{"key": "A|B|knows", "changes": {"weight": {"old": 1.0, "new": 2.0}}}]
  # result["unchanged"] -> 0
  ```

tests:
  ["relacion añadida","relacion eliminada","relacion con metadata modificada (mismo source/target/type, distinto weight)","key compuesta funciona correctamente"]

notes:
  La key compuesta se serializa como `source_id|target_id|relation_type`. Si alguno de los campos clave no existe en la relacion, se usa string vacio.

  Misma semantica que `diff_entities_py_datascience` pero adaptada para relaciones donde no hay un ID unico — la identidad se define por los tres campos de la key.

  Complemento natural de `diff_entities_py_datascience` para comparar grafos completos entre ejecuciones de pipelines.

code:
  """diff_relations — compara dos snapshots de relaciones con key compuesta."""


  def diff_relations(
      before: list[dict],
      after: list[dict],
      key: tuple[str, str, str] = ("source_id", "target_id", "relation_type"),
      ignore_fields: list[str] | None = None,
      compare_fields: list[str] | None = None,
  ) -> dict:
      """Compara relaciones entre dos snapshots usando key compuesta.

      Las relaciones se identifican por (source_id, target_id, relation_type)
      porque no tienen un ID unico propio. Detecta relaciones añadidas,
      eliminadas y modificadas (mismo source/target/type, distinta metadata).

      Args:
          before: Lista de relaciones del snapshot anterior.
          after: Lista de relaciones del snapshot posterior.
          key: Tupla de campos que forman la key compuesta.
               Default ("source_id", "target_id", "relation_type").
          ignore_fields: Campos a excluir de la comparacion.
                         Default ["created_at", "updated_at"].
          compare_fields: Si se da, solo compara estos campos.

      Returns:
          Dict con keys: added, removed, modified, unchanged, summary.
          modified contiene lista de {"key": str, "changes": {"field": {"old": ..., "new": ...}}}.
      """
      if ignore_fields is None:
          ignore_fields = ["created_at", "updated_at"]

      def make_key(rel: dict) -> str:
          return "|".join(str(rel.get(k, "")) for k in key)

      before_map = {make_key(r): r for r in before}
      after_map = {make_key(r): r for r in after}

      before_keys = set(before_map.keys())
      after_keys = set(after_map.keys())

      added = [after_map[k] for k in after_keys - before_keys]
      removed = [before_map[k] for k in before_keys - after_keys]

      modified = []
      unchanged = 0

      for k in before_keys & after_keys:
          b = before_map[k]
          a = after_map[k]

          if compare_fields is not None:
              fields_to_check = compare_fields
          else:
              all_fields = set(b.keys()) | set(a.keys())
              key_set = set(key)
              fields_to_check = [f for f in all_fields if f not in ignore_fields and f not in key_set]

          changes = {}
          for field in fields_to_check:
              old_val = b.get(field)
              new_val = a.get(field)
              if old_val != new_val:
                  changes[field] = {"old": old_val, "new": new_val}

          if changes:
              modified.append({"key": k, "changes": changes})
          else:
              unchanged += 1

      n_added = len(added)
      n_removed = len(removed)
      n_modified = len(modified)
      summary = f"{n_added} added, {n_removed} removed, {n_modified} modified, {unchanged} unchanged"

      return {
          "added": added,
          "removed": removed,
          "modified": modified,
          "unchanged": unchanged,
          "summary": summary,
      }

params_schema:
  {"params":[{"name":"before","desc":"lista de dicts con relaciones antes (ej: [{'source_id': 'A', 'target_id': 'B', 'relation_type': 'knows', 'weight': 1.0}, ...])"},{"name":"after","desc":"lista de dicts con relaciones despues, misma estructura que before"},{"name":"key","desc":"tupla de 3 nombres de campo que forman la identidad de una relacion (defecto: ('source_id', 'target_id', 'relation_type'))"},{"name":"ignore_fields","desc":"lista opcional de campos a ignorar en comparacion (ej: ['timestamp'])"},{"name":"compare_fields","desc":"lista opcional de campos SOLO a comparar (si se da, prioridad sobre ignore_fields)"}],"output":"dict con {added, removed, modified, unchanged} describiendo cambios en relaciones"}


id: estimate_hawkes_py_datascience
name: estimate_hawkes
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict
tags: ["estimation","hawkes","stochastic-process","microstructure","timeseries"]
uses_functions: []
uses_types: []
returns: []
imports: ["numpy","scipy"]
tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: fc3f1dfc15fd1079b36f7841371de90c47ffdf5c18db0a68c3c080eafc6cde7c
description:
  Estima parámetros de un proceso Hawkes (alpha, beta, branching_ratio) desde la autocorrelación de arrivals ajustando una exponencial decreciente sobre la ACF.

example:
  ```python
  arrivals = [0, 1, 3, 2, 0, 1, 4, 2, 1, 0] * 10
  result = estimate_hawkes(arrivals, max_lag=10)
  # {'alpha': 0.312, 'beta': 0.874, 'branching_ratio': 0.357, 'acf': [...]}
  ```

notes:
  Ajusta la función `a * exp(-b * lag)` sobre los lags 1..max_lag de la ACF usando `curve_fit` de scipy.
  Si el primer lag de la ACF es <= 0.01 (sin autocorrelación), retorna alpha=0, beta=1, branching_ratio=0.
  El branching_ratio = alpha/beta; si se acerca a 1, el proceso es explosivo.
  Función pura: requiere numpy y scipy instalados.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"arrivals","desc":"lista de conteos de eventos por periodo (ej: [0, 1, 3, 2, 0, 1, ...] eventos por tick). Reflect actividad temporal."},{"name":"max_lag","desc":"numero maximo de lags para calcular autocorrelacion (tipico: 30). Mayor = mas precision pero mas ruido."}],"output":"dict con {alpha, beta, branching_ratio, acf} estimados parametros del proceso Hawkes"}


id: estimate_pareto_alpha_py_datascience
name: estimate_pareto_alpha
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict
tags: ["estimation","pareto","power-law","heavy-tail","statistics"]
uses_functions: []
uses_types: []
returns: []
imports: ["numpy"]
tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 041e99454ce5ac3bbc35e7456c162071f8c68357712e78bf8cdc87a1ae2cb08b
description:
  Estima el exponente alpha de una distribución Pareto via MLE. Alpha bajo indica cola más pesada y mayor frecuencia de valores extremos.

example:
  ```python
  import numpy as np
  # Simular datos con cola pesada
  values = list(np.random.pareto(2.0, 1000) + 1)
  result = estimate_pareto_alpha(values, x_min_percentile=90.0)
  # {'alpha': ~2.0, 'x_min': ..., 'n_tail': 100}
  ```

notes:
  Usa el estimador MLE de Hill: α = n / Σ ln(xᵢ / x_min).
  x_min se determina como el percentil indicado de los valores positivos.
  Retorna alpha=0 si hay menos de 10 valores positivos o la cola tiene menos de 2 elementos.
  Función pura: requiere numpy instalado.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"values","desc":"lista de valores numericos positivos donde se sospecha cola pesada (ej: tamanios de ordenes, ingresos). Debe haber \u003e10 valores."},{"name":"x_min_percentile","desc":"percentil a partir del cual considerar la cola (tipico: 90.0 para considerar el 10% superior)"}],"output":"dict con {alpha, x_min, n_tail} donde alpha es el exponente estimado (menor = cola mas pesada)"}


id: extract_entities_llm_py_datascience
name: extract_entities_llm
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: impure
tags: ["llm","extraction","entity","nlp","osint","graph","fuzzygraph","datascience","prompt"]
uses_functions: []
uses_types: ["entity_candidate_py_datascience"]
returns: []
error_type: error_go_core
imports: ["warnings","typing.Callable"]
test_file_path: python/functions/datascience/extract_entities_llm_test.py
file_path: python/functions/datascience/extract_entities_llm.py
created_at: 2026-04-05T11:24:00Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f9970cd6482bc16058ebd78e56ec7e04c4d9ea664c9d6db835f87dc5e0432de4
signature:
  def extract_entities_llm(text: str, entity_schema: list[dict], llm_chat_json: Callable[[list[dict]], dict], language_instruction: str = 'Respond in English.') -> list[EntityCandidate]

description:
  Extrae entidades de un chunk de texto usando un LLM inyectado. Construye el system prompt con el schema, llama al LLM y valida la respuesta retornando EntityCandidate. JSON invalido o type_ref fuera del schema se descartan con warning.

example:
  ```python
  import json
  from extract_entities_llm import extract_entities_llm

  # LLM stub para tests — en produccion usar litellm o similar
  def mock_llm(messages: list[dict]) -> dict:
      return {
          "entities": [
              {
                  "name": "John Smith",
                  "type_ref": "osint_person_go_cybersecurity",
                  "attributes": {"full_name": "John Smith", "nationality": "US"},
                  "confidence": 0.95,
              },
              {
                  "name": "evil-corp.com",
                  "type_ref": "osint_domain_go_cybersecurity",
                  "attributes": {"fqdn": "evil-corp.com"},
                  "confidence": 0.88,
              },
          ]
      }

  schema = [
      {
          "type_ref": "osint_person_go_cybersecurity",
          "label": "Person",
          "metadata_fields": ["full_name", "alias", "nationality", "dob", "risk_score"],
      },
      {
          "type_ref": "osint_domain_go_cybersecurity",
          "label": "Domain",
          "metadata_fields": ["fqdn", "registrar", "created_date"],
      },
  ]

  text = "John Smith, a US citizen, was linked to the domain evil-corp.com."
  candidates = extract_entities_llm(text, schema, mock_llm)
  # [EntityCandidate(name='John Smith', type_ref='osint_person_go_cybersecurity', confidence=0.95),
  #  EntityCandidate(name='evil-corp.com', type_ref='osint_domain_go_cybersecurity', confidence=0.88)]
  ```

tests:
  ["texto con entidades claras retorna EntityCandidate","texto sin entidades retorna lista vacia","llm retorna json mal formado retorna lista vacia con warning","type_ref invalido en respuesta se descarta con warning","confidence se propaga correctamente","schema vacio lanza ValueError"]

notes:
  **Inyeccion de dependencia del LLM:** `llm_chat_json` recibe mensajes en formato OpenAI (`[{"role": "system", "content": "..."}, ...]`) y retorna un `dict` con la respuesta ya parseada como JSON. Esto desacopla la funcion de cualquier cliente especifico — puede usarse con OpenAI, Anthropic via litellm, o cualquier mock.

  **Validacion de type_ref:** Solo se aceptan entidades cuyo `type_ref` aparece en el `entity_schema`. Entidades con type_ref desconocido se descartan con `warnings.warn` (no lanzan excepcion) para ser resiliente ante alucinaciones del LLM.

  **Manejo de JSON invalido:** Si `llm_chat_json` lanza una excepcion o retorna un dict sin la clave `entities`, se retorna lista vacia y se emite un warning. El llamador puede decidir si reintentar.

  **Confidence clamping:** El valor de confidence se clampea al rango [0.0, 1.0] automaticamente.

  **Atributos null:** Los atributos con valor `None` se filtran del dict de atributos para mantener el output limpio.

  **source_chunk_indices:** Esta funcion no setea `source_chunk_indices` — ese campo lo llena el pipeline exterior que conoce el indice del chunk actual.

  Esta funcion es el bloque atomico de extraccion. El pipeline completo de grafos la llama por cada chunk del documento y luego deduplica los candidatos resultantes.

code:
  """Extrae entidades de un chunk de texto usando un LLM inyectado."""

  import sys
  import os
  import warnings
  from typing import Callable

  sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", ".."))

  from python.types.datascience.entity_candidate import EntityCandidate


  def _build_system_prompt(entity_schema: list[dict], language_instruction: str) -> str:
      """Construye el system prompt para extraccion de entidades."""
      lines = [
          "You are an entity extraction expert. Given text, extract all entities",
          "matching these types. For each entity, provide: name, type_ref,",
          "attributes (matching the metadata_fields for that type), and a",
          "confidence score (0.0-1.0).",
          "",
          "Entity types:",
      ]

      for schema_entry in entity_schema:
          label = schema_entry.get("label", "Unknown")
          type_ref = schema_entry.get("type_ref", "")
          metadata_fields = schema_entry.get("metadata_fields", [])
          lines.append(f"- {label} (type_ref: {type_ref})")
          if metadata_fields:
              lines.append(f"  fields: {', '.join(metadata_fields)}")

      lines += [
          "",
          'Output JSON: {"entities": [{"name": "...", "type_ref": "...", "attributes": {...}, "confidence": 0.9}]}',
          "",
          "Rules:",
          "- Only extract entities explicitly mentioned in the text",
          "- Use the exact type_ref from the schema",
          "- Leave unknown attributes as null",
          "- Confidence: 1.0 = explicitly named, 0.7 = strongly implied, 0.5 = weakly implied",
          f"- {language_instruction}",
      ]

      return "\n".join(lines)


  def extract_entities_llm(
      text: str,
      entity_schema: list[dict],
      llm_chat_json: Callable[[list[dict]], dict],
      language_instruction: str = "Respond in English.",
  ) -> list[EntityCandidate]:
      """Extrae entidades de un chunk de texto usando un LLM inyectado.

      Construye un system prompt con el schema de entity types, llama al LLM
      y valida la respuesta retornando una lista de EntityCandidate.

      Args:
          text: Chunk de texto a analizar.
          entity_schema: Lista de tipos con metadata fields. Cada entrada es un
              dict con las claves 'type_ref', 'label' y opcionalmente
              'metadata_fields'. Ejemplo:
              [{"type_ref": "osint_person_go_cybersecurity", "label": "Person",
                "metadata_fields": ["full_name", "alias"]}]
          llm_chat_json: Funcion que recibe una lista de mensajes OpenAI-style
              y retorna un dict con la respuesta JSON del LLM. Interfaz:
              llm_chat_json([{"role": "system", "content": "..."}, ...]) -> dict
          language_instruction: Instruccion de idioma para el LLM. Por defecto
              "Respond in English."

      Returns:
          Lista de EntityCandidate extraidos. Retorna lista vacia si el LLM
          no retorna JSON valido o si no se encuentran entidades.

      Raises:
          ValueError: Si entity_schema esta vacio.
      """
      if not entity_schema:
          raise ValueError("entity_schema no puede estar vacio")

      valid_type_refs = {entry.get("type_ref", "") for entry in entity_schema}
      type_ref_to_label = {
          entry.get("type_ref", ""): entry.get("label", "") for entry in entity_schema
      }

      system_prompt = _build_system_prompt(entity_schema, language_instruction)

      messages = [
          {"role": "system", "content": system_prompt},
          {"role": "user", "content": text},
      ]

      try:
          response = llm_chat_json(messages)
      except Exception as exc:
          warnings.warn(f"extract_entities_llm: error llamando al LLM: {exc}", stacklevel=2)
          return []

      raw_entities = response.get("entities", [])
      if not isinstance(raw_entities, list):
          warnings.warn(
              "extract_entities_llm: la respuesta del LLM no contiene 'entities' como lista",
              stacklevel=2,
          )
          return []

      candidates: list[EntityCandidate] = []
      for item in raw_entities:
          if not isinstance(item, dict):
              continue

          name = item.get("name", "")
          if not name:
              continue

          type_ref = item.get("type_ref", "")
          if type_ref not in valid_type_refs:
              warnings.warn(
                  f"extract_entities_llm: type_ref '{type_ref}' no esta en el schema, descartando entidad '{name}'",
                  stacklevel=2,
              )
              continue

          attributes = item.get("attributes", {})
          if not isinstance(attributes, dict):
              attributes = {}
          # Normalizar null values a None
          attributes = {k: v for k, v in attributes.items() if v is not None}

          confidence = item.get("confidence", 0.0)
          if not isinstance(confidence, (int, float)):
              confidence = 0.0
          confidence = float(max(0.0, min(1.0, confidence)))

          candidates.append(
              EntityCandidate(
                  name=name,
                  type_ref=type_ref,
                  type_label=type_ref_to_label.get(type_ref, ""),
                  attributes=attributes,
                  confidence=confidence,
              )
          )

      return candidates

params_schema:
  {"params":[{"name":"text","desc":"texto fuente para extraccion (ej: documento, parrafo, chunk de OSINT). Puede contener multiples entidades."},{"name":"entity_schema","desc":"lista de dicts describiendo tipos validos (ej: [{'type_ref': 'osint_person_go_cybersecurity', 'label': 'Person'}])"},{"name":"llm_chat_json","desc":"callable que toma list[dict] con messages OpenAI-format y retorna dict con clave 'entities'. Inyeccion de dependencia del LLM."},{"name":"language_instruction","desc":"instruccion de lenguaje para el LLM (defecto: 'Respond in English.'). Ej: 'Responde en Español.'"}],"output":"lista de EntityCandidate extraidas, cada una con name, type_ref, confidence, atributos"}


id: extract_relations_llm_py_datascience
name: extract_relations_llm
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: impure
tags: ["extraction","relation","llm","knowledge-graph","nlp","datascience","fuzzygraph","graph"]
uses_functions: []
uses_types: ["entity_candidate_py_datascience","relation_candidate_py_datascience"]
returns: ["relation_candidate_py_datascience"]
error_type: error_go_core
imports: ["logging","sys","os","typing"]
test_file_path: python/functions/datascience/extract_relations_llm_test.py
file_path: python/functions/datascience/extract_relations_llm.py
created_at: 2026-04-05T11:24:00Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 19cf96165dba8277bc9c7493d53df9596fdc82cda205efe351b6df82b5a470c9
signature:
  def extract_relations_llm(text: str, entities: list, relation_types: list[str], llm_chat_json: Callable[[list[dict]], dict], language_instruction: str = 'Respond in English.') -> list

description:
  Extrae relaciones entre entidades de un chunk de texto usando un LLM inyectado. Valida que from_name y to_name correspondan a entidades existentes, y usa 'related_to' como fallback para tipos de relacion no permitidos.

example:
  ```python
  from extract_relations_llm import extract_relations_llm
  from python.types.datascience.entity_candidate import EntityCandidate

  # Stub de llm_chat_json (en produccion usar llm_completion_retry o similar)
  def my_llm(messages: list[dict]) -> dict:
      # Llamar al LLM real aqui
      return {"relations": [...]}

  entities = [
      EntityCandidate(name="Acme Corp", type_label="Organization", confidence=0.95),
      EntityCandidate(name="John Smith", type_label="Person", confidence=0.9),
  ]

  relation_types = ["employs", "funds", "owns", "communicates_with", "related_to"]

  relations = extract_relations_llm(
      text="Acme Corp employs John Smith as CEO and funds his research.",
      entities=entities,
      relation_types=relation_types,
      llm_chat_json=my_llm,
  )

  for rel in relations:
      print(f"{rel.from_name} --[{rel.relation_type}]--> {rel.to_name} ({rel.confidence:.2f})")
  # Acme Corp --[employs]--> John Smith (0.90)
  # Acme Corp --[funds]--> John Smith (0.85)
  ```

tests:
  ["texto con dos entidades relacionadas","texto con entidades pero sin relacion","menos de dos entidades retorna lista vacia","llm inventa entidad que no existe se descarta"]

notes:
  **Inyeccion de dependencia del LLM:** `llm_chat_json` recibe una lista de mensajes en formato OpenAI (`[{"role": "system", "content": ...}, {"role": "user", "content": ...}]`) y retorna un dict con la clave `"relations"`. Esto desacopla la funcion de cualquier proveedor de LLM concreto.

  **Validacion de entidades:** Solo se aceptan relaciones donde `from_name` y `to_name` aparecen exactamente en los nombres de las entidades proporcionadas. Relaciones con nombres inventados por el LLM se descartan silenciosamente (con debug log).

  **Fallback de tipo:** Si el LLM propone un `relation_type` que no esta en la lista permitida, se reemplaza por `"related_to"`. Si `"related_to"` tampoco esta en la lista, se incluye igualmente como catch-all seguro.

  **Menos de 2 entidades:** La funcion retorna `[]` inmediatamente sin llamar al LLM, ya que no puede haber relaciones con menos de 2 participantes.

  **Error handling:** Si `llm_chat_json` lanza una excepcion, se captura con warning y retorna `[]`. Si la respuesta no contiene la clave `"relations"` o no es una lista, idem.

  **Confianza:** Los valores de confianza del LLM se clampean al rango `[0.0, 1.0]`. Valores no numericos se convierten a `0.0`.

  Disenado para fuzzygraph — se compone con `extract_entities_llm` (paso anterior) y `deduplicate_relations` (paso siguiente en el pipeline de extraccion).

code:
  """extract_relations_llm — extrae relaciones entre entidades usando un LLM."""

  import logging
  import sys
  import os
  from typing import Callable

  sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", ""))

  from python.types.datascience.entity_candidate import EntityCandidate
  from python.types.datascience.relation_candidate import RelationCandidate

  logger = logging.getLogger(__name__)


  def extract_relations_llm(
      text: str,
      entities: list[EntityCandidate],
      relation_types: list[str],
      llm_chat_json: Callable[[list[dict]], dict],
      language_instruction: str = "Respond in English.",
  ) -> list[RelationCandidate]:
      """Extrae relaciones entre entidades de un chunk de texto usando un LLM.

      Dado el texto original y las entidades ya extraidas, pide al LLM que
      identifique relaciones entre pares de entidades. Las relaciones cuyo
      from_name o to_name no coincidan con ninguna entidad existente se descartan.
      Los tipos de relacion no permitidos se reemplazan por "related_to".

      Args:
          text: chunk de texto (el mismo que se uso para extraer las entidades).
          entities: entidades ya extraidas del chunk.
          relation_types: tipos de relacion permitidos, ej: ["funds", "employs",
              "communicates_with", "owns", "related_to"].
          llm_chat_json: funcion inyectada que recibe una lista de mensajes
              (dicts con "role" y "content") y retorna un dict con la respuesta
              JSON del LLM.
          language_instruction: instruccion de idioma para el LLM.

      Returns:
          Lista de RelationCandidate validados. Vacia si hay menos de 2 entidades
          o si el LLM no encuentra relaciones.
      """
      if len(entities) < 2:
          return []

      entity_names = {e.name for e in entities}
      relation_types_set = set(relation_types)

      # Construir lista de entidades para el prompt
      entity_lines = "\n".join(
          f'- "{e.name}" ({e.type_label or e.type_ref or "Entity"})' for e in entities
      )

      # Construir tipos de relacion para el prompt
      relation_types_str = ", ".join(relation_types)

      system_prompt = f"""\
  You are a relation extraction expert. Given text and a list of entities already \
  extracted, identify relationships between them.

  Entities found in this text:
  {entity_lines}

  Allowed relation types: {relation_types_str}

  Output JSON: {{"relations": [
    {{"from_name": "Entity A", "to_name": "Entity B",
     "relation_type": "employs", "description": "...", "confidence": 0.8}}
  ]}}

  Rules:
  - Only extract relations explicitly stated or strongly implied in the text
  - from_name and to_name must match entity names exactly as listed above
  - relation_type must be one of the allowed types
  - Confidence: 1.0 = explicitly stated, 0.7 = strongly implied, 0.5 = weakly implied
  - Do not invent entities not in the list above
  - {language_instruction}"""

      messages = [
          {"role": "system", "content": system_prompt},
          {"role": "user", "content": text},
      ]

      try:
          response = llm_chat_json(messages)
      except Exception as exc:
          logger.warning("extract_relations_llm: LLM call failed: %s", exc)
          return []

      raw_relations = response.get("relations", [])
      if not isinstance(raw_relations, list):
          logger.warning("extract_relations_llm: 'relations' is not a list in LLM response")
          return []

      results: list[RelationCandidate] = []
      for item in raw_relations:
          if not isinstance(item, dict):
              continue

          from_name = item.get("from_name", "")
          to_name = item.get("to_name", "")

          # Validar que ambos nombres corresponden a entidades existentes
          if from_name not in entity_names:
              logger.debug(
                  "extract_relations_llm: from_name '%s' no coincide con ninguna entidad — descartando",
                  from_name,
              )
              continue
          if to_name not in entity_names:
              logger.debug(
                  "extract_relations_llm: to_name '%s' no coincide con ninguna entidad — descartando",
                  to_name,
              )
              continue

          relation_type = item.get("relation_type", "")
          if relation_type not in relation_types_set:
              logger.debug(
                  "extract_relations_llm: tipo '%s' no permitido — usando 'related_to'",
                  relation_type,
              )
              relation_type = "related_to"

          confidence = item.get("confidence", 0.0)
          if not isinstance(confidence, (int, float)):
              confidence = 0.0
          confidence = float(max(0.0, min(1.0, confidence)))

          results.append(
              RelationCandidate(
                  from_name=from_name,
                  to_name=to_name,
                  relation_type=relation_type,
                  description=item.get("description", ""),
                  confidence=confidence,
              )
          )

      return results

params_schema:
  {"params":[{"name":"text","desc":"texto fuente para extraccion de relaciones entre entidades (ej: documento OSINT, parrafo descriptivo)"},{"name":"entities","desc":"lista de EntityCandidate ya extraidas (ej: output de extract_entities_llm). Valida que las relaciones refieran entidades reales."},{"name":"relation_types","desc":"lista de nombres de relacion permitidos (ej: ['employs', 'funds', 'owns']). No permitidas se remapean a 'related_to'."},{"name":"llm_chat_json","desc":"callable que toma list[dict] mensajes y retorna dict con clave 'relations'. Inyeccion de dependencia del LLM."},{"name":"language_instruction","desc":"instruccion de lenguaje para el LLM (defecto: 'Respond in English.')"}],"output":"lista de RelationCandidate con from_name, to_name, relation_type, description, confidence"}


id: histogram_py_datascience
name: histogram
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def histogram(data: list, buckets: int) -> list
description: Calcula histograma con N buckets. Retorna lista de conteos por bucket.
tags: ["statistics","histogram","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  histogram([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 5)
  # [2, 2, 2, 2, 2]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 5523c90f59bd399840179c0047db315e9f8447f955d50cde3bdd9ad9669a7db3
notes:
  Los buckets cubren el rango [min, max] uniformemente. El ultimo bucket incluye el valor maximo. Si todos los valores son iguales, todos caen en el primer bucket.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos para agrupar en buckets"},{"name":"buckets","desc":"numero de buckets a crear (ej: 5, 10, 100). Mayor = histograma mas detallado."}],"output":"lista de conteos con len(resultado) == buckets, cada elemento es el numero de valores en ese bucket"}


id: hotness_score_py_datascience
name: hotness_score
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["ranking","decay","recency","frequency","scoring","cache","memory","datascience"]
uses_functions: []
uses_types: []
returns: []
imports: ["math","datetime"]
test_file_path: python/functions/datascience/hotness_score_test.py
file_path: python/functions/datascience/hotness_score.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 6430c0e7d781c9470fb69d6a84ba24465bc9a9fbf8dcd4a2f71849680b2e9649
signature:
  def hotness_score(active_count: int, updated_at: datetime | None, now: datetime | None = None, half_life_days: float = 7.0) -> float

description:
  Calcula un score de hotness combinando frecuencia de acceso y recencia temporal. Util para ranking de resultados, memoria hot/cold y cache eviction.

example:
  ```python
  from datetime import datetime, timedelta, timezone
  from datascience.hotness_score import hotness_score

  now = datetime.now(timezone.utc)

  # Item reciente con muchos accesos -> score alto
  score = hotness_score(active_count=150, updated_at=now - timedelta(hours=2), now=now)
  # score > 0.95

  # Item antiguo aunque muy accedido -> score bajo
  score = hotness_score(active_count=150, updated_at=now - timedelta(days=30), now=now)
  # score ~ 0.05

  # Item sin fecha -> siempre 0
  score = hotness_score(active_count=999, updated_at=None)
  # score == 0.0
  ```

tests:
  ["active_count=0, updated_at reciente","active_count=100, updated_at reciente (score alto)","active_count=100, updated_at hace 30 dias (score bajo)","updated_at=None (retorna 0.0)","now explicito (determinista para tests)","half_life_days custom"]

notes:
  Formula: `score = sigmoid(log1p(active_count)) * exp(-ln(2)/half_life_days * age_days)`

  **Componente de frecuencia** — `sigmoid(log1p(count))` mapea enteros no negativos al rango `(0.5, 1.0)`:
  - count=0 -> 0.5
  - count=10 -> ~0.92
  - count=100 -> ~0.99

  **Componente de recencia** — decaimiento exponencial con vida media configurable:
  - `half_life_days=7` (default): score se reduce a la mitad cada 7 dias
  - `half_life_days=1`: decaimiento agresivo (util para feeds en tiempo real)
  - `half_life_days=365`: decaimiento lento (util para contenido evergreen)

  **Propiedades del score:**
  - `updated_at=None` -> 0.0 siempre (item sin fecha no tiene hotness)
  - `active_count=0, reciente` -> ~0.5 (neutro pero fresco)
  - `active_count alto, reciente` -> ~1.0 (muy caliente)
  - `active_count alto, antiguo` -> ~0.0 (frio a pesar de popularidad pasada)

  Timestamps sin timezone se interpretan como UTC. Pasar `now` explicitamente garantiza determinismo en tests y reproducibilidad en pipelines batch.

  Fuente conceptual: openviking/retrieve/memory_lifecycle.py (AGPL-3.0). Reimplementado desde cero con formula equivalente.

code:
  """Hotness score — combining access frequency and recency decay."""

  import math
  from datetime import datetime, timezone


  def hotness_score(
      active_count: int,
      updated_at: datetime | None,
      now: datetime | None = None,
      half_life_days: float = 7.0,
  ) -> float:
      """Calcula un score de hotness combinando frecuencia de acceso y recencia.

      Formula: sigmoid(log1p(active_count)) * exp_decay(age_days, half_life_days)

      El componente de frecuencia mapea conteos enteros al rango (0, 1) via sigmoid(log1p).
      El componente de recencia decae exponencialmente con vida media configurable.

      Args:
          active_count: Numero de accesos o activaciones. Debe ser >= 0.
          updated_at: Timestamp de la ultima actualizacion. None retorna 0.0.
          now: Momento de referencia para calcular la edad. Si es None usa datetime.now(UTC).
          half_life_days: Dias para que la recencia se reduzca a la mitad. Default 7.

      Returns:
          float en [0.0, 1.0]. Valores mas cercanos a 1.0 indican mayor hotness.
      """
      if updated_at is None:
          return 0.0

      # Componente de frecuencia: sigmoid(log1p(count)) mapea 0..inf -> (0.5, 1.0)
      freq = 1.0 / (1.0 + math.exp(-math.log1p(active_count)))

      # Componente de recencia: decaimiento exponencial
      if now is None:
          now = datetime.now(timezone.utc)

      # Normalizar ambos timestamps a UTC para comparacion segura
      if updated_at.tzinfo is None:
          updated_at = updated_at.replace(tzinfo=timezone.utc)
      if now.tzinfo is None:
          now = now.replace(tzinfo=timezone.utc)

      age_days = max((now - updated_at).total_seconds() / 86400.0, 0.0)
      decay_rate = math.log(2) / half_life_days
      recency = math.exp(-decay_rate * age_days)

      return freq * recency

params_schema:
  {"params":[{"name":"active_count","desc":"numero de accesos/activaciones de un item (ej: 150 views). Mayor count = score mas alto."},{"name":"updated_at","desc":"datetime de la ultima actualizacion. Si None, retorna siempre 0.0."},{"name":"now","desc":"datetime actual (defecto: ahora). Usar parametro explicitamente para tests deterministas."},{"name":"half_life_days","desc":"vida media para el decaimiento exponencial (defecto: 7.0 dias). Menor = decaimiento mas rapido."}],"output":"hotness score en rango [0, 1]. Combina frecuencia (active_count) y recencia (dias desde updated_at)."}


id: impute_py_datascience
name: impute
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def impute(data: list) -> list
description: Reemplaza None y NaN con la media de los valores validos.
tags: ["imputation","missing","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
example:
  ```python
  impute([1.0, None, 3.0, float('nan'), 5.0])
  # [1.0, 3.0, 3.0, 3.0, 5.0]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Detecta tanto None como float('nan'). Si no hay valores validos, rellena con 0.0.
content_hash: 5cacced1e58a2da837920b3cbce0f64c32056756abc286deb244e306663c70c3
code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos con posibles None o NaN que requieren imputacion"}],"output":"lista de misma longitud con None y NaN reemplazados por la media de los valores validos"}


id: linspace_py_datascience
name: linspace
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def linspace(start: float, stop: float, num: int) -> list
description: Genera una lista de valores equiespaciados entre start y stop (inclusivos).
tags: ["numeric","range","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  linspace(0, 1, 5)
  # [0.0, 0.25, 0.5, 0.75, 1.0]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: d827ce463256c49f54cf85cc4d578e33b7580eafdd95e0546909ac9e50ddc11f
notes:
  Equivalente a numpy.linspace pero sin numpy. Si num=1, retorna [start]. Si num<=0, retorna lista vacia.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"start","desc":"valor inicial del rango (ej: 0.0)"},{"name":"stop","desc":"valor final del rango, incluido (ej: 1.0)"},{"name":"num","desc":"numero de puntos a generar (ej: 5). Si num=1, retorna [start]."}],"output":"lista de num valores equiespaciados entre start y stop (ambos inclusive)"}


id: melt_py_datascience
name: melt
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["datascience","tabular","melt","unpivot","transform","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/datascience/melt_test.py
file_path: python/functions/datascience/melt.py
created_at: 2026-04-05T11:24:20Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 3ac8a87ccb97843a16cde227ffd760296be3b83e4f4af9a6527a3abee2e4b425
signature:
  def melt(rows: list[dict], id_vars: list[str], value_vars: list[str] | None = None, var_name: str = 'variable', value_name: str = 'value') -> list[dict]

description:
  Inversa de pivot. Convierte columnas en filas (formato largo). Cada combinacion de id_vars + value_var genera una fila. Si value_vars es None, derrite todas las columnas no-id.

example:
  ```python
  rows = [{"region": "US", "q1": 10, "q2": 20}]
  melt(rows, id_vars=["region"], value_vars=["q1", "q2"])
  # [{"region": "US", "variable": "q1", "value": 10},
  #  {"region": "US", "variable": "q2", "value": 20}]
  ```

tests:
  ["Melt basico","Multiples id_vars","value_vars None derrite todas las columnas no-id","Fila con campo faltante en value_vars"]

notes:
  Funcion pura sin dependencias externas.
  Si un campo de value_vars no existe en la fila, su valor sera None.
  El parametro value_vars=None es util cuando se desconoce el schema exacto.

code:
  """Melt (unpivot) para datos tabulares list[dict]."""


  def melt(
      rows: list[dict],
      id_vars: list[str],
      value_vars: list[str] | None = None,
      var_name: str = "variable",
      value_name: str = "value",
  ) -> list[dict]:
      """Convierte columnas en filas (formato largo). Inversa de pivot.

      Cada combinacion de id_vars + value_var genera una fila nueva.
      Si value_vars es None, se usan todas las columnas que no esten en id_vars.

      Args:
          rows: Lista de dicts en formato ancho.
          id_vars: Columnas que se mantienen como identificadores en cada fila.
          value_vars: Columnas a convertir en filas. None = todas las no-id.
          var_name: Nombre de la columna que contendra los nombres de variables.
          value_name: Nombre de la columna que contendra los valores.

      Returns:
          Lista de dicts en formato largo con una fila por combinacion id+variable.
      """
      result = []
      for row in rows:
          # Determinar que columnas derretir
          if value_vars is None:
              vars_to_melt = [k for k in row if k not in id_vars]
          else:
              vars_to_melt = value_vars

          for var in vars_to_melt:
              new_row: dict = {k: row.get(k) for k in id_vars}
              new_row[var_name] = var
              new_row[value_name] = row.get(var)
              result.append(new_row)

      return result

params_schema:
  {"params":[{"name":"rows","desc":"lista de dicts en formato ancho (ej: [{'region': 'US', 'q1': 10, 'q2': 20}])"},{"name":"id_vars","desc":"lista de columnas a mantener como identificadores (ej: ['region']). No se derriten."},{"name":"value_vars","desc":"lista de columnas a derretir (ej: ['q1', 'q2']). Si None, derrite todas excepto id_vars."},{"name":"var_name","desc":"nombre de la columna que contendra los nombres de columnas derretidas (defecto: 'variable')"},{"name":"value_name","desc":"nombre de la columna que contendra los valores (defecto: 'value')"}],"output":"lista de dicts en formato largo, donde cada id_var + value_var genera una fila"}


id: merge_graphs_py_datascience
name: merge_graphs
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["graph","merge","deduplication","knowledge-graph","levenshtein","similarity","datascience"]
uses_functions: ["levenshtein_distance_py_cybersecurity"]
uses_types: []
returns: []
imports: ["sys","os"]
test_file_path: python/functions/datascience/merge_graphs_test.py
file_path: python/functions/datascience/merge_graphs.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c342adbb10829f29f565933fa1eaa28b8d8cc6c119d570bec0bdac4cf61f5553
signature:
  def merge_graphs(graphs: list[dict], entity_key: str = 'name', similarity_threshold: float = 0.85) -> dict

description:
  Mergea multiples grafos de conocimiento en uno deduplicando entities por similitud de nombre (Levenshtein normalizado). Relaciones se re-apuntan a las entities canonicas. Atributos se combinan por union.

example:
  ```python
  g1 = {
      "entities": [
          {"id": "1", "name": "Alice Corp", "type": "company"},
          {"id": "2", "name": "Bob", "type": "person"},
      ],
      "relations": [
          {"source_id": "2", "target_id": "1", "relation_type": "works_at"},
      ],
  }
  g2 = {
      "entities": [
          {"id": "3", "name": "Alice Corp.", "type": "company", "country": "US"},
      ],
      "relations": [],
  }

  result = merge_graphs([g1, g2], similarity_threshold=0.85)
  # result["entities"] -> 2 entities (Alice Corp mergeada, Bob)
  # result["merge_log"] -> [{"merged": ["3", "1"], "into": "1", "similarity": 0.909}]
  # "Alice Corp." mergeada en "Alice Corp" porque similitud > 0.85
  ```

tests:
  ["dos grafos con entity duplicada → merge","entities similares pero bajo threshold → no merge","relaciones re-apuntadas correctamente","merge log registra cada merge","tres grafos → merge transitivo","grafos sin overlap → concatenacion simple"]

notes:
  Funcion pura. Reutiliza `levenshtein_distance_py_cybersecurity` para calcular similitud normalizada entre nombres.

  **Algoritmo de merge transitivo**: si A~B y B~C, entonces A, B, C se mergean en uno solo. Se implementa via union-find (path compression simple).

  **Eleccion de canonical**: la entity con mas campos no-null gana. En caso de empate, la primera encontrada en el par.

  **Conflictos de atributos**: si ambas entities tienen un campo con valor, el canonical conserva el suyo (primero gana). Solo se copian campos que el canonical no tiene o tiene null.

  **Deduplicacion de relaciones**: por (source_id, target_id, relation_type). Si dos relaciones son identicas tras re-apuntar los IDs, se conserva la primera encontrada.

  **Complejidad**: O(n^2) en numero de entities por la comparacion de pares. Adecuado para grafos de knowledge tipicos (< 10K entities). Para grafos muy grandes, usar indexado por prefijo antes de comparar.

  **Importacion**: intenta importar `levenshtein_distance` desde el paquete `cybersecurity` del registry. Si no esta disponible, usa una reimplementacion inline equivalente.

code:
  """merge_graphs — mergea multiples grafos de conocimiento deduplicando entities por similitud."""

  import sys
  import os

  # Importar levenshtein_distance desde el registry
  sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "cybersecurity"))
  try:
      from cybersecurity import levenshtein_distance
  except ImportError:
      # Fallback: reimplementacion inline si el paquete no esta disponible
      def levenshtein_distance(a: str, b: str) -> int:
          """Calcula la distancia de Levenshtein entre dos strings."""
          if len(a) < len(b):
              return levenshtein_distance(b, a)
          if len(b) == 0:
              return len(a)
          prev_row = list(range(len(b) + 1))
          for i, ca in enumerate(a):
              curr_row = [i + 1]
              for j, cb in enumerate(b):
                  cost = 0 if ca == cb else 1
                  curr_row.append(
                      min(curr_row[j] + 1, prev_row[j + 1] + 1, prev_row[j] + cost)
                  )
              prev_row = curr_row
          return prev_row[-1]


  def _name_similarity(a: str, b: str) -> float:
      """Similitud de Levenshtein normalizada entre 0 y 1."""
      if not a and not b:
          return 1.0
      max_len = max(len(a), len(b))
      if max_len == 0:
          return 1.0
      dist = levenshtein_distance(a.lower(), b.lower())
      return 1.0 - dist / max_len


  def _count_non_null_fields(entity: dict) -> int:
      """Cuenta campos con valor no-None."""
      return sum(1 for v in entity.values() if v is not None)


  def _merge_two_entities(canonical: dict, other: dict) -> dict:
      """Combina dos entities: union de campos, ultimo gana en conflictos."""
      merged = dict(canonical)
      for k, v in other.items():
          if k not in merged or merged[k] is None:
              merged[k] = v
          # Si ambos tienen valor, el canonical (primero) gana — no sobreescribir
      return merged


  def merge_graphs(
      graphs: list[dict],
      entity_key: str = "name",
      similarity_threshold: float = 0.85,
  ) -> dict:
      """Mergea multiples grafos de conocimiento en uno, deduplicando entities por similitud.

      Algoritmo:
      1. Juntar todas las entities de todos los grafos (con ID de origen).
      2. Para cada par con similitud de nombre >= threshold, mergear.
      3. Elegir entity canonica (la que tiene mas campos no-null).
      4. Re-apuntar relaciones al ID canonico.
      5. Deduplicar relaciones identicas (mismo source, target, type).
      6. Registrar cada merge en merge_log.

      Args:
          graphs: Lista de grafos. Cada grafo es un dict con keys:
                  "entities" (list[dict]) y "relations" (list[dict]).
                  Las entities deben tener "id" y el campo entity_key.
          entity_key: Campo de texto usado para calcular similitud. Default "name".
          similarity_threshold: Umbral de similitud Levenshtein normalizada [0,1].
                                Default 0.85.

      Returns:
          Dict con keys: entities, relations, merge_log.
      """
      # Recopilar todas las entities y relaciones
      all_entities: list[dict] = []
      all_relations: list[dict] = []

      for graph in graphs:
          all_entities.extend(graph.get("entities", []))
          all_relations.extend(graph.get("relations", []))

      # Construir union-find para agrupar entities similares
      # id_map: entity_id original -> entity_id canonico
      id_map: dict[str, str] = {e["id"]: e["id"] for e in all_entities if "id" in e}
      entity_by_id: dict[str, dict] = {e["id"]: e for e in all_entities if "id" in e}

      merge_log: list[dict] = []

      def find_canonical(eid: str) -> str:
          while id_map.get(eid, eid) != eid:
              eid = id_map[eid]
          return eid

      entity_ids = [e["id"] for e in all_entities if "id" in e]

      # Comparar todos los pares (O(n^2) — aceptable para grafos de knowledge tipicos)
      for i in range(len(entity_ids)):
          for j in range(i + 1, len(entity_ids)):
              id_i = find_canonical(entity_ids[i])
              id_j = find_canonical(entity_ids[j])

              if id_i == id_j:
                  continue  # ya mergeados

              e_i = entity_by_id.get(id_i)
              e_j = entity_by_id.get(id_j)

              if e_i is None or e_j is None:
                  continue

              name_i = str(e_i.get(entity_key, ""))
              name_j = str(e_j.get(entity_key, ""))

              sim = _name_similarity(name_i, name_j)
              if sim >= similarity_threshold:
                  # Elegir canonical: el que tiene mas campos no-null
                  if _count_non_null_fields(e_i) >= _count_non_null_fields(e_j):
                      canonical_id, other_id = id_i, id_j
                  else:
                      canonical_id, other_id = id_j, id_i

                  # Mergear datos
                  merged = _merge_two_entities(entity_by_id[canonical_id], entity_by_id[other_id])
                  entity_by_id[canonical_id] = merged

                  # Redirigir other_id -> canonical_id
                  id_map[other_id] = canonical_id

                  merge_log.append({
                      "merged": [other_id, canonical_id],
                      "into": canonical_id,
                      "similarity": round(sim, 4),
                  })

      # Construir lista final de entities (solo canonicas)
      canonical_ids = {eid for eid in entity_ids if find_canonical(eid) == eid}
      final_entities = [entity_by_id[eid] for eid in canonical_ids if eid in entity_by_id]

      # Re-apuntar relaciones a IDs canonicos
      final_relations_set: dict[tuple, dict] = {}
      for rel in all_relations:
          new_rel = dict(rel)
          if "source_id" in new_rel:
              new_rel["source_id"] = find_canonical(new_rel["source_id"])
          if "target_id" in new_rel:
              new_rel["target_id"] = find_canonical(new_rel["target_id"])

          # Deduplicar por (source_id, target_id, relation_type)
          rel_key = (
              new_rel.get("source_id", ""),
              new_rel.get("target_id", ""),
              new_rel.get("relation_type", ""),
          )
          if rel_key not in final_relations_set:
              final_relations_set[rel_key] = new_rel

      return {
          "entities": final_entities,
          "relations": list(final_relations_set.values()),
          "merge_log": merge_log,
      }

params_schema:
  {"params":[{"name":"graphs","desc":"lista de grafos, cada uno con estructura {entities: [...], relations: [...]}"},{"name":"entity_key","desc":"campo de entity que se usa para fuzzy matching (defecto: 'name'). Determina similitud."},{"name":"similarity_threshold","desc":"umbral de similitud Levenshtein normalizado [0, 1] para mergear (tipico: 0.85). Mayor = mas estricto."}],"output":"dict con estructura {entities: [...], relations: [...], merge_log: [...]}, grafo mergeado y deduplicado"}


id: min_max_scale_py_datascience
name: min_max_scale
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def min_max_scale(data: list) -> list
description: Escala los valores al rango [0, 1] usando min-max normalization.
tags: ["normalization","scaling","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  min_max_scale([2, 4, 6, 8, 10])
  # [0.0, 0.25, 0.5, 0.75, 1.0]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Si todos los valores son iguales, retorna lista de ceros. No requiere imports externos.
content_hash: 6c15746334c2be6df2dcac5fd563470881b77bd40571ba1bb48b52c6ac502409
code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos a normalizar en el rango [0, 1]"}],"output":"lista de valores escalados al rango [0, 1] usando min-max normalization. Min-\u003e0, Max-\u003e1, valores intermedios proporcionales."}


id: ops_to_rdf_triples_py_datascience
name: ops_to_rdf_triples
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["rdf","graph","osint","knowledge-graph","triples","operations","semantic-web"]
uses_functions: []
uses_types: []
returns: []
imports: ["json","sqlite3"]
tests: []
file_path: python/functions/datascience/ops_to_rdf_triples.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1d8b1f8894873853916c2e97972a14e1a8da125687b72f95558507e48d08072b
signature:
  def ops_to_rdf_triples(db_path: str, namespace: str = 'http://osint.local/') -> list[tuple[str, str, str]]

description:
  Convierte entities y relations de operations.db a triples RDF (subject, predicate, object). Prefija IDs con namespace para formar URIs. Solo stdlib.

example:
  ```python
  from datascience.ops_to_rdf_triples import ops_to_rdf_triples

  triples = ops_to_rdf_triples("apps/my_analysis/operations.db")
  for s, p, o in triples[:5]:
      print(f"{s} -- {p} --> {o}")

  # Con namespace personalizado
  triples_ns = ops_to_rdf_triples(
      "apps/my_analysis/operations.db",
      namespace="http://mi-empresa.com/osint/"
  )
  ```

notes:
  Funcion pura — solo abre la DB en lectura, no escribe nada.

  Triples generados por entidad:
  - `(ns+id, rdf:type, type_ref)` — si type_ref no es None
  - `(ns+id, name, literal)` — si name no es None
  - `(ns+id, status, literal)` — si status no es None
  - `(ns+id, domain, literal)` — si domain no es None
  - `(ns+id, key, str(value))` — por cada clave en el JSON de metadata

  Triples generados por relacion:
  - `(ns+from_entity, relation_name, ns+to_entity)`

  Los subjects de relaciones tipo URI reciben el prefijo de namespace. Los predicados literales (name, status, etc.) no llevan prefijo. Esta separacion sigue la convencion RDF de distinguir recursos de literales sin introducir dependencias externas (rdflib u otras).

  Para exportar a Turtle (.ttl) o N-Triples, el notebook puede iterar la lista y formatear segun necesite.

code:
  """Convierte operations.db a triples RDF (subject, predicate, object)."""

  import json
  import sqlite3


  def ops_to_rdf_triples(
      db_path: str,
      namespace: str = "http://osint.local/",
  ) -> list[tuple[str, str, str]]:
      """Convierte entities y relations de operations.db a triples RDF.

      Genera triples para:
      - Tipo de entidad: (entity_uri, rdf:type, type_ref)
      - Nombre: (entity_uri, name, literal)
      - Status y domain: (entity_uri, status|domain, literal)
      - Cada clave de metadata: (entity_uri, key, str(value))
      - Relaciones: (from_entity_uri, relation_name, to_entity_uri)

      Los IDs de entidades se prefijarlos con el namespace para formar URIs.

      Args:
          db_path: Ruta al archivo operations.db.
          namespace: Prefijo de namespace para construir URIs. Default: "http://osint.local/".

      Returns:
          Lista de tuplas (subject, predicate, object) representando los triples RDF.
      """
      conn = sqlite3.connect(db_path)
      conn.row_factory = sqlite3.Row
      try:
          cur = conn.cursor()
          cur.execute(
              "SELECT id, name, type_ref, status, domain, metadata FROM entities"
          )
          raw_entities = [dict(row) for row in cur.fetchall()]

          cur.execute(
              "SELECT id, name, from_entity, to_entity FROM relations"
          )
          raw_relations = [dict(row) for row in cur.fetchall()]
      finally:
          conn.close()

      ns = namespace.rstrip("/") + "/"
      triples: list[tuple[str, str, str]] = []

      for entity in raw_entities:
          subject = ns + entity["id"]

          # rdf:type
          if entity["type_ref"]:
              triples.append((subject, "rdf:type", entity["type_ref"]))

          # name
          if entity["name"]:
              triples.append((subject, "name", entity["name"]))

          # status
          if entity["status"]:
              triples.append((subject, "status", entity["status"]))

          # domain
          if entity["domain"]:
              triples.append((subject, "domain", entity["domain"]))

          # metadata keys
          try:
              meta = json.loads(entity["metadata"]) if entity["metadata"] else {}
          except (json.JSONDecodeError, TypeError):
              meta = {}

          for key, value in meta.items():
              triples.append((subject, key, str(value)))

      for rel in raw_relations:
          from_uri = ns + rel["from_entity"]
          to_uri = ns + rel["to_entity"]
          predicate = rel["name"] or rel["id"]
          triples.append((from_uri, predicate, to_uri))

      return triples

params_schema:
  {"params":[{"name":"db_path","desc":"path a operations.db (ej: 'apps/my_analysis/operations.db')"},{"name":"namespace","desc":"prefijo URI para formar IRIs (defecto: 'http://osint.local/'). Ej: 'http://mi-empresa.com/osint/'"}],"output":"lista de tuplas (subject, predicate, object) en formato RDF. Subjects y objetos son URIs con namespace, predicados son literales."}


id: ops_to_sigma_json_py_datascience
name: ops_to_sigma_json
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def ops_to_sigma_json(db_path: str) -> dict
tags: ["graph","sigma","osint","visualization","operations","network"]
uses_functions: []
uses_types: []
returns: []
imports: ["json","sqlite3"]
tests: []
file_path: python/functions/datascience/ops_to_sigma_json.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1da886662918ebed1202ec2eebda7e5c1445f9808a1a11dbf2b783cb71b62a37
description:
  Convierte operations.db al formato JSON de sigma.js/graphology. Lee entities y relations, asigna colores por tipo y calcula tamanio de nodo combinando degree y risk_score.

example:
  ```python
  from datascience.ops_to_sigma_json import ops_to_sigma_json

  graph = ops_to_sigma_json("apps/my_analysis/operations.db")
  print(len(graph["nodes"]))  # numero de entidades
  print(len(graph["edges"]))  # numero de relaciones
  ```

notes:
  Funcion pura en el sentido de que no escribe ningun archivo ni tiene efectos secundarios observables mas alla de abrir y cerrar la conexion SQLite en modo lectura.

  COLOR_MAP cubre los tipos de entidad OSINT mas comunes. Tipos desconocidos reciben `#95a5a6` (gris).

  El tamanio del nodo (`size`) se calcula en el rango [5, 20]:
  - Si la entidad tiene `risk_score` en metadata: `(degree_norm + risk_norm) / 2`
  - Si no: `degree_norm` puro
  - degree_norm = `min(degree / 50, 1.0)`

  La metadata se aplana como atributos adicionales del nodo, sin sobrescribir campos reservados (`label`, `type`, `color`, `size`, `domain`, `status`).

code:
  """Convierte operations.db al formato JSON de sigma.js para visualizacion de grafos."""

  import json
  import sqlite3


  COLOR_MAP = {
      "person": "#e74c3c",
      "organization": "#3498db",
      "ip_address": "#2ecc71",
      "domain": "#f39c12",
      "crypto_wallet": "#f1c40f",
      "trading_signal": "#9b59b6",
      "vulnerability": "#e67e22",
      "malware": "#c0392b",
      "email": "#1abc9c",
  }

  DEFAULT_COLOR = "#95a5a6"
  SIZE_MIN = 5.0
  SIZE_MAX = 20.0


  def _calculate_degree(entity_id: str, relations: list[dict]) -> int:
      """Cuenta cuantas relaciones involucran a esta entidad."""
      return sum(
          1 for r in relations
          if r["from_entity"] == entity_id or r["to_entity"] == entity_id
      )


  def _calculate_size(entity_id: str, metadata: dict, relations: list[dict]) -> float:
      """Calcula el tamanio del nodo basado en degree y risk_score opcional."""
      degree = _calculate_degree(entity_id, relations)

      # Normalizar degree a rango [0, 1] asumiendo maximo razonable de 50
      degree_norm = min(degree / 50.0, 1.0)

      if "risk_score" in metadata:
          try:
              risk_norm = float(metadata["risk_score"]) / 100.0
              risk_norm = max(0.0, min(risk_norm, 1.0))
              score = (degree_norm + risk_norm) / 2.0
          except (ValueError, TypeError):
              score = degree_norm
      else:
          score = degree_norm

      return SIZE_MIN + score * (SIZE_MAX - SIZE_MIN)


  def ops_to_sigma_json(db_path: str) -> dict:
      """Convierte operations.db al formato JSON esperado por sigma.js.

      Lee entities y relations de la base de datos de operaciones y construye
      el dict con nodos y aristas en el formato de graphology/sigma.js.
      El tamanio de cada nodo se calcula a partir de su degree en el grafo
      y, si esta disponible, de su risk_score en metadata (media 50/50).

      Args:
          db_path: Ruta al archivo operations.db.

      Returns:
          Dict con claves 'nodes' y 'edges' compatible con sigma.js / graphology.
      """
      conn = sqlite3.connect(db_path)
      conn.row_factory = sqlite3.Row
      try:
          cur = conn.cursor()

          cur.execute(
              "SELECT id, name, type_ref, status, domain, metadata FROM entities"
          )
          raw_entities = [dict(row) for row in cur.fetchall()]

          cur.execute(
              "SELECT id, name, from_entity, to_entity, weight FROM relations"
          )
          raw_relations = [dict(row) for row in cur.fetchall()]
      finally:
          conn.close()

      nodes = []
      for entity in raw_entities:
          try:
              meta = json.loads(entity["metadata"]) if entity["metadata"] else {}
          except (json.JSONDecodeError, TypeError):
              meta = {}

          type_ref = entity["type_ref"] or "unknown"
          color = COLOR_MAP.get(type_ref, DEFAULT_COLOR)
          size = _calculate_size(entity["id"], meta, raw_relations)

          attributes = {
              "label": entity["name"] or entity["id"],
              "entity_type": type_ref,
              "color": color,
              "size": size,
              "domain": entity["domain"] or "",
              "status": entity["status"] or "",
          }
          # Aplana metadata como atributos adicionales (sin sobrescribir campos reservados)
          reserved = {"label", "entity_type", "color", "size", "domain", "status", "type", "x", "y", "hidden", "zIndex"}
          for k, v in meta.items():
              if k not in reserved:
                  attributes[k] = v

          nodes.append({"key": entity["id"], "attributes": attributes})

      edges = []
      for rel in raw_relations:
          edges.append({
              "key": rel["id"],
              "source": rel["from_entity"],
              "target": rel["to_entity"],
              "attributes": {
                  "label": rel["name"] or "",
                  "weight": rel["weight"] if rel["weight"] is not None else 1.0,
                  "type": "arrow",
              },
          })

      return {"nodes": nodes, "edges": edges}

params_schema:
  {"params":[{"name":"db_path","desc":"path a operations.db (ej: 'apps/my_analysis/operations.db')"}],"output":"dict con estructura {nodes: [...], edges: [...]} compatible con sigma.js/graphology. Nodos tienen id, label, color, size, type."}


id: pearson_py_datascience
name: pearson
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def pearson(xs: list, ys: list) -> float
description: Calcula el coeficiente de correlacion de Pearson entre dos listas de floats.
tags: ["statistics","correlation","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
example:
  ```python
  r = pearson([1, 2, 3], [2, 4, 6])
  # r = 1.0
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: add599f370be793b9c4e93ae97e0354ff635c1f9c360e69f82a5c58ebaec4344
notes:
  Usa solo math stdlib. No requiere numpy. Retorna 0.0 si las listas tienen longitud diferente, estan vacias, o la desviacion es cero.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista de valores numericos de la primera variable (ej: [1, 2, 3])"},{"name":"ys","desc":"lista de valores numericos de la segunda variable, misma longitud que xs"}],"output":"coeficiente de correlacion de Pearson en rango [-1, 1]. 1.0=correlacion perfecta positiva, -1.0=negativa, 0.0=sin correlacion"}


id: pivot_py_datascience
name: pivot
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def pivot(rows: list[dict], index: str, columns: str, values: str, agg: str = 'sum') -> list[dict]
tags: ["datascience","tabular","pivot","transform","aggregation","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["collections"]
test_file_path: python/functions/datascience/pivot_test.py
file_path: python/functions/datascience/pivot.py
created_at: 2026-04-05T11:24:00Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f37c0f72afd2e874273638c296a8bf7f6e933237fe75c0b999fc28401c2a4a21
description:
  Pivot table sin pandas. Agrupa por index, expande valores unicos de columns como nuevas columnas y agrega values con la funcion indicada (sum, count, mean, min, max, first, last).

example:
  ```python
  rows = [
      {"region": "US", "product": "A", "sales": 10},
      {"region": "US", "product": "B", "sales": 20},
      {"region": "EU", "product": "A", "sales": 15},
  ]
  pivot(rows, index="region", columns="product", values="sales")
  # [{"region": "US", "A": 10, "B": 20}, {"region": "EU", "A": 15, "B": 0}]
  ```

tests:
  ["Pivot basico con sum","Pivot con count y mean","Valores faltantes rellenados con 0","Una sola fila","Multiples valores por celda requieren agregacion"]

notes:
  Funcion pura sin dependencias externas (solo collections.defaultdict de stdlib).
  Preserva el orden de aparicion de los valores de index y columns.
  Valores numericos faltantes se rellenan con 0; no numericos con None.

code:
  """Pivot table sin pandas para datos tabulares list[dict]."""

  from collections import defaultdict


  def pivot(
      rows: list[dict],
      index: str,
      columns: str,
      values: str,
      agg: str = "sum",
  ) -> list[dict]:
      """Transforma datos del formato largo al formato ancho (pivot table).

      Agrupa por `index`, expande los valores unicos de `columns` como nuevas
      columnas y agrega la columna `values` con la funcion indicada.

      Args:
          rows: Lista de dicts con los datos en formato largo.
          index: Nombre de la columna que actua como indice de filas.
          columns: Nombre de la columna cuyos valores unicos se convierten en columnas.
          values: Nombre de la columna cuyos valores se agregan.
          agg: Funcion de agregacion: sum, count, mean, min, max, first, last.

      Returns:
          Lista de dicts con una fila por valor unico de index y una columna
          por cada valor unico de columns. Valores numericos faltantes rellenados
          con 0, valores no numericos con None.
      """
      # Recopilar valores unicos de columns (orden de aparicion)
      col_values: list = []
      seen_cols: set = set()
      index_order: list = []
      seen_index: set = set()

      for row in rows:
          idx = row.get(index)
          col = row.get(columns)
          if idx not in seen_index:
              seen_index.add(idx)
              index_order.append(idx)
          if col not in seen_cols:
              seen_cols.add(col)
              col_values.append(col)

      # Acumular: groups[index_val][col_val] = lista de values
      groups: dict[any, dict[any, list]] = defaultdict(lambda: defaultdict(list))
      for row in rows:
          idx = row.get(index)
          col = row.get(columns)
          val = row.get(values)
          if val is not None:
              groups[idx][col].append(val)

      # Determinar si los valores son numericos (para relleno de 0)
      sample_vals = [v for g in groups.values() for vs in g.values() for v in vs]
      is_numeric = all(isinstance(v, (int, float)) for v in sample_vals) if sample_vals else True

      def _aggregate(vals: list, func: str):
          if not vals:
              return 0 if is_numeric else None
          if func == "sum":
              return sum(vals)
          if func == "count":
              return len(vals)
          if func == "mean":
              return sum(vals) / len(vals)
          if func == "min":
              return min(vals)
          if func == "max":
              return max(vals)
          if func == "first":
              return vals[0]
          if func == "last":
              return vals[-1]
          raise ValueError(f"Funcion de agregacion no soportada: {func}")

      result = []
      for idx in index_order:
          record: dict = {index: idx}
          for col in col_values:
              vals = groups[idx][col]
              if vals:
                  record[col] = _aggregate(vals, agg)
              else:
                  record[col] = 0 if is_numeric else None
          result.append(record)

      return result

params_schema:
  {"params":[{"name":"rows","desc":"lista de dicts en formato largo (ej: [{'region': 'US', 'product': 'A', 'sales': 10}, ...])"},{"name":"index","desc":"columna que sera el indice de filas en la tabla pivote (ej: 'region'). Valores unicos -\u003e filas."},{"name":"columns","desc":"columna cuyos valores unicos se expanden como nuevas columnas (ej: 'product'). Valores unicos -\u003e columnas."},{"name":"values","desc":"columna con los valores a agregar en las celdas (ej: 'sales')"},{"name":"agg","desc":"funcion de agregacion si hay multiples valores por celda (defecto: 'sum'). Otras: count, mean, min, max, first, last."}],"output":"lista de dicts en formato ancho, donde el indice y las columnas expandidas forman el schema"}


id: render_sigma_html_py_datascience
name: render_sigma_html
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: impure
signature: def render_sigma_html(graph_data: dict, output_path: str, title: str = 'OSINT Graph') -> str
tags: ["graph","sigma","osint","visualization","html","forceatlas2","network","dark-theme"]
uses_functions: ["ops_to_sigma_json_py_datascience"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["json","os"]
tests: []
file_path: python/functions/datascience/render_sigma_html.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 987efdc99385270e6dae28ea71bd359376f81e2b98eb27e131de3024474d5d79
description:
  Genera un archivo HTML standalone con sigma.js v2.4 que visualiza un grafo OSINT. Aplica ForceAtlas2, dark theme, filtros por tipo de nodo y tooltip con metadata. Retorna el path absoluto del archivo escrito.

example:
  ```python
  from datascience.ops_to_sigma_json import ops_to_sigma_json
  from datascience.render_sigma_html import render_sigma_html

  graph = ops_to_sigma_json("apps/osint_analysis/operations.db")
  path = render_sigma_html(graph, "/tmp/osint_graph.html", title="Red OSINT")
  print(f"Abre en el browser: {path}")
  ```

notes:
  Funcion impura — escribe un archivo en disco.

  El HTML es completamente standalone: no necesita servidor web ni backend. Todos los assets se cargan desde jsDelivr CDN:
  - graphology 0.25.4
  - graphology-layout-forceatlas2 0.10.1
  - sigma 2.4.0

  El JSON del grafo se embebe directamente en el `<script>` para evitar peticiones CORS al abrir desde `file://`.

  ForceAtlas2 se ejecuta sincrono con 500 iteraciones antes de instanciar el renderer. Para grafos grandes (>500 nodos) considerar reducir iteraciones o usar `barnesHutOptimize: true` (que se activa automaticamente para grafos >300 nodos).

  Panel lateral (fijo, top-right):
  - Titulo del grafo
  - Contador de nodos y aristas
  - Checkboxes de filtro por tipo (con color dot)

  Tooltip (hover sobre nodo):
  - Label y tipo del nodo
  - status, domain
  - Todos los campos adicionales de metadata aplanados

  Los caracteres especiales en labels y metadata se escapan correctamente para evitar XSS al abrir el HTML.

code:
  """Renderiza un grafo sigma.js como HTML standalone con dark theme y layout ForceAtlas2."""

  import json
  import os


  _HTML_TEMPLATE = """\
  <!DOCTYPE html>
  <html lang="en">
  <head>
      <meta charset="UTF-8" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>{title}</title>
      <script src="https://cdn.jsdelivr.net/npm/graphology@0.25.4/dist/graphology.umd.min.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/graphology-library@0.8.0/dist/graphology-library.min.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/sigma@2.4.0/build/sigma.min.js"></script>
      <style>
          * {{ box-sizing: border-box; margin: 0; padding: 0; }}
          body {{ background: #1a1a2e; color: #eee; font-family: 'Segoe UI', system-ui, sans-serif; overflow: hidden; }}
          #container {{ width: 100vw; height: 100vh; }}
          #panel {{
              position: absolute; top: 12px; right: 12px;
              background: rgba(10, 10, 30, 0.88);
              border: 1px solid rgba(255,255,255,0.12);
              padding: 16px; border-radius: 10px;
              z-index: 10; min-width: 200px; max-width: 260px;
              backdrop-filter: blur(6px);
          }}
          #panel h3 {{ font-size: 14px; font-weight: 600; margin-bottom: 12px; color: #a0c4ff; letter-spacing: 0.5px; }}
          #stats {{ font-size: 11px; color: #888; margin-bottom: 12px; }}
          #filters {{ display: flex; flex-direction: column; gap: 6px; }}
          .filter-item {{ display: flex; align-items: center; gap: 8px; font-size: 12px; cursor: pointer; }}
          .filter-item input {{ cursor: pointer; accent-color: #a0c4ff; }}
          .color-dot {{ width: 10px; height: 10px; border-radius: 50%; flex-shrink: 0; }}
          #tooltip {{
              position: absolute; display: none;
              background: rgba(5, 5, 20, 0.95);
              border: 1px solid rgba(255,255,255,0.15);
              padding: 10px 14px; border-radius: 8px;
              pointer-events: none; z-index: 20;
              max-width: 300px; font-size: 12px; line-height: 1.6;
          }}
          #tooltip .tt-title {{ font-weight: 600; color: #a0c4ff; margin-bottom: 6px; font-size: 13px; }}
          #tooltip .tt-row {{ display: flex; gap: 6px; }}
          #tooltip .tt-key {{ color: #888; min-width: 80px; }}
          #tooltip .tt-val {{ color: #eee; word-break: break-all; }}
      </style>
  </head>
  <body>
      <div id="container"></div>
      <div id="panel">
          <h3>{title}</h3>
          <div id="stats"></div>
          <div id="filters"></div>
      </div>
      <div id="tooltip"></div>

      <script>
      (function () {{
          const graphData = {json_data};

          // ── Build graphology graph ──────────────────────────────────────────────
          const Graph = graphology.Graph || graphology;
          const g = new Graph({{ multi: true, type: 'directed' }});

          // Assign random initial positions
          graphData.nodes.forEach(function (n) {{
              g.addNode(n.key, Object.assign({{
                  x: (Math.random() - 0.5) * 10,
                  y: (Math.random() - 0.5) * 10,
              }}, n.attributes));
          }});

          graphData.edges.forEach(function (e) {{
              try {{
                  g.addEdgeWithKey(e.key, e.source, e.target, e.attributes || {{}});
              }} catch (err) {{
                  // skip duplicate edge keys gracefully
              }}
          }});

          // ── ForceAtlas2 layout (synchronous, 500 iterations) ───────────────────
          const FA2 = graphologyLibrary.layoutForceAtlas2;
          FA2.assign(g, {{
              iterations: 500,
              settings: {{
                  gravity: 1,
                  scalingRatio: 2,
                  slowDown: 5,
                  barnesHutOptimize: g.order > 300,
              }},
          }});

          // ── Sigma renderer ──────────────────────────────────────────────────────
          const renderer = new Sigma(g, document.getElementById('container'), {{
              renderEdgeLabels: false,
              defaultEdgeColor: '#444',
              defaultNodeColor: '#95a5a6',
              labelColor: {{ color: '#ccc' }},
              labelSize: 11,
              edgeReducer: function (edge, data) {{
                  return Object.assign({{}}, data, {{ size: Math.max(1, (data.weight || 1) * 0.8) }});
              }},
          }});

          // ── Stats panel ─────────────────────────────────────────────────────────
          document.getElementById('stats').textContent =
              graphData.nodes.length + ' nodes · ' + graphData.edges.length + ' edges';

          // ── Filter panel by node type ───────────────────────────────────────────
          const typeColors = {{}};
          graphData.nodes.forEach(function (n) {{
              const t = n.attributes.entity_type || 'unknown';
              typeColors[t] = n.attributes.color || '#95a5a6';
          }});

          const hiddenTypes = new Set();
          const filtersDiv = document.getElementById('filters');

          Object.keys(typeColors).sort().forEach(function (type) {{
              const color = typeColors[type];
              const label = document.createElement('label');
              label.className = 'filter-item';

              const cb = document.createElement('input');
              cb.type = 'checkbox';
              cb.checked = true;
              cb.addEventListener('change', function () {{
                  if (cb.checked) hiddenTypes.delete(type);
                  else hiddenTypes.add(type);
                  renderer.refresh();
              }});

              const dot = document.createElement('span');
              dot.className = 'color-dot';
              dot.style.background = color;

              label.appendChild(cb);
              label.appendChild(dot);
              label.appendChild(document.createTextNode(type));
              filtersDiv.appendChild(label);
          }});

          // Node reducer applies type filter
          renderer.setSetting('nodeReducer', function (node, data) {{
              if (hiddenTypes.has(data.entity_type)) return Object.assign({{}}, data, {{ hidden: true }});
              return data;
          }});

          // ── Tooltip on hover ────────────────────────────────────────────────────
          const tooltip = document.getElementById('tooltip');

          renderer.on('enterNode', function (ref) {{
              const nodeAttrs = g.getNodeAttributes(ref.node);
              const reserved = new Set(['x', 'y', 'size', 'color', 'label', 'type', 'hidden']);

              let html = '<div class="tt-title">' + escHtml(nodeAttrs.label || ref.node) + '</div>';
              html += '<div class="tt-row"><span class="tt-key">type</span><span class="tt-val">' + escHtml(nodeAttrs.entity_type || '') + '</span></div>';
              html += '<div class="tt-row"><span class="tt-key">status</span><span class="tt-val">' + escHtml(nodeAttrs.status || '') + '</span></div>';
              html += '<div class="tt-row"><span class="tt-key">domain</span><span class="tt-val">' + escHtml(nodeAttrs.domain || '') + '</span></div>';

              Object.keys(nodeAttrs).sort().forEach(function (k) {{
                  if (!reserved.has(k) && !['status', 'domain', 'type', 'label'].includes(k)) {{
                      html += '<div class="tt-row"><span class="tt-key">' + escHtml(k) + '</span><span class="tt-val">' + escHtml(String(nodeAttrs[k])) + '</span></div>';
                  }}
              }});

              tooltip.innerHTML = html;
              tooltip.style.display = 'block';
          }});

          renderer.on('leaveNode', function () {{
              tooltip.style.display = 'none';
          }});

          document.getElementById('container').addEventListener('mousemove', function (e) {{
              tooltip.style.left = (e.clientX + 16) + 'px';
              tooltip.style.top = (e.clientY + 16) + 'px';
          }});

          function escHtml(str) {{
              return String(str)
                  .replace(/&/g, '&amp;')
                  .replace(/</g, '&lt;')
                  .replace(/>/g, '&gt;')
                  .replace(/"/g, '&quot;');
          }}
      }})();
      </script>
  </body>
  </html>
  """


  def render_sigma_html(
      graph_data: dict,
      output_path: str,
      title: str = "OSINT Graph",
  ) -> str:
      """Genera un HTML standalone con sigma.js que visualiza el grafo OSINT.

      Recibe el dict producido por ops_to_sigma_json, embebe los datos como JSON
      en el HTML, aplica ForceAtlas2 (500 iteraciones sincrono) y renderiza con
      sigma.js v2.4. Incluye dark theme, panel de filtros por tipo de nodo y
      tooltip con metadata al hacer hover.

      Args:
          graph_data: Dict con claves 'nodes' y 'edges' en formato graphology/sigma.
          output_path: Ruta del archivo HTML a escribir.
          title: Titulo del grafo mostrado en el panel y la pestana.

      Returns:
          Ruta absoluta del archivo HTML escrito.

      Raises:
          Exception: Si no se puede escribir el archivo en output_path.
      """
      json_data = json.dumps(graph_data, ensure_ascii=False)

      html = _HTML_TEMPLATE.format(
          title=title,
          json_data=json_data,
      )

      abs_path = os.path.abspath(output_path)
      os.makedirs(os.path.dirname(abs_path) or ".", exist_ok=True)

      try:
          with open(abs_path, "w", encoding="utf-8") as f:
              f.write(html)
      except OSError as exc:
          raise Exception(f"render_sigma_html: no se pudo escribir '{abs_path}': {exc}") from exc

      return abs_path

params_schema:
  {"params":[{"name":"graph_data","desc":"dict en formato sigma.js con {nodes: [...], edges: [...]} (ej: output de ops_to_sigma_json_py_datascience)"},{"name":"output_path","desc":"path absoluto donde guardar el HTML (ej: '/tmp/osint_graph.html'). Directorio debe existir."},{"name":"title","desc":"titulo del grafo que aparece en la visualizacion (defecto: 'OSINT Graph')"}],"output":"string con el path absoluto del archivo HTML escrito. Archivo es standalone y abre en cualquier navegador."}


id: rolling_window_py_datascience
name: rolling_window
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def rolling_window(xs: list, size: int) -> list
description: Genera ventanas deslizantes de tamanio fijo sobre una lista.
tags: ["windowing","timeseries","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  rolling_window([1, 2, 3, 4, 5], 3)
  # [[1, 2, 3], [2, 3, 4], [3, 4, 5]]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ef161056f83bca9f4c4e0df216f33c876753cfad0c9b3c5809c71747d766c409
notes:
  Retorna lista vacia si size <= 0 o size > len(xs). Util para calcular medias moviles u otras metricas sobre ventanas.

code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista de valores para crear ventanas (ej: [1, 2, 3, 4, 5])"},{"name":"size","desc":"tamanio de cada ventana (ej: 3). Debe ser positivo y \u003c= len(xs)."}],"output":"lista de listas, cada una una ventana de tamanio size. Resultado tiene len(xs) - size + 1 ventanas."}


id: standardize_py_datascience
name: standardize
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def standardize(data: list) -> list
description: Estandarizacion Z-score: transforma los datos a media=0 y desviacion=1.
tags: ["statistics","normalization","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
example:
  ```python
  standardize([10, 20, 30])
  # [-1.2247..., 0.0, 1.2247...]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Si la desviacion estandar es cero, retorna lista de ceros. Usa desviacion poblacional (N, no N-1).
content_hash: 68639708b50c319b00689ee447d443bfec00222e3012b7736f24bd27178dbb16
code:
[cell repeats an earlier one in this file: 5679 characters]

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos a estandarizar"}],"output":"lista de misma longitud con datos transformados a media=0 y desviacion estandar=1 (z-scores)"}


id: embedding_encode_py_infra
name: embedding_encode
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_encode(model: SentenceTransformer, texts: list, mode: str = 'document') -> list
tags: ["embedding","encode","e5","multilingual","python"]
uses_functions: ["embedding_load_model_py_infra"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["sentence_transformers"]
tests: []
file_path: python/functions/embedding/model.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 3d1c80b921aba6c0e358f2babf9d175191a285864da33b5ead8ab5d8a86b90c9
description:
  Genera embeddings normalizados para textos. Aplica prefijos e5 automaticamente segun mode (document/query).

example:
  ```python
  model = embedding_load_model(".local/models/e5-small")

  # Indexar documentos
  doc_embs = embedding_encode(model, ["La IA transforma la industria", "Python es versatil"], mode="document")

  # Buscar
  query_embs = embedding_encode(model, ["¿Que es machine learning?"], mode="query")
  ```

notes:
  mode="document" agrega prefijo "passage: ", mode="query" agrega "query: ".
  Estos prefijos son requeridos por modelos e5 para retrieval optimo.
  Los embeddings retornados son float32 normalizados (norma L2 = 1).
  Para e5-small la dimension es 384. Throughput ~1900 docs/s en CPU.

code:
  """Embedding model management — save, load, and encode with multilingual-e5-small."""

  import os

  from sentence_transformers import SentenceTransformer


  def embedding_save_model(model_id: str, path: str) -> str:
      """Descarga modelo de HuggingFace y lo guarda en path local.

      Args:
          model_id: ID del modelo en HuggingFace (ej: "intfloat/multilingual-e5-small").
          path: Directorio destino para guardar el modelo.

      Returns:
          Path absoluto donde se guardo el modelo.

      Raises:
          OSError: Si no se puede escribir en el path.
          Exception: Si el modelo no existe en HuggingFace.
      """
      os.makedirs(path, exist_ok=True)
      model = SentenceTransformer(model_id)
      model.save(path)
      return os.path.abspath(path)


  def embedding_load_model(path: str) -> SentenceTransformer:
      """Carga modelo de embeddings desde path local.

      Args:
          path: Directorio con el modelo guardado por embedding_save_model.

      Returns:
          Instancia de SentenceTransformer lista para encode.

      Raises:
          OSError: Si el path no existe o no contiene un modelo valido.
      """
      return SentenceTransformer(path)


  def embedding_encode(model: SentenceTransformer, texts: list, mode: str = "document") -> list:
      """Genera embeddings normalizados para una lista de textos.

      Aplica automaticamente los prefijos requeridos por modelos e5:
      - mode="document" -> "passage: " prefix
      - mode="query"    -> "query: " prefix

      Args:
          model: Modelo cargado con embedding_load_model.
          texts: Lista de strings a codificar.
          mode: "document" para indexar, "query" para buscar.

      Returns:
          Lista de arrays numpy float32 normalizados (dim depende del modelo).

      Raises:
          ValueError: Si mode no es "document" ni "query".
      """
      if mode not in ("document", "query"):
          raise ValueError(f"mode must be 'document' or 'query', got '{mode}'")

      prefix = "passage: " if mode == "document" else "query: "
      prefixed = [f"{prefix}{t}" for t in texts]
      embeddings = model.encode(prefixed, normalize_embeddings=True, show_progress_bar=False)
      return embeddings

params_schema:
  {"params":[{"name":"model","desc":"instancia SentenceTransformer cargada con embedding_load_model"},{"name":"texts","desc":"lista de strings a codificar como embeddings"},{"name":"mode","desc":"contexto semántico: 'document' para indexación, 'query' para búsqueda (aplica prefijos e5)"}],"output":"list[list[float]]: embeddings normalizados (L2=1), dimensión 384 para e5-small"}


id: embedding_load_model_py_infra
name: embedding_load_model
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_load_model(path: str) -> SentenceTransformer
description: Carga modelo de embeddings desde path local. Retorna instancia lista para encode.
tags: ["embedding","model","load","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["sentence_transformers"]
tests: []
file_path: python/functions/embedding/model.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: c54b9c97b3eb859590e5773787425729c3210c23bb13d4b0e66f85c3df7eab1e
example:
  ```python
  model = embedding_load_model(".local/models/e5-small")
  # model listo para usar con embedding_encode
  ```

notes:
  Carga desde path local (~1.8s) es mas rapida que desde HF cache (~4.1s).
  El modelo retornado es compatible con embedding_encode.

code:
  """Embedding model management — save, load, and encode with multilingual-e5-small."""

  import os

  from sentence_transformers import SentenceTransformer


  def embedding_save_model(model_id: str, path: str) -> str:
      """Descarga modelo de HuggingFace y lo guarda en path local.

      Args:
          model_id: ID del modelo en HuggingFace (ej: "intfloat/multilingual-e5-small").
          path: Directorio destino para guardar el modelo.

      Returns:
          Path absoluto donde se guardo el modelo.

      Raises:
          OSError: Si no se puede escribir en el path.
          Exception: Si el modelo no existe en HuggingFace.
      """
      os.makedirs(path, exist_ok=True)
      model = SentenceTransformer(model_id)
      model.save(path)
      return os.path.abspath(path)


  def embedding_load_model(path: str) -> SentenceTransformer:
      """Carga modelo de embeddings desde path local.

      Args:
          path: Directorio con el modelo guardado por embedding_save_model.

      Returns:
          Instancia de SentenceTransformer lista para encode.

      Raises:
          OSError: Si el path no existe o no contiene un modelo valido.
      """
      return SentenceTransformer(path)


  def embedding_encode(model: SentenceTransformer, texts: list, mode: str = "document") -> list:
      """Genera embeddings normalizados para una lista de textos.

      Aplica automaticamente los prefijos requeridos por modelos e5:
      - mode="document" -> "passage: " prefix
      - mode="query"    -> "query: " prefix

      Args:
          model: Modelo cargado con embedding_load_model.
          texts: Lista de strings a codificar.
          mode: "document" para indexar, "query" para buscar.

      Returns:
          Lista de arrays numpy float32 normalizados (dim depende del modelo).

      Raises:
          ValueError: Si mode no es "document" ni "query".
      """
      if mode not in ("document", "query"):
          raise ValueError(f"mode must be 'document' or 'query', got '{mode}'")

      prefix = "passage: " if mode == "document" else "query: "
      prefixed = [f"{prefix}{t}" for t in texts]
      embeddings = model.encode(prefixed, normalize_embeddings=True, show_progress_bar=False)
      return embeddings

params_schema:
  {"params":[{"name":"path","desc":"ruta al directorio donde está guardado el modelo (ej: .local/models/e5-small)"}],"output":"SentenceTransformer: instancia lista para embedding_encode"}


id: embedding_save_model_py_infra
name: embedding_save_model
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_save_model(model_id: str, path: str) -> str
description: Descarga modelo de embeddings de HuggingFace y lo guarda en path local para carga rapida sin red.
tags: ["embedding","model","save","huggingface","e5","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["sentence_transformers"]
tests: []
file_path: python/functions/embedding/model.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 2bbf7c530bf0487da8b888825e2458e364a4665e95ffdc128a9342653c364830
example:
  ```python
  path = embedding_save_model("intfloat/multilingual-e5-small", ".local/models/e5-small")
  # path = "/home/lucas/fn_registry/.local/models/e5-small"
  ```

notes:
  El modelo se guarda en formato sentence-transformers (safetensors + tokenizer).
  Para multilingual-e5-small ocupa ~465 MB en disco.
  Carga local es ~2.3x mas rapida que desde HF cache.

code:
  """Embedding model management — save, load, and encode with multilingual-e5-small."""

  import os

  from sentence_transformers import SentenceTransformer


  def embedding_save_model(model_id: str, path: str) -> str:
      """Descarga modelo de HuggingFace y lo guarda en path local.

      Args:
          model_id: ID del modelo en HuggingFace (ej: "intfloat/multilingual-e5-small").
          path: Directorio destino para guardar el modelo.

      Returns:
          Path absoluto donde se guardo el modelo.

      Raises:
          OSError: Si no se puede escribir en el path.
          Exception: Si el modelo no existe en HuggingFace.
      """
      os.makedirs(path, exist_ok=True)
      model = SentenceTransformer(model_id)
      model.save(path)
      return os.path.abspath(path)


  def embedding_load_model(path: str) -> SentenceTransformer:
      """Carga modelo de embeddings desde path local.

      Args:
          path: Directorio con el modelo guardado por embedding_save_model.

      Returns:
          Instancia de SentenceTransformer lista para encode.

      Raises:
          OSError: Si el path no existe o no contiene un modelo valido.
      """
      return SentenceTransformer(path)


  def embedding_encode(model: SentenceTransformer, texts: list, mode: str = "document") -> list:
      """Genera embeddings normalizados para una lista de textos.

      Aplica automaticamente los prefijos requeridos por modelos e5:
      - mode="document" -> "passage: " prefix
      - mode="query"    -> "query: " prefix

      Args:
          model: Modelo cargado con embedding_load_model.
          texts: Lista de strings a codificar.
          mode: "document" para indexar, "query" para buscar.

      Returns:
          Lista de arrays numpy float32 normalizados (dim depende del modelo).

      Raises:
          ValueError: Si mode no es "document" ni "query".
      """
      if mode not in ("document", "query"):
          raise ValueError(f"mode must be 'document' or 'query', got '{mode}'")

      prefix = "passage: " if mode == "document" else "query: "
      prefixed = [f"{prefix}{t}" for t in texts]
      embeddings = model.encode(prefixed, normalize_embeddings=True, show_progress_bar=False)
      return embeddings

params_schema:
  {"params":[{"name":"model_id","desc":"identificador de modelo en HuggingFace (ej: intfloat/multilingual-e5-small)"},{"name":"path","desc":"ruta local donde guardar el modelo descargado"}],"output":"string: ruta absoluta donde se guardó el modelo"}


id: embedding_search_sqlvec_py_infra
name: embedding_search_sqlvec
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_search_sqlvec(db_path: str, table: str, query_embedding: list, k: int = 10) -> list
description: Busca los k vecinos mas cercanos en tabla sqlite-vec. Retorna rowids y distancias ordenados.
tags: ["embedding","sqlite","vector","search","retrieval","sqlite-vec","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["sqlite3","sqlite_vec","numpy"]
tests: []
file_path: python/functions/embedding/sqlvec.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: a0d3793f46ce1fed772d16b44e69df12d90c5fd0624510577eccda10e18ab6f5
example:
  ```python
  model = embedding_load_model(".local/models/e5-small")
  q_emb = embedding_encode(model, ["¿Que es machine learning?"], mode="query")[0]

  results = embedding_search_sqlvec("vectors.db", "doc_embeddings", q_emb, k=5)
  # [{"rowid": 0, "distance": 0.23}, {"rowid": 1, "distance": 0.45}, ...]
  ```

notes:
  Busqueda brute-force (exacta, no aproximada). Para 50k vectores tarda ~19ms/query.
  El campo distance es distancia coseno (menor = mas similar) porque los embeddings estan normalizados.
  Cold start rapido (~18ms) porque SQLite no carga todo el indice a RAM.

code:
  """Vector storage and retrieval with sqlite-vec."""

  import sqlite3

  import numpy as np
  import sqlite_vec


  def embedding_store_sqlvec(db_path: str, table: str, ids: list, embeddings: list, dim: int = 384) -> int:
      """Inserta embeddings en una tabla sqlite-vec.

      Crea la tabla virtual si no existe. Inserta en batches de 500.

      Args:
          db_path: Path al archivo SQLite.
          table: Nombre de la tabla virtual vec0.
          ids: Lista de IDs (int) para cada embedding.
          embeddings: Lista de arrays numpy float32.
          dim: Dimension de los embeddings (default 384 para e5-small).

      Returns:
          Numero de embeddings insertados.

      Raises:
          sqlite3.Error: Si hay error de escritura en la BD.
      """
      db = sqlite3.connect(db_path)
      db.enable_load_extension(True)
      sqlite_vec.load(db)

      db.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS [{table}] USING vec0(embedding float[{dim}])")

      batch_size = 500
      count = 0
      for i in range(0, len(ids), batch_size):
          batch = [
              (int(ids[j]), np.asarray(embeddings[j], dtype=np.float32).tobytes())
              for j in range(i, min(i + batch_size, len(ids)))
          ]
          db.executemany(f"INSERT INTO [{table}](rowid, embedding) VALUES (?, ?)", batch)
          count += len(batch)

      db.commit()
      db.close()
      return count


  def embedding_search_sqlvec(db_path: str, table: str, query_embedding: list, k: int = 10) -> list:
      """Busca los k vecinos mas cercanos en una tabla sqlite-vec.

      Args:
          db_path: Path al archivo SQLite con la tabla vec0.
          table: Nombre de la tabla virtual.
          query_embedding: Array numpy float32 del query.
          k: Numero de resultados a retornar.

      Returns:
          Lista de dicts con 'rowid' y 'distance' ordenados por cercania.

      Raises:
          sqlite3.Error: Si la tabla no existe o hay error de lectura.
      """
      db = sqlite3.connect(db_path)
      db.enable_load_extension(True)
      sqlite_vec.load(db)

      q_bytes = np.asarray(query_embedding, dtype=np.float32).tobytes()
      rows = db.execute(
          f"SELECT rowid, distance FROM [{table}] WHERE embedding MATCH ? ORDER BY distance LIMIT ?",
          (q_bytes, k),
      ).fetchall()

      db.close()
      return [{"rowid": r[0], "distance": r[1]} for r in rows]

params_schema:
  {"params":[{"name":"db_path","desc":"ruta a la base de datos SQLite con tabla virtual sqlite-vec"},{"name":"table","desc":"nombre de la tabla virtual donde buscar vectores"},{"name":"query_embedding","desc":"vector de embedding (list[float]) a usar como query"},{"name":"k","desc":"cantidad de vecinos más cercanos a retornar"}],"output":"list[dict]: resultados ordenados con 'rowid' e 'distance' (coseno, menor=más similar)"}


id: embedding_search_usearch_py_infra
name: embedding_search_usearch
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_search_usearch(path: str, query_embedding: list, k: int = 10, dim: int = 384) -> list
description: Busca los k vecinos mas cercanos en indice USearch persistido. Busqueda sub-milisegundo.
tags: ["embedding","usearch","vector","search","retrieval","ann","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["usearch","numpy"]
tests: []
file_path: python/functions/embedding/usearch_store.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 476b72b106271583a7850c193fa94b40caef7849d7a1b8734c10bb4efdcd7a8f
example:
  ```python
  model = embedding_load_model(".local/models/e5-small")
  q_emb = embedding_encode(model, ["¿Que es machine learning?"], mode="query")[0]

  results = embedding_search_usearch("docs.usearch", q_emb, k=5)
  # [{"key": 0, "distance": 0.82}, {"key": 1, "distance": 0.65}, ...]
  ```

notes:
  Carga el indice completo a RAM antes de buscar. Cold start ~190ms para 50k vectores.
  Busqueda aproximada (HNSW) — puede no encontrar el vecino exacto pero es 150x mas rapido que brute-force.
  Distance es inner product (mayor = mas similar, al reves que sqlite-vec).

code:
  """Vector storage and retrieval with USearch."""

  import numpy as np
  from usearch.index import Index


  def embedding_store_usearch(path: str, ids: list, embeddings: list, dim: int = 384) -> int:
      """Crea o sobreescribe un indice USearch con los embeddings dados.

      Args:
          path: Path del archivo .usearch para persistir el indice.
          ids: Lista de IDs (int) para cada embedding.
          embeddings: Lista de arrays numpy float32.
          dim: Dimension de los embeddings (default 384 para e5-small).

      Returns:
          Numero de embeddings insertados.

      Raises:
          OSError: Si no se puede escribir en el path.
      """
      index = Index(ndim=dim, metric="ip", dtype="f32")
      keys = np.array(ids, dtype=np.uint64)
      vecs = np.array(embeddings, dtype=np.float32)
      index.add(keys, vecs)
      index.save(path)
      return len(ids)


  def embedding_search_usearch(path: str, query_embedding: list, k: int = 10, dim: int = 384) -> list:
      """Busca los k vecinos mas cercanos en un indice USearch persistido.

      Args:
          path: Path del archivo .usearch.
          query_embedding: Array numpy float32 del query.
          k: Numero de resultados a retornar.
          dim: Dimension de los embeddings.

      Returns:
          Lista de dicts con 'key' y 'distance' ordenados por cercania.

      Raises:
          OSError: Si el archivo no existe.
      """
      index = Index(ndim=dim, metric="ip", dtype="f32")
      index.load(path)
      q = np.asarray(query_embedding, dtype=np.float32)
      results = index.search(q, k)
      keys = np.atleast_1d(results.keys)
      distances = np.atleast_1d(results.distances)
      return [{"key": int(keys[i]), "distance": float(distances[i])} for i in range(len(keys))]

params_schema:
  {"params":[{"name":"path","desc":"ruta al archivo de índice USearch persistido"},{"name":"query_embedding","desc":"vector de embedding (list[float]) a usar como query"},{"name":"k","desc":"cantidad de vecinos aproximados a retornar"},{"name":"dim","desc":"dimensión del espacio de embeddings (por defecto 384 para e5-small)"}],"output":"list[dict]: resultados con 'key' y 'distance' (inner product, mayor=más similar)"}


id: embedding_store_sqlvec_py_infra
name: embedding_store_sqlvec
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
description: Inserta embeddings en tabla sqlite-vec. Crea la tabla virtual si no existe. Insercion en batches.
tags: ["embedding","sqlite","vector","store","sqlite-vec","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["sqlite3","sqlite_vec","numpy"]
tests: []
file_path: python/functions/embedding/sqlvec.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 390090a73220fbe6887db3849af27f913a6a7039677cfc9b39357ff2f7b75701
signature:
  def embedding_store_sqlvec(db_path: str, table: str, ids: list, embeddings: list, dim: int = 384) -> int

example:
  ```python
  model = embedding_load_model(".local/models/e5-small")
  docs = ["La IA transforma la industria", "Python es versatil"]
  embs = embedding_encode(model, docs, mode="document")

  n = embedding_store_sqlvec("vectors.db", "doc_embeddings", [0, 1], embs)
  # n = 2
  ```

notes:
  Usa sqlite-vec (extension pura C para SQLite). Los vectores se almacenan como blobs float32.
  Compatible con cualquier SQLite — se puede usar el mismo archivo para metadata con tablas normales.
  Insercion en batches de 500 para evitar limits de SQLite.
  Para 50k vectores dim=384: ~75 MB en disco, busqueda ~19ms/query.

code:
  """Vector storage and retrieval with sqlite-vec."""

  import sqlite3

  import numpy as np
  import sqlite_vec


  def embedding_store_sqlvec(db_path: str, table: str, ids: list, embeddings: list, dim: int = 384) -> int:
      """Inserta embeddings en una tabla sqlite-vec.

      Crea la tabla virtual si no existe. Inserta en batches de 500.

      Args:
          db_path: Path al archivo SQLite.
          table: Nombre de la tabla virtual vec0.
          ids: Lista de IDs (int) para cada embedding.
          embeddings: Lista de arrays numpy float32.
          dim: Dimension de los embeddings (default 384 para e5-small).

      Returns:
          Numero de embeddings insertados.

      Raises:
          sqlite3.Error: Si hay error de escritura en la BD.
      """
      db = sqlite3.connect(db_path)
      db.enable_load_extension(True)
      sqlite_vec.load(db)

      db.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS [{table}] USING vec0(embedding float[{dim}])")

      batch_size = 500
      count = 0
      for i in range(0, len(ids), batch_size):
          batch = [
              (int(ids[j]), np.asarray(embeddings[j], dtype=np.float32).tobytes())
              for j in range(i, min(i + batch_size, len(ids)))
          ]
          db.executemany(f"INSERT INTO [{table}](rowid, embedding) VALUES (?, ?)", batch)
          count += len(batch)

      db.commit()
      db.close()
      return count


  def embedding_search_sqlvec(db_path: str, table: str, query_embedding: list, k: int = 10) -> list:
      """Busca los k vecinos mas cercanos en una tabla sqlite-vec.

      Args:
          db_path: Path al archivo SQLite con la tabla vec0.
          table: Nombre de la tabla virtual.
          query_embedding: Array numpy float32 del query.
          k: Numero de resultados a retornar.

      Returns:
          Lista de dicts con 'rowid' y 'distance' ordenados por cercania.

      Raises:
          sqlite3.Error: Si la tabla no existe o hay error de lectura.
      """
      db = sqlite3.connect(db_path)
      db.enable_load_extension(True)
      sqlite_vec.load(db)

      q_bytes = np.asarray(query_embedding, dtype=np.float32).tobytes()
      rows = db.execute(
          f"SELECT rowid, distance FROM [{table}] WHERE embedding MATCH ? ORDER BY distance LIMIT ?",
          (q_bytes, k),
      ).fetchall()

      db.close()
      return [{"rowid": r[0], "distance": r[1]} for r in rows]

params_schema:
  {"params":[{"name":"db_path","desc":"ruta a la base de datos SQLite (se crea si no existe)"},{"name":"table","desc":"nombre de la tabla virtual sqlite-vec a crear"},{"name":"ids","desc":"lista de identificadores enteros para los embeddings"},{"name":"embeddings","desc":"lista de vectores (list[list[float]]) a almacenar"},{"name":"dim","desc":"dimensión de los vectores (por defecto 384 para e5-small)"}],"output":"int: cantidad de vectores insertados"}


id: embedding_store_usearch_py_infra
name: embedding_store_usearch
kind: function
lang: py
domain: infra
version: 1.0.0
purity: impure
signature: def embedding_store_usearch(path: str, ids: list, embeddings: list, dim: int = 384) -> int
description: Crea indice USearch con embeddings y lo persiste a archivo. Busqueda sub-milisegundo.
tags: ["embedding","usearch","vector","store","ann","python"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["usearch","numpy"]
tests: []
file_path: python/functions/embedding/usearch_store.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 67b27b4d2806329009ff704d80d23c3336b68788eb746c05d44eb6d7a3ca7a84
example:
  ```python
  model = embedding_load_model(".local/models/e5-small")
  docs = ["La IA transforma la industria", "Python es versatil"]
  embs = embedding_encode(model, docs, mode="document")

  n = embedding_store_usearch("docs.usearch", [0, 1], embs)
  # n = 2
  ```

notes:
  USearch usa HNSW (approximate nearest neighbors). Para 50k vectores dim=384:
  ~80 MB en disco, busqueda ~0.13ms/query (150x mas rapido que sqlite-vec).
  El tradeoff es que no soporta metadata nativa — usar junto con SQLite para metadata.
  Sobreescribe el archivo si ya existe.

code:
  """Vector storage and retrieval with USearch."""

  import numpy as np
  from usearch.index import Index


  def embedding_store_usearch(path: str, ids: list, embeddings: list, dim: int = 384) -> int:
      """Crea o sobreescribe un indice USearch con los embeddings dados.

      Args:
          path: Path del archivo .usearch para persistir el indice.
          ids: Lista de IDs (int) para cada embedding.
          embeddings: Lista de arrays numpy float32.
          dim: Dimension de los embeddings (default 384 para e5-small).

      Returns:
          Numero de embeddings insertados.

      Raises:
          OSError: Si no se puede escribir en el path.
      """
      index = Index(ndim=dim, metric="ip", dtype="f32")
      keys = np.array(ids, dtype=np.uint64)
      vecs = np.array(embeddings, dtype=np.float32)
      index.add(keys, vecs)
      index.save(path)
      return len(ids)


  def embedding_search_usearch(path: str, query_embedding: list, k: int = 10, dim: int = 384) -> list:
      """Busca los k vecinos mas cercanos en un indice USearch persistido.

      Args:
          path: Path del archivo .usearch.
          query_embedding: Array numpy float32 del query.
          k: Numero de resultados a retornar.
          dim: Dimension de los embeddings.

      Returns:
          Lista de dicts con 'key' y 'distance' ordenados por cercania.

      Raises:
          OSError: Si el archivo no existe.
      """
      index = Index(ndim=dim, metric="ip", dtype="f32")
      index.load(path)
      q = np.asarray(query_embedding, dtype=np.float32)
      results = index.search(q, k)
      keys = np.atleast_1d(results.keys)
      distances = np.atleast_1d(results.distances)
      return [{"key": int(keys[i]), "distance": float(distances[i])} for i in range(len(keys))]

params_schema:
  {"params":[{"name":"path","desc":"ruta donde guardar el archivo de índice USearch"},{"name":"ids","desc":"lista de identificadores enteros para los embeddings"},{"name":"embeddings","desc":"lista de vectores (list[list[float]]) a indexar"},{"name":"dim","desc":"dimensión de los vectores (por defecto 384 para e5-small)"}],"output":"int: cantidad de vectores indexados"}


id: annualized_volatility_py_finance
name: annualized_volatility
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
signature: def annualized_volatility(returns: list, periods_per_year: float) -> float
description: Calcula la volatilidad anualizada de una serie de retornos.
tags: ["finance","volatility","risk","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 396ebc359a1a4e6176e8546120e0a7da756dff98e14dcbd0cb34306408e30e95
example:
  ```python
  daily_returns = [0.01, -0.005, 0.008, 0.003, -0.002, 0.006, 0.004]
  vol = annualized_volatility(daily_returns, 252.0)
  # Volatilidad anualizada (std * sqrt(252))
  ```

notes:
  Formula: std_muestral(returns) * sqrt(periods_per_year).
  Usa desviacion estandar muestral (n-1) para ser consistente con la practica financiera.
  Retorna 0.0 si hay menos de 2 retornos o periods_per_year es menor o igual a cero.

code:
  """Finance domain — pure functions for financial indicators and calculations."""

  import math


  def sma(data: list, period: int) -> list:
      """Calcula la media movil simple (SMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      result = []
      for i in range(period - 1, len(data)):
          window = data[i - period + 1 : i + 1]
          result.append(sum(window) / period)
      return result


  def ema(data: list, period: int) -> list:
      """Calcula la media movil exponencial (EMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      multiplier = 2.0 / (period + 1)
      # Primer valor es SMA del primer periodo
      first_sma = sum(data[:period]) / period
      result = [first_sma]
      for i in range(period, len(data)):
          val = (data[i] - result[-1]) * multiplier + result[-1]
          result.append(val)
      return result


  def rsi(data: list, period: int) -> list:
      """Calcula el Relative Strength Index (RSI) de una serie de precios."""
      if period <= 0 or len(data) < period + 1:
          return []
      deltas = [data[i] - data[i - 1] for i in range(1, len(data))]
      gains = [d if d > 0 else 0.0 for d in deltas]
      losses = [-d if d < 0 else 0.0 for d in deltas]

      avg_gain = sum(gains[:period]) / period
      avg_loss = sum(losses[:period]) / period

      result = []
      if avg_loss == 0:
          result.append(100.0)
      else:
          rs = avg_gain / avg_loss
          result.append(100.0 - 100.0 / (1.0 + rs))

      for i in range(period, len(deltas)):
          avg_gain = (avg_gain * (period - 1) + gains[i]) / period
          avg_loss = (avg_loss * (period - 1) + losses[i]) / period
          if avg_loss == 0:
              result.append(100.0)
          else:
              rs = avg_gain / avg_loss
              result.append(100.0 - 100.0 / (1.0 + rs))

      return result


  def bollinger_bands(data: list, period: int, num_std: float) -> tuple:
      """Calcula las Bandas de Bollinger (upper, middle, lower)."""
      if period <= 0 or period > len(data):
          return ([], [], [])
      middle = sma(data, period)
      upper = []
      lower = []
      for i in range(len(middle)):
          window = data[i : i + period]
          mean = middle[i]
          variance = sum((x - mean) ** 2 for x in window) / period
          std = math.sqrt(variance)
          upper.append(mean + num_std * std)
          lower.append(mean - num_std * std)
      return (upper, middle, lower)


  def sharpe_ratio(returns: list, risk_free_rate: float, periods_per_year: float) -> float:
      """Calcula el Sharpe Ratio anualizado."""
      if len(returns) == 0 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean_return = sum(returns) / n
      excess = mean_return - risk_free_rate / periods_per_year
      variance = sum((r - mean_return) ** 2 for r in returns) / n
      std = math.sqrt(variance)
      if std == 0:
          return 0.0
      return (excess / std) * math.sqrt(periods_per_year)


  def max_drawdown(values: list) -> tuple:
      """Calcula el max drawdown y los indices de inicio y fin."""
      if len(values) < 2:
          return (0.0, 0, 0)
      peak = values[0]
      peak_idx = 0
      max_dd = 0.0
      dd_start = 0
      dd_end = 0
      for i in range(1, len(values)):
          if values[i] > peak:
              peak = values[i]
              peak_idx = i
          dd = (peak - values[i]) / peak if peak != 0 else 0.0
          if dd > max_dd:
              max_dd = dd
              dd_start = peak_idx
              dd_end = i
      return (max_dd, dd_start, dd_end)


  def vwap(prices: list, volumes: list) -> float:
      """Calcula el Volume-Weighted Average Price (VWAP)."""
      if len(prices) == 0 or len(prices) != len(volumes):
          return 0.0
      total_volume = sum(volumes)
      if total_volume == 0:
          return 0.0
      return sum(p * v for p, v in zip(prices, volumes)) / total_volume


  def log_return(price_start: float, price_end: float) -> float:
      """Calcula el retorno logaritmico entre dos precios."""
      if price_start <= 0 or price_end <= 0:
          return 0.0
      return math.log(price_end / price_start)


  def annualized_volatility(returns: list, periods_per_year: float) -> float:
      """Calcula la volatilidad anualizada de una serie de retornos."""
      if len(returns) < 2 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean = sum(returns) / n
      variance = sum((r - mean) ** 2 for r in returns) / (n - 1)
      return math.sqrt(variance) * math.sqrt(periods_per_year)


  def generate_gbm_prices(
      initial_price: float,
      n_ticks: int,
      sigma: float,
      mu: float = 0.0,
      jump_intensity: float = 0.0,
      jump_size_std: float = 0.05,
      seed: int = 42,
  ) -> list:
      """Genera serie de precios fundamentales con Geometric Brownian Motion + jump-diffusion.

      S(t+1) = S(t) * exp((mu - sigma^2/2)*dt + sigma*sqrt(dt)*Z + J*N)
      donde Z ~ N(0,1), N ~ Bernoulli(jump_intensity), J ~ N(0, jump_size_std)
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      prices = [0.0] * n_ticks
      prices[0] = initial_price
      dt = 1.0
      for t in range(1, n_ticks):
          z = rng.standard_normal()
          gbm = (mu - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * z
          jump = 0.0
          if jump_intensity > 0 and rng.random() < jump_intensity:
              jump = rng.normal(0, jump_size_std)
          prices[t] = prices[t - 1] * np.exp(gbm + jump)
      return prices


  def avellaneda_stoikov_quotes(
      mid_price: float,
      inventory: float,
      gamma: float,
      sigma: float,
      spread_base: float,
      n_levels: int = 3,
      qty_base: float = 10.0,
  ) -> list:
      """Genera ordenes de market maker usando el modelo Avellaneda-Stoikov.

      Precio de reserva: r = mid - inventory * gamma * sigma^2
      Half spread: delta = spread_base/2 + gamma * sigma^2/2

      Retorna lista de dicts con keys: side, price, qty
      """
      reservation = mid_price - inventory * gamma * sigma**2
      half_spread = spread_base / 2 + gamma * sigma**2 / 2
      orders = []
      for level in range(n_levels):
          offset = level * half_spread * 0.5
          qty = qty_base * (1 + level * 0.5)
          bid_price = round(reservation - half_spread - offset, 2)
          ask_price = round(reservation + half_spread + offset, 2)
          if bid_price > 0:
              orders.append({'side': 'buy', 'price': bid_price, 'qty': qty})
          if ask_price > 0:
              orders.append({'side': 'sell', 'price': ask_price, 'qty': qty})
      return orders


  def generate_taker_order(
      alpha: float = 2.0,
      size_min: float = 1.0,
      size_max: float = 100.0,
      buy_prob: float = 0.5,
      seed: int | None = None,
  ) -> dict:
      """Genera una market order de taker con tamano power-law (Pareto).

      P(size > x) ~ x^(-alpha). Alpha bajo = mas ballenas.
      Retorna dict con keys: side, qty
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      side = 'buy' if rng.random() < buy_prob else 'sell'
      raw_size = (rng.pareto(alpha) + 1) * size_min
      size = min(round(raw_size, 1), size_max)
      return {'side': side, 'qty': size}


  def hawkes_intensity(
      base_rate: float,
      hawkes_alpha: float,
      hawkes_beta: float,
      event_times: list,
      current_time: float,
  ) -> float:
      """Calcula la intensidad lambda(t) de un proceso de Hawkes en el tiempo actual.

      lambda(t) = base_rate + sum(alpha * exp(-beta * (t - ti)))
      donde ti son los tiempos de eventos pasados.
      """
      import numpy as np
      excitation = sum(
          hawkes_alpha * np.exp(-hawkes_beta * (current_time - ti))
          for ti in event_times
          if ti < current_time
      )
      return max(0.0, base_rate + excitation)

params_schema:
  {"params":[{"name":"returns","desc":"lista de retornos diarios o periodicos (ej: [0.01, -0.005, 0.008]). Valores en rango [-1, 1] tipicamente."},{"name":"periods_per_year","desc":"numero de periodos por año (252 para datos diarios, 12 para mensuales, 52 para semanales)"}],"output":"volatilidad anualizada en forma decimal (ej: 0.15 para 15% de volatilidad anualizada)"}


id: avellaneda_stoikov_quotes_py_finance
name: avellaneda_stoikov_quotes
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
tags: ["simulation","market-making","avellaneda-stoikov","montecarlo","finance","order-book"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1a660b7339369317adcb3987aa82f9e6d32efdd780a3612fed5b93c30dfbb41c
signature:
  avellaneda_stoikov_quotes(mid_price: float, inventory: float, gamma: float, sigma: float, spread_base: float, n_levels: int, qty_base: float) -> list[dict]

description:
  Genera ordenes de market maker usando el modelo Avellaneda-Stoikov. Calcula precio de reserva y half spread optimos segun inventario y volatilidad.

example:
  ```python
  orders = avellaneda_stoikov_quotes(
      mid_price=100.0,
      inventory=0.0,
      gamma=0.1,
      sigma=0.02,
      spread_base=0.5,
      n_levels=3,
      qty_base=10.0,
  )
  # [
  #   {'side': 'buy',  'price': 99.75, 'qty': 10.0},
  #   {'side': 'sell', 'price': 100.25, 'qty': 10.0},
  #   ...
  # ]
  ```

notes:
  Funcion pura — sin aleatoriedad.
  `gamma` controla la aversion al riesgo de inventario: mayor gamma = spreads mas amplios.
  `inventory` positivo sesga los quotes hacia venta (reduce inventario largo).
  Cada nivel adicional ensancha el spread en `half_spread * 0.5` y aumenta la cantidad en `qty_base * 0.5`.
  Ordenes con precio <= 0 se descartan automaticamente.

code:
  """Finance domain — pure functions for financial indicators and calculations."""

  import math


  def sma(data: list, period: int) -> list:
      """Calcula la media movil simple (SMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      result = []
      for i in range(period - 1, len(data)):
          window = data[i - period + 1 : i + 1]
          result.append(sum(window) / period)
      return result


  def ema(data: list, period: int) -> list:
      """Calcula la media movil exponencial (EMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      multiplier = 2.0 / (period + 1)
      # Primer valor es SMA del primer periodo
      first_sma = sum(data[:period]) / period
      result = [first_sma]
      for i in range(period, len(data)):
          val = (data[i] - result[-1]) * multiplier + result[-1]
          result.append(val)
      return result


  def rsi(data: list, period: int) -> list:
      """Calcula el Relative Strength Index (RSI) de una serie de precios."""
      if period <= 0 or len(data) < period + 1:
          return []
      deltas = [data[i] - data[i - 1] for i in range(1, len(data))]
      gains = [d if d > 0 else 0.0 for d in deltas]
      losses = [-d if d < 0 else 0.0 for d in deltas]

      avg_gain = sum(gains[:period]) / period
      avg_loss = sum(losses[:period]) / period

      result = []
      if avg_loss == 0:
          result.append(100.0)
      else:
          rs = avg_gain / avg_loss
          result.append(100.0 - 100.0 / (1.0 + rs))

      for i in range(period, len(deltas)):
          avg_gain = (avg_gain * (period - 1) + gains[i]) / period
          avg_loss = (avg_loss * (period - 1) + losses[i]) / period
          if avg_loss == 0:
              result.append(100.0)
          else:
              rs = avg_gain / avg_loss
              result.append(100.0 - 100.0 / (1.0 + rs))

      return result


  def bollinger_bands(data: list, period: int, num_std: float) -> tuple:
      """Calcula las Bandas de Bollinger (upper, middle, lower)."""
      if period <= 0 or period > len(data):
          return ([], [], [])
      middle = sma(data, period)
      upper = []
      lower = []
      for i in range(len(middle)):
          window = data[i : i + period]
          mean = middle[i]
          variance = sum((x - mean) ** 2 for x in window) / period
          std = math.sqrt(variance)
          upper.append(mean + num_std * std)
          lower.append(mean - num_std * std)
      return (upper, middle, lower)


  def sharpe_ratio(returns: list, risk_free_rate: float, periods_per_year: float) -> float:
      """Calcula el Sharpe Ratio anualizado."""
      if len(returns) == 0 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean_return = sum(returns) / n
      excess = mean_return - risk_free_rate / periods_per_year
      variance = sum((r - mean_return) ** 2 for r in returns) / n
      std = math.sqrt(variance)
      if std == 0:
          return 0.0
      return (excess / std) * math.sqrt(periods_per_year)


  def max_drawdown(values: list) -> tuple:
      """Calcula el max drawdown y los indices de inicio y fin."""
      if len(values) < 2:
          return (0.0, 0, 0)
      peak = values[0]
      peak_idx = 0
      max_dd = 0.0
      dd_start = 0
      dd_end = 0
      for i in range(1, len(values)):
          if values[i] > peak:
              peak = values[i]
              peak_idx = i
          dd = (peak - values[i]) / peak if peak != 0 else 0.0
          if dd > max_dd:
              max_dd = dd
              dd_start = peak_idx
              dd_end = i
      return (max_dd, dd_start, dd_end)


  def vwap(prices: list, volumes: list) -> float:
      """Calcula el Volume-Weighted Average Price (VWAP)."""
      if len(prices) == 0 or len(prices) != len(volumes):
          return 0.0
      total_volume = sum(volumes)
      if total_volume == 0:
          return 0.0
      return sum(p * v for p, v in zip(prices, volumes)) / total_volume


  def log_return(price_start: float, price_end: float) -> float:
      """Calcula el retorno logaritmico entre dos precios."""
      if price_start <= 0 or price_end <= 0:
          return 0.0
      return math.log(price_end / price_start)


  def annualized_volatility(returns: list, periods_per_year: float) -> float:
      """Calcula la volatilidad anualizada de una serie de retornos."""
      if len(returns) < 2 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean = sum(returns) / n
      variance = sum((r - mean) ** 2 for r in returns) / (n - 1)
      return math.sqrt(variance) * math.sqrt(periods_per_year)


  def generate_gbm_prices(
      initial_price: float,
      n_ticks: int,
      sigma: float,
      mu: float = 0.0,
      jump_intensity: float = 0.0,
      jump_size_std: float = 0.05,
      seed: int = 42,
  ) -> list:
      """Genera serie de precios fundamentales con Geometric Brownian Motion + jump-diffusion.

      S(t+1) = S(t) * exp((mu - sigma^2/2)*dt + sigma*sqrt(dt)*Z + J*N)
      donde Z ~ N(0,1), N ~ Bernoulli(jump_intensity), J ~ N(0, jump_size_std)
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      prices = [0.0] * n_ticks
      prices[0] = initial_price
      dt = 1.0
      for t in range(1, n_ticks):
          z = rng.standard_normal()
          gbm = (mu - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * z
          jump = 0.0
          if jump_intensity > 0 and rng.random() < jump_intensity:
              jump = rng.normal(0, jump_size_std)
          prices[t] = prices[t - 1] * np.exp(gbm + jump)
      return prices


  def avellaneda_stoikov_quotes(
      mid_price: float,
      inventory: float,
      gamma: float,
      sigma: float,
      spread_base: float,
      n_levels: int = 3,
      qty_base: float = 10.0,
  ) -> list:
      """Genera ordenes de market maker usando el modelo Avellaneda-Stoikov.

      Precio de reserva: r = mid - inventory * gamma * sigma^2
      Half spread: delta = spread_base/2 + gamma * sigma^2/2

      Retorna lista de dicts con keys: side, price, qty
      """
      reservation = mid_price - inventory * gamma * sigma**2
      half_spread = spread_base / 2 + gamma * sigma**2 / 2
      orders = []
      for level in range(n_levels):
          offset = level * half_spread * 0.5
          qty = qty_base * (1 + level * 0.5)
          bid_price = round(reservation - half_spread - offset, 2)
          ask_price = round(reservation + half_spread + offset, 2)
          if bid_price > 0:
              orders.append({'side': 'buy', 'price': bid_price, 'qty': qty})
          if ask_price > 0:
              orders.append({'side': 'sell', 'price': ask_price, 'qty': qty})
      return orders


  def generate_taker_order(
      alpha: float = 2.0,
      size_min: float = 1.0,
      size_max: float = 100.0,
      buy_prob: float = 0.5,
      seed: int | None = None,
  ) -> dict:
      """Genera una market order de taker con tamano power-law (Pareto).

      P(size > x) ~ x^(-alpha). Alpha bajo = mas ballenas.
      Retorna dict con keys: side, qty
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      side = 'buy' if rng.random() < buy_prob else 'sell'
      raw_size = (rng.pareto(alpha) + 1) * size_min
      size = min(round(raw_size, 1), size_max)
      return {'side': side, 'qty': size}


  def hawkes_intensity(
      base_rate: float,
      hawkes_alpha: float,
      hawkes_beta: float,
      event_times: list,
      current_time: float,
  ) -> float:
      """Calcula la intensidad lambda(t) de un proceso de Hawkes en el tiempo actual.

      lambda(t) = base_rate + sum(alpha * exp(-beta * (t - ti)))
      donde ti son los tiempos de eventos pasados.
      """
      import numpy as np
      excitation = sum(
          hawkes_alpha * np.exp(-hawkes_beta * (current_time - ti))
          for ti in event_times
          if ti < current_time
      )
      return max(0.0, base_rate + excitation)

params_schema:
  {"params":[{"name":"mid_price","desc":"precio medio actual del activo subyacente (punto de referencia para la cotizacion)"},{"name":"inventory","desc":"inventario actual neto del market maker (cantidad de activos tenidos, positivo=long)"},{"name":"gamma","desc":"coeficiente de aversion al riesgo de inventario (rango tipico: 0.01-1.0). Mayor gamma = spreads mas amplios."},{"name":"sigma","desc":"volatilidad estimada del activo en la ventana de tiempo actual (ej: 0.02 para 2% diario)"},{"name":"spread_base","desc":"spread base en unidades de precio para el nivel 0 (ej: 0.5 para +/-0.5 alrededor del mid_price)"},{"name":"n_levels","desc":"numero de niveles de cotizacion a generar (tipico: 3-5). Cada nivel se ensancha a partir del anterior."},{"name":"qty_base","desc":"cantidad base de unidades por nivel (se reduce en cada nivel: qty_base, qty_base*0.5, qty_base*0.25, ...)"}],"output":"lista de ordenes con campos {'side': 'buy'|'sell', 'price': float, 'qty': float}"}


id: bollinger_bands_py_finance
name: bollinger_bands
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
signature: def bollinger_bands(data: list, period: int, num_std: float) -> tuple
description: Calcula las Bandas de Bollinger (upper, middle, lower) de una serie de precios.
tags: ["finance","bollinger","volatility","indicator","python"]
uses_functions: ["sma_py_finance"]
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1cd4a9bd8e2cffc9447a2a04a53af487afb974e2f9bfdf7b7c49ee265009fd42
example:
  ```python
  prices = [10, 11, 12, 13, 14, 15, 14, 13, 12, 11]
  upper, middle, lower = bollinger_bands(prices, 5, 2.0)
  # middle es la SMA(5), upper/lower son middle +/- 2*std
  ```

notes:
  Retorna tupla de tres listas (upper, middle, lower). Cada lista tiene len(data) - period + 1 elementos.
  La desviacion estandar se calcula sobre la ventana de tamanio period (poblacional, no muestral).
  Usa internamente la funcion sma para la banda media.

code:
  """Finance domain — pure functions for financial indicators and calculations."""

  import math


  def sma(data: list, period: int) -> list:
      """Calcula la media movil simple (SMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      result = []
      for i in range(period - 1, len(data)):
          window = data[i - period + 1 : i + 1]
          result.append(sum(window) / period)
      return result


  def ema(data: list, period: int) -> list:
      """Calcula la media movil exponencial (EMA) de una serie de precios."""
      if period <= 0 or period > len(data):
          return []
      multiplier = 2.0 / (period + 1)
      # Primer valor es SMA del primer periodo
      first_sma = sum(data[:period]) / period
      result = [first_sma]
      for i in range(period, len(data)):
          val = (data[i] - result[-1]) * multiplier + result[-1]
          result.append(val)
      return result


  def rsi(data: list, period: int) -> list:
      """Calcula el Relative Strength Index (RSI) de una serie de precios."""
      if period <= 0 or len(data) < period + 1:
          return []
      deltas = [data[i] - data[i - 1] for i in range(1, len(data))]
      gains = [d if d > 0 else 0.0 for d in deltas]
      losses = [-d if d < 0 else 0.0 for d in deltas]

      avg_gain = sum(gains[:period]) / period
      avg_loss = sum(losses[:period]) / period

      result = []
      if avg_loss == 0:
          result.append(100.0)
      else:
          rs = avg_gain / avg_loss
          result.append(100.0 - 100.0 / (1.0 + rs))

      for i in range(period, len(deltas)):
          avg_gain = (avg_gain * (period - 1) + gains[i]) / period
          avg_loss = (avg_loss * (period - 1) + losses[i]) / period
          if avg_loss == 0:
              result.append(100.0)
          else:
              rs = avg_gain / avg_loss
              result.append(100.0 - 100.0 / (1.0 + rs))

      return result


  def bollinger_bands(data: list, period: int, num_std: float) -> tuple:
      """Calcula las Bandas de Bollinger (upper, middle, lower)."""
      if period <= 0 or period > len(data):
          return ([], [], [])
      middle = sma(data, period)
      upper = []
      lower = []
      for i in range(len(middle)):
          window = data[i : i + period]
          mean = middle[i]
          variance = sum((x - mean) ** 2 for x in window) / period
          std = math.sqrt(variance)
          upper.append(mean + num_std * std)
          lower.append(mean - num_std * std)
      return (upper, middle, lower)


  def sharpe_ratio(returns: list, risk_free_rate: float, periods_per_year: float) -> float:
      """Calcula el Sharpe Ratio anualizado."""
      if len(returns) == 0 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean_return = sum(returns) / n
      excess = mean_return - risk_free_rate / periods_per_year
      variance = sum((r - mean_return) ** 2 for r in returns) / n
      std = math.sqrt(variance)
      if std == 0:
          return 0.0
      return (excess / std) * math.sqrt(periods_per_year)


  def max_drawdown(values: list) -> tuple:
      """Calcula el max drawdown y los indices de inicio y fin."""
      if len(values) < 2:
          return (0.0, 0, 0)
      peak = values[0]
      peak_idx = 0
      max_dd = 0.0
      dd_start = 0
      dd_end = 0
      for i in range(1, len(values)):
          if values[i] > peak:
              peak = values[i]
              peak_idx = i
          dd = (peak - values[i]) / peak if peak != 0 else 0.0
          if dd > max_dd:
              max_dd = dd
              dd_start = peak_idx
              dd_end = i
      return (max_dd, dd_start, dd_end)


  def vwap(prices: list, volumes: list) -> float:
      """Calcula el Volume-Weighted Average Price (VWAP)."""
      if len(prices) == 0 or len(prices) != len(volumes):
          return 0.0
      total_volume = sum(volumes)
      if total_volume == 0:
          return 0.0
      return sum(p * v for p, v in zip(prices, volumes)) / total_volume


  def log_return(price_start: float, price_end: float) -> float:
      """Calcula el retorno logaritmico entre dos precios."""
      if price_start <= 0 or price_end <= 0:
          return 0.0
      return math.log(price_end / price_start)


  def annualized_volatility(returns: list, periods_per_year: float) -> float:
      """Calcula la volatilidad anualizada de una serie de retornos."""
      if len(returns) < 2 or periods_per_year <= 0:
          return 0.0
      n = len(returns)
      mean = sum(returns) / n
      variance = sum((r - mean) ** 2 for r in returns) / (n - 1)
      return math.sqrt(variance) * math.sqrt(periods_per_year)


  def generate_gbm_prices(
      initial_price: float,
      n_ticks: int,
      sigma: float,
      mu: float = 0.0,
      jump_intensity: float = 0.0,
      jump_size_std: float = 0.05,
      seed: int = 42,
  ) -> list:
      """Genera serie de precios fundamentales con Geometric Brownian Motion + jump-diffusion.

      S(t+1) = S(t) * exp((mu - sigma^2/2)*dt + sigma*sqrt(dt)*Z + J*N)
      donde Z ~ N(0,1), N ~ Bernoulli(jump_intensity), J ~ N(0, jump_size_std)
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      prices = [0.0] * n_ticks
      prices[0] = initial_price
      dt = 1.0
      for t in range(1, n_ticks):
          z = rng.standard_normal()
          gbm = (mu - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * z
          jump = 0.0
          if jump_intensity > 0 and rng.random() < jump_intensity:
              jump = rng.normal(0, jump_size_std)
          prices[t] = prices[t - 1] * np.exp(gbm + jump)
      return prices


  def avellaneda_stoikov_quotes(
      mid_price: float,
      inventory: float,
      gamma: float,
      sigma: float,
      spread_base: float,
      n_levels: int = 3,
      qty_base: float = 10.0,
  ) -> list:
      """Genera ordenes de market maker usando el modelo Avellaneda-Stoikov.

      Precio de reserva: r = mid - inventory * gamma * sigma^2
      Half spread: delta = spread_base/2 + gamma * sigma^2/2

      Retorna lista de dicts con keys: side, price, qty
      """
      reservation = mid_price - inventory * gamma * sigma**2
      half_spread = spread_base / 2 + gamma * sigma**2 / 2
      orders = []
      for level in range(n_levels):
          offset = level * half_spread * 0.5
          qty = qty_base * (1 + level * 0.5)
          bid_price = round(reservation - half_spread - offset, 2)
          ask_price = round(reservation + half_spread + offset, 2)
          if bid_price > 0:
              orders.append({'side': 'buy', 'price': bid_price, 'qty': qty})
          if ask_price > 0:
              orders.append({'side': 'sell', 'price': ask_price, 'qty': qty})
      return orders


  def generate_taker_order(
      alpha: float = 2.0,
      size_min: float = 1.0,
      size_max: float = 100.0,
      buy_prob: float = 0.5,
      seed: int | None = None,
  ) -> dict:
      """Genera una market order de taker con tamano power-law (Pareto).

      P(size > x) ~ x^(-alpha). Alpha bajo = mas ballenas.
      Retorna dict con keys: side, qty
      """
      import numpy as np
      rng = np.random.default_rng(seed)
      side = 'buy' if rng.random() < buy_prob else 'sell'
      raw_size = (rng.pareto(alpha) + 1) * size_min
      size = min(round(raw_size, 1), size_max)
      return {'side': side, 'qty': size}


  def hawkes_intensity(
      base_rate: float,
      hawkes_alpha: float,
      hawkes_beta: float,
      event_times: list,
      current_time: float,
  ) -> float:
      """Calcula la intensidad lambda(t) de un proceso de Hawkes en el tiempo actual.

      lambda(t) = base_rate + sum(alpha * exp(-beta * (t - ti)))
      donde ti son los tiempos de eventos pasados.
      """
      import numpy as np
      excitation = sum(
          hawkes_alpha * np.exp(-hawkes_beta * (current_time - ti))
          for ti in event_times
          if ti < current_time
      )
      return max(0.0, base_rate + excitation)

params_schema:
  {"params":[{"name":"data","desc":"lista de precios o valores (ej: precios de cierre diarios)"},{"name":"period","desc":"tamanio de la ventana movil en numero de periodos (tipico: 5, 10, 20). Mayor periodo = bandas mas suave."},{"name":"num_std","desc":"numero de desviaciones estandar para las bandas (tipico: 2.0 para 95% de los datos en rango)"}],"output":"tupla (upper, middle, lower) donde cada elemento es una lista de floats con len(data) - period + 1 elementos"}


id: ema_py_finance
name: ema
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
signature: def ema(data: list, period: int) -> list
description: Calcula la media movil exponencial (EMA) de una serie de precios.
tags: ["finance","moving-average","exponential","indicator","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 68bb5458ca2deff2e4ab9ee59ee905cb3ea1921765f1969ccd8806c676598726
example:
  ```python
  prices = [10, 11, 12, 13, 14, 15]
  result = ema(prices, 3)
  # [11.0, 11.5, 12.25, 13.125]
  ```

notes:
  El primer valor de la EMA es el SMA del primer periodo. El multiplicador es 2 / (period + 1).
  Retorna len(data) - period + 1 elementos. Lista vacia si period invalido.

code:
[cell repeats an earlier one in this file: 7656 characters]

params_schema:
  {"params":[{"name":"data","desc":"lista de precios o valores a suavizar (ej: precios de cierre diarios)"},{"name":"period","desc":"numero de periodos para el calculo de EMA (tipico: 5, 12, 26, 50, 200). Mayor periodo = mas suavizado."}],"output":"lista de valores EMA con len(data) - period + 1 elementos (mas corta que la entrada)"}


id: generate_gbm_prices_py_finance
name: generate_gbm_prices
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
tags: ["simulation","gbm","price","montecarlo","finance","stochastic"]
uses_functions: []
uses_types: []
returns: []
imports: ["numpy"]
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 98157b4a92515b8c41c694d7ed909ae894d13aeae5ca8c8db29c61e5e9fb35f6
signature:
  generate_gbm_prices(initial_price: float, n_ticks: int, sigma: float, mu: float, jump_intensity: float, jump_size_std: float, seed: int) -> list[float]

description:
  Genera serie de precios fundamentales con Geometric Brownian Motion + jump-diffusion. S(t+1) = S(t) * exp((mu - sigma^2/2)*dt + sigma*sqrt(dt)*Z + J*N).

example:
  ```python
  prices = generate_gbm_prices(
      initial_price=100.0,
      n_ticks=1000,
      sigma=0.02,
      mu=0.0,
      jump_intensity=0.01,
      jump_size_std=0.05,
      seed=42,
  )
  # prices[0] == 100.0
  # len(prices) == 1000
  ```

notes:
  Funcion pura — el seed fija el resultado deterministicamente.
  `jump_intensity=0.0` desactiva los saltos (GBM puro).
  `dt=1.0` por tick (tiempo discreto). Para tiempo continuo, ajustar sigma y mu en consecuencia.
  Requiere numpy para la generacion de numeros aleatorios y el calculo de exp.

code:
[cell repeats an earlier one in this file: 7656 characters]

params_schema:
  {"params":[{"name":"initial_price","desc":"precio inicial S(0) del activo subyacente (ej: 100.0)"},{"name":"n_ticks","desc":"numero de ticks/pasos en la simulacion (ej: 1000). Determina len(resultado)."},{"name":"sigma","desc":"volatilidad anualizada (ej: 0.02 para 2% por tick). Controlador clave del ruido."},{"name":"mu","desc":"drift anualizado (ej: 0.0 para random walk, 0.05 para uptrend). Retorno esperado por tick."},{"name":"jump_intensity","desc":"probabilidad de salto por tick en rango [0, 1] (ej: 0.01 para 1% por tick). 0.0 = GBM puro."},{"name":"jump_size_std","desc":"desviacion estandar del tamanio de los saltos (ej: 0.05 para jumps de ~5%). Normal(0, jump_size_std)."},{"name":"seed","desc":"semilla para reproducibilidad deterministica (ej: 42)"}],"output":"lista de precios con len(resultado) == n_ticks. resultado[0] == initial_price."}


id: generate_taker_order_py_finance
name: generate_taker_order
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
tags: ["simulation","taker","power-law","montecarlo","finance","order-book"]
uses_functions: []
uses_types: []
returns: []
imports: ["numpy"]
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e0d06eef1c6fbc8a21f40bd684c6eb46e355853190cefc5de92a3205c843d095
signature:
  generate_taker_order(alpha: float, size_min: float, size_max: float, buy_prob: float, seed: int | None) -> dict

description:
  Genera una market order de taker con tamano distribuido segun power-law (Pareto). Alpha bajo produce ordenes mas grandes (ballenas).

example:
  ```python
  order = generate_taker_order(
      alpha=2.0,
      size_min=1.0,
      size_max=100.0,
      buy_prob=0.5,
      seed=42,
  )
  # {'side': 'buy', 'qty': 3.7}
  ```

notes:
  Funcion pura cuando se fija seed. Con seed=None el resultado es no deterministico.
  La distribucion Pareto con alpha=2 modela bien la distribucion empirica de tamaños de ordenes en mercados reales.
  `size_max` actua como techo (clipping) para evitar ordenes extremas.
  Retorna dict con keys: `side` ('buy' o 'sell') y `qty` (float redondeado a 1 decimal).

code:
[cell repeats an earlier one in this file: 7656 characters]

params_schema:
  {"params":[{"name":"alpha","desc":"exponente de la distribucion Pareto (rango tipico: 1.5-3.0). Menor alpha = colas mas pesadas y ordenes mas grandes."},{"name":"size_min","desc":"tamanio minimo de orden (ej: 1.0)"},{"name":"size_max","desc":"tamanio maximo de orden (clipping superior, ej: 100.0)"},{"name":"buy_prob","desc":"probabilidad de que la orden sea BUY en rango [0, 1] (ej: 0.5 para igual BUY/SELL)"},{"name":"seed","desc":"semilla para reproducibilidad (int o None). Si None, no deterministica."}],"output":"dict con campos {'side': 'buy'|'sell', 'qty': float redondeado a 1 decimal}"}


id: hawkes_intensity_py_finance
name: hawkes_intensity
kind: function
lang: py
domain: finance
version: 1.0.0
purity: pure
tags: ["simulation","hawkes","stochastic-process","montecarlo","finance","point-process"]
uses_functions: []
uses_types: []
returns: []
imports: ["numpy"]
tests: []
file_path: python/functions/finance/finance.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 867368e2340bf7570a8319cd2b5cf53df00ca60c63108ab290dd09565f4410cd
signature:
  hawkes_intensity(base_rate: float, hawkes_alpha: float, hawkes_beta: float, event_times: list[float], current_time: float) -> float

description:
  Calcula la intensidad lambda(t) de un proceso de Hawkes en el tiempo actual. Modela la autocorrelacion temporal de eventos de mercado (rafagas de ordenes).

example:
  ```python
  intensity = hawkes_intensity(
      base_rate=1.0,
      hawkes_alpha=0.8,
      hawkes_beta=2.0,
      event_times=[0.5, 1.2, 1.8],
      current_time=2.5,
  )
  # Intensidad > base_rate por excitacion de eventos pasados
  ```

notes:
  Funcion pura — determinista dado el mismo historial de eventos.
  `hawkes_alpha` controla la magnitud del salto de intensidad por evento.
  `hawkes_beta` controla la velocidad de decaimiento (mayor beta = decaimiento mas rapido).
  La condicion de estabilidad del proceso es hawkes_alpha < hawkes_beta.
  Eventos con ti >= current_time se ignoran automaticamente.
  Retorna max(0.0, ...) para garantizar intensidad no negativa.

code:
[cell repeats an earlier one in this file: 7656 characters]

params_schema:
  {"params":[{"name":"base_rate","desc":"intensidad de linea base lambda(0) del proceso Hawkes (ej: 1.0 orden/minuto). Tasa base de eventos sin influencia."},{"name":"hawkes_alpha","desc":"magnitud del salto de intensidad por evento (rango: 0.0-1.0). Mayor alpha = mayor excitacion por eventos pasados."},{"name":"hawkes_beta","desc":"velocidad de decaimiento de la memoria (rango tipico: 0.5-3.0). Mayor beta = decaimiento mas rapido, memoria mas corta."},{"name":"event_times","desc":"lista de timestamps de eventos pasados (ej: [0.5, 1.2, 1.8]). Milisegundos o segundos segun escala."},{"name":"current_time","desc":"timestamp actual en la misma escala que event_times (ej: 2.5)"}],"output":"intensidad lambda(current_time) en rango [0.0, infinito). Valores \u003e base_rate indican excitacion por eventos recientes."}


