# functions

id: estimate_token_count_py_core
name: estimate_token_count
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def estimate_token_count(content: str) -> int
tags: ["tokens","estimation","nlp","cjk","text"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: ["texto vacio retorna cero","solo latin","solo CJK","texto mixto"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: bad61df097deb73febc71ce39d69f4473925cdc6620e538cd0a1c9ccdfa4488b
description:
  Estimacion rapida de tokens sin tokenizer. CJK chars cuentan ~0.7 token/char, otros non-whitespace ~0.3 token/char.

example:
  ```python
  estimate_token_count("hello world")  # 3
  estimate_token_count("中文语")        # 2  (3 * 0.7 = 2)
  estimate_token_count("")             # 0
  ```

notes:
  Funcion pura. No requiere ninguna dependencia externa. Precision aproximada: util para guardianes de limite de contexto antes de llamar a LLMs, no para conteo exacto de tokens BPE. CJK range: `[\u4e00-\u9fff\u3040-\u30ff\uac00-\ud7af]` (CJK unificado, Hiragana/Katakana, Hangul).

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"string de contenido a contar"}],"output":"entero estimando el numero de tokens en el contenido"}


id: excel_to_markdown_py_core
name: excel_to_markdown
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: excel_to_markdown(path: str, max_rows_per_sheet: int = 1000) -> str
tags: ["excel","markdown","xlsx","xls","conversion","parser","io"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["openpyxl","xlrd"]
test_file_path: python/functions/core/excel_to_markdown_test.py
file_path: python/functions/core/excel_to_markdown.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 298367fec629a02550817c1ea82b7ce0dbe79d93cbe6ac75b792387e57af41a3
description:
  Convierte un archivo Excel (.xlsx, .xls, .xlsm) a markdown con cada sheet como seccion H2. Soporta tipos de celda: fechas ISO, booleanos, errores Excel, numeros enteros y flotantes. Trunca sheets que superen max_rows_per_sheet.

example:
  ```python
  from excel_to_markdown import excel_to_markdown

  md = excel_to_markdown("report.xlsx")
  print(md)
  # ## Sheet: Ventas
  #
  # **Dimensions:** 101 x 4
  #
  # | Producto | Precio | Cantidad | Total |
  # | --- | --- | --- | --- |
  # | Manzana | 1 | 100 | 100 |
  # ...

  # Con limite de filas
  md = excel_to_markdown("big_file.xlsx", max_rows_per_sheet=50)
  ```

tests:
  ["xlsx con multiples sheets produce una seccion H2 por sheet","sheet vacio produce nota de sheet vacio","sheet truncado con nota de filas omitidas","sheet con formulas data_only muestra valores calculados","extension no soportada lanza ValueError","archivo inexistente lanza FileNotFoundError","dimensiones del sheet en metadata","tabla markdown con formato correcto"]

notes:
  - `.xlsx` y `.xlsm`: usa `openpyxl` con `data_only=True` (lee valores calculados, no formulas).
  - `.xls` (legacy): usa `xlrd`. Manejo de tipos especiales: EMPTY/BLANK → "", DATE → ISO 8601, BOOLEAN → "TRUE"/"FALSE", ERROR → codigo Excel (#NULL!, #DIV/0!, etc.), NUMBER → entero si no tiene decimales.
  - Fechas sin hora se formatean como `YYYY-MM-DD`; con hora como `YYYY-MM-DDTHH:MM:SS`.
  - Los pipes `|` dentro de celdas se escapan como `\|`.
  - Si `xlwt` no esta disponible, los tests .xls se saltan (xlwt solo se necesita para crear fixtures, no para leer).
  - Reimplementacion desde cero, inspirada conceptualmente en OpenViking (AGPL-3.0). Sin codigo copiado.

code:
  """Convierte archivos Excel a Markdown con cada sheet como seccion H2."""

  import os
  from pathlib import Path


  # Codigos de error Excel para xlrd
  _XL_ERROR_CODES = {
      0: "#NULL!",
      7: "#DIV/0!",
      15: "#VALUE!",
      23: "#REF!",
      29: "#NAME?",
      36: "#NUM!",
      42: "#N/A",
  }


  def _rows_to_markdown_table(rows: list[list[str]]) -> str:
      """Convierte filas de strings a tabla markdown."""
      if not rows:
          return ""

      header = rows[0]
      col_count = len(header)

      # Normalizar todas las filas al mismo numero de columnas
      normalized = []
      for row in rows:
          if len(row) < col_count:
              row = row + [""] * (col_count - len(row))
          normalized.append(row[:col_count])

      # Escapar pipes en celdas
      def escape(cell: str) -> str:
          return cell.replace("|", "\\|").replace("\n", " ")

      lines = []
      # Header
      lines.append("| " + " | ".join(escape(c) for c in normalized[0]) + " |")
      # Separator
      lines.append("| " + " | ".join("---" for _ in range(col_count)) + " |")
      # Data rows
      for row in normalized[1:]:
          lines.append("| " + " | ".join(escape(c) for c in row) + " |")

      return "\n".join(lines)


  def _cell_value_xlrd(cell, workbook) -> str:
      """Convierte una celda xlrd a string segun su tipo."""
      import xlrd

      ctype = cell.ctype

      if ctype in (xlrd.XL_CELL_EMPTY, xlrd.XL_CELL_BLANK):
          return ""
      elif ctype == xlrd.XL_CELL_DATE:
          try:
              dt = xlrd.xldate_as_datetime(cell.value, workbook.datemode)
              if dt.hour == 0 and dt.minute == 0 and dt.second == 0:
                  return dt.date().isoformat()
              return dt.isoformat()
          except Exception:
              return str(cell.value)
      elif ctype == xlrd.XL_CELL_BOOLEAN:
          return "TRUE" if cell.value else "FALSE"
      elif ctype == xlrd.XL_CELL_ERROR:
          return _XL_ERROR_CODES.get(int(cell.value), "#ERROR!")
      elif ctype == xlrd.XL_CELL_NUMBER:
          v = cell.value
          if v == int(v):
              return str(int(v))
          return str(v)
      elif ctype == xlrd.XL_CELL_TEXT:
          return str(cell.value)
      else:
          return str(cell.value)


  def _sheet_xlrd(sheet, workbook, max_rows: int) -> str:
      """Convierte un sheet xlrd a markdown."""
      nrows = sheet.nrows
      ncols = sheet.ncols

      lines = []
      lines.append(f"## Sheet: {sheet.name}")
      lines.append("")
      lines.append(f"**Dimensions:** {nrows} x {ncols}")
      lines.append("")

      if nrows == 0 or ncols == 0:
          lines.append("*(empty sheet)*")
          return "\n".join(lines)

      display_rows = min(nrows, max_rows)
      rows = []
      for r in range(display_rows):
          row_data = [_cell_value_xlrd(sheet.cell(r, c), workbook) for c in range(ncols)]
          rows.append(row_data)

      lines.append(_rows_to_markdown_table(rows))

      if nrows > max_rows:
          omitted = nrows - max_rows
          lines.append("")
          lines.append(f"*{omitted} rows omitted (max_rows_per_sheet={max_rows})*")

      return "\n".join(lines)


  def _cell_value_openpyxl(cell) -> str:
      """Convierte una celda openpyxl a string."""
      v = cell.value
      if v is None:
          return ""
      if isinstance(v, bool):
          return "TRUE" if v else "FALSE"
      if isinstance(v, float):
          if v == int(v):
              return str(int(v))
          return str(v)
      if isinstance(v, int):
          return str(v)
      # Fechas y datetimes
      import datetime
      if isinstance(v, datetime.datetime):
          if v.hour == 0 and v.minute == 0 and v.second == 0:
              return v.date().isoformat()
          return v.isoformat()
      if isinstance(v, datetime.date):
          return v.isoformat()
      return str(v)


  def _sheet_openpyxl(ws, max_rows: int) -> str:
      """Convierte un worksheet openpyxl a markdown."""
      all_rows = list(ws.iter_rows())
      nrows = len(all_rows)
      ncols = ws.max_column or 0

      lines = []
      lines.append(f"## Sheet: {ws.title}")
      lines.append("")
      lines.append(f"**Dimensions:** {nrows} x {ncols}")
      lines.append("")

      if nrows == 0 or ncols == 0:
          lines.append("*(empty sheet)*")
          return "\n".join(lines)

      display_rows = min(nrows, max_rows)
      rows = []
      for row in all_rows[:display_rows]:
          row_data = [_cell_value_openpyxl(cell) for cell in row]
          rows.append(row_data)

      lines.append(_rows_to_markdown_table(rows))

      if nrows > max_rows:
          omitted = nrows - max_rows
          lines.append("")
          lines.append(f"*{omitted} rows omitted (max_rows_per_sheet={max_rows})*")

      return "\n".join(lines)


  def excel_to_markdown(path: str, max_rows_per_sheet: int = 1000) -> str:
      """Convierte un archivo Excel (.xlsx, .xls, .xlsm) a markdown.

      Cada sheet se convierte en una seccion H2. Las filas se representan
      como tablas markdown. Si el numero de filas supera max_rows_per_sheet,
      el sheet se trunca y se añade una nota.

      Args:
          path: Ruta al archivo Excel (.xlsx, .xls, .xlsm).
          max_rows_per_sheet: Maximo de filas a incluir por sheet (default 1000).

      Returns:
          String markdown con todos los sheets del archivo.

      Raises:
          ValueError: Si la extension no es soportada.
          FileNotFoundError: Si el archivo no existe.
          Exception: Si hay errores leyendo el archivo.
      """
      p = Path(path)
      if not p.exists():
          raise FileNotFoundError(f"File not found: {path}")

      ext = p.suffix.lower()

      if ext == ".xls":
          import xlrd
          wb = xlrd.open_workbook(path)
          sections = []
          for sheet_name in wb.sheet_names():
              sheet = wb.sheet_by_name(sheet_name)
              sections.append(_sheet_xlrd(sheet, wb, max_rows_per_sheet))
          return "\n\n".join(sections)

      elif ext in (".xlsx", ".xlsm"):
          import openpyxl
          wb = openpyxl.load_workbook(path, data_only=True)
          sections = []
          for ws in wb.worksheets:
              sections.append(_sheet_openpyxl(ws, max_rows_per_sheet))
          return "\n\n".join(sections)

      else:
          raise ValueError(f"Unsupported extension '{ext}'. Use .xlsx, .xls, or .xlsm.")

params_schema:
  {"params":[{"name":"path","desc":"ruta al archivo Excel (.xlsx, .xls, .xlsm)"},{"name":"max_rows_per_sheet","desc":"limite maximo de filas a procesar por sheet"}],"output":"string markdown con cada sheet como seccion H2 y sus datos en tablas"}


id: extract_frontmatter_py_core
name: extract_frontmatter
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def extract_frontmatter(content: str) -> tuple[str, dict | None]
tags: ["markdown","frontmatter","yaml","parsing"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","yaml"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 23f664a0b4212ac611ece53c0cc43dcfe145617eabc961d3e574d5156fd9674f
description:
  Extrae YAML frontmatter (delimitado por ---) del inicio de un string markdown. Retorna el contenido sin frontmatter y el dict parseado (o None si no hay).

example:
  ```python
  content = "---\ntitle: Hello\nauthor: Alice\n---\n# Body\n"
  remaining, data = extract_frontmatter(content)
  # remaining = "# Body\n"
  # data = {"title": "Hello", "author": "Alice"}

  no_fm = "# Just markdown\n\nNo frontmatter."
  remaining, data = extract_frontmatter(no_fm)
  # remaining == no_fm
  # data is None
  ```

tests:
  ["contenido con frontmatter","sin frontmatter retorna None","frontmatter vacio","frontmatter con listas"]

notes:
  Funcion pura. Usa `yaml.safe_load` si PyYAML esta disponible; si no, cae back a un parser simple de `key: value`. Solo reconoce frontmatter al inicio estricto del string (posicion 0). El bloque debe estar delimitado por `---\n` de apertura y `\n---\n` de cierre.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"string con contenido markdown potencialmente precedido por YAML frontmatter"}],"output":"tupla (contenido_sin_frontmatter, dict_parseado_o_None)"}


id: extract_json_from_llm_py_core
name: extract_json_from_llm
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def extract_json_from_llm(content: str) -> dict
description: Extrae y parsea JSON de respuestas LLM. Maneja bloques ```json, trailing commas, None->null.
tags: ["json","llm","parsing","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: ["json"]
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f40b8c6c0368e47a5e6714394d5177cc0090d6d8099b8b81aa1526021c3145fd
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  raw = '```json\n{"key": "value", "items": [1, 2, 3,]}\n```'
  result = extract_json_from_llm(raw)
  # {"key": "value", "items": [1, 2, 3]}
  ```

notes:
  Funcion pura. Maneja errores comunes de LLMs: trailing commas, `None` en lugar de `null`, whitespace extra. Retorna dict vacio si el JSON es irrecuperable.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"string con respuesta LLM que contiene JSON (puede estar en bloque ```json)"}],"output":"dict parseado desde el JSON; dict vacio si no se puede recuperar"}


id: extract_markdown_headers_py_core
name: extract_markdown_headers
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def extract_markdown_headers(markdown_content: str) -> tuple[list[dict], list[str]]
description: Extrae todos los headers (h1-h6) de markdown con nivel y numero de linea, ignorando code blocks.
tags: ["markdown","parsing","headers","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c5aba41a9019c311a968cb8bb6e614f7617e0d3b13702ce18afd191c45e86009
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/page_index_md.py
example:
  ```python
  md = "# Title\n\nSome text\n\n## Section\n\n```\n# not a header\n```"
  headers, lines = extract_markdown_headers(md)
  # headers = [{"title": "Title", "level": 1, "line_num": 1}, {"title": "Section", "level": 2, "line_num": 5}]
  ```

notes:
  Funcion pura. Detecta y omite bloques de codigo (triple backtick). Retorna tupla: (lista de headers, lista de lineas originales).

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"markdown_content","desc":"string con contenido markdown"}],"output":"tupla (lista_de_headers, lista_de_lineas_originales) donde headers son dicts con title/level/line_num"}


id: extract_pdf_bookmarks_py_core
name: extract_pdf_bookmarks
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def extract_pdf_bookmarks(pdf) -> list[dict]
tags: ["pdf","bookmarks","outlines","parsing","pdfplumber"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["pdfplumber"]
tests: []
file_path: python/functions/core/extract_pdf_bookmarks.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 35b3635274d868d71f7de81a97feed0c3582e3ac40eb948e13cbc7b047d623c8
description:
  Extrae la estructura de bookmarks/outlines de un PDF abierto con pdfplumber. Retorna lista de dicts con level (1-6), title y page_num.

example:
  ```python
  import pdfplumber
  from extract_pdf_bookmarks import extract_pdf_bookmarks

  with pdfplumber.open("document.pdf") as pdf:
      bookmarks = extract_pdf_bookmarks(pdf)
      for bm in bookmarks:
          print(f"{'#' * bm['level']} {bm['title']} (page {bm['page_num']})")
  ```

notes:
  Recibe un objeto `pdfplumber.PDF` ya abierto (no un path). Construye un mapping interno `objid -> page_number` desde `pdf.pages` para resolver los destinos de outline. El nivel se limita al rango [1, 6] para compatibilidad markdown. Retorna lista vacia si el PDF no tiene outlines o si `get_outlines()` falla. Impure porque accede al estado interno de un objeto PDF ya abierto.

code:
  """Extract the bookmark/outline structure from a PDF opened with pdfplumber."""

  import pdfplumber


  def extract_pdf_bookmarks(pdf: pdfplumber.PDF) -> list[dict]:
      """Extract bookmarks/outlines from an open pdfplumber PDF object.

      Args:
          pdf: An open pdfplumber.PDF object.

      Returns:
          list[dict]: List of {"level": int, "title": str, "page_num": int | None}.
                      Level is clamped to [1, 6]. Returns empty list if no outlines.
      """
      try:
          outlines = pdf.doc.get_outlines()
      except Exception:
          return []

      if not outlines:
          return []

      # Build objid -> page_number mapping
      objid_to_page: dict[int, int] = {}
      for i, page in enumerate(pdf.pages):
          try:
              obj = page.page_obj
              objid_to_page[obj.objid] = i + 1  # 1-indexed page numbers
          except Exception:
              pass

      bookmarks = []
      for item in outlines:
          try:
              level = item[0]  # integer level from get_outlines
              title = item[1]
              dest = item[2]  # destination: page object or list

              # Clamp level to [1, 6]
              level = max(1, min(6, level))

              # Resolve destination to page number
              page_num = None
              if dest is not None:
                  if isinstance(dest, list) and len(dest) > 0:
                      # dest[0] is the page object
                      page_obj = dest[0]
                      try:
                          page_num = objid_to_page.get(page_obj.objid)
                      except Exception:
                          pass
                  else:
                      try:
                          page_num = objid_to_page.get(dest.objid)
                      except Exception:
                          pass

              bookmarks.append({"level": level, "title": str(title), "page_num": page_num})
          except Exception:
              continue

      return bookmarks

params_schema:
  {"params":[{"name":"pdf","desc":"objeto pdfplumber.PDF abierto"}],"output":"lista de dicts con campos level (1-6), title y page_num para cada bookmark"}


id: extract_pdf_text_py_core
name: extract_pdf_text
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def extract_pdf_text(pdf_path: str) -> str
description: Extrae todo el texto de un PDF concatenando todas las paginas. Usa PyPDF2.
tags: ["pdf","text","extraction","parsing"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["PyPDF2"]
example:
  ```python
  text = extract_pdf_text("/path/to/document.pdf")
  print(len(text)) # total characters
  ```

tests: []
file_path: python/functions/core/extract_pdf_text.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:44:15Z
props: []
emits: []
variant: []
content_hash: 8e77a771fd9414fc17a034dfd24eda926e20a0be55ca9269c3490002826970e2
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
notes:
  Requiere `pip install PyPDF2`. Extraccion basica de texto — no maneja OCR ni PDFs escaneados. Para PDFs complejos considerar PyMuPDF.

code:
  """Extract all text from a PDF file using PyPDF2."""

  import PyPDF2


  def extract_pdf_text(pdf_path: str) -> str:
      """Extract all text from a PDF file.

      Args:
          pdf_path: Path to the PDF file.

      Returns:
          str: Concatenated text from all pages.
      """
      pdf_reader = PyPDF2.PdfReader(pdf_path)
      text = ""
      for page in pdf_reader.pages:
          text += page.extract_text() or ""
      return text

params_schema:
  {"params":[{"name":"pdf_path","desc":"ruta al archivo PDF desde el cual extraer el texto"}],"output":"string: texto concatenado de todas las páginas del PDF"}


id: extract_text_from_file_py_core
name: extract_text_from_file
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: extract_text_from_file(file_path: str) -> str
tags: ["text","pdf","markdown","txt","encoding","extraction","file","io"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["os","fitz (PyMuPDF)","charset_normalizer","chardet"]
test_file_path: python/functions/core/extract_text_from_file_test.py
file_path: python/functions/core/extract_text_from_file.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 87cbf8b50c5dbf344779540ab951ce225c2f924da6516f26579e36fc096f2fc6
description:
  Extrae texto plano de un archivo. Soporta PDF (PyMuPDF), Markdown y TXT con deteccion automatica de encoding.

example:
  ```python
  # PDF
  text = extract_text_from_file("report.pdf")

  # Markdown
  text = extract_text_from_file("README.md")

  # TXT con encoding desconocido
  text = extract_text_from_file("notes.txt")
  ```

tests:
  ["PDF con texto extrae contenido correctamente","archivo MD UTF-8 retorna contenido","archivo TXT latin-1 detecta encoding","archivo inexistente lanza FileNotFoundError","extension no soportada lanza ValueError"]

notes:
  Para PDF usa PyMuPDF (`fitz`) que produce mejor texto que PyPDF2, especialmente en PDFs con columnas o layout complejo. Las paginas se unen con `\n\n`.

  La deteccion de encoding para archivos de texto sigue este orden de prioridad:
  1. Intenta UTF-8 directamente
  2. `charset_normalizer.from_bytes().best().encoding`
  3. `chardet.detect(data)["encoding"]`
  4. UTF-8 con `errors='replace'` como ultimo recurso

  Diferencia con `extract_pdf_text_py_core`: esa funcion usa PyPDF2 y solo soporta PDF. Esta funcion usa PyMuPDF y soporta ademas MD y TXT con deteccion de encoding.

code:
  """Extract plain text from PDF, Markdown, or TXT files."""


  SUPPORTED_EXTENSIONS = {".pdf", ".md", ".markdown", ".txt"}


  def _detect_encoding(data: bytes) -> str:
      """Detect encoding of raw bytes using multiple fallback strategies."""
      # Strategy 1: UTF-8
      try:
          data.decode("utf-8")
          return "utf-8"
      except UnicodeDecodeError:
          pass

      # Strategy 2: charset_normalizer
      try:
          from charset_normalizer import from_bytes

          result = from_bytes(data).best()
          if result is not None and result.encoding:
              return result.encoding
      except ImportError:
          pass

      # Strategy 3: chardet
      try:
          import chardet

          detected = chardet.detect(data)
          if detected and detected.get("encoding"):
              return detected["encoding"]
      except ImportError:
          pass

      # Last resort: UTF-8 with replacement
      return "utf-8"


  def extract_text_from_file(file_path: str) -> str:
      """Extract plain text from a file. Supports PDF, Markdown and TXT.

      For PDF files uses PyMuPDF (fitz) to extract text from each page,
      joining them with double newlines. For text-based files (.md, .markdown,
      .txt) reads the file with automatic encoding detection.

      Args:
          file_path: Absolute or relative path to the file.

      Returns:
          str: Extracted plain text content.

      Raises:
          FileNotFoundError: If the file does not exist.
          ValueError: If the file extension is not supported.
          ImportError: If PyMuPDF is not installed and a PDF is provided.
      """
      import os

      if not os.path.exists(file_path):
          raise FileNotFoundError(f"File not found: {file_path}")

      _, ext = os.path.splitext(file_path.lower())

      if ext == ".pdf":
          try:
              import fitz  # PyMuPDF
          except ImportError as e:
              raise ImportError(
                  "PyMuPDF is required for PDF extraction. "
                  "Install it with: pip install PyMuPDF"
              ) from e

          doc = fitz.open(file_path)
          pages = [page.get_text() for page in doc]
          return "\n\n".join(pages)

      elif ext in {".md", ".markdown", ".txt"}:
          with open(file_path, "rb") as f:
              raw = f.read()

          encoding = _detect_encoding(raw)
          try:
              return raw.decode(encoding)
          except (UnicodeDecodeError, LookupError):
              return raw.decode("utf-8", errors="replace")

      else:
          raise ValueError(
              f"Unsupported file extension: '{ext}'. "
              f"Supported: {', '.join(sorted(SUPPORTED_EXTENSIONS))}"
          )

params_schema:
  {"params":[{"name":"file_path","desc":"ruta al archivo (PDF, MD o TXT)"}],"output":"string con texto extraido del archivo"}


id: fetch_and_parse_url_py_core
name: fetch_and_parse_url
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: fetch_and_parse_url(url: str, timeout: float = 30.0) -> str
tags: ["http","fetch","html","markdown","parse","url","scraping"]
uses_functions: ["detect_url_type_py_core","html_to_markdown_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["httpx"]
tests: []
file_path: python/functions/core/fetch_and_parse_url.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c8329d93009d3c57222061e9f3da03e8378979c2d36c04c65d630fef7de47ba7
description:
  Descarga una pagina web y la convierte a markdown. Combina detect_url_type + fetch HTML + html_to_markdown en una sola operacion.

example:
  ```python
  from core.fetch_and_parse_url import fetch_and_parse_url

  # Descargar y convertir una pagina web
  md = fetch_and_parse_url("https://example.com")
  print(md)

  # Con timeout personalizado
  md = fetch_and_parse_url("https://en.wikipedia.org/wiki/Python", timeout=15.0)
  ```

notes:
  Algoritmo:
  1. `detect_url_type(url)` determina el tipo de contenido (por patron, extension o HEAD request).
  2. Si es `code_repository` → lanza Exception (requiere git clone, no HTTP fetch).
  3. Si es `pdf` → lanza Exception (requiere pdfminer/pypdf, no incluido).
  4. `httpx.get(url)` descarga el contenido con follow_redirects.
  5. Si es `webpage` o Content-Type HTML → `html_to_markdown(raw_html)`.
  6. Si es `markdown`, `text` o codigo → retorna el texto directamente.

  Lanza `Exception` con mensaje descriptivo en cualquier fallo de red o tipo no soportado.

  Funcion impura: hace I/O (HTTP requests).

code:
  """Descarga una pagina web y la convierte a markdown."""

  from __future__ import annotations


  def fetch_and_parse_url(url: str, timeout: float = 30.0) -> str:
      """Descarga una pagina web y la convierte a markdown.

      Detecta el tipo de URL con detect_url_type, descarga el contenido con
      httpx y lo convierte al formato apropiado:
      - webpage: fetch HTML → html_to_markdown
      - markdown: retorna el texto directamente
      - text/code: retorna el texto directamente
      - pdf: retorna stub (requiere dependencia externa)
      - code_repository: retorna stub (requiere clonar repo)

      Args:
          url: URL a descargar y parsear.
          timeout: Timeout en segundos para las peticiones HTTP.

      Returns:
          Contenido de la URL en formato markdown.

      Raises:
          Exception: Si falla la descarga (timeout, DNS, HTTP error) o el tipo
                     de URL no es soportado.
      """
      import httpx

      from detect_url_type import detect_url_type
      from html_to_markdown import html_to_markdown

      # Detectar tipo de URL (puede hacer HEAD request)
      url_type, _meta = detect_url_type(url, timeout=timeout)

      if url_type == "code_repository":
          raise Exception(
              f"fetch_and_parse_url: code_repository URLs require git clone, not supported. url={url!r}"
          )

      if url_type == "pdf":
          raise Exception(
              f"fetch_and_parse_url: PDF parsing requires external dependency (pdfminer/pypdf). url={url!r}"
          )

      # Fetch content via GET
      try:
          response = httpx.get(url, timeout=timeout, follow_redirects=True)
          response.raise_for_status()
      except httpx.HTTPStatusError as exc:
          raise Exception(
              f"fetch_and_parse_url: HTTP {exc.response.status_code} for {url!r}"
          ) from exc
      except Exception as exc:
          raise Exception(f"fetch_and_parse_url: request failed for {url!r}: {exc}") from exc

      content_type = response.headers.get("content-type", "").lower()
      raw_text = response.text

      if url_type == "webpage" or "text/html" in content_type:
          return html_to_markdown(raw_text)

      # markdown, text, or code files — return as-is
      return raw_text

params_schema:
  {"params":[{"name":"url","desc":"URL a descargar y convertir"},{"name":"timeout","desc":"timeout en segundos para el HTTP request"}],"output":"string markdown con el contenido convertido desde la pagina web"}


id: filter_list_py_core
name: filter_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def filter_list(xs: list, pred: callable) -> list
description: Filtra una lista aplicando un predicado sin mutar la original.
tags: ["list","functional","filter","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  evens = filter_list([1, 2, 3, 4], lambda n: n % 2 == 0)
  # [2, 4]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. No muta la lista original. Equivalente a `[x for x in xs if pred(x)]`.
content_hash: 300f8efb03f8a888b2abd1d68657254a40e50467089c1010b72318bbb0a8bbd9
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista a filtrar"},{"name":"pred","desc":"predicado (funcion que retorna bool) para seleccionar elementos"}],"output":"lista con solo los elementos que cumplen el predicado"}


id: find_py_core
name: find
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def find(xs: list, pred: callable)
description: Encuentra el primer elemento que cumple el predicado. Retorna None si no hay coincidencia.
tags: ["list","functional","find","search","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = find([1, 2, 3, 4], lambda n: n > 2)
  # 3
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Retorna None si ningun elemento cumple el predicado. Cortocircuita al primer match.
content_hash: 522cafc65271b3d851763f91df8a0aedc268b5926a01acbdfa1483899be17a60
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista donde buscar"},{"name":"pred","desc":"predicado (funcion que retorna bool) para encontrar un elemento"}],"output":"primer elemento que cumple el predicado, o None si no hay coincidencia"}


id: find_headings_py_core
name: find_headings
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def find_headings(content: str) -> list[tuple[int, int, str, int]]
tags: ["markdown","headings","parsing","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 05c40c3c4b0071820d34026166eb0b64c60d7411997671bc527e79cb487e6c39
description:
  Encuentra todos los headings markdown (# a ######), excluyendo los que estan dentro de code blocks, HTML comments y bloques indentados. Retorna lista de (start_pos, end_pos, title, level).

example:
  ```python
  content = "# Title\n\nSome text\n\n## Section\n\n```\n# Ignored\n```\n"
  headings = find_headings(content)
  # [(0, 7, "Title", 1), (22, 33, "Section", 2)]
  # (positions approximated)
  ```

tests:
  ["headings normales detectados correctamente","headings dentro de code blocks no detectados","headings escapados ignorados","headings en HTML comments ignorados"]

notes:
  Funcion pura. Excluye tres tipos de contextos: bloques de codigo triple backtick, comentarios HTML (`<!-- ... -->`), y lineas indentadas con 4 espacios o tabulacion. Tambien filtra headings precedidos por backslash (`\#`). Diferencia clave respecto a `extract_markdown_headers`: esta funcion retorna posiciones de caracter, no numeros de linea, lo que facilita la extraccion de contenido entre headings.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"string con contenido markdown"}],"output":"lista de tuplas (start_pos, end_pos, title, level) para cada heading detectado"}


id: find_index_py_core
name: find_index
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def find_index(xs: list, pred: callable) -> int
description: Encuentra el indice del primer elemento que cumple el predicado. Retorna -1 si no hay coincidencia.
tags: ["list","functional","find","index","search","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  idx = find_index([10, 20, 30, 40], lambda n: n > 25)
  # 2
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Retorna -1 si ningun elemento cumple el predicado.
content_hash: ded9f0432c6d2bed815b60acc4b1d175ff041cb3533f18318bf0210cf7700c75
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista donde buscar"},{"name":"pred","desc":"predicado (funcion que retorna bool) para encontrar un indice"}],"output":"entero con el indice del primer elemento que cumple, o -1 si no hay"}


id: flat_map_py_core
name: flat_map
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def flat_map(xs: list, fn: callable) -> list
description: Aplica una funcion que retorna listas a cada elemento y aplana el resultado un nivel.
tags: ["list","functional","flatmap","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = flat_map([1, 2, 3], lambda n: [n, n * 10])
  # [1, 10, 2, 20, 3, 30]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Equivalente a flatten(map_list(xs, fn)). Solo aplana un nivel.
content_hash: 813c024a0699cd37db249749bf43ab15f1d0b6bd784988e108f2a8f79f118bfd
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista a transformar"},{"name":"fn","desc":"funcion que recibe un elemento y retorna una lista"}],"output":"lista plana resultado de aplicar fn a cada elemento y aplanar un nivel"}


id: flatten_py_core
name: flatten
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def flatten(xss: list) -> list
description: Aplana una lista de listas un nivel, concatenando las sublistas.
tags: ["list","functional","flatten","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = flatten([[1, 2], [3], [4, 5]])
  # [1, 2, 3, 4, 5]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Solo aplana un nivel de anidamiento.
content_hash: 96915997f8c298382e1f9228cfb197156b1efd52d1c8a6275b6daa688ffe7f1c
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xss","desc":"lista de listas a aplanar"}],"output":"lista plana con todos los elementos concatenados un nivel"}


id: flatten_tree_py_core
name: flatten_tree
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def flatten_tree(structure: Any) -> list[dict]
description: Aplana un arbol jerarquico (dict con 'nodes') a lista plana sin hijos. Deep copy de cada nodo.
tags: ["tree","flatten","hierarchy","functional"]
uses_functions: []
uses_types: []
returns: []
imports: ["copy"]
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 94e862a8db22802dce148e40532f04c1acdb9b2cd3bb9c6384f2ae7ecea299a9
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  tree = [{"title": "A", "nodes": [{"title": "A1", "nodes": []}]}]
  flatten_tree(tree)
  # [{"title": "A"}, {"title": "A1"}]
  ```

notes:
  Funcion pura. Usa deep copy para no mutar el arbol original. Elimina el campo 'nodes' de cada nodo aplanado.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"structure","desc":"arbol jerarquico (dict con campo 'nodes' anidado recursivamente)"}],"output":"lista plana de dicts sin campo 'nodes', deep copy de cada nodo"}


id: format_iso8601_py_core
name: format_iso8601
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: format_iso8601(dt: datetime) -> str
tags: ["datetime","iso8601","format","time","utc"]
uses_functions: []
uses_types: []
returns: []
imports: ["datetime"]
test_file_path: python/functions/core/format_iso8601_test.py
file_path: python/functions/core/format_iso8601.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 762c1f38f78a8ac2d49074a18bafef87612af53ead0cbe49b48e915336566453
description:
  Formatea un datetime a ISO 8601 UTC con milisegundos. Formato: yyyy-MM-ddTHH:mm:ss.SSSZ. Si naive asume UTC, si aware convierte a UTC.

example:
  ```python
  from datetime import datetime, timezone, timedelta
  from format_iso8601 import format_iso8601

  # Naive (asume UTC)
  s = format_iso8601(datetime(2026, 2, 21, 13, 20, 23, 147000))
  # "2026-02-21T13:20:23.147Z"

  # Con timezone +8
  tz8 = timezone(timedelta(hours=8))
  s = format_iso8601(datetime(2026, 2, 21, 21, 20, 23, 147000, tzinfo=tz8))
  # "2026-02-21T13:20:23.147Z"
  ```

tests:
  ["datetime naive formateado como UTC","datetime con timezone convertido a UTC","datetime UTC sin conversion"]

notes:
  Algoritmo:
  1. Si naive: `dt.replace(tzinfo=timezone.utc)`.
  2. Si aware: `dt.astimezone(timezone.utc)`.
  3. `dt.isoformat(timespec="milliseconds").replace("+00:00", "Z")`.

  Funcion pura. No hace I/O ni tiene efectos secundarios.

code:
  """Formatea un datetime a ISO 8601 UTC con milisegundos."""

  from datetime import datetime, timezone


  def format_iso8601(dt: datetime) -> str:
      """Formatea un datetime a ISO 8601 UTC con milisegundos.

      Formato de salida: ``yyyy-MM-ddTHH:mm:ss.SSSZ``

      Si el datetime es naive (sin tzinfo), se asume UTC.
      Si el datetime es aware, se convierte a UTC antes de formatear.

      Args:
          dt: datetime a formatear. Puede ser naive o aware.

      Returns:
          String ISO 8601 en UTC con milisegundos, terminando en 'Z'.
      """
      if dt.tzinfo is None:
          dt = dt.replace(tzinfo=timezone.utc)
      else:
          dt = dt.astimezone(timezone.utc)
      return dt.isoformat(timespec="milliseconds").replace("+00:00", "Z")

params_schema:
  {"params":[{"name":"dt","desc":"datetime object (puede ser naive o aware)"}],"output":"string ISO 8601 UTC con milisegundos formato yyyy-MM-ddTHH:mm:ss.SSSZ"}


id: format_simplified_py_core
name: format_simplified
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: format_simplified(dt: datetime, now: datetime) -> str
tags: ["datetime","format","time","human","display"]
uses_functions: []
uses_types: []
returns: []
imports: ["datetime"]
test_file_path: python/functions/core/format_simplified_test.py
file_path: python/functions/core/format_simplified.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e3606391d5246c178478ba865432a70864a7dc0bf768d82c44bc7efc4ec822cf
description:
  Formato humano simplificado: si dt es del mismo dia que now muestra HH:MM:SS, si no muestra YYYY-MM-DD.

example:
  ```python
  from datetime import datetime
  from format_simplified import format_simplified

  now = datetime(2026, 2, 21, 15, 0, 0)

  # Mismo dia
  s = format_simplified(datetime(2026, 2, 21, 9, 30, 0), now)
  # "09:30:00"

  # Dia anterior
  s = format_simplified(datetime(2026, 2, 20, 9, 30, 0), now)
  # "2026-02-20"
  ```

tests:
  ["mismo dia muestra formato hora","dia anterior muestra formato fecha","exactamente 24h muestra formato fecha"]

notes:
  Algoritmo:
  1. Remover tzinfo de ambos datetimes para comparacion simple (`replace(tzinfo=None)`).
  2. Si `(now - dt).days < 1`: retornar `dt.strftime("%H:%M:%S")`.
  3. Si no: retornar `dt.strftime("%Y-%m-%d")`.

  El umbral de 1 dia en `timedelta.days` significa que cualquier diferencia
  menor a 24 horas se muestra como hora. Un dt exactamente 24h atras
  tendra `days == 1`, mostrando fecha.

  Funcion pura. No hace I/O ni tiene efectos secundarios.

code:
  """Formato humano simplificado de datetime: hora si es hoy, fecha si es otro dia."""

  from datetime import datetime


  def format_simplified(dt: datetime, now: datetime) -> str:
      """Formato humano simplificado de datetime.

      Si ``dt`` es del mismo dia que ``now`` (diferencia < 1 dia), retorna
      la hora en formato ``HH:MM:SS``. En caso contrario retorna la fecha
      en formato ``YYYY-MM-DD``.

      Args:
          dt: datetime a formatear.
          now: datetime de referencia (el momento actual).

      Returns:
          String ``HH:MM:SS`` si mismo dia, ``YYYY-MM-DD`` si otro dia.
      """
      dt_naive = dt.replace(tzinfo=None)
      now_naive = now.replace(tzinfo=None)
      diff = now_naive - dt_naive
      if diff.days < 1:
          return dt.strftime("%H:%M:%S")
      return dt.strftime("%Y-%m-%d")

params_schema:
  {"params":[{"name":"dt","desc":"datetime a formatear"},{"name":"now","desc":"datetime actual para comparacion"}],"output":"string con formato humano simplificado: HH:MM:SS si mismo dia, YYYY-MM-DD si otro dia"}


id: format_table_to_markdown_py_core
name: format_table_to_markdown
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def format_table_to_markdown(rows: list[list[str]], has_header: bool = True) -> str
tags: ["markdown","table","formatting","text","pure"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/format_table_to_markdown_test.py
file_path: python/functions/core/format_table_to_markdown.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f00dd3baebc5a88bd9be888e6982124f18622ead3ae5bf1f590ade99f294c686
description:
  Convierte una lista 2D de celdas a tabla markdown con alineacion de columnas. Escapa pipes en celdas y añade separador header.

example:
  ```python
  rows = [["Name", "Age"], ["Alice", "30"], ["Bob", "25"]]
  md = format_table_to_markdown(rows)
  # | Name  | Age |
  # | ----- | --- |
  # | Alice | 30  |
  # | Bob   | 25  |
  ```

tests:
  ["tabla normal","tabla con celdas vacias","tabla con 1 fila","tabla vacia","celdas con pipes","sin header"]

notes:
  Funcion pura. No tiene dependencias externas. Calcula el ancho maximo por columna para alinear. El separador usa minimo 3 guiones por columna para cumplir con la especificacion markdown. Escapa los pipes dentro de celdas con `\|`. Si `has_header=False`, omite la fila separadora.

code:
  """Convert a 2D list of cells to a markdown table with column alignment."""


  def format_table_to_markdown(rows: list[list[str]], has_header: bool = True) -> str:
      """Convert a 2D list of cells to a markdown table.

      Args:
          rows: 2D list where each inner list is a row of cell strings.
          has_header: If True, the first row is treated as the header row.

      Returns:
          str: Markdown table string. Returns empty string for empty input.
      """
      if not rows:
          return ""

      def escape_cell(cell: str) -> str:
          return str(cell).replace("|", "\\|")

      # Determine column count from widest row
      col_count = max(len(row) for row in rows)

      # Pad rows to same column count
      padded = [row + [""] * (col_count - len(row)) for row in rows]

      # Escape pipe characters in all cells
      escaped = [[escape_cell(cell) for cell in row] for row in padded]

      # Calculate max width per column
      col_widths = [
          max(len(escaped[r][c]) for r in range(len(escaped)))
          for c in range(col_count)
      ]
      col_widths = [max(w, 3) for w in col_widths]  # minimum width of 3 for separator

      def format_row(row: list[str]) -> str:
          cells = [cell.ljust(col_widths[i]) for i, cell in enumerate(row)]
          return "| " + " | ".join(cells) + " |"

      lines = []

      if has_header and len(escaped) >= 1:
          lines.append(format_row(escaped[0]))
          separator = "| " + " | ".join("-" * col_widths[i] for i in range(col_count)) + " |"
          lines.append(separator)
          for row in escaped[1:]:
              lines.append(format_row(row))
      else:
          for row in escaped:
              lines.append(format_row(row))

      return "\n".join(lines)

params_schema:
  {"params":[{"name":"rows","desc":"lista 2D de listas con celdas (cada fila es una lista de strings)"},{"name":"has_header","desc":"si True la primera fila se trata como header y se anade separador"}],"output":"string con tabla markdown formateada con pipes y alineacion de columnas"}


id: format_tree_structure_py_core
name: format_tree_structure
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def format_tree_structure(structure: Any, order: list[str] = None) -> Any
description: Reordena campos de cada nodo de un arbol segun orden de claves especificado.
tags: ["tree","format","order","structure"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 75eccec381c981f23d487e35894b3a3d56c4c66d856b9e187d3ec2d2d6843b4f
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  tree = [{"text": "...", "title": "Intro", "node_id": "0001"}]
  format_tree_structure(tree, order=["title", "node_id", "text"])
  # [{"title": "Intro", "node_id": "0001", "text": "..."}]
  ```

notes:
  Funcion pura. Elimina nodos vacios (nodes=[]) automaticamente. Claves no listadas en order se descartan.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"structure","desc":"arbol jerarquico (dict con campo 'nodes' anidado recursivamente)"},{"name":"order","desc":"lista de nombres de campos en el orden deseado para reordenar"}],"output":"estructura arbol con los campos de cada nodo reordenados segun order"}


id: from_csv_py_core
name: from_csv
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: from_csv(text: str, delimiter: str = ',', has_header: bool = True) -> list[dict]
tags: ["csv","parser","import","tabular","format"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/from_csv_test.py
file_path: python/functions/core/from_csv.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 04293f5f76f14588e0aebe756ea36a85d0c28c2a2085228387190545c3531062
description:
  Parser CSV a datos tabulares. Complemento de to_csv. Soporta campos entre comillas con escaping RFC 4180. Si has_header=False, genera keys col_0, col_1, etc.

example:
  ```python
  text = "nombre,edad\r\nAna,30\r\nBob,25"
  rows = from_csv(text)
  # [{"nombre": "Ana", "edad": "30"}, {"nombre": "Bob", "edad": "25"}]

  # Sin header
  text = "Ana,30\nBob,25"
  rows = from_csv(text, has_header=False)
  # [{"col_0": "Ana", "col_1": "30"}, {"col_0": "Bob", "col_1": "25"}]

  # Con escaping
  text = 'a,b\r\n"dijo ""hola""","uno,dos"'
  rows = from_csv(text)
  # [{"a": 'dijo "hola"', "b": "uno,dos"}]
  ```

tests:
  ["csv simple con header","campos con escaping","sin header keys generadas","lineas vacias ignoradas","un solo campo por fila"]

notes:
  Parser manual sin el modulo csv de stdlib. Normaliza CRLF y LF antes de procesar.
  Ignora lineas vacias. Todos los valores son strings — la conversion de tipos queda a cargo del caller.

code:
  """Parser CSV a datos tabulares (RFC 4180). Complemento de to_csv."""


  def _parse_row(line: str, delimiter: str) -> list[str]:
      """Parsea una linea CSV respetando campos entre comillas (RFC 4180)."""
      fields: list[str] = []
      field_chars: list[str] = []
      in_quotes = False
      i = 0

      while i < len(line):
          ch = line[i]

          if in_quotes:
              if ch == '"':
                  # Comilla doble escapada o cierre de campo
                  if i + 1 < len(line) and line[i + 1] == '"':
                      field_chars.append('"')
                      i += 2
                      continue
                  else:
                      in_quotes = False
              else:
                  field_chars.append(ch)
          else:
              if ch == '"' and not field_chars:
                  in_quotes = True
              elif ch == delimiter:
                  fields.append("".join(field_chars))
                  field_chars = []
              else:
                  field_chars.append(ch)
          i += 1

      fields.append("".join(field_chars))
      return fields


  def from_csv(
      text: str,
      delimiter: str = ",",
      has_header: bool = True,
  ) -> list[dict]:
      """Parser CSV a lista de dicts.

      Complemento de to_csv. Soporta campos entre comillas con escaping RFC 4180.
      Si has_header=False, genera keys col_0, col_1, etc.

      Args:
          text: Contenido CSV completo como string.
          delimiter: Separador de campos. Por defecto coma.
          has_header: Si True, primera fila es el encabezado.
                      Si False, genera keys col_0, col_1, ...

      Returns:
          Lista de dicts. Lista vacia si el texto esta vacio o solo tiene header.
      """
      # Normalizar line endings
      normalized = text.replace("\r\n", "\n").replace("\r", "\n")
      lines = [l for l in normalized.split("\n") if l.strip() != ""]

      if not lines:
          return []

      if has_header:
          headers = _parse_row(lines[0], delimiter)
          data_lines = lines[1:]
      else:
          # Determinar numero de columnas desde la primera fila
          sample = _parse_row(lines[0], delimiter)
          headers = [f"col_{i}" for i in range(len(sample))]
          data_lines = lines

      result: list[dict] = []
      for line in data_lines:
          fields = _parse_row(line, delimiter)
          # Alinear con headers (rellenar con "" si faltan campos)
          row = {}
          for i, header in enumerate(headers):
              row[header] = fields[i] if i < len(fields) else ""
          result.append(row)

      return result

params_schema:
  {"params":[{"name":"text","desc":"string con contenido CSV"},{"name":"delimiter","desc":"caracter delimitador de campos (default coma)"},{"name":"has_header","desc":"si True la primera fila se usa como nombres de columnas"}],"output":"lista de dicts donde cada dict es una fila y las keys son los nombres de columnas"}


id: from_jsonl_py_core
name: from_jsonl
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: from_jsonl(text: str) -> list[dict]
tags: ["jsonl","json","parser","import","streaming","format"]
uses_functions: []
uses_types: []
returns: []
imports: ["json"]
tests: ["jsonl valido","lineas vacias intercaladas","linea invalida raise con numero"]
test_file_path: python/functions/core/from_jsonl_test.py
file_path: python/functions/core/from_jsonl.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 43f44c7feaebc55cbc025f41e43cad1d9b518867fb2a4f4338721de272b19b60
description:
  Parser JSONL a lista de dicts. Ignora lineas vacias. Lanza ValueError con el numero de linea si una linea contiene JSON invalido. Complemento de to_jsonl.

example:
  ```python
  text = '{"id": 1}\n{"id": 2}'
  rows = from_jsonl(text)
  # [{"id": 1}, {"id": 2}]

  # Lineas vacias ignoradas
  text = '{"id": 1}\n\n{"id": 2}\n'
  rows = from_jsonl(text)
  # [{"id": 1}, {"id": 2}]

  # JSON invalido levanta error con numero de linea
  try:
      from_jsonl('{"ok": 1}\nnot-json')
  except ValueError as e:
      print(e)  # "JSON invalido en linea 2: ..."
  ```

notes:
  Aunque se declara pure (no hace I/O), puede lanzar ValueError para JSON invalido.
  Esto es consistente con la convencion del registry: funciones puras pueden lanzar
  excepciones de validacion — solo las funciones impuras retornan error como valor.

code:
  """Parser JSON Lines (JSONL) a lista de dicts. Complemento de to_jsonl."""

  import json


  def from_jsonl(text: str) -> list[dict]:
      """Parser JSONL a lista de dicts.

      Complemento de to_jsonl. Ignora lineas vacias. Lanza ValueError si
      una linea contiene JSON invalido, indicando el numero de linea.

      Args:
          text: Contenido JSONL como string (una linea JSON por linea).

      Returns:
          Lista de dicts parseados.

      Raises:
          ValueError: Si una linea no es JSON valido, con el numero de linea.
      """
      result: list[dict] = []

      for line_num, line in enumerate(text.splitlines(), start=1):
          stripped = line.strip()
          if not stripped:
              continue
          try:
              parsed = json.loads(stripped)
          except json.JSONDecodeError as exc:
              raise ValueError(
                  f"JSON invalido en linea {line_num}: {exc}"
              ) from exc
          result.append(parsed)

      return result

params_schema:
  {"params":[{"name":"text","desc":"string con contenido JSONL (JSON Lines, una linea por objeto)"}],"output":"lista de dicts, cada uno parseado desde una linea del JSONL"}


id: generate_html_report_py_core
name: generate_html_report
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: generate_html_report(title: str, sections: list[dict]) -> str
tags: ["html","report","export","table","kpi","template","format"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/generate_html_report_test.py
file_path: python/functions/core/generate_html_report.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 3307f5c9483a50c364d26b495d54d466f6d56f3744836040ebda429bf4169f81
description:
  Genera un reporte HTML autocontenido con CSS inline. Soporta secciones de tipo table (list[dict]), text (str con markdown basico), kpi (cards con label/value/delta) y list (list[str]). Para exportar resultados de pipelines sin servidor.

example:
  ```python
  sections = [
      {
          "heading": "Resumen ejecutivo",
          "type": "kpi",
          "data": [
              {"label": "Revenue", "value": "$1.2M", "delta": "+15%"},
              {"label": "Churn", "value": "3.2%", "delta": "-0.5%"},
          ],
      },
      {
          "heading": "Top usuarios",
          "type": "table",
          "data": [
              {"usuario": "ana@example.com", "compras": 42},
              {"usuario": "bob@example.com", "compras": 38},
          ],
      },
      {
          "heading": "Notas",
          "type": "text",
          "data": "Datos del **trimestre Q1**. Ver [dashboard](https://example.com).",
      },
  ]

  html = generate_html_report("Reporte Mensual", sections)
  # Retorna string HTML completo con DOCTYPE, head con CSS inline, body con secciones
  ```

tests:
  ["reporte con una tabla","reporte con multiples secciones mixtas","kpi con deltas positivos y negativos","caracteres especiales html escapados en data","titulo con caracteres especiales"]

notes:
  CSS completamente inline en `<style>`. Tema minimalista con max-width 960px, sans-serif,
  tabla con zebra stripes, cards KPI con colores verde/rojo para deltas.
  Todo el contenido del usuario pasa por HTML escape para proteger contra XSS.

documentation:
  ## Tipos de seccion

  - **table**: `data` es `list[dict]` — renderiza `<table>` con headers extraidos de las keys
  - **text**: `data` es `str` — soporta `**bold**` y `[text](url)`, escapa HTML
  - **kpi**: `data` es `list[{"label", "value", "delta"}]` — cards con colores para delta positivo/negativo
  - **list**: `data` es `list[str]` — renderiza `<ul><li>...</li></ul>`

code:
  """Genera reportes HTML autocontenidos con CSS inline."""


  _HTML_ESCAPES = {
      "&": "&amp;",
      "<": "&lt;",
      ">": "&gt;",
      '"': "&quot;",
      "'": "&#x27;",
  }


  def _esc(value: str) -> str:
      for ch, entity in _HTML_ESCAPES.items():
          value = value.replace(ch, entity)
      return value


  def _render_table(data: list[dict]) -> str:
      if not data:
          return "<p><em>(sin datos)</em></p>"
      headers = list(data[0].keys())
      rows_html = ""
      for i, row in enumerate(data):
          cls = ' class="zebra"' if i % 2 == 1 else ""
          cells = "".join(f"<td>{_esc(str(row.get(h, '')))}</td>" for h in headers)
          rows_html += f"<tr{cls}>{cells}</tr>\n"
      headers_html = "".join(f"<th>{_esc(h)}</th>" for h in headers)
      return (
          f"<table>\n<thead><tr>{headers_html}</tr></thead>\n"
          f"<tbody>\n{rows_html}</tbody>\n</table>"
      )


  def _render_text(data: str) -> str:
      # Markdown basico: **bold** y [text](url)
      import re

      text = _esc(str(data))
      # Bold: **text** (despues de escapar, & no interfiere)
      text = re.sub(r"\*\*(.+?)\*\*", r"<strong>\1</strong>", text)
      # Links: [text](url)
      text = re.sub(r"\[(.+?)\]\((.+?)\)", r'<a href="\2">\1</a>', text)
      return f"<p>{text}</p>"


  def _render_kpi(data: list[dict]) -> str:
      cards = ""
      for kpi in data:
          label = _esc(str(kpi.get("label", "")))
          value = _esc(str(kpi.get("value", "")))
          delta = kpi.get("delta")
          delta_html = ""
          if delta is not None:
              delta_str = str(delta)
              if delta_str.startswith("+"):
                  delta_html = f'<span class="delta-pos">{_esc(delta_str)}</span>'
              elif delta_str.startswith("-"):
                  delta_html = f'<span class="delta-neg">{_esc(delta_str)}</span>'
              else:
                  delta_html = f'<span class="delta-neutral">{_esc(delta_str)}</span>'
          cards += (
              f'<div class="kpi-card">'
              f'<div class="kpi-label">{label}</div>'
              f'<div class="kpi-value">{value}</div>'
              f"{delta_html}"
              f"</div>\n"
          )
      return f'<div class="kpi-grid">\n{cards}</div>'


  def _render_list(data: list[str]) -> str:
      items = "".join(f"<li>{_esc(str(item))}</li>\n" for item in data)
      return f"<ul>\n{items}</ul>"


  _CSS = """
  body {
      font-family: sans-serif;
      max-width: 960px;
      margin: 2rem auto;
      padding: 0 1rem;
      color: #222;
      background: #fff;
  }
  h1 { font-size: 1.8rem; border-bottom: 2px solid #ddd; padding-bottom: .5rem; }
  h2 { font-size: 1.3rem; margin-top: 2rem; color: #333; }
  table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: .95rem; }
  th { background: #f0f0f0; text-align: left; padding: .5rem .75rem; border: 1px solid #ddd; }
  td { padding: .45rem .75rem; border: 1px solid #ddd; }
  tr.zebra { background: #f9f9f9; }
  ul { padding-left: 1.5rem; }
  li { margin: .3rem 0; }
  p { line-height: 1.6; }
  a { color: #0066cc; }
  .kpi-grid { display: flex; flex-wrap: wrap; gap: 1rem; margin: 1rem 0; }
  .kpi-card {
      border: 1px solid #ddd;
      border-radius: 6px;
      padding: 1rem 1.5rem;
      min-width: 140px;
      background: #fafafa;
  }
  .kpi-label { font-size: .85rem; color: #666; margin-bottom: .25rem; }
  .kpi-value { font-size: 1.6rem; font-weight: bold; }
  .delta-pos { color: #16a34a; font-size: .9rem; }
  .delta-neg { color: #dc2626; font-size: .9rem; }
  .delta-neutral { color: #888; font-size: .9rem; }
  """.strip()


  def generate_html_report(title: str, sections: list[dict]) -> str:
      """Genera un reporte HTML autocontenido con CSS inline.

      Cada seccion es un dict con:
          heading: str — titulo de la seccion
          type: "table" | "text" | "kpi" | "list" — tipo de contenido
          data: contenido segun el tipo:
              table -> list[dict]
              text  -> str (soporta **bold** y [links](url))
              kpi   -> list[{"label": str, "value": str|number, "delta": str|None}]
              list  -> list[str]

      No requiere servidor — todo el CSS va inline en <style>.

      Args:
          title: Titulo del reporte (mostrado en <h1> y <title>).
          sections: Lista de secciones a incluir en el reporte.

      Returns:
          String HTML completo con DOCTYPE.
      """
      sections_html = ""
      for section in sections:
          heading = _esc(str(section.get("heading", "")))
          kind = section.get("type", "text")
          data = section.get("data")

          if kind == "table":
              content = _render_table(data or [])
          elif kind == "kpi":
              content = _render_kpi(data or [])
          elif kind == "list":
              content = _render_list(data or [])
          else:
              content = _render_text(str(data or ""))

          sections_html += f"<section>\n<h2>{heading}</h2>\n{content}\n</section>\n"

      return (
          "<!DOCTYPE html>\n"
          "<html lang='es'>\n"
          "<head>\n"
          "<meta charset='UTF-8'>\n"
          "<meta name='viewport' content='width=device-width, initial-scale=1'>\n"
          f"<title>{_esc(title)}</title>\n"
          f"<style>\n{_CSS}\n</style>\n"
          "</head>\n"
          "<body>\n"
          f"<h1>{_esc(title)}</h1>\n"
          f"{sections_html}"
          "</body>\n"
          "</html>"
      )

params_schema:
  {"params":[{"name":"title","desc":"titulo del reporte HTML"},{"name":"sections","desc":"lista de dicts con campos heading, type, data para cada seccion"}],"output":"string HTML completo autocontenido con CSS inline y todas las secciones renderizadas"}


id: generate_pwa_manifest_py_core
name: generate_pwa_manifest
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["pwa","manifest","web","json","frontend"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/generate_pwa_manifest_test.py
file_path: python/functions/core/generate_pwa_manifest.py
created_at: 2026-04-06T16:39:56Z
updated_at: 2026-04-06T16:39:56Z
props: []
emits: []
variant: []
content_hash: 610fcff3f70f6928736f99b71521f3a2b9c6436a264f275a3b245b008eb2a0e1
signature:
  def generate_pwa_manifest(name: str, short_name: str, description: str = "", start_url: str = "/", display: str = "standalone", orientation: str = "portrait", theme_color: str = "#000000", background_color: str = "#000000", icons: list[dict] | None = None, categories: list[str] | None = None, lang: str = "es") -> dict

description:
  Genera el contenido de un manifest.json para Progressive Web Apps segun el estandar W3C Web App Manifest. Retorna un dict listo para json.dumps(). Si icons es None genera placeholders estandar para 192x192 y 512x512.

example:
  ```python
  manifest = generate_pwa_manifest(
      name="Voice Guide",
      short_name="VoiceGuide",
      description="Guia por voz con POIs",
      theme_color="#12b886",
      background_color="#1a1b1e",
  )
  # {
  #   "name": "Voice Guide",
  #   "short_name": "VoiceGuide",
  #   "start_url": "/",
  #   "scope": "/",
  #   "display": "standalone",
  #   "orientation": "portrait",
  #   "theme_color": "#12b886",
  #   "background_color": "#1a1b1e",
  #   "icons": [
  #     {"src": "/icons/icon-192x192.png", "sizes": "192x192", "type": "image/png", "purpose": "any maskable"},
  #     {"src": "/icons/icon-512x512.png", "sizes": "512x512", "type": "image/png", "purpose": "any maskable"},
  #   ],
  #   "categories": [],
  #   "lang": "es",
  # }

  import json
  with open("public/manifest.json", "w") as f:
      json.dump(manifest, f, indent=2, ensure_ascii=False)
  ```

tests:
  ["valores por defecto generan manifest valido con icons placeholder","icons custom se respetan","todos los campos se incluyen correctamente","categories none produce lista vacia","json dumps compatible"]

notes:
  Funcion pura — sin I/O, sin efectos secundarios. El caller es responsable de escribir el archivo con `json.dumps()` o `json.dump()`.

  Los valores de `display` validos segun W3C son: `fullscreen`, `standalone`, `minimal-ui`, `browser`. La funcion no valida el valor — acepta cualquier string para permitir valores futuros del estandar.

  El campo `scope` siempre se fija a `"/"`. Si se necesita un scope distinto, el caller puede modificar el dict resultante antes de serializar.

code:
  """Genera el contenido de un manifest.json para Progressive Web Apps (W3C Web App Manifest)."""


  def generate_pwa_manifest(
      name: str,
      short_name: str,
      description: str = "",
      start_url: str = "/",
      display: str = "standalone",
      orientation: str = "portrait",
      theme_color: str = "#000000",
      background_color: str = "#000000",
      icons: list[dict] | None = None,
      categories: list[str] | None = None,
      lang: str = "es",
  ) -> dict:
      """Genera el contenido de un manifest.json para Progressive Web Apps.

      Args:
          name: Nombre completo de la aplicacion.
          short_name: Nombre corto para launchers y pantallas de inicio.
          description: Descripcion de la aplicacion.
          start_url: URL de inicio de la PWA.
          display: Modo de visualizacion (standalone, fullscreen, minimal-ui, browser).
          orientation: Orientacion preferida (portrait, landscape, any).
          theme_color: Color del tema en hex.
          background_color: Color de fondo en hex.
          icons: Lista de iconos con src, sizes, type y purpose. Si es None, usa placeholders estandar.
          categories: Lista de categorias de la app. Si es None, usa lista vacia.
          lang: Idioma principal de la aplicacion (BCP 47).

      Returns:
          Dict listo para json.dumps() con la estructura estandar W3C Web App Manifest.
      """
      if icons is None:
          icons = [
              {
                  "src": "/icons/icon-192x192.png",
                  "sizes": "192x192",
                  "type": "image/png",
                  "purpose": "any maskable",
              },
              {
                  "src": "/icons/icon-512x512.png",
                  "sizes": "512x512",
                  "type": "image/png",
                  "purpose": "any maskable",
              },
          ]

      if categories is None:
          categories = []

      return {
          "name": name,
          "short_name": short_name,
          "description": description,
          "start_url": start_url,
          "scope": "/",
          "display": display,
          "orientation": orientation,
          "theme_color": theme_color,
          "background_color": background_color,
          "icons": icons,
          "categories": categories,
          "lang": lang,
      }

params_schema:
  {"params":[{"name":"name","desc":"Nombre completo de la aplicacion que aparece en la pantalla de instalacion."},{"name":"short_name","desc":"Nombre corto para launchers y pantallas de inicio (maximo 12 caracteres recomendado)."},{"name":"description","desc":"Descripcion de la aplicacion. Cadena vacia por defecto."},{"name":"start_url","desc":"URL de inicio al lanzar la PWA desde la pantalla de inicio. Por defecto '/'."},{"name":"display","desc":"Modo de visualizacion: standalone, fullscreen, minimal-ui o browser. Por defecto standalone."},{"name":"orientation","desc":"Orientacion preferida de pantalla: portrait, landscape o any. Por defecto portrait."},{"name":"theme_color","desc":"Color del tema en formato hex (#RRGGBB) para la barra de estado y UI del SO. Por defecto #000000."},{"name":"background_color","desc":"Color de fondo en formato hex (#RRGGBB) mostrado durante la carga. Por defecto #000000."},{"name":"icons","desc":"Lista de dicts con src, sizes, type y purpose. Si es None genera placeholders para 192x192 y 512x512."},{"name":"categories","desc":"Lista de categorias W3C (navigation, shopping, travel, etc). Si es None usa lista vacia."},{"name":"lang","desc":"Codigo de idioma BCP 47 del contenido principal de la app. Por defecto 'es'."}],"output":"Dict con la estructura completa W3C Web App Manifest: name, short_name, description, start_url, scope, display, orientation, theme_color, background_color, icons, categories y lang."}


id: generate_service_worker_py_core
name: generate_service_worker
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["pwa","service-worker","javascript","cache","offline","codegen"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/generate_service_worker_test.py
file_path: python/functions/core/generate_service_worker.py
created_at: 2026-04-06T16:40:15Z
updated_at: 2026-04-06T16:40:15Z
props: []
emits: []
variant: []
content_hash: 7a8fb361c8b06ed2481fa4b1593057ae486e9e7e8c8fe5d07f79b9465c5c3202
signature:
  def generate_service_worker(cache_name: str = 'app-cache-v1', precache_urls: list[str] | None = None, network_first_patterns: list[str] | None = None) -> str

description:
  Genera el codigo JavaScript de un Service Worker basico para PWAs con estrategia cache-first para assets estaticos y network-first para API calls.

example:
  ```python
  sw_code = generate_service_worker(
      cache_name="voice-guide-v1",
      precache_urls=["/", "/index.html", "/manifest.json"],
      network_first_patterns=["/api/"],
  )
  # Escribir al filesystem (operacion impura, fuera de esta funcion):
  # with open("public/sw.js", "w") as f:
  #     f.write(sw_code)
  ```

tests:
  ["SW por defecto contiene install/activate/fetch handlers","Custom precache_urls aparecen en el codigo","Custom network_first_patterns aparecen en el fetch handler","El cache_name personalizado aparece en el codigo generado","Los valores por defecto se usan cuando los parametros son None","La funcion retorna un string no vacio","El evento activate limpia caches antiguos","La estrategia cache-first guarda respuestas de red en cache"]

notes:
  Funcion pura: no hace I/O, no tiene efectos secundarios, retorna siempre el mismo string dado los mismos inputs.

  El codigo JS generado usa `self.skipWaiting()` en install y `self.clients.claim()` en activate para activacion inmediata del SW sin necesidad de refrescar la pagina.

  La estrategia de fetch:
  - **Network-first** (para `/api/` por defecto): intenta la red, guarda la respuesta exitosa en cache, y usa el cache como fallback si la red falla. Garantiza datos frescos cuando hay conexion.
  - **Cache-first** (para todo lo demas): sirve desde cache si existe, si no va a red y guarda en cache. Optimo para assets estaticos (JS, CSS, imagenes) que cambian con el nombre de version.

  Para invalidar el cache al desplegar una nueva version, basta con cambiar `cache_name` (ej: `app-cache-v2`). El evento activate limpia automaticamente cualquier cache con nombre distinto al actual.

code:
  """Genera el codigo JavaScript de un Service Worker basico para PWAs."""

  from __future__ import annotations


  def generate_service_worker(
      cache_name: str = "app-cache-v1",
      precache_urls: list[str] | None = None,
      network_first_patterns: list[str] | None = None,
  ) -> str:
      """Genera el codigo JavaScript de un Service Worker con estrategia cache-first
      para assets estaticos y network-first para API calls.

      Args:
          cache_name: Nombre del cache a usar para esta version del SW.
          precache_urls: URLs a pre-cachear en el evento install.
          network_first_patterns: Patrones de URL que usan estrategia network-first.

      Returns:
          String con codigo JavaScript valido del Service Worker.
      """
      if precache_urls is None:
          precache_urls = ["/", "/index.html"]
      if network_first_patterns is None:
          network_first_patterns = ["/api/"]

      precache_urls_js = _list_to_js_array(precache_urls, indent=2)
      network_patterns_js = _list_to_js_array(network_first_patterns, indent=2)
      network_check_js = _build_network_check(network_first_patterns)

      return f"""\
  // Service Worker — generado automaticamente
  // Estrategia: cache-first para assets, network-first para API

  const CACHE_NAME = '{cache_name}';

  const PRECACHE_URLS = {precache_urls_js};

  const NETWORK_FIRST_PATTERNS = {network_patterns_js};

  // ── install ──────────────────────────────────────────────────────────────────
  // Pre-cachea los assets estaticos esenciales al instalar el SW.
  self.addEventListener('install', (event) => {{
    event.waitUntil(
      caches.open(CACHE_NAME).then((cache) => {{
        return cache.addAll(PRECACHE_URLS);
      }})
    );
    self.skipWaiting();
  }});

  // ── activate ─────────────────────────────────────────────────────────────────
  // Limpia caches de versiones anteriores al activar la nueva version del SW.
  self.addEventListener('activate', (event) => {{
    event.waitUntil(
      caches.keys().then((cacheNames) => {{
        return Promise.all(
          cacheNames
            .filter((name) => name !== CACHE_NAME)
            .map((name) => caches.delete(name))
        );
      }})
    );
    self.clients.claim();
  }});

  // ── fetch ─────────────────────────────────────────────────────────────────────
  // Intercepta peticiones:
  //   - URLs que coinciden con NETWORK_FIRST_PATTERNS → network-first con fallback a cache
  //   - El resto → cache-first con fallback a network (y guarda en cache la respuesta)
  self.addEventListener('fetch', (event) => {{
    const url = event.request.url;

    const isNetworkFirst = {network_check_js};

    if (isNetworkFirst) {{
      // Network-first: intenta red, si falla usa cache
      event.respondWith(
        fetch(event.request)
          .then((response) => {{
            if (response && response.status === 200) {{
              const responseClone = response.clone();
              caches.open(CACHE_NAME).then((cache) => {{
                cache.put(event.request, responseClone);
              }});
            }}
            return response;
          }})
          .catch(() => caches.match(event.request))
      );
    }} else {{
      // Cache-first: sirve desde cache, si no existe va a red y guarda en cache
      event.respondWith(
        caches.match(event.request).then((cached) => {{
          if (cached) {{
            return cached;
          }}
          return fetch(event.request).then((response) => {{
            if (response && response.status === 200) {{
              const responseClone = response.clone();
              caches.open(CACHE_NAME).then((cache) => {{
                cache.put(event.request, responseClone);
              }});
            }}
            return response;
          }});
        }})
      );
    }}
  }});
  """


  def _list_to_js_array(items: list[str], indent: int = 0) -> str:
      """Convierte una lista Python a un array JS multilínea."""
      if not items:
          return "[]"
      pad = " " * indent
      inner = f",\n{pad}  ".join(f"'{item}'" for item in items)
      return f"[\n{pad}  {inner},\n{pad}]"


  def _build_network_check(patterns: list[str]) -> str:
      """Genera la expresion JS que comprueba si una URL matchea algun patron."""
      if not patterns:
          return "false"
      checks = " || ".join(f"url.includes('{p}')" for p in patterns)
      return checks

params_schema:
  {"params":[{"name":"cache_name","desc":"Nombre del cache para esta version del SW. Se usa como clave para identificar el cache activo y limpiar los anteriores en activate."},{"name":"precache_urls","desc":"Lista de URLs a pre-cachear durante el evento install. Por defecto: ['/', '/index.html']."},{"name":"network_first_patterns","desc":"Lista de patrones de URL (substrings) que usan estrategia network-first. Por defecto: ['/api/']. Cualquier URL que contenga alguno de estos patrones va a red primero."}],"output":"String con codigo JavaScript valido del Service Worker listo para escribir como sw.js. Incluye handlers de install (precacheo), activate (limpieza de caches antiguos) y fetch (cache-first / network-first segun patron)."}


id: get_leaf_nodes_py_core
name: get_leaf_nodes
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def get_leaf_nodes(structure: Any) -> list[dict]
description: Extrae solo nodos hoja (sin hijos) de un arbol jerarquico. Deep copy de cada nodo.
tags: ["tree","leaf","hierarchy","functional"]
uses_functions: []
uses_types: []
returns: []
imports: ["copy"]
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Usa deep copy. Un nodo es hoja si su campo 'nodes' es falsy (vacio o ausente).
content_hash: 8c1103eeb5ce3a33e73ff94fc94636d9b2f41a91824b781e4ee81122131bfbfc
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  tree = [{"title": "A", "nodes": [{"title": "A1", "nodes": []}, {"title": "A2", "nodes": []}]}]
  get_leaf_nodes(tree)
  # [{"title": "A1"}, {"title": "A2"}]
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"structure","desc":"arbol jerarquico (dict con campo 'nodes' anidado recursivamente)"}],"output":"lista de dicts representando solo los nodos hoja (sin hijos)"}


id: get_pdf_page_tokens_py_core
name: get_pdf_page_tokens
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
description: Extrae texto y cuenta tokens por pagina de un PDF. Soporta PyPDF2 y PyMuPDF como backends.
tags: ["pdf","tokens","extraction","litellm","parsing"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["litellm","PyPDF2"]
tests: []
file_path: python/functions/core/get_pdf_page_tokens.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 69dfabef70ac2301b193f7307e3e438ea969a78dcbe5681e30ee1df958daeb97
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
signature:
  def get_pdf_page_tokens(pdf_path, model: str = None, pdf_parser: str = 'PyPDF2') -> list[tuple[str, int]]

example:
  ```python
  pages = get_pdf_page_tokens("report.pdf", model="gpt-4o")
  for text, tokens in pages:
      print(f"{tokens} tokens")

  # Con PyMuPDF (mejor para PDFs complejos)
  pages = get_pdf_page_tokens("report.pdf", pdf_parser="PyMuPDF")
  total = sum(t for _, t in pages)
  ```

notes:
  Requiere `pip install litellm PyPDF2` (o `pymupdf` para backend PyMuPDF). Acepta path string o BytesIO. Util para estimar costos de procesamiento LLM y para page_list_to_groups.

code:
  """Extract text and token count per page from a PDF. Supports PyPDF2 and PyMuPDF."""

  import os
  from io import BytesIO

  import litellm


  def get_pdf_page_tokens(pdf_path, model: str = None,
                          pdf_parser: str = "PyPDF2") -> list[tuple[str, int]]:
      """Extract text and token count for each page of a PDF.

      Args:
          pdf_path: Path to PDF file, or BytesIO object.
          model: Model name for token counting (passed to litellm.token_counter).
          pdf_parser: Parser backend — 'PyPDF2' or 'PyMuPDF'.

      Returns:
          list[tuple[str, int]]: List of (page_text, token_count) per page.
      """
      if pdf_parser == "PyPDF2":
          import PyPDF2
          pdf_reader = PyPDF2.PdfReader(pdf_path)
          page_list = []
          for page in pdf_reader.pages:
              page_text = page.extract_text() or ""
              token_length = litellm.token_counter(model=model, text=page_text)
              page_list.append((page_text, token_length))
          return page_list

      elif pdf_parser == "PyMuPDF":
          import pymupdf
          if isinstance(pdf_path, BytesIO):
              doc = pymupdf.open(stream=pdf_path, filetype="pdf")
          elif isinstance(pdf_path, str) and os.path.isfile(pdf_path):
              doc = pymupdf.open(pdf_path)
          else:
              raise ValueError(f"Invalid pdf_path: {pdf_path}")
          page_list = []
          for page in doc:
              page_text = page.get_text()
              token_length = litellm.token_counter(model=model, text=page_text)
              page_list.append((page_text, token_length))
          return page_list

      else:
          raise ValueError(f"Unsupported PDF parser: {pdf_parser}. Use 'PyPDF2' or 'PyMuPDF'.")

params_schema:
  {"params":[{"name":"pdf_path","desc":"ruta al archivo PDF (string o BytesIO)"},{"name":"model","desc":"nombre del modelo LLM para contar tokens (p.ej. 'gpt-4o')"},{"name":"pdf_parser","desc":"backend para extraer texto: 'PyPDF2' (default) o 'PyMuPDF'"}],"output":"lista de tuplas (text, token_count) uno por pagina del PDF"}


id: get_text_stats_py_core
name: get_text_stats
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def get_text_stats(text: str) -> dict
description: Estadisticas basicas de un texto: total de caracteres, lineas y palabras.
tags: ["text","statistics","stats","characters","words","lines"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["texto normal con palabras y lineas","texto vacio retorna ceros","texto con solo newlines"]
test_file_path: python/functions/core/get_text_stats_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 5d66780d6ed3368842288d6ba6567037a6a9dea9d6d341af873454bc784186cb
example:
  ```python
  stats = get_text_stats("hello world\nfoo bar")
  # {"total_chars": 19, "total_lines": 2, "total_words": 4}
  ```

notes:
  Funcion pura sin dependencias externas. `total_lines` cuenta newlines + 1, por lo que un texto vacio cuenta como 1 linea (comportamiento consistente con `wc -l` + 1). `total_words` usa `str.split()` que separa por cualquier whitespace y descarta vacios, equivalente a contar tokens separados por espacios.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"text","desc":"string de texto a analizar"}],"output":"dict con claves total_chars, total_lines, total_words conteniendo conteos basicos"}


id: group_by_py_core
name: group_by
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def group_by(xs: list, key_fn: callable) -> dict
description: Agrupa elementos de una lista por una funcion clave. Retorna dict de clave a lista.
tags: ["list","functional","group","classify","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. El orden de los elementos dentro de cada grupo se preserva.
content_hash: 95fbf958d07cf5ce351d71bce5ea0a80eb0834607e35a181cb8a8108edc5849c
example:
  ```python
  result = group_by(["hi", "hey", "bye"], lambda s: s[0])
  # {"h": ["hi", "hey"], "b": ["bye"]}
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"lista de elementos a agrupar"},{"name":"key_fn","desc":"funcion que extrae la clave de agrupacion de cada elemento"}],"output":"dict donde las keys son los resultados de key_fn y los values son listas de elementos"}


id: html_to_markdown_py_core
name: html_to_markdown
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: html_to_markdown(html: str) -> str
tags: ["html","markdown","parse","convert","readabilipy","markdownify","content-extraction"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","html.parser"]
test_file_path: python/functions/core/html_to_markdown_test.py
file_path: python/functions/core/html_to_markdown.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1d465f86cdb511528b2d27b785a01f6f5643e2365a2f53f0e80a4765b67e9ed3
description:
  Convierte HTML a markdown. Usa readabilipy para extraer contenido principal (filtra nav, ads, boilerplate), luego markdownify para convertir a markdown. Si las librerias opcionales no estan disponibles, usa un parser stdlib como fallback.

example:
  ```python
  from core.html_to_markdown import html_to_markdown

  html = """
  <html>
    <body>
      <nav><a href="/">Home</a></nav>
      <main>
        <h1>Titulo del articulo</h1>
        <p>Contenido <strong>relevante</strong> aqui.</p>
      </main>
      <footer>Copyright 2026</footer>
    </body>
  </html>
  """

  md = html_to_markdown(html)
  # "# Titulo del articulo\n\nContenido **relevante** aqui."
  ```

tests:
  ["HTML con nav/footer filtra boilerplate","HTML limpio se convierte correctamente","HTML con imagenes lazy-loaded"]

notes:
  Algoritmo:
  1. Preprocesar HTML: manejar contenido oculto WeChat (`js_content` con display:none),
     lazy loading images (`data-src` → `src`).
  2. Extraer contenido principal con `readabilipy` (basado en Mozilla Readability).
     Si no esta disponible, usa el HTML completo.
  3. Convertir a markdown con `markdownify` (headings ATX, strip script/style).
     Si no esta disponible, usa el parser stdlib de la misma funcion.

  Dependencias opcionales (mejoran la calidad si estan instaladas):
  - `readabilipy` — extraccion del contenido principal (filtra nav, ads, boilerplate)
  - `markdownify` — conversion HTML→markdown de alta fidelidad
  - `beautifulsoup4` — requerida por readabilipy

  Sin las dependencias opcionales la funcion sigue siendo pura y funcional,
  usando `html.parser` de stdlib como fallback.

  Funcion pura. No hace I/O ni tiene efectos secundarios.

code:
  """Convierte HTML a markdown usando readabilipy + markdownify, con fallback a stdlib."""

  import re
  from html.parser import HTMLParser
  from typing import Optional


  # ---------------------------------------------------------------------------
  # Stdlib fallback parser (no external deps)
  # ---------------------------------------------------------------------------

  _BLOCK_TAGS = {
      "p", "div", "article", "section", "main", "header", "footer", "aside",
      "nav", "figure", "figcaption", "blockquote", "pre", "ul", "ol", "li",
      "table", "thead", "tbody", "tr", "th", "td", "h1", "h2", "h3",
      "h4", "h5", "h6", "br", "hr",
  }

  _SKIP_TAGS = {
      "script", "style", "noscript", "iframe", "svg", "canvas",
      "nav", "footer", "header", "aside",
  }

  _HEADING_TAGS = {"h1": 1, "h2": 2, "h3": 3, "h4": 4, "h5": 5, "h6": 6}


  class _HTMLToMarkdownParser(HTMLParser):
      """Minimal HTML → Markdown parser using only stdlib."""

      def __init__(self) -> None:
          super().__init__(convert_charrefs=True)
          self._parts: list[str] = []
          self._skip_depth = 0
          self._in_pre = False
          self._tag_stack: list[str] = []
          self._list_stack: list[str] = []

      def handle_starttag(self, tag: str, attrs: list) -> None:
          tag = tag.lower()
          self._tag_stack.append(tag)

          if self._skip_depth > 0:
              if tag in _SKIP_TAGS:
                  self._skip_depth += 1
              return

          if tag in _SKIP_TAGS:
              self._skip_depth += 1
              return

          attrs_dict = dict(attrs)

          if tag in _HEADING_TAGS:
              level = _HEADING_TAGS[tag]
              self._parts.append(f"\n\n{'#' * level} ")

          elif tag == "p":
              self._parts.append("\n\n")

          elif tag == "br":
              self._parts.append("  \n")

          elif tag == "hr":
              self._parts.append("\n\n---\n\n")

          elif tag == "pre":
              self._in_pre = True
              self._parts.append("\n\n```\n")

          elif tag == "code" and not self._in_pre:
              self._parts.append("`")

          elif tag in ("strong", "b"):
              self._parts.append("**")

          elif tag in ("em", "i"):
              self._parts.append("*")

          elif tag == "a":
              href = attrs_dict.get("href", "")
              self._parts.append("[")
              self._parts.append(f"_href:{href}_")

          elif tag == "img":
              # Handle lazy-loaded images: prefer data-src over src
              src = attrs_dict.get("data-src") or attrs_dict.get("src", "")
              alt = attrs_dict.get("alt", "")
              self._parts.append(f"\n\n![{alt}]({src})\n\n")

          elif tag == "ul":
              self._list_stack.append("ul")
              self._parts.append("\n")

          elif tag == "ol":
              self._list_stack.append("ol")
              self._parts.append("\n")

          elif tag == "li":
              prefix = "-" if (not self._list_stack or self._list_stack[-1] == "ul") else "1."
              self._parts.append(f"\n{prefix} ")

          elif tag in ("blockquote",):
              self._parts.append("\n\n> ")

          elif tag in ("th", "td"):
              self._parts.append("| ")

          elif tag == "tr":
              self._parts.append("\n")

      def handle_endtag(self, tag: str) -> None:
          tag = tag.lower()
          if self._tag_stack and self._tag_stack[-1] == tag:
              self._tag_stack.pop()

          if self._skip_depth > 0:
              if tag in _SKIP_TAGS:
                  self._skip_depth -= 1
              return

          if tag in _HEADING_TAGS:
              self._parts.append("\n\n")

          elif tag == "p":
              self._parts.append("\n\n")

          elif tag == "pre":
              self._in_pre = False
              self._parts.append("\n```\n\n")

          elif tag == "code" and not self._in_pre:
              self._parts.append("`")

          elif tag in ("strong", "b"):
              self._parts.append("**")

          elif tag in ("em", "i"):
              self._parts.append("*")

          elif tag == "a":
              # Find the matching _href: placeholder and rebuild [text](href)
              text_parts: list[str] = []
              href = ""
              while self._parts:
                  part = self._parts.pop()
                  if part.startswith("_href:") and part.endswith("_"):
                      href = part[6:-1]
                      # collected text_parts in reverse, also the "[" opener
                      if self._parts and self._parts[-1] == "[":
                          self._parts.pop()
                      break
                  text_parts.insert(0, part)
              link_text = "".join(text_parts).strip()
              self._parts.append(f"[{link_text}]({href})")

          elif tag in ("ul", "ol"):
              if self._list_stack:
                  self._list_stack.pop()
              self._parts.append("\n")

      def handle_data(self, data: str) -> None:
          if self._skip_depth > 0:
              return
          if self._in_pre:
              self._parts.append(data)
          else:
              self._parts.append(data)

      def get_markdown(self) -> str:
          raw = "".join(self._parts)
          # Collapse 3+ consecutive newlines to 2
          raw = re.sub(r"\n{3,}", "\n\n", raw)
          return raw.strip()


  def _stdlib_html_to_markdown(html: str) -> str:
      """Convert HTML to markdown using only Python stdlib."""
      parser = _HTMLToMarkdownParser()
      parser.feed(html)
      return parser.get_markdown()


  # ---------------------------------------------------------------------------
  # Public function
  # ---------------------------------------------------------------------------


  def html_to_markdown(html: str) -> str:
      """Convierte HTML a markdown.

      Usa readabilipy para extraer el contenido principal (filtra nav, ads,
      boilerplate) y markdownify para convertir a markdown. Si alguna de esas
      librerias no esta disponible, usa un parser stdlib como fallback.

      Pasos:
      1. Preprocesar HTML: manejar contenido oculto (WeChat js_content),
         lazy loading images (data-src → src).
      2. Extraer contenido principal con readabilipy (basado en Mozilla
         Readability). Fallback: usar el HTML completo.
      3. Convertir a markdown con markdownify (headings ATX, strip
         script/style). Fallback: parser stdlib.

      Args:
          html: HTML completo de la pagina.

      Returns:
          Contenido de la pagina en formato markdown.
      """
      # Step 1: preprocess — handle WeChat hidden content and lazy-loaded images
      html = _preprocess_html(html)

      # Step 2: extract main content with readabilipy (optional dep)
      main_html = _extract_main_content(html)

      # Step 3: convert to markdown
      return _convert_to_markdown(main_html)


  def _preprocess_html(html: str) -> str:
      """Preprocesar HTML antes de extraer contenido.

      - Expande contenido oculto de WeChat (js_content).
      - Reemplaza data-src por src en imagenes lazy-loaded.
      """
      # WeChat js_content: replace hidden wrapper divs
      html = re.sub(
          r'<div[^>]*id=["\']js_content["\'][^>]*style=["\'][^"\']*display\s*:\s*none[^"\']*["\'][^>]*>',
          '<div id="js_content">',
          html,
          flags=re.IGNORECASE,
      )

      # Lazy loading: copy data-src to src for img tags
      def replace_lazy_src(m: re.Match) -> str:
          tag = m.group(0)
          data_src_match = re.search(r'data-src=["\']([^"\']*)["\']', tag)
          if data_src_match:
              data_src = data_src_match.group(1)
              # Replace or add src attribute
              if re.search(r'\bsrc=["\']', tag):
                  tag = re.sub(r'\bsrc=["\'][^"\']*["\']', f'src="{data_src}"', tag)
              else:
                  tag = tag.replace("<img", f'<img src="{data_src}"', 1)
          return tag

      html = re.sub(r"<img[^>]+>", replace_lazy_src, html, flags=re.IGNORECASE)
      return html


  def _extract_main_content(html: str) -> str:
      """Extraer contenido principal usando readabilipy si esta disponible."""
      try:
          from readabilipy import simple_json_from_html_string  # type: ignore

          article = simple_json_from_html_string(html, use_readability=True)
          return article.get("content") or html
      except ImportError:
          return html


  def _convert_to_markdown(html: str) -> str:
      """Convertir HTML a markdown usando markdownify si esta disponible."""
      try:
          import markdownify  # type: ignore

          return markdownify.markdownify(
              html,
              heading_style="ATX",
              strip=["script", "style"],
          )
      except ImportError:
          return _stdlib_html_to_markdown(html)

params_schema:
  {"params":[{"name":"html","desc":"string con contenido HTML a convertir"}],"output":"string con markdown equivalente al contenido HTML"}


id: is_git_repo_url_py_core
name: is_git_repo_url
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def is_git_repo_url(url: str, known_hosts: list[str] | None = None) -> bool
tags: ["git","url","validation","github","gitlab","repository"]
uses_functions: []
uses_types: []
returns: []
imports: ["urllib.parse"]
tests: ["URL repo valida","URL de issue (False)","URL de blob/file (False)","URL con tree/branch (True)"]
test_file_path: python/functions/core/parse_git_url_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 5aab8611784b855c1a96fffc93455ac697f4feb9206889d1ec32535a05b8fbce
description:
  Verifica si una URL apunta a un repositorio git clonable. Acepta org/repo y org/repo/tree/<ref>. Rechaza issues, blobs, PRs y otros sub-recursos.

example:
  ```python
  is_git_repo_url("https://github.com/psf/requests")
  # True

  is_git_repo_url("https://github.com/psf/requests/issues/123")
  # False

  is_git_repo_url("https://github.com/psf/requests/blob/main/README.md")
  # False

  is_git_repo_url("https://github.com/psf/requests/tree/main")
  # True

  is_git_repo_url("git@github.com:psf/requests.git")
  # True
  ```

notes:
  Funcion pura. Para SSH y git:// se acepta cualquier path siempre que el host sea conocido (los protocolos de clonacion no navegan a sub-recursos). Para HTTP/HTTPS se exige exactamente 2 segmentos (org/repo) o 4 segmentos con `tree` en posicion 3.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"url","desc":"URL del repositorio de código a validar (HTTPS, HTTP, SSH)"},{"name":"known_hosts","desc":"Lista opcional de hosts conocidos para validar URLs SSH"}],"output":"True si la URL apunta a un repositorio clonable, False en caso contrario"}


id: join_by_key_py_core
name: join_by_key
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def join_by_key(left: list[dict], right: list[dict], key: str, how: str = 'inner') -> list[dict]
tags: ["tabular","join","merge","python","core"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/join_by_key_test.py
file_path: python/functions/core/join_by_key.py
created_at: 2026-04-05T11:24:52Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 038c35f71009d9f059a816900b4a6b073a44bb08df2c8561e804c66ed4d6ac05
description:
  Join de dos listas de dicts por una clave comun. Soporta inner, left, right y outer. Campos duplicados del right se sufijan con _right. Algoritmo O(n+m).

example:
  ```python
  left = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
  right = [{"id": 1, "dept": "eng"}, {"id": 3, "dept": "sales"}]

  join_by_key(left, right, key="id", how="inner")
  # [{"id": 1, "name": "Alice", "dept": "eng"}]

  join_by_key(left, right, key="id", how="left")
  # [{"id": 1, "name": "Alice", "dept": "eng"},
  #  {"id": 2, "name": "Bob", "dept": None}]
  ```

tests:
  ["Inner join solo matches","Left join todos los left con None para right sin match","Right join","Outer join","Campos duplicados con sufijo _right","Key ausente en alguna fila"]

notes:
  Funcion pura sin dependencias externas.
  El algoritmo indexa right en O(n) y luego itera left en O(m), total O(n+m).
  Los campos de right que colisionan con campos de left (excepto la clave) se renombran con sufijo _right.

code:
  """Join de dos tablas tabulares por una clave comun."""


  def join_by_key(
      left: list[dict],
      right: list[dict],
      key: str,
      how: str = "inner",
  ) -> list[dict]:
      """Une dos listas de dicts por una clave comun.

      Soporta los cuatro tipos de join: inner, left, right, outer.
      Campos duplicados del lado right (distintos a la clave) se sufijan con _right.

      Algoritmo O(n+m): indexa right por key, luego itera left buscando matches.

      Args:
          left: Lista de dicts del lado izquierdo.
          right: Lista de dicts del lado derecho.
          key: Nombre del campo clave para el join.
          how: Tipo de join: inner, left, right, outer.

      Returns:
          Lista de dicts con campos de ambos lados mergeados.
          Campos del right ausentes en un match left se rellenan con None.
          Campos del left ausentes en un match right se rellenan con None.
      """
      # Indexar right por key
      right_index: dict[any, list[dict]] = {}
      for row in right:
          k = row.get(key)
          right_index.setdefault(k, []).append(row)

      # Determinar campos del right que podrian colisionar con left
      left_keys = {k for row in left for k in row}
      right_only_keys = {k for row in right for k in row if k != key}
      conflicting = right_only_keys & left_keys - {key}

      def _merge(l_row: dict | None, r_row: dict | None) -> dict:
          result: dict = {}
          if l_row is not None:
              result.update(l_row)
          if r_row is not None:
              for k, v in r_row.items():
                  if k == key:
                      continue
                  if k in conflicting:
                      result[f"{k}_right"] = v
                  else:
                      result[k] = v
          return result

      def _empty_left(left_sample: dict | None) -> dict:
          if left_sample is None:
              return {}
          return {k: None for k in left_sample}

      def _empty_right() -> dict:
          result: dict = {}
          for row in right:
              for k in row:
                  if k == key:
                      continue
                  dest = f"{k}_right" if k in conflicting else k
                  result[dest] = None
          return result

      matched_right_keys: set = set()
      output: list[dict] = []

      for l_row in left:
          k = l_row.get(key)
          r_rows = right_index.get(k)
          if r_rows:
              matched_right_keys.add(k)
              for r_row in r_rows:
                  output.append(_merge(l_row, r_row))
          else:
              if how in ("left", "outer"):
                  output.append(_merge(l_row, None) | _empty_right())

      if how in ("right", "outer"):
          for r_row in right:
              k = r_row.get(key)
              if k not in matched_right_keys:
                  base = _empty_right()
                  base[key] = k
                  for rk, rv in r_row.items():
                      if rk == key:
                          continue
                      dest = f"{rk}_right" if rk in conflicting else rk
                      base[dest] = rv
                  output.append(base)

      return output

params_schema:
  {"params":[{"name":"left","desc":"Lista izquierda de diccionarios a joinear"},{"name":"right","desc":"Lista derecha de diccionarios a joinear"},{"name":"key","desc":"Nombre de la clave común para hacer matching"},{"name":"how","desc":"Tipo de join: 'inner', 'left', 'right' u 'outer'"}],"output":"Lista de diccionarios resultantes del join con campos renombrados si hay conflicto"}


id: list_to_tree_py_core
name: list_to_tree
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def list_to_tree(data: list[dict]) -> list[dict]
description: Convierte lista plana con codigos de estructura ('1.2.3') a arbol jerarquico anidado.
tags: ["tree","hierarchy","structure","conversion"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 958d2f62de9dba3b954400a8f6c55f21d40c922586182ff684fd70b062d1d1bc
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  flat = [
      {"structure": "1", "title": "Intro", "start_index": 1, "end_index": 5},
      {"structure": "1.1", "title": "Background", "start_index": 1, "end_index": 3},
      {"structure": "1.2", "title": "Scope", "start_index": 3, "end_index": 5},
      {"structure": "2", "title": "Methods", "start_index": 5, "end_index": 10},
  ]
  tree = list_to_tree(flat)
  # [{"title": "Intro", "nodes": [{"title": "Background"}, {"title": "Scope"}]}, {"title": "Methods"}]
  ```

notes:
  Funcion pura. Cada item necesita campo 'structure' con codigo jerarquico separado por puntos. Nodos huerfanos se promueven a raiz.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"data","desc":"Lista plana de diccionarios con campo 'structure' (ej: '1.2.3')"}],"output":"Árbol jerárquico anidado donde nodos padres contienen su lista 'nodes'"}


id: llm_acompletion_retry_py_core
name: llm_acompletion_retry
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["llm","completion","retry","async","litellm","api"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["litellm","asyncio","logging"]
tests: []
file_path: python/functions/core/llm_acompletion_retry.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c1919dd7d4bbafcb4ca0cf164dc65142b0d5e4bcccd4b7b2f1e597d4a76c1222
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
signature:
  async def llm_acompletion_retry(model: str, prompt: str, max_retries: int = 10, temperature: float = 0) -> str

description:
  Completion LLM asincrono con retry automatico. Soporte multi-modelo via litellm (OpenAI, Anthropic, etc.).

example:
  ```python
  import asyncio

  async def main():
      response = await llm_acompletion_retry("gpt-4o", "Summarize this text: ...")
      print(response)

  asyncio.run(main())
  ```

notes:
  Requiere `pip install litellm`. Version async de llm_completion_retry. Usa asyncio.sleep entre retries. Ideal para procesar multiples prompts en paralelo con asyncio.gather.

code:
  """Async LLM completion with retry logic via litellm. Supports 100+ models."""

  import asyncio
  import logging

  import litellm

  litellm.drop_params = True


  async def llm_acompletion_retry(model: str, prompt: str, max_retries: int = 10,
                                  temperature: float = 0) -> str:
      """Asynchronous LLM completion with retry. Multi-model support via litellm.

      Args:
          model: Model name (e.g. 'gpt-4o', 'claude-sonnet-4-20250514'). Strips 'litellm/' prefix.
          prompt: User prompt text.
          max_retries: Max retry attempts on failure.
          temperature: Sampling temperature.

      Returns:
          str: Response content. Empty string if all retries fail.
      """
      if model:
          model = model.removeprefix("litellm/")

      messages = [{"role": "user", "content": prompt}]

      for i in range(max_retries):
          try:
              response = await litellm.acompletion(
                  model=model,
                  messages=messages,
                  temperature=temperature,
              )
              return response.choices[0].message.content
          except Exception as e:
              logging.error(f"Async LLM completion error (attempt {i+1}/{max_retries}): {e}")
              if i < max_retries - 1:
                  await asyncio.sleep(1)
              else:
                  logging.error(f"Max retries reached for model={model}")
                  return ""

params_schema:
  {"params":[{"name":"model","desc":"Nombre del modelo LLM (ej: 'gpt-4o', 'claude-sonnet-4-20250514')"},{"name":"prompt","desc":"Mensaje de usuario a enviar al modelo"},{"name":"max_retries","desc":"Número máximo de reintentos ante errores transitorios"},{"name":"temperature","desc":"Temperatura para la generación (0.0 a 1.0)"}],"output":"Texto de respuesta del modelo LLM"}


id: llm_completion_retry_py_core
name: llm_completion_retry
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["llm","completion","retry","litellm","api"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["litellm","logging","time"]
tests: []
file_path: python/functions/core/llm_completion_retry.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 93da6872007658e6f8fc13bbd5e158a71bff0c2b25323721995a680ac19d9577
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
signature:
  def llm_completion_retry(model: str, prompt: str, chat_history: list = None, return_finish_reason: bool = False, max_retries: int = 10, temperature: float = 0) -> str

description:
  Completion LLM sincrono con retry automatico (max 10). Soporte multi-modelo via litellm (OpenAI, Anthropic, etc.).

example:
  ```python
  response = llm_completion_retry("gpt-4o", "Explain quantum computing in one sentence")
  # "Quantum computing uses quantum bits..."

  # Con historial de chat
  history = [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]
  response = llm_completion_retry("claude-sonnet-4-20250514", "What's 2+2?", chat_history=history)

  # Con finish reason
  content, reason = llm_completion_retry("gpt-4o", "...", return_finish_reason=True)
  # reason: "finished" | "max_output_reached" | "error"
  ```

notes:
  Requiere `pip install litellm`. Soporta 100+ modelos via litellm. Retry con sleep(1) entre intentos. Retorna string vacio si todos los intentos fallan.

code:
  """LLM completion with retry logic via litellm. Supports 100+ models."""

  import logging
  import time

  import litellm

  litellm.drop_params = True


  def llm_completion_retry(model: str, prompt: str, chat_history: list = None,
                           return_finish_reason: bool = False, max_retries: int = 10,
                           temperature: float = 0):
      """Synchronous LLM completion with retry. Multi-model support via litellm.

      Args:
          model: Model name (e.g. 'gpt-4o', 'claude-sonnet-4-20250514'). Strips 'litellm/' prefix.
          prompt: User prompt text.
          chat_history: Optional list of prior messages [{"role": ..., "content": ...}].
          return_finish_reason: If True, returns (content, reason) tuple.
          max_retries: Max retry attempts on failure.
          temperature: Sampling temperature.

      Returns:
          str or (str, str): Response content, optionally with finish reason.
      """
      if model:
          model = model.removeprefix("litellm/")

      messages = list(chat_history or []) + [{"role": "user", "content": prompt}]

      for i in range(max_retries):
          try:
              response = litellm.completion(
                  model=model,
                  messages=messages,
                  temperature=temperature,
              )
              content = response.choices[0].message.content
              if return_finish_reason:
                  reason = "max_output_reached" if response.choices[0].finish_reason == "length" else "finished"
                  return content, reason
              return content
          except Exception as e:
              logging.error(f"LLM completion error (attempt {i+1}/{max_retries}): {e}")
              if i < max_retries - 1:
                  time.sleep(1)
              else:
                  logging.error(f"Max retries reached for model={model}")
                  if return_finish_reason:
                      return "", "error"
                  return ""

params_schema:
  {"params":[{"name":"model","desc":"Nombre del modelo LLM soportado por litellm"},{"name":"prompt","desc":"Prompt del usuario a procesar"},{"name":"chat_history","desc":"Historial de chat opcional con formato [{'role': '...', 'content': '...'}]"},{"name":"return_finish_reason","desc":"Si True, retorna tupla (content, finish_reason) en lugar de solo content"},{"name":"max_retries","desc":"Número máximo de reintentos ante fallos"},{"name":"temperature","desc":"Temperatura para la generación"}],"output":"String con la respuesta del modelo, o tupla (str, str) si return_finish_reason=True"}


id: load_translations_py_core
name: load_translations
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def load_translations(locales_dir: str) -> dict[str, dict]
tags: ["i18n","translation","locale","json","files"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["json","os"]
test_file_path: python/functions/core/load_translations_test.py
file_path: python/functions/core/load_translations.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 843fb3c59250898043ddc58efe16b69705608e7648185632b159af01529581a8
description:
  Carga todos los archivos JSON de un directorio de locales. Cada archivo {locale}.json se indexa por nombre sin extension. Retorna {} si el directorio no existe o esta vacio.

example:
  ```python
  from load_translations import load_translations
  from t import _set_translations, t

  # Estructura de archivos:
  # locales/
  #   en.json  →  {"report": {"done": "Done", "sectionStart": "Section: {title}"}}
  #   es.json  →  {"report": {"done": "Listo"}}

  translations = load_translations("locales/")
  _set_translations(translations, default_locale="en")

  t("report.done", locale="es")
  # → "Listo"
  ```

tests:
  ["carga multiples locales","directorio inexistente retorna dict vacio","ignora archivos no json","locale con estructura anidada"]

notes:
  Lee el filesystem, por eso es impura. Los errores de JSON malformado se propagan directamente (`json.JSONDecodeError`). Los errores de acceso al directorio se propagan como `OSError`. Companera natural de `t_py_core` — el flujo tipico es: `load_translations` al inicio de la app → `_set_translations` → llamadas a `t` durante la ejecucion. Inspirada conceptualmente en el modulo `locale.py` de MiroFish (AGPL-3.0); reimplementada desde cero.

code:
  """Carga de archivos JSON de un directorio de locales."""

  import json
  import os


  def load_translations(locales_dir: str) -> dict[str, dict]:
      """Carga todos los archivos JSON de un directorio de locales.

      Cada archivo `{locale}.json` se carga como diccionario y se indexa
      por el nombre del archivo sin extension (el locale).

      Args:
          locales_dir: Ruta al directorio que contiene los archivos JSON de locales.

      Returns:
          Diccionario {locale: dict_de_traducciones}. Retorna {} si el directorio
          no existe o no contiene archivos JSON.

      Raises:
          OSError: Si el directorio no es accesible.
          json.JSONDecodeError: Si un archivo JSON esta malformado.

      Example:
          >>> # locales/en.json = {"greeting": "Hello"}
          >>> # locales/es.json = {"greeting": "Hola"}
          >>> translations = load_translations("locales/")
          >>> translations["en"]["greeting"]
          'Hello'
          >>> translations["es"]["greeting"]
          'Hola'
      """
      translations: dict[str, dict] = {}

      if not os.path.isdir(locales_dir):
          return translations

      for filename in os.listdir(locales_dir):
          if not filename.endswith(".json"):
              continue
          locale = filename[:-5]  # quitar ".json"
          filepath = os.path.join(locales_dir, filename)
          with open(filepath, encoding="utf-8") as f:
              translations[locale] = json.load(f)

      return translations

params_schema:
  {"params":[{"name":"locales_dir","desc":"Ruta al directorio con archivos de traducción JSON ({locale}.json)"}],"output":"Diccionario con clave=nombre_locale y valor=contenido del JSON"}


id: map_list_py_core
name: map_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def map_list(xs: list, fn: callable) -> list
description: Aplica una funcion a cada elemento de una lista, retornando una nueva lista.
tags: ["list","functional","map","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  doubled = map_list([1, 2, 3], lambda n: n * 2)
  # [2, 4, 6]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. No muta la lista original. Equivalente a `[fn(x) for x in xs]`.
content_hash: a3c5808241deefb5870a616b8676fcb6d17296e1c9b48531b0f80426d865af56
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista de elementos a transformar"},{"name":"fn","desc":"Función a aplicar a cada elemento"}],"output":"Nueva lista con el resultado de aplicar fn a cada elemento"}


id: match_pois_to_interests_py_core
name: match_pois_to_interests
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["poi","ranking","filtering","interests","geospatial","recommendation","pure"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/match_pois_to_interests_test.py
file_path: python/functions/core/match_pois_to_interests.py
created_at: 2026-04-06T16:23:12Z
updated_at: 2026-04-06T16:23:12Z
props: []
emits: []
variant: []
content_hash: 31b4d462aa010a5acde0f2e18683933b27346d32cadb090a629571a1448aa976
signature:
  def match_pois_to_interests(pois: list[dict], interests: list[dict], max_results: int = 10) -> list[dict]

description:
  Filtra y rankea una lista de POIs segun un perfil de intereses del usuario. Calcula un score por categoria (0.5), nombre (0.3) y tags (0.2), multiplicado por el weight del interes. Retorna los top max_results POIs con score > 0, enriquecidos con score y matched_interests.

example:
  ```python
  pois = [
      {"id": 1, "lat": 40.41, "lon": -3.70, "name": "Museo del Prado", "category": "museum", "tags": {"tourism": "museum"}},
      {"id": 2, "lat": 40.42, "lon": -3.71, "name": "Bar El Tigre", "category": "bar", "tags": {"amenity": "bar"}},
  ]
  interests = [{"name": "arte", "keywords": ["museo", "museum", "arte"], "weight": 1.0}]
  result = match_pois_to_interests(pois, interests)
  # [{"id": 1, ..., "score": 1.0, "matched_interests": ["arte"]}]
  ```

tests:
  ["pois que matchean parcialmente por categoria y nombre","pois sin match retorna lista vacia","multiples intereses con diferentes weights acumulan score"]

notes:
  Funcion pura: no hace I/O, no muta los inputs, es determinista.

  **Calculo de score por interes:**
  - Match de categoria: 0.5 si `poi["category"]` esta en los keywords del interes
  - Match de nombre: 0.3 si algun keyword aparece como substring en `poi["name"]` (case-insensitive)
  - Match de tags: 0.2 si algun keyword aparece en los valores de `poi["tags"]` (case-insensitive)
  - El subtotal del interes se multiplica por `weight`

  Si un POI matchea multiples intereses, los scores se suman. Un POI puede obtener score maximo de `sum(weight_i * 1.0)` sobre todos los intereses que matcheen.

  El campo `tags` puede ser dict con valores string o numericos — todos se convierten a string para la comparacion.

code:
  """Filtra y rankea POIs segun un perfil de intereses del usuario."""


  def match_pois_to_interests(
      pois: list[dict],
      interests: list[dict],
      max_results: int = 10,
  ) -> list[dict]:
      """Filtra y rankea una lista de POIs segun un perfil de intereses.

      Calcula un score para cada POI basado en matches de categoria, nombre
      y tags contra los keywords de cada interes. Solo retorna POIs con score > 0,
      ordenados por score descendente, hasta max_results.

      Args:
          pois: Lista de POIs con campos id, lat, lon, name, category, tags.
          interests: Lista de intereses con campos name, keywords y weight.
          max_results: Numero maximo de resultados a retornar.

      Returns:
          Lista de POIs enriquecidos con score y matched_interests, ordenados
          por score descendente. Lista vacia si ningun POI matchea.
      """
      scored: list[dict] = []

      for poi in pois:
          total_score = 0.0
          matched: list[str] = []

          category = (poi.get("category") or "").lower()
          name = (poi.get("name") or "").lower()
          tags = poi.get("tags") or {}
          tag_values = " ".join(str(v) for v in tags.values()).lower()

          for interest in interests:
              keywords: list[str] = [kw.lower() for kw in (interest.get("keywords") or [])]
              weight: float = float(interest.get("weight", 1.0))
              interest_name: str = interest.get("name", "")

              interest_score = 0.0

              # Match de categoria: 0.5 puntos si la categoria aparece en keywords
              if any(kw == category for kw in keywords):
                  interest_score += 0.5

              # Match de nombre: 0.3 puntos si algun keyword es substring del nombre
              if any(kw in name for kw in keywords):
                  interest_score += 0.3

              # Match de tags: 0.2 puntos si algun keyword aparece en valores de tags
              if any(kw in tag_values for kw in keywords):
                  interest_score += 0.2

              if interest_score > 0:
                  total_score += interest_score * weight
                  matched.append(interest_name)

          if total_score > 0:
              scored.append({**poi, "score": round(total_score, 6), "matched_interests": matched})

      scored.sort(key=lambda p: p["score"], reverse=True)
      return scored[:max_results]

params_schema:
  {"params":[{"name":"pois","desc":"Lista de POIs, cada uno con campos id, lat, lon, name, category y tags (dict). Formato compatible con la salida de overpass_nearby_pois."},{"name":"interests","desc":"Lista de intereses del usuario, cada uno con name (str), keywords (list[str]) y weight (float entre 0 y 1). Los keywords se comparan contra categoria, nombre y tags del POI."},{"name":"max_results","desc":"Numero maximo de POIs a retornar. Por defecto 10."}],"output":"Lista de POIs filtrados y ordenados por score descendente. Cada POI incluye todos sus campos originales mas score (float) y matched_interests (list[str] con los nombres de intereses que matchearon). Lista vacia si ningun POI supera score 0."}


id: merge_entity_attributes_py_core
name: merge_entity_attributes
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def merge_entity_attributes(attr_list: list[dict]) -> dict
tags: ["merge","entity","attributes","resolution","deduplication","fuzzygraph","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/merge_entity_attributes_test.py
file_path: python/functions/core/merge_entity_attributes.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c873027aea78de507dab0bf47af91e9e5704487041246d29fc973326c0ac9904
description:
  Combina atributos de multiples candidatos de la misma entidad. Aplica heuristicas de resolucion por tipo de campo: max para numericos, min/max para fechas, union para listas, OR para booleanos, mas largo para strings.

example:
  ```python
  a = {"risk_score": 3.5, "first_seen": "2022-05-15", "verified": False}
  b = {"risk_score": 7.2, "first_seen": "2023-01-01", "verified": True, "alias": "Alice"}

  result = merge_entity_attributes([a, b])
  # {
  #   "risk_score": 7.2,           # max
  #   "first_seen": "2022-05-15",  # min (mas antigua)
  #   "verified": True,            # OR logico
  #   "alias": "Alice"             # solo en b
  # }
  ```

tests:
  ["Atributos complementarios (A tiene full_name, B tiene nationality) -\u003e ambos","Atributos conflictivos en risk_score -\u003e max","Atributos first_seen conflictivos -\u003e min","Todos null -\u003e null","Listas -\u003e union sin duplicados","Boolean verified -\u003e True si alguno es True","String conflictivo -\u003e usar el mas largo","Valores iguales -\u003e usar ese valor","Un solo candidato -\u003e retorna sus atributos tal cual","Lista vacia -\u003e retorna dict vacio","last_seen conflictivo -\u003e max (mas reciente)","Un candidato tiene null, otro tiene valor -\u003e usar el valor"]

notes:
  Funcion pura. No tiene dependencias externas. Las listas conocidas de campos especiales (`_NUMERIC_FIELDS`, `_DATE_MIN_FIELDS`, etc.) pueden extenderse si el dominio crece.

  Disenada originalmente para el grafo de entidades de fuzzygraph, donde multiples fuentes pueden describir la misma entidad con datos complementarios o contradictorios.

documentation:
  ## Heuristicas de resolucion

  | Campo / tipo | Conflicto | Resolucion |
  |---|---|---|
  | `risk_score`, `balance`, `cvss` | numerico | `max` |
  | `first_seen`, `created_date` | fecha | `min` (mas antigua) |
  | `last_seen`, `expires_date` | fecha | `max` (mas reciente) |
  | `verified`, `exploited` | booleano | `any` (OR logico) |
  | cualquier `list` | lista | union sin duplicados |
  | cualquier `str` u otro | string | el mas largo |

  Los campos fuera de las listas conocidas usan la heuristica por tipo Python (`list`, `bool`, luego `str`/otro).

code:
  """Combina atributos de multiples candidatos de la misma entidad."""

  from __future__ import annotations

  _NUMERIC_FIELDS = {"risk_score", "balance", "cvss"}
  _DATE_MIN_FIELDS = {"first_seen", "created_date"}
  _DATE_MAX_FIELDS = {"last_seen", "expires_date"}
  _BOOL_FIELDS = {"verified", "exploited"}


  def merge_entity_attributes(attr_list: list[dict]) -> dict:
      """Combina atributos de multiples candidatos de la misma entidad.

      Para cada campo presente en cualquier candidato recopila todos los valores
      non-null y aplica heuristicas de resolucion por tipo de campo:
      - Numerico (risk_score, balance, cvss): max
      - Fecha min (first_seen, created_date): min (mas antigua)
      - Fecha max (last_seen, expires_date): max (mas reciente)
      - Lista (cualquier valor de tipo list): union sin duplicados
      - Boolean (verified, exploited): OR logico
      - String: el mas largo

      Args:
          attr_list: Lista de dicts con los atributos de cada candidato.

      Returns:
          Dict con los atributos fusionados.
      """
      if not attr_list:
          return {}

      # Recopilar todas las claves presentes en cualquier candidato
      all_keys: set[str] = set()
      for attrs in attr_list:
          all_keys.update(attrs.keys())

      merged: dict = {}

      for key in all_keys:
          # Recopilar valores non-null
          values = [attrs[key] for attrs in attr_list if key in attrs and attrs[key] is not None]

          if not values:
              merged[key] = None
              continue

          if len(values) == 1:
              merged[key] = values[0]
              continue

          # Todos iguales
          if all(v == values[0] for v in values):
              merged[key] = values[0]
              continue

          # Resolver conflicto segun tipo de campo
          if key in _NUMERIC_FIELDS:
              merged[key] = max(values)
          elif key in _DATE_MIN_FIELDS:
              merged[key] = min(values)
          elif key in _DATE_MAX_FIELDS:
              merged[key] = max(values)
          elif key in _BOOL_FIELDS:
              merged[key] = any(values)
          elif isinstance(values[0], list):
              # Union de listas sin duplicados, preservando orden de aparicion
              seen: list = []
              for lst in values:
                  for item in lst:
                      if item not in seen:
                          seen.append(item)
              merged[key] = seen
          else:
              # String u otro: usar el mas largo
              str_values = [str(v) for v in values]
              merged[key] = max(str_values, key=len)

      return merged

params_schema:
  {"params":[{"name":"attr_list","desc":"Lista de diccionarios con atributos de candidatos de la misma entidad"}],"output":"Diccionario con atributos merged aplicando heurísticas de resolución por tipo"}


id: next_cron_time_py_core
name: next_cron_time
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: next_cron_time(schedule: dict, after: datetime) -> datetime | None
tags: ["cron","scheduling","time","next","pure"]
uses_functions: ["parse_cron_expr_py_core"]
uses_types: []
returns: []
imports: ["datetime"]
test_file_path: python/functions/core/next_cron_time_test.py
file_path: python/functions/core/next_cron_time.py
created_at: 2026-04-05T14:44:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: cea085d6fa611c9a9e89b8ccb757608301afb19f1f4b46578a5958f1e3b83e6d
description:
  Calcula la proxima ejecucion de un cron schedule despues de un tiempo dado. Avanza minuto a minuto saltando campos no coincidentes. Retorna None si no hay match en 366 dias (schedule imposible).

example:
  ```python
  from parse_cron_expr import parse_cron_expr
  from next_cron_time import next_cron_time
  from datetime import datetime

  sched = parse_cron_expr("0 * * * *")
  after = datetime(2024, 1, 15, 14, 30)
  nxt = next_cron_time(sched, after)
  # nxt = datetime(2024, 1, 15, 15, 0)

  weekdays = parse_cron_expr("0 9 * * 1-5")
  friday = datetime(2024, 1, 19, 10, 0)  # viernes
  nxt2 = next_cron_time(weekdays, friday)
  # nxt2 = datetime(2024, 1, 22, 9, 0)  # lunes
  ```

tests:
  ["0 * * * * desde :30 retorna la proxima hora en punto","0 9 * * 1-5 desde viernes retorna proximo lunes a las 9","schedule imposible retorna None","paso */15 desde :05 retorna :15"]

notes:
  Misma semantica que la version Go (next_cron_time_go_core). Usa AND para day_of_month y day_of_week.
  El campo day_of_week usa convencion cron (0=domingo) convertido internamente a weekday() de Python.
  Retorna None en lugar de lanzar excepcion para mantener purity (sin side effects).

code:
  """Calcula la proxima ejecucion de un cron schedule."""

  from __future__ import annotations

  from datetime import datetime, timedelta


  def next_cron_time(schedule: dict, after: datetime) -> datetime | None:
      """Calcula la proxima ejecucion de un cron schedule despues de `after`.

      Avanza minuto a minuto saltando campos no coincidentes. Retorna None
      si no hay match en 366 dias (schedule imposible).

      Args:
          schedule: Dict con claves minute, hour, day_of_month, month, day_of_week,
                    cada una con lista de enteros validos. Producido por parse_cron_expr.
          after: Tiempo de referencia. El proximo tick sera estrictamente posterior.

      Returns:
          Proximo datetime que satisface el schedule, o None si no existe en 366 dias.
      """
      minutes: list[int] = schedule["minute"]
      hours: list[int] = schedule["hour"]
      doms: list[int] = schedule["day_of_month"]
      months: list[int] = schedule["month"]
      dows: list[int] = schedule["day_of_week"]

      # Truncate to minute and advance by 1.
      t = after.replace(second=0, microsecond=0) + timedelta(minutes=1)
      limit = after + timedelta(days=366)

      while t < limit:
          # Check month.
          if t.month not in months:
              t = _next_valid_month(t, months)
              if t is None:
                  return None
              continue

          # Check day_of_month AND day_of_week (AND semantics).
          if t.day not in doms or t.weekday() not in _python_dows(dows):
              t = t.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
              continue

          # Check hour.
          if t.hour not in hours:
              nxt = _next_valid_hour(t, hours)
              if nxt is None:
                  t = t.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
              else:
                  t = nxt
              continue

          # Check minute.
          if t.minute not in minutes:
              nxt = _next_valid_minute(t, minutes)
              if nxt is None:
                  t = t.replace(minute=0, second=0, microsecond=0) + timedelta(hours=1)
              else:
                  t = nxt
              continue

          return t

      return None


  def _python_dows(dows: list[int]) -> set[int]:
      """Convert cron day-of-week (0=Sun) to Python weekday() (0=Mon, 6=Sun)."""
      result = set()
      for d in dows:
          # cron: 0=Sun,1=Mon,...,6=Sat
          # python weekday: 0=Mon,...,6=Sun
          result.add((d - 1) % 7)
      return result


  def _next_valid_month(t: datetime, months: list[int]) -> datetime | None:
      month = t.month
      for m in months:
          if m > month:
              return t.replace(month=m, day=1, hour=0, minute=0, second=0, microsecond=0)
      # Wrap to next year.
      if months:
          try:
              return t.replace(year=t.year + 1, month=months[0], day=1, hour=0, minute=0, second=0, microsecond=0)
          except ValueError:
              return None
      return None


  def _next_valid_hour(t: datetime, hours: list[int]) -> datetime | None:
      h = t.hour
      for hh in hours:
          if hh > h:
              return t.replace(hour=hh, minute=0, second=0, microsecond=0)
      return None


  def _next_valid_minute(t: datetime, minutes: list[int]) -> datetime | None:
      m = t.minute
      for mm in minutes:
          if mm > m:
              return t.replace(minute=mm, second=0, microsecond=0)
      return None

params_schema:
  {"params":[{"name":"schedule","desc":"Diccionario de cron schedule parsado con parse_cron_expr"},{"name":"after","desc":"Datetime base desde el cual calcular la próxima ejecución"}],"output":"Próximo datetime cuando se ejecute el cron, o None si el schedule es imposible"}


id: normalize_entity_name_py_core
name: normalize_entity_name
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def normalize_entity_name(name: str, entity_type: str = "") -> str
tags: ["normalize","entity","name","deduplication","osint","fuzzygraph","nlp","text"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/normalize_entity_name_test.py
file_path: python/functions/core/normalize_entity_name.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 44c807e4d3f7817ed54a0879733921183a4549eaf4ef41cce606b4ee720f6d17
description:
  Normaliza el nombre de una entidad para comparacion y deduplicacion. Aplica reglas distintas segun el tipo: ip/email/domain/crypto_wallet/phone usan normalizacion tecnica; person remueve titulos y convierte formato Apellido-Nombre; organization elimina sufijos legales; default aplica lower+strip+colapso de espacios.

example:
  ```python
  # Persona con titulo y whitespace
  normalize_entity_name("  Dr. Jane Doe  ", "person")   # "Jane Doe"

  # Formato apellido-nombre
  normalize_entity_name("Smith, John", "person")          # "John Smith"

  # Organizacion con sufijo legal
  normalize_entity_name("Google LLC", "organization")     # "Google"
  normalize_entity_name("ACME Corp.", "organization")     # "Acme"

  # Tecnico: dominio con www y trailing dot
  normalize_entity_name("www.example.com.", "domain")     # "example.com"

  # Email case-insensitive
  normalize_entity_name("user@GMAIL.com", "email")        # "user@gmail.com"

  # Telefono: solo digitos y +
  normalize_entity_name("+1 (555) 123-4567", "phone")     # "+15551234567"

  # Crypto wallet es case-sensitive
  normalize_entity_name("1A1zP1eP5QGefi2DMPTfTL5SLmv7Divf", "crypto_wallet")
  # "1A1zP1eP5QGefi2DMPTfTL5SLmv7Divf" (sin cambio de case)

  # Default
  normalize_entity_name("  Hello   WORLD  ")              # "hello world"
  ```

tests:
  ["person strip whitespace","person apellido nombre","person titulo","organization llc","organization corp","ip sin cambio","email lowercase","domain www y trailing dot","phone formato","default lowercase","default colapsa espacios","crypto wallet case sensitive","domain solo trailing dot","person prof titulo"]

notes:
  Disenada para el modulo fuzzygraph donde entidades de distintos tipos necesitan normalizacion previa a la comparacion y deduplicacion.

  El parametro `entity_type` es case-insensitive. Valores soportados: `ip`, `email`, `domain`, `crypto_wallet`, `phone`, `person`, `organization`. Cualquier otro valor (incluido string vacio) activa el comportamiento default.

  Bitcoin y otras crypto addresses son case-sensitive por diseno — el tipo `crypto_wallet` solo aplica strip sin cambiar el case.

  Los titulos de persona reconocidos incluyen: Dr, Mr, Mrs, Ms, Miss, Prof, Sr, Jr, Ing, Lic, Gen, Col, Maj, Capt, Sgt, Rev, Hon (con o sin punto final).

  Los sufijos legales de organizacion reconocidos incluyen: Inc, LLC, Ltd, Corp, Co, SA, GmbH, BV, NV, PLC, AG, SRL, SL, Pty, LP, LLP, LLLP, PC, PA, PLLC, Foundation, Group, Holdings, Enterprises, International, Industries, Services, Solutions, Systems, Technologies.

code:
  """Normaliza el nombre de una entidad para comparacion y deduplicacion."""

  import re


  _TITLES = re.compile(
      r"^\b(?:Dr|Mr|Mrs|Ms|Miss|Prof|Sr|Jr|Ing|Lic|Gen|Col|Maj|Capt|Sgt|Rev|Hon)\.?\s+",
      re.IGNORECASE,
  )

  _LEGAL_SUFFIXES = re.compile(
      r"\b(?:Inc|LLC|Ltd|Corp|Co|S\.?A|GmbH|B\.?V|N\.?V|PLC|AG|SRL|S\.?L|Pty|"
      r"LP|LLP|LLLP|PC|PA|PLLC|Foundation|Group|Holdings|Enterprises?|"
      r"International|Industries|Services?|Solutions?|Systems?|Technologies?)\.?\s*$",
      re.IGNORECASE,
  )

  _MULTI_SPACE = re.compile(r"\s+")


  def normalize_entity_name(name: str, entity_type: str = "") -> str:
      """Normaliza el nombre de una entidad para comparacion y deduplicacion.

      Aplica reglas diferentes segun el tipo de entidad:
      - ip / email / domain / crypto_wallet / phone: normalizacion tecnica
      - person: normalizacion de nombre humano (titulos, formato apellido-nombre)
      - organization: normalizacion corporativa (sufijos legales)
      - default: lower + strip + colapsar espacios

      Args:
          name: nombre de la entidad a normalizar.
          entity_type: tipo de entidad (ip, email, domain, crypto_wallet, phone,
                       person, organization). Vacio = default.

      Returns:
          nombre normalizado como string.
      """
      name = name.strip()
      et = entity_type.lower().strip()

      if et == "ip":
          return name.lower()

      if et == "email":
          return name.lower()

      if et == "domain":
          result = name.lower().rstrip(".")
          if result.startswith("www."):
              result = result[4:]
          return result

      if et == "crypto_wallet":
          # Bitcoin addresses son case-sensitive — solo strip
          return name

      if et == "phone":
          # Mantener solo digitos y el signo +
          return re.sub(r"[^\d+]", "", name)

      if et == "person":
          # Remover titulos al inicio
          result = _TITLES.sub("", name).strip()
          # Detectar formato "Apellido, Nombre"
          if "," in result:
              parts = result.split(",", 1)
              last = parts[0].strip()
              first = parts[1].strip()
              result = f"{first} {last}"
          # Colapsar espacios y title case
          result = _MULTI_SPACE.sub(" ", result).strip()
          return result.title()

      if et == "organization":
          result = _LEGAL_SUFFIXES.sub("", name).strip()
          result = _MULTI_SPACE.sub(" ", result).strip()
          # Title case para consistencia
          return result.title()

      # Default: lower, strip, colapsar espacios
      return _MULTI_SPACE.sub(" ", name.lower()).strip()

params_schema:
  {"params":[{"name":"name","desc":"Nombre de la entidad a normalizar"},{"name":"entity_type","desc":"Tipo de entidad para aplicar reglas específicas (person, organization, email, domain, ip, etc.)"}],"output":"Nombre normalizado según el tipo de entidad"}


id: page_list_to_groups_py_core
name: page_list_to_groups
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["chunking","tokens","pagination","grouping"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 31a28ba9ecaf69f6796c057332980e39425762bfb5854e9099a9b9593355e0db
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/page_index.py
signature:
  def page_list_to_groups(page_contents: list[str], token_lengths: list[int], max_tokens: int = 20000, overlap_pages: int = 1) -> list[str]

description:
  Agrupa paginas en chunks de texto respetando limite de tokens con overlap configurable entre grupos.

example:
  ```python
  contents = ["page1 text", "page2 text", "page3 text", "page4 text"]
  tokens = [5000, 6000, 7000, 4000]
  groups = page_list_to_groups(contents, tokens, max_tokens=12000, overlap_pages=1)
  # 2 groups con overlap de 1 pagina entre ellos
  ```

notes:
  Funcion pura. Si el total de tokens cabe en max_tokens, retorna un solo grupo. El overlap permite contexto compartido entre chunks adyacentes. Util para procesar documentos largos con LLMs.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"page_contents","desc":"Lista de strings con contenido de cada página"},{"name":"token_lengths","desc":"Lista paralela con número de tokens por página"},{"name":"max_tokens","desc":"Límite máximo de tokens por grupo"},{"name":"overlap_pages","desc":"Número de páginas a solapar entre grupos consecutivos"}],"output":"Lista de strings donde cada string es un grupo de páginas combinadas"}


id: parse_code_ast_py_core
name: parse_code_ast
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def parse_code_ast(source_code: str, language: str) -> list[CodeEntity]
tags: ["ast","parsing","tree-sitter","code","symbol","extraction","analysis","multi-language"]
uses_functions: []
uses_types: ["code_entity_py_core"]
returns: ["code_entity_py_core"]
test_file_path: python/functions/core/parse_code_ast_test.py
file_path: python/functions/core/parse_code_ast.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a4c0c0b698cfd32d349598298d55cba7e43dce0a52cb6d06413be12e78b823f1
description:
  Extrae entidades de codigo (funciones, clases, metodos, tipos, interfaces, structs, traits) de codigo fuente usando tree-sitter. Soporta Python, JavaScript, TypeScript, Go, Rust, Java y C++. Opera sobre strings: no accede al disco.

imports:
  ["tree-sitter","tree-sitter-python","tree-sitter-go","tree-sitter-javascript","tree-sitter-typescript","tree-sitter-rust","tree-sitter-java","tree-sitter-cpp"]

example:
  ```python
  import sys
  sys.path.insert(0, "python/functions/core")
  sys.path.insert(0, "python/types/core")

  from parse_code_ast import parse_code_ast

  source = '''
  def greet(name: str) -> str:
      """Saluda al usuario."""
      return f"Hello, {name}!"

  class Animal:
      def speak(self) -> str:
          return ""
  '''

  entities = parse_code_ast(source, "python")
  for e in entities:
      print(e.kind, e.name, f"lines {e.start_line}-{e.end_line}")
      for child in e.children:
          print("  ->", child.kind, child.name)

  # function greet lines 2-4
  # class Animal lines 6-8
  #   -> method speak
  ```

tests:
  ["lenguaje no soportado lanza value error","lenguaje no soportado lanza value error vacio","codigo vacio retorna lista vacia","codigo solo espacios retorna lista vacia","alias py funciona","alias ts funciona","lenguajes soportados incluyen go y python","python extrae funcion simple","python extrae clase con metodos","python extrae docstring","go extrae funciones y tipos","typescript extrae interfaces","codigo con errores de sintaxis es tolerante"]

notes:
  - **Pura sobre strings**: opera exclusivamente sobre `source_code: str`. No accede a disco, red ni estado global. Es deterministicamente pura dada una entrada.
  - **Tolerancia a errores**: tree-sitter parsea incluso codigo con errores de sintaxis, produciendo nodos ERROR en el AST pero continuando el parsing. La funcion retorna lo que pueda extraer.
  - **Imports lazy**: `tree-sitter` y los grammars se importan en el momento de llamar a la funcion, no al importar el modulo. Esto permite importar `parse_code_ast` sin tener tree-sitter instalado.
  - **Firma completa**: la firma incluye decoradores (Python), generic params (Go/TypeScript), visibility modifiers (Rust/Java). Es todo el texto desde el inicio de la definicion hasta el inicio del body.
  - **Children un nivel**: solo se extraen metodos directos de clases. No se hace recursion profunda (un impl de Rust con traits anidados producira children del impl, no sub-children).
  - **API tree-sitter >= 0.22**: usa `Language(mod.language())` en lugar del antiguo `Language.build_library()`. Compatible con los paquetes pip actuales.
  - Reimplementado desde cero. Inspirado conceptualmente en `openviking/parse/parsers/code/ast/` (AGPL-3.0) pero sin copiar codigo.

documentation:
  ## Lenguajes soportados

  | Lenguaje | Alias | Grammar pip | Entidades extraidas |
  |---|---|---|---|
  | `python` | `py` | `tree-sitter-python` | function, class, method |
  | `javascript` | `js` | `tree-sitter-javascript` | function, class, method |
  | `typescript` | `ts` | `tree-sitter-typescript` | function, class, method, interface, type |
  | `go` | — | `tree-sitter-go` | function, method, type |
  | `rust` | `rs` | `tree-sitter-rust` | function, struct, class (impl), trait |
  | `java` | — | `tree-sitter-java` | class, method, interface, function (constructor) |
  | `cpp` | `c++`, `cxx` | `tree-sitter-cpp` | function, class, struct, template |

  ## Algoritmo

  1. Normalizar alias de lenguaje (`py` → `python`, `ts` → `typescript`, etc.)
  2. Importar `tree-sitter` y el grammar del lenguaje (falla con ImportError claro si no instalado)
  3. Construir `Language` desde el modulo del grammar (API tree-sitter >= 0.22)
  4. Parsear `source_code.encode("utf-8")` → AST (tree-sitter tolera errores de sintaxis)
  5. Walk recursivo del AST — para cada nodo relevante:
     - Extraer nombre desde hijo `identifier` / `type_identifier`
     - Extraer firma: texto del nodo hasta el primer nodo body
     - Extraer docstring: primer string literal en Python; comentario `//` / `///` / `/** */` previo en otros lenguajes
     - Para clases/structs/traits/interfaces: extraer metodos como `children`
  6. Retornar lista plana de entidades de nivel superior

  ## Instalacion de dependencias

  ```bash
  # Base (obligatorio)
  pip install tree-sitter

  # Grammars (instalar solo los que necesites)
  pip install tree-sitter-python
  pip install tree-sitter-go
  pip install tree-sitter-javascript
  pip install tree-sitter-typescript
  pip install tree-sitter-rust
  pip install tree-sitter-java
  pip install tree-sitter-cpp
  ```

code:
  """parse_code_ast — extrae entidades de codigo fuente usando tree-sitter."""

  from __future__ import annotations

  import sys
  import os

  # Permite importar CodeEntity desde python/types/core/
  _TYPES_DIR = os.path.join(os.path.dirname(__file__), "..", "..", "..", "python", "types", "core")
  _TYPES_DIR = os.path.abspath(_TYPES_DIR)
  if _TYPES_DIR not in sys.path:
      sys.path.insert(0, _TYPES_DIR)

  try:
      from code_entity import CodeEntity
  except ImportError:
      # Fallback: define localmente para que el modulo pueda importarse
      from dataclasses import dataclass, field

      @dataclass
      class CodeEntity:  # type: ignore[no-redef]
          kind: str
          name: str
          start_line: int
          end_line: int
          signature: str
          docstring: str | None
          language: str
          children: list["CodeEntity"] = field(default_factory=list)


  # ---------------------------------------------------------------------------
  # Lenguajes soportados
  # ---------------------------------------------------------------------------

  SUPPORTED_LANGUAGES: dict[str, str] = {
      "python": "tree_sitter_python",
      "javascript": "tree_sitter_javascript",
      "typescript": "tree_sitter_typescript",
      "go": "tree_sitter_go",
      "rust": "tree_sitter_rust",
      "java": "tree_sitter_java",
      "cpp": "tree_sitter_cpp",
  }

  # Alias normalizados
  _ALIASES: dict[str, str] = {
      "py": "python",
      "js": "javascript",
      "ts": "typescript",
      "rs": "rust",
      "c++": "cpp",
      "cxx": "cpp",
  }

  # Nodos AST relevantes por lenguaje
  _ENTITY_NODES: dict[str, dict[str, str]] = {
      "python": {
          "function_definition": "function",
          "class_definition": "class",
          "decorated_definition": "function",  # resuelto en runtime
      },
      "javascript": {
          "function_declaration": "function",
          "function_expression": "function",
          "arrow_function": "function",
          "class_declaration": "class",
          "method_definition": "method",
      },
      "typescript": {
          "function_declaration": "function",
          "function_expression": "function",
          "arrow_function": "function",
          "class_declaration": "class",
          "method_definition": "method",
          "interface_declaration": "interface",
          "type_alias_declaration": "type",
      },
      "go": {
          "function_declaration": "function",
          "method_declaration": "method",
          "type_declaration": "type",
      },
      "rust": {
          "function_item": "function",
          "struct_item": "struct",
          "impl_item": "class",  # impl block tratado como class con metodos
          "trait_item": "trait",
      },
      "java": {
          "class_declaration": "class",
          "method_declaration": "method",
          "interface_declaration": "interface",
          "constructor_declaration": "function",
      },
      "cpp": {
          "function_definition": "function",
          "class_specifier": "class",
          "template_declaration": "function",
          "struct_specifier": "struct",
      },
  }


  # ---------------------------------------------------------------------------
  # Helpers de extraccion
  # ---------------------------------------------------------------------------

  def _node_text(node, source_bytes: bytes) -> str:
      """Retorna el texto crudo del nodo."""
      return source_bytes[node.start_byte:node.end_byte].decode("utf-8", errors="replace")


  def _extract_name(node, source_bytes: bytes, lang: str) -> str:
      """Extrae el nombre de la entidad desde el nodo AST."""
      # Buscar nodo hijo con type 'identifier' o 'name'
      for child in node.children:
          if child.type in ("identifier", "name", "type_identifier", "field_identifier"):
              return _node_text(child, source_bytes)
      # Para decorated_definition en Python, ir al nodo interno
      if node.type == "decorated_definition":
          for child in node.children:
              if child.type in ("function_definition", "class_definition"):
                  return _extract_name(child, source_bytes, lang)
      return "<anonymous>"


  def _extract_signature(node, source_bytes: bytes, lang: str) -> str:
      """Extrae la firma del nodo: todo antes del body."""
      body_types = {
          "block", "body", "statement_block", "compound_statement",
          "declaration_list", "field_declaration_list", "class_body",
          "enum_body", "struct_body", "impl_body",
      }
      # Acumular texto hasta encontrar el body
      start = node.start_byte
      body_start = None
      for child in node.children:
          if child.type in body_types:
              body_start = child.start_byte
              break
      if body_start is not None:
          sig = source_bytes[start:body_start].decode("utf-8", errors="replace").strip()
      else:
          # Sin body distinguible (ej: type alias, interface en algunos grammars)
          sig = _node_text(node, source_bytes).split("\n")[0].strip()
      return sig


  def _extract_docstring_python(node, source_bytes: bytes) -> str | None:
      """Extrae docstring Python: primer string literal en el body."""
      body_types = {"block", "body"}
      for child in node.children:
          if child.type in body_types:
              for stmt in child.children:
                  if stmt.type == "expression_statement":
                      for inner in stmt.children:
                          if inner.type in ("string", "concatenated_string"):
                              raw = _node_text(inner, source_bytes)
                              # Limpiar comillas triples / simples
                              for q in ('"""', "'''", '"', "'"):
                                  if raw.startswith(q) and raw.endswith(q) and len(raw) > 2 * len(q):
                                      return raw[len(q):-len(q)].strip()
                              return raw.strip()
                  # Solo miramos el primer statement real
                  if stmt.type not in ("comment", "\n", "pass_statement"):
                      break
      return None


  def _extract_docstring_comment(node, source_bytes: bytes, all_bytes: bytes) -> str | None:
      """Extrae comentario previo al nodo (para Go, Rust, Java, C++)."""
      # Buscar comentario en siblings anteriores del nodo en el tree
      # Usamos el inicio del nodo y buscamos hacia atras en el texto
      start = node.start_byte
      preceding = all_bytes[:start].decode("utf-8", errors="replace")
      lines = preceding.split("\n")
      # Recolectar lineas de comentario inmediatamente anteriores
      comment_lines: list[str] = []
      for line in reversed(lines):
          stripped = line.strip()
          if stripped.startswith("///") or stripped.startswith("//"):
              comment_lines.insert(0, stripped.lstrip("/").strip())
          elif stripped.startswith("*") or stripped.startswith("/**") or stripped.startswith("*/"):
              content = stripped.lstrip("/*").rstrip("*/").strip()
              if content:
                  comment_lines.insert(0, content)
          elif stripped == "":
              # Linea en blanco: detener busqueda solo si ya tenemos algo
              if comment_lines:
                  break
          else:
              break
      return "\n".join(comment_lines) if comment_lines else None


  def _extract_docstring(node, source_bytes: bytes, lang: str) -> str | None:
      """Extrae docstring/comentario segun el lenguaje."""
      if lang == "python":
          # Para decorated_definition, extraer del nodo interno
          if node.type == "decorated_definition":
              for child in node.children:
                  if child.type in ("function_definition", "class_definition"):
                      return _extract_docstring_python(child, source_bytes)
          return _extract_docstring_python(node, source_bytes)
      else:
          return _extract_docstring_comment(node, source_bytes, source_bytes)


  def _resolve_kind(node_type: str, node, source_bytes: bytes, lang: str, entity_map: dict) -> str:
      """Resuelve el kind de la entidad (maneja decorated_definition en Python)."""
      base_kind = entity_map.get(node_type, "function")
      if node_type == "decorated_definition":
          for child in node.children:
              if child.type == "class_definition":
                  return "class"
              if child.type == "function_definition":
                  return "function"
      return base_kind


  def _extract_children(node, source_bytes: bytes, lang: str, entity_map: dict) -> list[CodeEntity]:
      """Extrae metodos/funciones hijos dentro de una clase/impl/trait."""
      children: list[CodeEntity] = []
      method_kinds = {
          "method_definition", "method_declaration", "function_definition",
          "function_declaration", "function_item",
      }
      body_types = {
          "block", "body", "statement_block", "class_body", "declaration_list",
          "impl_body", "field_declaration_list",
      }

      def _walk_for_methods(n):
          for child in n.children:
              if child.type in body_types:
                  _walk_for_methods(child)
              elif child.type in method_kinds and child.type in entity_map:
                  kind = entity_map[child.type]
                  name = _extract_name(child, source_bytes, lang)
                  start_line = child.start_point[0] + 1
                  end_line = child.end_point[0] + 1
                  sig = _extract_signature(child, source_bytes, lang)
                  doc = _extract_docstring(child, source_bytes, lang)
                  children.append(CodeEntity(
                      kind=kind,
                      name=name,
                      start_line=start_line,
                      end_line=end_line,
                      signature=sig,
                      docstring=doc,
                      language=lang,
                      children=[],
                  ))

      _walk_for_methods(node)
      return children


  # ---------------------------------------------------------------------------
  # Funcion principal
  # ---------------------------------------------------------------------------

  def parse_code_ast(source_code: str, language: str) -> list[CodeEntity]:
      """Extrae entidades de codigo fuente usando tree-sitter.

      Parsea el codigo fuente y retorna una lista plana de entidades de nivel
      superior (funciones, clases, tipos, interfaces). Las clases incluyen sus
      metodos como `children`.

      Args:
          source_code: Codigo fuente como string.
          language: Lenguaje del codigo. Valores soportados: "python",
              "javascript", "typescript", "go", "rust", "java", "cpp".
              Se aceptan alias: "py", "js", "ts", "rs", "c++", "cxx".

      Returns:
          Lista de CodeEntity con las entidades extraidas. Lista vacia si el
          codigo no contiene entidades reconocibles o es vacio.

      Raises:
          ValueError: Si el lenguaje no esta soportado.
          ImportError: Si tree-sitter o el grammar del lenguaje no estan
              instalados. Instalar con:
              `pip install tree-sitter tree-sitter-python tree-sitter-go` etc.
      """
      if not source_code or not source_code.strip():
          return []

      # Normalizar lenguaje
      lang = language.lower().strip()
      lang = _ALIASES.get(lang, lang)

      if lang not in SUPPORTED_LANGUAGES:
          supported = ", ".join(sorted(SUPPORTED_LANGUAGES))
          raise ValueError(
              f"Lenguaje '{language}' no soportado. "
              f"Soportados: {supported}"
          )

      # Importar tree-sitter
      try:
          import tree_sitter  # noqa: F401
          from tree_sitter import Language, Parser
      except ImportError as exc:
          raise ImportError(
              "tree-sitter no esta instalado. Instalar con: "
              "pip install tree-sitter tree-sitter-python tree-sitter-go "
              "tree-sitter-javascript tree-sitter-typescript tree-sitter-rust "
              "tree-sitter-java tree-sitter-cpp"
          ) from exc

      # Importar grammar del lenguaje
      grammar_module = SUPPORTED_LANGUAGES[lang]
      try:
          import importlib
          mod = importlib.import_module(grammar_module)
      except ImportError as exc:
          pip_name = grammar_module.replace("_", "-")
          raise ImportError(
              f"Grammar '{grammar_module}' no instalado. Instalar con: "
              f"pip install {pip_name}"
          ) from exc

      # Construir Language segun la API de tree-sitter >= 0.22
      try:
          # API nueva: el modulo tiene una funcion language() que retorna el Language
          if lang == "typescript":
              # tree-sitter-typescript expone typescript y tsx por separado
              ts_lang = Language(mod.language_typescript())
          elif hasattr(mod, "language"):
              ts_lang = Language(mod.language())
          else:
              # Fallback: algunos grammars exponen Language directamente
              ts_lang = Language(mod.Language)
      except Exception as exc:
          raise ImportError(
              f"No se pudo inicializar el grammar para '{lang}': {exc}. "
              f"Verifica que {grammar_module} es compatible con esta version de tree-sitter."
          ) from exc

      parser = Parser(ts_lang)

      source_bytes = source_code.encode("utf-8")
      tree = parser.parse(source_bytes)

      entity_map = _ENTITY_NODES.get(lang, {})
      top_level_kinds = {"class", "struct", "trait", "interface"}

      results: list[CodeEntity] = []

      def _walk(node, depth: int = 0) -> None:
          """Walk recursivo del AST para extraer entidades de nivel superior."""
          if node.type in entity_map and depth <= 2:
              kind = _resolve_kind(node.type, node, source_bytes, lang, entity_map)
              name = _extract_name(node, source_bytes, lang)
              start_line = node.start_point[0] + 1
              end_line = node.end_point[0] + 1
              sig = _extract_signature(node, source_bytes, lang)
              doc = _extract_docstring(node, source_bytes, lang)

              # Extraer children solo para contenedores
              children: list[CodeEntity] = []
              if kind in top_level_kinds:
                  children = _extract_children(node, source_bytes, lang, entity_map)

              results.append(CodeEntity(
                  kind=kind,
                  name=name,
                  start_line=start_line,
                  end_line=end_line,
                  signature=sig,
                  docstring=doc,
                  language=lang,
                  children=children,
              ))
              # No seguir descendiendo si ya extrajimos la entidad (children ya incluidos)
              return

          for child in node.children:
              _walk(child, depth + 1)

      _walk(tree.root_node)
      return results

params_schema:
  {"params":[{"name":"source_code","desc":"String con el código fuente a analizar"},{"name":"language","desc":"Lenguaje del código (python/py, go, javascript/js, typescript/ts, rust/rs, java, cpp/c++)"}],"output":"Lista de entidades CodeEntity extraídas del AST (funciones, clases, métodos, interfaces)"}


id: parse_cron_expr_py_core
name: parse_cron_expr
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: parse_cron_expr(expr: str) -> dict
tags: ["cron","scheduling","parsing","time","pure"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/parse_cron_expr_test.py
file_path: python/functions/core/parse_cron_expr.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 8d6c5ba0e54e616c0487cf75b6f6d88f995fd77ffd20938cafa96a2dcc8921b3
description:
  Parsea una expresion cron estandar de 5 campos. Soporta *, rangos (1-5), listas (1,3,5), pasos (*/15) y aliases (@hourly, @daily, @weekly, @monthly, @yearly). Retorna dict con claves minute, hour, day_of_month, month, day_of_week con listas de enteros validos.

example:
  ```python
  from parse_cron_expr import parse_cron_expr

  sched = parse_cron_expr("*/15 * * * *")
  # sched["minute"] = [0, 15, 30, 45]
  # sched["hour"]   = list(range(24))

  sched2 = parse_cron_expr("@daily")
  # sched2["minute"] = [0]
  # sched2["hour"]   = [0]

  sched3 = parse_cron_expr("0 9 * * 1-5")
  # sched3["day_of_week"] = [1, 2, 3, 4, 5]
  ```

tests:
  ["*/15 expande minutos a [0 15 30 45]","@daily resuelve a minuto 0 hora 0","0 9 1,15 * * expande dias a [1 15]","0 9 * * 1-5 expande dia de semana a [1 2 3 4 5]","expresion con 4 campos lanza ValueError","minuto fuera de rango lanza ValueError"]

notes:
  Misma semantica que la version Go (parse_cron_expr_go_core). Usa dict en lugar de struct. Limites de campos: minute [0,59], hour [0,23], day_of_month [1,31], month [1,12], day_of_week [0,6] (0=domingo). Lanza ValueError con mensaje descriptivo para entradas invalidas.

code:
  """Parser de expresiones cron estandar de 5 campos."""

  from __future__ import annotations

  _ALIASES: dict[str, str] = {
      "@yearly": "0 0 1 1 *",
      "@annually": "0 0 1 1 *",
      "@monthly": "0 0 1 * *",
      "@weekly": "0 0 * * 0",
      "@daily": "0 0 * * *",
      "@midnight": "0 0 * * *",
      "@hourly": "0 * * * *",
  }

  # (lo, hi) inclusive for each field: minute, hour, day_of_month, month, day_of_week
  _FIELD_LIMITS: list[tuple[int, int]] = [
      (0, 59),
      (0, 23),
      (1, 31),
      (1, 12),
      (0, 6),
  ]
  _FIELD_NAMES = ["minute", "hour", "day_of_month", "month", "day_of_week"]


  def parse_cron_expr(expr: str) -> dict[str, list[int]]:
      """Parsea una expresion cron estandar de 5 campos.

      Soporta *, rangos (1-5), listas (1,3,5), pasos (*/15) y aliases
      (@hourly, @daily, @weekly, @monthly, @yearly).

      Args:
          expr: Expresion cron de 5 campos o alias.

      Returns:
          Dict con claves minute, hour, day_of_month, month, day_of_week,
          cada una con la lista de enteros validos para ese campo.

      Raises:
          ValueError: Si la expresion es invalida o un valor esta fuera de rango.
      """
      expr = expr.strip()
      expr = _ALIASES.get(expr, expr)
      fields = expr.split()
      if len(fields) != 5:
          raise ValueError(
              f"parse_cron_expr: expected 5 fields, got {len(fields)} in {expr!r}"
          )
      result: dict[str, list[int]] = {}
      for i, field in enumerate(fields):
          lo, hi = _FIELD_LIMITS[i]
          key = _FIELD_NAMES[i]
          result[key] = _parse_field(field, lo, hi, key)
      return result


  def _parse_field(field: str, lo: int, hi: int, name: str) -> list[int]:
      if field == "*":
          return list(range(lo, hi + 1))
      seen: set[int] = set()
      values: list[int] = []
      for part in field.split(","):
          for v in _parse_part(part, lo, hi):
              if v not in seen:
                  seen.add(v)
                  values.append(v)
      values.sort()
      return values


  def _parse_part(part: str, lo: int, hi: int) -> list[int]:
      if "/" in part:
          base, step_str = part.split("/", 1)
          try:
              step = int(step_str)
          except ValueError:
              raise ValueError(f"invalid step {step_str!r}")
          if step <= 0:
              raise ValueError(f"step must be positive, got {step}")
          if base == "*":
              start, end = lo, hi
          elif "-" in base:
              start, end = _parse_range(base, lo, hi)
          else:
              start = _parse_value(base, lo, hi)
              end = hi
          return list(range(start, end + 1, step))
      if "-" in part:
          start, end = _parse_range(part, lo, hi)
          return list(range(start, end + 1))
      return [_parse_value(part, lo, hi)]


  def _parse_range(s: str, lo: int, hi: int) -> tuple[int, int]:
      parts = s.split("-", 1)
      if len(parts) != 2:
          raise ValueError(f"invalid range {s!r}")
      start = _parse_value(parts[0], lo, hi)
      end = _parse_value(parts[1], lo, hi)
      if start > end:
          raise ValueError(f"range start {start} > end {end}")
      return start, end


  def _parse_value(s: str, lo: int, hi: int) -> int:
      try:
          v = int(s)
      except ValueError:
          raise ValueError(f"invalid value {s!r}: not an integer")
      if v < lo or v > hi:
          raise ValueError(f"value {v} out of range [{lo}, {hi}]")
      return v

params_schema:
  {"params":[{"name":"expr","desc":"Expresión cron estándar de 5 campos o alias como @daily, @hourly"}],"output":"Diccionario con claves minute, hour, day_of_month, month, day_of_week conteniendo listas de enteros válidos"}


id: parse_git_url_py_core
name: parse_git_url
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def parse_git_url(url: str, known_hosts: list[str] | None = None) -> str | None
tags: ["git","url","parsing","github","gitlab","repository"]
uses_functions: []
uses_types: []
returns: []
imports: ["urllib.parse"]
test_file_path: python/functions/core/parse_git_url_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 4ee5e80ade9730ccf6f0a9ba3c7b2713bdc6ac39ab322eceebdca817b27ea0f5
description:
  Parsea una URL de code hosting (GitHub, GitLab, etc.) y retorna el path org/repo. Soporta HTTPS, HTTP, SSH (git@), git:// y ssh://.

example:
  ```python
  parse_git_url("https://github.com/psf/requests")
  # "psf/requests"

  parse_git_url("git@github.com:psf/requests.git")
  # "psf/requests"

  parse_git_url("https://github.com/psf/requests/issues/123")
  # "psf/requests"  (ignora segmentos extra)

  parse_git_url("https://bitbucket.org/org/repo")
  # None  (host desconocido)
  ```

tests:
  ["URL HTTPS GitHub","URL SSH git@","URL con .git suffix","URL con path extra (issues/123)","URL no reconocida"]

notes:
  Funcion pura. Sanitiza org y repo dejando solo `[a-zA-Z0-9_-]`. Los hosts por defecto son `github.com` y `gitlab.com`; se pueden extender via `known_hosts`. No hace requests de red.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"url","desc":"URL de repositorio de código (HTTPS, SSH, git://, ssh://)"},{"name":"known_hosts","desc":"Lista opcional de hosts conocidos a reconocer"}],"output":"String con formato 'org/repo' si la URL es válida, None si no se puede parsear"}


id: parse_iso_datetime_py_core
name: parse_iso_datetime
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: parse_iso_datetime(value: str) -> datetime
tags: ["datetime","iso8601","parse","time","windows"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","datetime"]
test_file_path: python/functions/core/parse_iso_datetime_test.py
file_path: python/functions/core/parse_iso_datetime.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 9c003264cccefa59999cc793dbfaef318b9fc7b0f01c6853bdc50b1c0a58a823
description:
  Parsea un datetime ISO 8601 tolerando fracciones de segundo con mas de 6 digitos. Windows produce timestamps como 2026-02-21T13:20:23.1470042+08:00 donde la fraccion excede los 6 digitos que acepta datetime.fromisoformat.

example:
  ```python
  from parse_iso_datetime import parse_iso_datetime

  # Timestamp normal
  dt = parse_iso_datetime("2026-02-21T13:20:23+08:00")

  # Timestamp Windows con 7 digitos en fraccion
  dt = parse_iso_datetime("2026-02-21T13:20:23.1470042+08:00")

  # UTC con Z
  dt = parse_iso_datetime("2026-02-21T13:20:23Z")
  ```

tests:
  ["ISO normal sin fraccion","ISO con Z como UTC","ISO con fraccion de mas de 6 digitos","ISO con timezone offset"]

notes:
  Algoritmo:
  1. Regex `(\.\d{6})\d+` trunca la fraccion de segundo a exactamente 6 digitos.
  2. Reemplaza `Z` final por `+00:00` para compatibilidad con `fromisoformat`.
  3. Llama a `datetime.fromisoformat(normalized)`.

  Funcion pura. No hace I/O ni tiene efectos secundarios.

code:
  """Parsea un datetime ISO 8601 tolerando fracciones de segundo con mas de 6 digitos."""

  import re
  from datetime import datetime

  _FRAC_RE = re.compile(r"(\.\d{6})\d+")


  def parse_iso_datetime(value: str) -> datetime:
      """Parsea un datetime ISO 8601 tolerando fracciones de segundo con >6 digitos.

      Windows produce timestamps como ``2026-02-21T13:20:23.1470042+08:00`` donde
      la fraccion de segundo excede los 6 digitos que acepta ``datetime.fromisoformat``.
      Esta funcion trunca la fraccion a 6 digitos antes de parsear.

      Args:
          value: String ISO 8601. Puede contener 'Z' como offset UTC o
                 fracciones de segundo con mas de 6 digitos.

      Returns:
          datetime parseado, con tzinfo si el string incluia offset.
      """
      normalized = _FRAC_RE.sub(r"\1", value)
      normalized = normalized.replace("Z", "+00:00")
      return datetime.fromisoformat(normalized)

params_schema:
  {"params":[{"name":"value","desc":"String en formato ISO 8601 (posiblemente con fracciones de segundo excesivas)"}],"output":"Objeto datetime parseado y normalizado"}


id: parse_llm_json_py_core
name: parse_llm_json
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def parse_llm_json(response: str) -> dict
tags: ["llm","json","parsing","cleaning","think","codeblock"]
uses_functions: ["strip_think_tags_py_core","strip_markdown_codeblock_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["json"]
test_file_path: python/functions/core/parse_llm_json_test.py
file_path: python/functions/core/parse_llm_json.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: d37286401cf3f83416587c6bf4e06938e6e3224663de3fe1b25da4c7650568fb
description:
  Parsea una respuesta LLM como JSON, limpiando primero think tags y markdown codeblocks. Combina strip_think_tags + strip_markdown_codeblock + json.loads.

example:
  ```python
  # Respuesta limpia
  result = parse_llm_json('{"status": "ok"}')
  # {"status": "ok"}

  # Respuesta con think tags y codeblock
  raw = '<think>I need to return JSON.</think>\n```json\n{"status": "ok"}\n```'
  result = parse_llm_json(raw)
  # {"status": "ok"}

  # JSON invalido lanza ValueError
  parse_llm_json("not json at all")
  # ValueError: parse_llm_json: invalid JSON after cleaning. Cleaned text: 'not json at all'
  ```

tests:
  ["JSON limpio es parseado correctamente","JSON con think tags y codeblock es parseado correctamente","JSON invalido lanza ValueError"]

notes:
  Funcion impura porque puede lanzar ValueError. El mensaje de error incluye el texto ya limpiado (post strip_think_tags y strip_markdown_codeblock) para facilitar el debugging. Composicion de tres funciones: strip_think_tags → strip_markdown_codeblock → json.loads.

code:
  """Parse JSON from LLM responses, cleaning think tags and markdown code blocks."""

  import json

  from strip_think_tags import strip_think_tags
  from strip_markdown_codeblock import strip_markdown_codeblock


  def parse_llm_json(response: str) -> dict:
      """Parse an LLM response as JSON, cleaning think tags and markdown code blocks first.

      Combines strip_think_tags + strip_markdown_codeblock + json.loads into a
      single pipeline. Raises ValueError with the cleaned text included in the
      message to aid debugging.

      Args:
          response: Raw LLM response string.

      Returns:
          Parsed JSON as a dict.

      Raises:
          ValueError: If the cleaned text is not valid JSON.
      """
      cleaned = strip_think_tags(response)
      cleaned = strip_markdown_codeblock(cleaned)
      try:
          return json.loads(cleaned)
      except json.JSONDecodeError as exc:
          raise ValueError(
              f"parse_llm_json: invalid JSON after cleaning. "
              f"Cleaned text: {cleaned!r}"
          ) from exc

params_schema:
  {"params":[{"name":"response","desc":"String con respuesta del LLM (posiblemente con think tags y markdown codeblocks)"}],"output":"Diccionario parseado del JSON limpio"}


id: parse_page_range_py_core
name: parse_page_range
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def parse_page_range(pages: str) -> list[int]
tags: ["parsing","range","pages","string"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Lanza ValueError si un rango tiene start > end.
content_hash: 4d47b364c066da092db85fe81c6fcfa0dc1367cdd27110338e346d246e32a027
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/retrieve.py
description:
  Parsea string de rangos de paginas ('5-7', '3,8', '12') a lista de enteros ordenada y sin duplicados.

example:
  ```python
  parse_page_range("1-3,7,10-12")
  # [1, 2, 3, 7, 10, 11, 12]

  parse_page_range("5")
  # [5]
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"pages","desc":"String con especificación de páginas (ej: '5-7,3,10-12')"}],"output":"Lista ordenada de números de página sin duplicados"}


id: parser_registry_py_core
name: parser_registry
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["parser","registry","plugin","extensible","dispatch","pattern","singleton"]
uses_functions: []
uses_types: ["base_parser_py_core","parse_result_py_core"]
returns: ["parse_result_py_core"]
error_type: error_go_core
imports: []
test_file_path: python/functions/core/parser_registry_test.py
file_path: python/functions/core/parser_registry.py
created_at: 2026-04-05T11:25:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: cd508c918240be87ff06ece09e27f1c82b70106995dee9e41371f5eaed31ad86
params_schema: {"output":"ParseResult con contenido parseado y metadatos del parsing"}
signature:
  class ParserRegistry: register(name: str, parser: BaseParser) -> None; unregister(name: str) -> None; get_parser(name: str) -> BaseParser | None; get_parser_for_file(path: str) -> BaseParser | None; async parse(source: str, **kwargs) -> ParseResult; list_parsers() -> list[str]; list_supported_extensions() -> list[str]

description:
  Registry extensible que despacha parsing de archivos al parser correcto basado en extension. Patron plugin: registrar parsers por nombre y extensiones, resolver automaticamente. Mantiene estado mutable (mapa extension→parser). Singleton global disponible via get_registry().

example:
  ```python
  from parser_registry import ParserRegistry, get_registry

  # Usar instancia propia
  registry = ParserRegistry()
  registry.register("markdown", MarkdownParser())
  registry.register("pdf", PDFParser())
  registry.register("text", TextParser())

  result = await registry.parse("documento.md")
  result2 = await registry.parse("Contenido directo\ncon varias lineas")

  # Usar singleton global
  global_registry = get_registry()
  global_registry.register("markdown", MarkdownParser())
  ```

tests:
  ["registrar parser custom","resolver parser por extension de archivo","parsear archivo existente con extension registrada","parsear string de contenido sin path","desregistrar parser elimina nombre y extensiones","extension desconocida hace fallback al parser text"]

notes:
  Patron plugin extensible: cada parser se registra por nombre y declara sus extensiones. El mapa extension→nombre se construye automaticamente al registrar. Unregister limpia el mapa solo si el parser registrado sigue siendo el mismo (evita colisiones si dos parsers comparten extension).

  El singleton `get_registry()` usa lazy initialization thread-unsafe — suficiente para la mayoria de casos. Para uso multi-threaded, crear instancias propias de ParserRegistry.

  Los parsers especiales `"text"`, `"directory"` y `"code"` son convencion — el registry no los crea automaticamente, deben registrarse explicitamente.

documentation:
  ## Algoritmo de despacho

  `parse(source)` sigue este orden de decision:

  1. Si `source` es URL de repositorio de codigo (github.com, gitlab.com, bitbucket.org) → delega al parser `"code"`.
  2. Si `source` parece path de archivo (longitud ≤ 512, sin newlines) y existe en disco:
     - Si es directorio → delega al parser `"directory"`.
     - Resuelve parser por extension → delega.
     - Sin parser para la extension → fallback al parser `"text"` via `parse_content`.
  3. Si no parece path (contenido largo o con newlines) → parsea como string con parser `"text"` via `parse_content`.

code:
  """ParserRegistry — registry extensible que despacha parsing por extension de archivo."""

  from __future__ import annotations

  import re
  from pathlib import Path
  from typing import TYPE_CHECKING

  if TYPE_CHECKING:
      from base_parser import BaseParser
      from parse_result import ParseResult

  # Patron para detectar URLs de repositorios de codigo.
  _CODE_REPO_PATTERN = re.compile(
      r"^https?://(github\.com|gitlab\.com|bitbucket\.org)/[\w.\-]+/[\w.\-]+"
  )

  # Longitud maxima que puede tener un path de archivo valido (heuristica).
  _MAX_PATH_LEN = 512


  def _looks_like_path(source: str) -> bool:
      """Heuristica: True si source parece un path de archivo, no contenido."""
      return len(source) <= _MAX_PATH_LEN and "\n" not in source


  class ParserRegistry:
      """Registry extensible que despacha parsing al parser correcto segun extension.

      Patron plugin: registrar parsers por nombre y lista de extensiones,
      resolver automaticamente al parsear un archivo. El registry mantiene
      un mapa de extension→nombre para O(1) lookup.

      Uso tipico:
          registry = ParserRegistry()
          registry.register("markdown", MarkdownParser())
          registry.register("pdf", PDFParser())
          result = await registry.parse("doc.md")
      """

      def __init__(self) -> None:
          # nombre → instancia de parser
          self._parsers: dict[str, "BaseParser"] = {}
          # extension → nombre del parser
          self._ext_map: dict[str, str] = {}

      # ------------------------------------------------------------------
      # Registro
      # ------------------------------------------------------------------

      def register(self, name: str, parser: "BaseParser") -> None:
          """Registra un parser con el nombre dado.

          Sobreescribe cualquier parser previamente registrado con el mismo nombre.
          Actualiza el mapa de extensiones automaticamente.

          Args:
              name:   Nombre unico del parser (ej: "markdown", "pdf").
              parser: Instancia de BaseParser a registrar.
          """
          self._parsers[name] = parser
          for ext in parser.supported_extensions:
              self._ext_map[ext.lower()] = name

      def unregister(self, name: str) -> None:
          """Elimina un parser del registry.

          Tambien elimina sus extensiones del mapa. Si el nombre no existe,
          no hace nada.

          Args:
              name: Nombre del parser a eliminar.
          """
          parser = self._parsers.pop(name, None)
          if parser is None:
              return
          for ext in parser.supported_extensions:
              key = ext.lower()
              if self._ext_map.get(key) == name:
                  del self._ext_map[key]

      # ------------------------------------------------------------------
      # Resolucion
      # ------------------------------------------------------------------

      def get_parser(self, name: str) -> "BaseParser | None":
          """Retorna el parser registrado bajo el nombre dado.

          Args:
              name: Nombre del parser.

          Returns:
              Instancia de BaseParser o None si no existe.
          """
          return self._parsers.get(name)

      def get_parser_for_file(self, path: str) -> "BaseParser | None":
          """Resuelve el parser correcto para un archivo dado su extension.

          Args:
              path: Path del archivo (puede ser relativo o absoluto).

          Returns:
              Instancia de BaseParser o None si no hay parser para esa extension.
          """
          ext = Path(path).suffix.lower()
          name = self._ext_map.get(ext)
          if name is None:
              return None
          return self._parsers.get(name)

      # ------------------------------------------------------------------
      # Parseo
      # ------------------------------------------------------------------

      async def parse(self, source: str, **kwargs) -> "ParseResult":
          """Parsea source despachando al parser correcto segun el tipo de entrada.

          Algoritmo de despacho:
          1. Si source es URL de repo de codigo → delega al parser "code".
          2. Si source parece path de archivo (corto, sin newlines) y existe en disco:
             - Si es directorio → delega al parser "directory".
             - Resuelve parser por extension → delega.
             - Si no hay parser para la extension → fallback al parser "text".
          3. Si no parece path (contenido largo o con newlines) → parsea como
             string de contenido con el parser "text".

          Args:
              source:   Path de archivo, directorio, URL o string de contenido.
              **kwargs: Argumentos adicionales pasados al parser seleccionado.

          Returns:
              ParseResult con el arbol de nodos del documento.

          Raises:
              ValueError: Si el parser requerido no esta registrado.
          """
          # 1. URL de repositorio de codigo
          if _CODE_REPO_PATTERN.match(source):
              return await self._dispatch("code", source, **kwargs)

          # 2. Parece un path de archivo
          if _looks_like_path(source):
              p = Path(source)
              if p.exists():
                  if p.is_dir():
                      return await self._dispatch("directory", source, **kwargs)
                  parser = self.get_parser_for_file(source)
                  if parser is not None:
                      return await parser.parse(source, **kwargs)
                  # Fallback a text si no hay parser para la extension
                  return await self._dispatch_content("text", source, **kwargs)

          # 3. Es contenido directo (string largo o con newlines)
          text_parser = self._parsers.get("text")
          if text_parser is None:
              raise ValueError(
                  "No hay parser 'text' registrado. Registra un TextParser primero."
              )
          return await text_parser.parse_content(source, **kwargs)

      # ------------------------------------------------------------------
      # Introspeccion
      # ------------------------------------------------------------------

      def list_parsers(self) -> list[str]:
          """Retorna los nombres de todos los parsers registrados.

          Returns:
              Lista de nombres de parsers, ordenada alfabeticamente.
          """
          return sorted(self._parsers.keys())

      def list_supported_extensions(self) -> list[str]:
          """Retorna todas las extensiones soportadas por parsers registrados.

          Returns:
              Lista de extensiones (con punto, lowercase), ordenada alfabeticamente.
          """
          return sorted(self._ext_map.keys())

      # ------------------------------------------------------------------
      # Helpers internos
      # ------------------------------------------------------------------

      async def _dispatch(self, name: str, source: str, **kwargs) -> "ParseResult":
          """Despacha a un parser por nombre via parse()."""
          parser = self._parsers.get(name)
          if parser is None:
              raise ValueError(
                  f"Parser '{name}' no esta registrado. "
                  f"Parsers disponibles: {self.list_parsers()}"
              )
          return await parser.parse(source, **kwargs)

      async def _dispatch_content(
          self, name: str, content: str, **kwargs
      ) -> "ParseResult":
          """Despacha a un parser por nombre via parse_content()."""
          parser = self._parsers.get(name)
          if parser is None:
              raise ValueError(
                  f"Parser '{name}' no esta registrado. "
                  f"Parsers disponibles: {self.list_parsers()}"
              )
          return await parser.parse_content(content, **kwargs)


  # ------------------------------------------------------------------
  # Singleton global
  # ------------------------------------------------------------------

  _registry: ParserRegistry | None = None


  def get_registry() -> ParserRegistry:
      """Retorna la instancia global del ParserRegistry (lazy-initialized).

      Returns:
          Instancia singleton de ParserRegistry.
      """
      global _registry
      if _registry is None:
          _registry = ParserRegistry()
      return _registry


id: partition_py_core
name: partition
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def partition(xs: list, pred: callable) -> tuple
description: Divide una lista en dos: (elementos que cumplen el predicado, elementos que no).
tags: ["list","functional","partition","split","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Retorna tupla de dos listas: (matches, non_matches).
content_hash: afde859b97ca114c587756d63bd8a7005c92729e49dad6d950e2ac7eee83446f
example:
  ```python
  evens, odds = partition([1, 2, 3, 4, 5], lambda n: n % 2 == 0)
  # evens = [2, 4], odds = [1, 3, 5]
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista a dividir"},{"name":"pred","desc":"Función predicado que retorna True/False"}],"output":"Tupla (elementos_que_cumplen, elementos_que_no_cumplen)"}


id: pdf_to_markdown_py_core
name: pdf_to_markdown
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def pdf_to_markdown(pdf_path: str, heading_detection: str = 'auto') -> tuple[str, dict]
tags: ["pdf","markdown","conversion","headings","tables","pdfplumber","parsing"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["pdfplumber","os","sys"]
tests: []
file_path: python/functions/core/pdf_to_markdown.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 440e3d88d63ac293d2815e1a880ce883a697d7ade029a6702c75c23142b003d5
description:
  Convierte un PDF a markdown. Extrae texto, tablas e inyecta headings detectados desde bookmarks o analisis de fuentes. Retorna (markdown_content, metadata_dict).

uses_functions:
  ["extract_pdf_bookmarks_py_core","detect_headings_by_font_py_core","format_table_to_markdown_py_core"]

example:
  ```python
  from pdf_to_markdown import pdf_to_markdown

  md, meta = pdf_to_markdown("report.pdf")
  print(f"Pages: {meta['pages']}, Headings: {meta['headings_found']}, Tables: {meta['tables_found']}")
  print(md[:500])

  # Con estrategia especifica
  md, meta = pdf_to_markdown("report.pdf", heading_detection="font")
  print(f"Strategy used: {meta['heading_strategy']}")
  ```

notes:
  Estrategias de heading_detection:
  - `"bookmarks"`: extrae outlines/bookmarks del PDF via pdfplumber. Rapido y preciso cuando el PDF tiene bookmarks bien formados.
  - `"font"`: analiza distribucion de font sizes para detectar headings. Util para PDFs sin bookmarks.
  - `"auto"`: intenta bookmarks primero; si no hay, usa analisis de font. Estrategia por defecto.

  El metadata_dict retornado contiene: `pages` (int), `heading_strategy` (str: "bookmarks"|"font"|"none"), `headings_found` (int), `tables_found` (int).

  Tablas se extraen via `page.extract_tables()` de pdfplumber y se formatean como tablas markdown. Requiere `pip install pdfplumber`.

code:
  """Convert a PDF file to markdown, with heading detection and table extraction."""

  import os
  import sys

  import pdfplumber

  # Allow import from same directory when run directly
  _DIR = os.path.dirname(os.path.abspath(__file__))
  if _DIR not in sys.path:
      sys.path.insert(0, _DIR)

  from extract_pdf_bookmarks import extract_pdf_bookmarks
  from detect_headings_by_font import detect_headings_by_font
  from format_table_to_markdown import format_table_to_markdown


  def pdf_to_markdown(
      pdf_path: str,
      heading_detection: str = "auto",
  ) -> tuple[str, dict]:
      """Convert a PDF file to markdown.

      Extracts text, tables, and injects detected headings from bookmarks or
      font size analysis.

      Args:
          pdf_path: Path to the PDF file.
          heading_detection: Strategy for detecting headings:
              - "bookmarks": use PDF outline/bookmarks only.
              - "font": analyze font size distribution.
              - "auto": try bookmarks first, fall back to font analysis.

      Returns:
          tuple[str, dict]: (markdown_content, metadata_dict).
              metadata_dict contains: {"pages": int, "heading_strategy": str,
              "headings_found": int, "tables_found": int}

      Raises:
          FileNotFoundError: If pdf_path does not exist.
          Exception: If the PDF cannot be opened or is corrupted.
      """
      if not os.path.exists(pdf_path):
          raise FileNotFoundError(f"PDF not found: {pdf_path}")

      with pdfplumber.open(pdf_path) as pdf:
          total_pages = len(pdf.pages)

          # Step 1: Detect headings
          headings: list[dict] = []
          strategy_used = heading_detection

          if heading_detection in ("bookmarks", "auto"):
              headings = extract_pdf_bookmarks(pdf)
              if headings:
                  strategy_used = "bookmarks"
              elif heading_detection == "auto":
                  headings = detect_headings_by_font(pdf)
                  strategy_used = "font" if headings else "none"
          elif heading_detection == "font":
              headings = detect_headings_by_font(pdf)
              strategy_used = "font"

          # Step 2: Group headings by page number
          headings_by_page: dict[int, list[dict]] = {}
          for h in headings:
              pn = h.get("page_num")
              if pn is not None:
                  headings_by_page.setdefault(pn, []).append(h)

          # Step 3: Build markdown page by page
          sections: list[str] = []
          tables_found = 0

          for page_idx, page in enumerate(pdf.pages):
              page_num = page_idx + 1
              page_parts: list[str] = []

              # Inject headings for this page
              for h in headings_by_page.get(page_num, []):
                  level = h["level"]
                  title = h["title"]
                  page_parts.append(f"{'#' * level} {title}")

              # Extract page text
              text = page.extract_text() or ""
              if text.strip():
                  page_parts.append(text.strip())

              # Extract tables
              try:
                  tables = page.extract_tables()
              except Exception:
                  tables = []

              for table in tables:
                  if not table:
                      continue
                  # Convert all cells to strings, treating None as empty
                  str_table = [
                      [str(cell) if cell is not None else "" for cell in row]
                      for row in table
                  ]
                  md_table = format_table_to_markdown(str_table)
                  if md_table:
                      page_parts.append(md_table)
                      tables_found += 1

              if page_parts:
                  sections.append("\n\n".join(page_parts))

          markdown_content = "\n\n".join(sections)

          metadata = {
              "pages": total_pages,
              "heading_strategy": strategy_used,
              "headings_found": len(headings),
              "tables_found": tables_found,
          }

          return markdown_content, metadata

params_schema:
  {"params":[{"name":"pdf_path","desc":"Ruta al archivo PDF a convertir"},{"name":"heading_detection","desc":"Estrategia de detección de headings: 'auto', 'bookmarks', 'font'"}],"output":"Tupla (markdown_content, metadata_dict) con el PDF convertido a markdown y metadatos"}


id: pipe_py_core
name: pipe
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def pipe(value, *fns)
description: Pasa un valor a traves de una secuencia de funciones de izquierda a derecha.
tags: ["functional","pipe","composition","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a3950c6e7d628ef3dd17ec6ef31b5a87f701260a9ddf46e9f8caa4a6a6c0000b
example:
  ```python
  result = pipe(
      [1, 2, 3, 4, 5],
      lambda xs: filter_list(xs, lambda n: n > 2),
      lambda xs: map_list(xs, lambda n: n * 10),
  )
  # [30, 40, 50]
  ```

notes:
  Funcion pura. Ejecuta las funciones en orden de izquierda a derecha: f1(value), luego f2(result), etc.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"value","desc":"Valor inicial a pasar por las funciones"},{"name":"fns","desc":"Funciones a aplicar secuencialmente de izquierda a derecha"}],"output":"Resultado final después de aplicar todas las funciones en orden"}


id: preprocess_text_py_core
name: preprocess_text
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def preprocess_text(text: str) -> str
description: Normaliza whitespace y newlines de un texto crudo. Util como paso previo a chunking o indexing.
tags: ["text","preprocessing","normalization","whitespace","newlines"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/preprocess_text_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 1ea3ef46077bb8518bacfe823abf1c5669466407f3348258d13ecb8fcfeff373
example:
  ```python
  raw = "  hello  \r\n  world  \r\n\n\n\n  end  "
  result = preprocess_text(raw)
  # "hello\nworld\n\nend"
  ```

tests:
  ["texto con saltos de linea Windows CRLF","texto con multiples newlines consecutivos","texto con espacios leading y trailing en cada linea"]

notes:
  Funcion pura. Aplica tres pasos en orden: normalizar line endings (CRLF y CR a LF), colapsar 3+ newlines consecutivos a 2, strip de cada linea. El strip global final elimina whitespace al inicio y al final del texto completo.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"text","desc":"Texto crudo a normalizar"}],"output":"Texto con whitespace normalizado y newlines ajustados"}


id: react_loop_py_core
name: react_loop
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["llm","agent","react","reasoning","tools","loop"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["json","re","typing"]
test_file_path: python/functions/core/react_loop_test.py
file_path: python/functions/core/react_loop.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2503ed1aeec783924cb5e7d9bd1279013f6c1ff7afe822b6b1578ca6fb8b4e2
signature:
  def react_loop(llm_chat: Callable[[list[dict]], str], tools: dict[str, Callable[..., str]], system_prompt: str, user_prompt: str, max_iterations: int = 5, on_thought: Callable[[str], None] | None = None, on_action: Callable[[str, dict], None] | None = None, on_observation: Callable[[str], None] | None = None) -> str

description:
  Implementa el patron ReACT (Reasoning + Acting) para agentes LLM. El agente razona, decide usar herramientas, observa resultados, y repite hasta producir una respuesta final.

example:
  ```python
  import openai

  client = openai.OpenAI()

  def llm_chat(messages: list[dict]) -> str:
      resp = client.chat.completions.create(model="gpt-4o", messages=messages)
      return resp.choices[0].message.content

  def search(query: str) -> str:
      """Search the knowledge base."""
      return f"Results for: {query}"

  def analyze(entity_id: str) -> str:
      """Deep dive into an entity."""
      return f"Analysis of entity {entity_id}"

  answer = react_loop(
      llm_chat=llm_chat,
      tools={"search": search, "analyze": analyze},
      system_prompt="You are a research assistant.",
      user_prompt="What is quantum entanglement?",
      max_iterations=5,
      on_thought=lambda t: print(f"[thought] {t}"),
      on_action=lambda a, i: print(f"[action] {a}({i})"),
      on_observation=lambda o: print(f"[obs] {o}"),
  )
  print(answer)
  ```

tests:
  ["agente que usa 1 tool y da respuesta final","agente que usa multiples tools iterativamente","agente que da respuesta directa sin tools","max iterations alcanzado fallback","tool que falla error handling"]

notes:
  El formato de parseo sigue el patron ReACT clasico (Thought/Action/Action Input/Observation/Final Answer), compatible con LangChain y agentes basados en este patron.

  Los callbacks (on_thought, on_action, on_observation) permiten logging, UI y tracing sin acoplar al agente.

  La funcion es completamente generica: funciona con cualquier LLM (OpenAI, Anthropic, litellm) y cualquier conjunto de tools. El tool description se extrae del primer renglon del docstring de cada callable.

  Si max_iterations se agota sin Final Answer, retorna el ultimo response crudo del LLM como fallback.

  Los errores de tools se capturan y se entregan como Observation al LLM para que pueda recuperarse. Los errores del LLM propio se propagan hacia arriba.

code:
  """ReACT (Reasoning + Acting) agent loop for LLM-based agents."""

  import json
  import re
  from typing import Callable


  def react_loop(
      llm_chat: Callable[[list[dict]], str],
      tools: dict[str, Callable[..., str]],
      system_prompt: str,
      user_prompt: str,
      max_iterations: int = 5,
      on_thought: Callable[[str], None] | None = None,
      on_action: Callable[[str, dict], None] | None = None,
      on_observation: Callable[[str], None] | None = None,
  ) -> str:
      """Execute a ReACT (Reasoning + Acting) agent loop.

      The agent reasons, selects tools, observes results, and repeats until
      producing a Final Answer or exhausting max_iterations.

      Args:
          llm_chat: Callable that takes a list of messages and returns a string response.
          tools: Mapping of tool name to callable. Each tool returns a string.
          system_prompt: System instruction for the agent.
          user_prompt: Initial user request.
          max_iterations: Maximum reasoning iterations before fallback.
          on_thought: Optional callback invoked with each Thought string.
          on_action: Optional callback invoked with (action_name, action_input_dict).
          on_observation: Optional callback invoked with each Observation string.

      Returns:
          The Final Answer string, or the last raw LLM response as fallback.
      """
      tools_description = _format_tools(tools)

      augmented_system = (
          f"{system_prompt}\n\n"
          "You have access to the following tools:\n"
          f"{tools_description}\n\n"
          "Use this format:\n"
          "Thought: <your reasoning>\n"
          "Action: <tool name>\n"
          "Action Input: <JSON object with tool arguments>\n"
          "Observation: <tool result will appear here>\n"
          "... (repeat as needed)\n"
          "Final Answer: <your final response to the user>"
      )

      messages: list[dict] = [
          {"role": "system", "content": augmented_system},
          {"role": "user", "content": user_prompt},
      ]

      last_response = ""

      for _ in range(max_iterations):
          response = llm_chat(messages)
          last_response = response

          final_answer = _parse_final_answer(response)
          if final_answer is not None:
              return final_answer

          thought = _parse_section(response, "Thought")
          action = _parse_section(response, "Action")
          action_input_raw = _parse_section(response, "Action Input")

          if thought and on_thought:
              on_thought(thought)

          if not action:
              # No action and no final answer — return as-is
              return response

          action_input: dict = {}
          if action_input_raw:
              try:
                  action_input = json.loads(action_input_raw)
              except json.JSONDecodeError:
                  action_input = {"input": action_input_raw}

          if on_action:
              on_action(action, action_input)

          tool_fn = tools.get(action)
          if tool_fn is None:
              observation = f"Error: tool '{action}' not found. Available tools: {list(tools.keys())}"
          else:
              try:
                  observation = tool_fn(**action_input)
              except Exception as exc:
                  observation = f"Error executing tool '{action}': {exc}"

          if on_observation:
              on_observation(observation)

          # Append assistant turn + observation to continue the conversation
          messages.append({"role": "assistant", "content": response})
          messages.append({"role": "user", "content": f"Observation: {observation}"})

      return last_response


  # ---------------------------------------------------------------------------
  # Private helpers
  # ---------------------------------------------------------------------------

  def _format_tools(tools: dict[str, Callable[..., str]]) -> str:
      """Build a human-readable tool list from the tools dict."""
      lines = []
      for name, fn in tools.items():
          doc = (fn.__doc__ or "").strip().split("\n")[0]
          lines.append(f"- {name}: {doc}" if doc else f"- {name}")
      return "\n".join(lines)


  def _parse_section(text: str, label: str) -> str | None:
      """Extract the value after a 'Label:' marker, stopping at the next label."""
      pattern = rf"^{label}:\s*(.+?)(?=\n(?:Thought|Action Input|Action|Observation|Final Answer):|$)"
      match = re.search(pattern, text, re.IGNORECASE | re.MULTILINE | re.DOTALL)
      if match:
          return match.group(1).strip()
      return None


  def _parse_final_answer(text: str) -> str | None:
      """Extract Final Answer from agent response."""
      match = re.search(r"Final Answer:\s*(.+)", text, re.IGNORECASE | re.DOTALL)
      if match:
          return match.group(1).strip()
      return None

params_schema:
  {"params":[{"name":"llm_chat","desc":"Función que envía mensajes al LLM y retorna la respuesta"},{"name":"tools","desc":"Diccionario de funciones disponibles como tools {nombre: callable}"},{"name":"system_prompt","desc":"Instrucciones del sistema para el agente"},{"name":"user_prompt","desc":"Pregunta/tarea inicial del usuario"},{"name":"max_iterations","desc":"Máximo número de iteraciones antes de fallback"},{"name":"on_thought","desc":"Callback opcional para logging de pensamientos"},{"name":"on_action","desc":"Callback opcional para logging de acciones"},{"name":"on_observation","desc":"Callback opcional para logging de observaciones"}],"output":"String con la respuesta final del agente"}


id: reduce_list_py_core
name: reduce_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def reduce_list(xs: list, initial, fn: callable)
description: Reduce una lista con un acumulador y una funcion binaria fn(acc, x).
tags: ["list","functional","reduce","fold","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  total = reduce_list([1, 2, 3, 4], 0, lambda acc, x: acc + x)
  # 10
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 93e872393ca31a41f7deb8f2296e7eb221f5125464224d53b5b185dc581b6d28
notes:
  Funcion pura. Usa functools.reduce internamente. El valor inicial es obligatorio para evitar errores con listas vacias.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista a reducir"},{"name":"initial","desc":"Valor inicial del acumulador"},{"name":"fn","desc":"Función binaria fn(acumulador, elemento)"}],"output":"Valor final del acumulador después de procesar todos los elementos"}


id: remove_tree_fields_py_core
name: remove_tree_fields
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def remove_tree_fields(data: Any, fields: list[str] = None) -> Any
description: Elimina campos especificados recursivamente de un arbol (dict/list). Por defecto elimina 'text'.
tags: ["tree","filter","fields","recursive"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Crea nueva estructura sin los campos indicados. No muta el original.
content_hash: 21e57aa89cb454b7926d07ad124eb4a0a37edf7d238df6a4285407565d159bb4
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  data = {"title": "A", "text": "largo...", "nodes": [{"title": "B", "text": "otro..."}]}
  remove_tree_fields(data, ["text"])
  # {"title": "A", "nodes": [{"title": "B"}]}
  ```

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"data","desc":"Estructura árbol (dict/list/valor) a procesar"},{"name":"fields","desc":"Lista de nombres de campo a eliminar recursivamente (default: ['text'])"}],"output":"Copia de la estructura sin los campos especificados"}


id: render_template_py_core
name: render_template
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: render_template(template: str, context: dict, missing: str = '') -> str
tags: ["template","render","html","string","format","jinja"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/render_template_test.py
file_path: python/functions/core/render_template.py
created_at: 2026-04-05T14:43:57Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 06947a4d35706ad4b354da5321505e33bbfaaa05cf26e4fa71a5f0d1d6f1ce79
description:
  Motor de templates minimalista sin dependencias. Soporta {{var}}, {{{raw}}}, {{obj.field}} dot-path, {% for x in list %}...{% endfor %}, {% if cond %}...{% endif %}, {% if not cond %}...{% endif %}. HTML-escaped por defecto.

example:
  ```python
  # Sustitucion simple
  result = render_template("Hola {{nombre}}!", {"nombre": "Ana"})
  # "Hola Ana!"

  # Dot-path
  ctx = {"user": {"name": "Bob", "city": "Madrid"}}
  result = render_template("{{user.name}} en {{user.city}}", ctx)
  # "Bob en Madrid"

  # For loop
  ctx = {"items": ["a", "b", "c"]}
  result = render_template("{% for x in items %}-{{x}}{% endfor %}", ctx)
  # "-a-b-c"

  # If condicional
  result = render_template("{% if show %}visible{% endif %}", {"show": True})
  # "visible"

  # Sin escape (raw)
  result = render_template("{{{html}}}", {"html": "<b>bold</b>"})
  # "<b>bold</b>"

  # Variable ausente con valor configurable
  result = render_template("{{falta}}", {}, missing="N/A")
  # "N/A"
  ```

tests:
  ["sustitucion simple","dot path en nested dicts","for loop con lista de strings","for loop con lista de dicts","if endif condicional","variable missing retorna string vacio","variable missing configurable","html escaping por defecto","triple braces sin escape"]

notes:
  No pretende reemplazar Jinja2. Cubre el 80% de casos simples sin dependencias.
  HTML escape por defecto protege contra XSS al generar HTML desde datos de usuario.
  Usa {{{triple}}} cuando el valor ya es HTML confiable.
  Los bloques for/if pueden anidarse pero el parsing es iterativo, no recursivo profundo.

code:
  """Motor de templates minimalista sin dependencias externas."""

  import re


  _HTML_ESCAPES = {
      "&": "&amp;",
      "<": "&lt;",
      ">": "&gt;",
      '"': "&quot;",
      "'": "&#x27;",
  }


  def _html_escape(value: str) -> str:
      for ch, entity in _HTML_ESCAPES.items():
          value = value.replace(ch, entity)
      return value


  def _resolve_path(context: dict, path: str, missing: str) -> str:
      """Resuelve un dot-path en un dict anidado."""
      parts = path.strip().split(".")
      current = context
      for part in parts:
          if isinstance(current, dict) and part in current:
              current = current[part]
          else:
              return missing
      return str(current) if current is not None else ""


  def _process_block(template: str, context: dict, missing: str) -> str:
      """Procesa un bloque de template (puede contener for/if anidados)."""
      result = template

      # Procesar bloques for: {% for x in items %}...{% endfor %}
      for_pattern = re.compile(
          r"\{%\s*for\s+(\w+)\s+in\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endfor\s*%\}",
          re.DOTALL,
      )
      while for_pattern.search(result):
          def replace_for(m: re.Match) -> str:
              var_name = m.group(1)
              list_path = m.group(2)
              body = m.group(3)
              items = _resolve_raw(context, list_path)
              if not isinstance(items, list):
                  return ""
              parts = []
              for item in items:
                  loop_context = dict(context)
                  loop_context[var_name] = item
                  parts.append(_process_block(body, loop_context, missing))
              return "".join(parts)

          result = for_pattern.sub(replace_for, result)

      # Procesar bloques if not: {% if not cond %}...{% endif %}
      if_not_pattern = re.compile(
          r"\{%\s*if\s+not\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endif\s*%\}",
          re.DOTALL,
      )
      while if_not_pattern.search(result):
          def replace_if_not(m: re.Match) -> str:
              cond_path = m.group(1)
              body = m.group(2)
              value = _resolve_raw(context, cond_path)
              if not value:
                  return _process_block(body, context, missing)
              return ""

          result = if_not_pattern.sub(replace_if_not, result)

      # Procesar bloques if: {% if cond %}...{% endif %}
      if_pattern = re.compile(
          r"\{%\s*if\s+(\w+(?:\.\w+)*)\s*%\}(.*?)\{%\s*endif\s*%\}",
          re.DOTALL,
      )
      while if_pattern.search(result):
          def replace_if(m: re.Match) -> str:
              cond_path = m.group(1)
              body = m.group(2)
              value = _resolve_raw(context, cond_path)
              if value:
                  return _process_block(body, context, missing)
              return ""

          result = if_pattern.sub(replace_if, result)

      # Sustitucion triple braces (sin escape): {{{var}}}
      raw_pattern = re.compile(r"\{\{\{(\s*[\w.]+\s*)\}\}\}")
      result = raw_pattern.sub(
          lambda m: _resolve_path(context, m.group(1), missing), result
      )

      # Sustitucion doble braces (HTML-escaped): {{var}}
      escaped_pattern = re.compile(r"\{\{(\s*[\w.]+\s*)\}\}")
      result = escaped_pattern.sub(
          lambda m: _html_escape(_resolve_path(context, m.group(1), missing)), result
      )

      return result


  def _resolve_raw(context: dict, path: str):
      """Resuelve un dot-path y retorna el valor raw (no convertido a string)."""
      parts = path.strip().split(".")
      current = context
      for part in parts:
          if isinstance(current, dict) and part in current:
              current = current[part]
          else:
              return None
      return current


  def render_template(template: str, context: dict, missing: str = "") -> str:
      """Motor de templates minimalista sin dependencias.

      Soporta sustitucion de variables, dot-path traversal, bucles for
      e condicionales if/if not. No pretende reemplazar Jinja2 — cubre
      el 80% de casos simples.

      Sintaxis:
          {{var}}             — sustitucion con HTML escape
          {{{var}}}           — sustitucion sin escape (raw)
          {{obj.key.subkey}}  — dot-path traversal en dicts anidados
          {% for x in items %}...{% endfor %} — iteracion sobre lista
          {% if cond %}...{% endif %}          — condicional truthy
          {% if not cond %}...{% endif %}      — negacion

      Args:
          template: String con la plantilla.
          context: Dict con los valores disponibles para sustitucion.
          missing: Valor por defecto para variables no encontradas.
                   Por defecto string vacio.

      Returns:
          String con todos los placeholders procesados.
      """
      return _process_block(template, context, missing)

params_schema:
  {"params":[{"name":"template","desc":"String con template usando {{var}}, {{{raw}}}, {% for %}, {% if %}"},{"name":"context","desc":"Diccionario con valores para las variables"},{"name":"missing","desc":"Valor a usar cuando falta una variable (default: '')"}],"output":"String renderizado con variables sustituidas y lógica aplicada"}


id: retry_async_py_core
name: retry_async
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","async","resilience","api","asyncio"]
uses_functions: ["classify_api_error_py_core","compute_backoff_delay_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["asyncio","typing"]
tests: []
file_path: python/functions/core/retry_async.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e2232c77a0f2085d0a151f77b95a39da971e6edb83748a857841ac18fba56516
signature:
  async def retry_async(func: Callable[[], Awaitable[T]], max_retries: int, base_delay: float = 0.5, max_delay: float = 8.0, jitter: bool = True, is_retryable: Callable[[Exception], bool] | None = None) -> T

description:
  Version async de retry_sync. Reintenta una corrutina en errores transitorios con exponential backoff. Usa asyncio.sleep en vez de time.sleep.

example:
  ```python
  import httpx

  async def fetch():
      async with httpx.AsyncClient() as client:
          return await client.get("https://api.example.com/data")

  # Reintentar hasta 3 veces con backoff
  response = await retry_async(func=fetch, max_retries=3)

  # Con predicado personalizado
  result = await retry_async(
      func=my_async_db_query,
      max_retries=5,
      is_retryable=lambda e: "deadlock" in str(e),
  )
  ```

notes:
  Version async de retry_sync: usa `asyncio.sleep` en lugar de `time.sleep`, permitiendo que el event loop atienda otras tareas durante la espera. El comportamiento de clasificacion y backoff es identico a retry_sync.

code:
  """Retry an async callable with exponential backoff."""

  import asyncio
  from typing import Awaitable, Callable, TypeVar

  from classify_api_error import classify_api_error
  from compute_backoff_delay import compute_backoff_delay

  T = TypeVar("T")


  async def retry_async(
      func: Callable[[], Awaitable[T]],
      max_retries: int,
      base_delay: float = 0.5,
      max_delay: float = 8.0,
      jitter: bool = True,
      is_retryable: Callable[[Exception], bool] | None = None,
  ) -> T:
      """Retry an async function on transient errors with exponential backoff.

      Args:
          func: Zero-argument async callable to execute and retry.
          max_retries: Maximum number of retry attempts after the first failure.
          base_delay: Base delay in seconds for backoff calculation.
          max_delay: Maximum delay cap in seconds.
          jitter: If True, adds random jitter to backoff delay.
          is_retryable: Optional predicate; if None, uses classify_api_error to
                        classify transient errors as retryable.

      Returns:
          The return value of func on success.

      Raises:
          Exception: The last exception raised if all attempts are exhausted or
                     if the error is not retryable.
      """
      if is_retryable is None:
          is_retryable = lambda e: classify_api_error(e) == "transient"

      last_exc: Exception | None = None
      for attempt in range(max_retries + 1):
          try:
              return await func()
          except Exception as exc:
              last_exc = exc
              if not is_retryable(exc) or attempt >= max_retries:
                  raise
              delay = compute_backoff_delay(attempt, base_delay, max_delay, jitter)
              await asyncio.sleep(delay)

      raise last_exc  # unreachable but satisfies type checkers

params_schema:
  {"params":[{"name":"func","desc":"Corrutina a reintentar"},{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"base_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"is_retryable","desc":"Predicado opcional para decidir si reintentar una excepción"}],"output":"Resultado de ejecutar la función si tiene éxito"}


id: retry_sync_py_core
name: retry_sync
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","sync","resilience","api"]
uses_functions: ["classify_api_error_py_core","compute_backoff_delay_py_core"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["time","typing"]
tests: []
file_path: python/functions/core/retry_sync.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 70ff8396d5324c071f29ee769f47d444f351f55a89fecbbc3b79848a66b6ffff
signature:
  def retry_sync(func: Callable[[], T], max_retries: int, base_delay: float = 0.5, max_delay: float = 8.0, jitter: bool = True, is_retryable: Callable[[Exception], bool] | None = None) -> T

description:
  Reintenta una funcion sincrona en errores transitorios con exponential backoff. Por defecto usa classify_api_error para decidir si reintentar. Hace time.sleep entre intentos.

example:
  ```python
  import httpx

  # Reintentar una llamada HTTP transitoria hasta 3 veces
  response = retry_sync(
      func=lambda: httpx.get("https://api.example.com/data"),
      max_retries=3,
  )

  # Con predicado personalizado
  retry_sync(
      func=my_db_query,
      max_retries=5,
      is_retryable=lambda e: "deadlock" in str(e),
  )
  ```

notes:
  Impure: hace `time.sleep` entre reintentos. Usa `classify_api_error` como clasificador por defecto — solo reintenta errores marcados como "transient". Si `is_retryable` devuelve False en la primera excepcion, re-raise inmediato sin esperar. El parametro `max_retries` es el numero de reintentos despues del primer intento (total de llamadas = max_retries + 1).

code:
  """Retry a synchronous callable with exponential backoff."""

  import time
  from typing import Callable, TypeVar

  from classify_api_error import classify_api_error
  from compute_backoff_delay import compute_backoff_delay

  T = TypeVar("T")


  def retry_sync(
      func: Callable[[], T],
      max_retries: int,
      base_delay: float = 0.5,
      max_delay: float = 8.0,
      jitter: bool = True,
      is_retryable: Callable[[Exception], bool] | None = None,
  ) -> T:
      """Retry a synchronous function on transient errors with exponential backoff.

      Args:
          func: Zero-argument callable to execute and retry.
          max_retries: Maximum number of retry attempts after the first failure.
          base_delay: Base delay in seconds for backoff calculation.
          max_delay: Maximum delay cap in seconds.
          jitter: If True, adds random jitter to backoff delay.
          is_retryable: Optional predicate; if None, uses classify_api_error to
                        classify transient errors as retryable.

      Returns:
          The return value of func on success.

      Raises:
          Exception: The last exception raised if all attempts are exhausted or
                     if the error is not retryable.
      """
      if is_retryable is None:
          is_retryable = lambda e: classify_api_error(e) == "transient"

      last_exc: Exception | None = None
      for attempt in range(max_retries + 1):
          try:
              return func()
          except Exception as exc:
              last_exc = exc
              if not is_retryable(exc) or attempt >= max_retries:
                  raise
              delay = compute_backoff_delay(attempt, base_delay, max_delay, jitter)
              time.sleep(delay)

      raise last_exc  # unreachable but satisfies type checkers

params_schema:
  {"params":[{"name":"func","desc":"Función a reintentar"},{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"base_delay","desc":"Delay inicial en segundos entre reintentos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"is_retryable","desc":"Predicado opcional para decidir si reintentar"}],"output":"Resultado de ejecutar la función si tiene éxito"}


id: retry_with_backoff_py_core
name: retry_with_backoff
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","decorator","sync","resilience","jitter"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["functools","time","random","asyncio","collections.abc","typing"]
test_file_path: python/functions/core/retry_with_backoff_test.py
file_path: python/functions/core/retry_with_backoff.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2ec2aa207e45f592ea1caa92f5c03164b63a405400d3a33256e42e65bec30bb
signature:
  def retry_with_backoff(max_retries: int = 3, initial_delay: float = 1.0, max_delay: float = 30.0, backoff_factor: float = 2.0, jitter: bool = True, exceptions: tuple[type[Exception], ...] = (Exception,), on_retry: Callable[[Exception, int], None] | None = None) -> Callable

description:
  Decorador que reintenta una funcion sincrona con exponential backoff cuando lanza excepciones del tipo especificado. Soporta jitter, callback on_retry y filtrado por tipo de excepcion.

example:
  ```python
  from retry_with_backoff import retry_with_backoff

  @retry_with_backoff(max_retries=3, exceptions=(ConnectionError, TimeoutError))
  def call_api():
      response = requests.get("https://api.example.com/data")
      response.raise_for_status()
      return response.json()

  # Con callback para logging
  def log_retry(exc: Exception, attempt: int) -> None:
      print(f"Attempt {attempt} failed: {exc}. Retrying...")

  @retry_with_backoff(max_retries=5, on_retry=log_retry)
  def fetch_resource():
      ...
  ```

tests:
  ["funcion que falla dos veces y luego exito","funcion que siempre falla agota retries y lanza","on_retry callback se llama","jitter produce delays variables","solo reintenta excepciones especificadas"]

notes:
  Impure: hace `time.sleep` entre reintentos. El delay entre intentos se calcula como `min(delay, max_delay)`, opcionalmente multiplicado por un factor aleatorio en `[0.5, 1.0]` si `jitter=True`. Tras cada intento fallido, `delay *= backoff_factor`. El parametro `max_retries` es el numero de reintentos despues del primer intento (total de llamadas = max_retries + 1). Solo reintenta excepciones que sean instancias de los tipos en `exceptions`. Implementado en el mismo archivo que `retry_with_backoff_async`.

code:
  """Retry decorators with exponential backoff for sync and async functions."""

  import asyncio
  import functools
  import random
  import time
  from collections.abc import Callable
  from typing import Any


  def retry_with_backoff(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries a synchronous function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      time.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator


  def retry_with_backoff_async(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries an async function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target async function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          async def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return await func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      await asyncio.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator

params_schema:
  {"params":[{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"initial_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"backoff_factor","desc":"Factor multiplicativo para el delay exponencial"},{"name":"jitter","desc":"Si True, añade variabilidad aleatoria al delay"},{"name":"exceptions","desc":"Tupla de tipos de excepción a reintentar"},{"name":"on_retry","desc":"Callback opcional llamado cuando se reintenta"}],"output":"Decorador que envuelve la función con lógica de retry"}


id: retry_with_backoff_async_py_core
name: retry_with_backoff_async
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["retry","backoff","exponential","decorator","async","asyncio","resilience","jitter"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["functools","asyncio","random","collections.abc","typing"]
test_file_path: python/functions/core/retry_with_backoff_test.py
file_path: python/functions/core/retry_with_backoff.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 182bb56a243c9c38de789be672fbdf28de81fc6fd00533e7855bd5aaa18aa672
signature:
  def retry_with_backoff_async(max_retries: int = 3, initial_delay: float = 1.0, max_delay: float = 30.0, backoff_factor: float = 2.0, jitter: bool = True, exceptions: tuple[type[Exception], ...] = (Exception,), on_retry: Callable[[Exception, int], None] | None = None) -> Callable

description:
  Decorador que reintenta una funcion async con exponential backoff cuando lanza excepciones del tipo especificado. Usa asyncio.sleep en vez de time.sleep.

example:
  ```python
  from retry_with_backoff import retry_with_backoff_async

  @retry_with_backoff_async(max_retries=3, exceptions=(ConnectionError, TimeoutError))
  async def fetch_data():
      async with httpx.AsyncClient() as client:
          response = await client.get("https://api.example.com/data")
          response.raise_for_status()
          return response.json()

  # Con callback para logging
  def log_retry(exc: Exception, attempt: int) -> None:
      print(f"Async attempt {attempt} failed: {exc}. Retrying...")

  @retry_with_backoff_async(max_retries=5, on_retry=log_retry)
  async def call_service():
      ...
  ```

tests:
  ["async funcion que falla dos veces y luego exito","async funcion que siempre falla agota retries y lanza","async on_retry callback se llama","async jitter produce delays variables","async solo reintenta excepciones especificadas"]

notes:
  Impure: usa `asyncio.sleep` entre reintentos — no bloquea el event loop. Tiene la misma logica de backoff que `retry_with_backoff` pero para coroutines. El wrapper generado es una coroutine (`async def`), por lo que es compatible con `await` y `asyncio.gather`. Definido en el mismo archivo que `retry_with_backoff`.

code:
  """Retry decorators with exponential backoff for sync and async functions."""

  import asyncio
  import functools
  import random
  import time
  from collections.abc import Callable
  from typing import Any


  def retry_with_backoff(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries a synchronous function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      time.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator


  def retry_with_backoff_async(
      max_retries: int = 3,
      initial_delay: float = 1.0,
      max_delay: float = 30.0,
      backoff_factor: float = 2.0,
      jitter: bool = True,
      exceptions: tuple[type[Exception], ...] = (Exception,),
      on_retry: Callable[[Exception, int], None] | None = None,
  ) -> Callable:
      """Decorator that retries an async function with exponential backoff.

      Args:
          max_retries: Maximum number of retry attempts after the first failure.
          initial_delay: Initial delay in seconds before the first retry.
          max_delay: Maximum delay cap in seconds.
          backoff_factor: Multiplier applied to delay after each retry.
          jitter: If True, multiplies delay by a random factor in [0.5, 1.0].
          exceptions: Tuple of exception types that trigger a retry.
          on_retry: Optional callback invoked with (exception, attempt) before sleeping.

      Returns:
          Decorator that wraps the target async function with retry logic.

      Raises:
          Exception: The last exception raised after all retries are exhausted.
      """
      def decorator(func: Callable) -> Callable:
          @functools.wraps(func)
          async def wrapper(*args: Any, **kwargs: Any) -> Any:
              delay = initial_delay
              for attempt in range(max_retries + 1):
                  try:
                      return await func(*args, **kwargs)
                  except exceptions as exc:
                      if attempt >= max_retries:
                          raise
                      capped = min(delay, max_delay)
                      actual_delay = capped * (0.5 + random.random()) if jitter else capped
                      if on_retry is not None:
                          on_retry(exc, attempt)
                      await asyncio.sleep(actual_delay)
                      delay *= backoff_factor
          return wrapper
      return decorator

params_schema:
  {"params":[{"name":"max_retries","desc":"Número máximo de reintentos"},{"name":"initial_delay","desc":"Delay inicial en segundos"},{"name":"max_delay","desc":"Delay máximo en segundos"},{"name":"backoff_factor","desc":"Factor multiplicativo para exponential backoff"},{"name":"jitter","desc":"Si True, añade variabilidad al delay"},{"name":"exceptions","desc":"Tupla de tipos de excepción a reintentar"},{"name":"on_retry","desc":"Callback opcional para logging de reintentos"}],"output":"Decorador que envuelve función async con lógica de retry"}


id: sanitize_for_path_py_core
name: sanitize_for_path
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def sanitize_for_path(text: str, max_length: int = 50) -> str
tags: ["path","sanitize","string","filesystem","slug"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","hashlib"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: c7230e66c9e245e8c30e4be0c11c632924d635e44f28b7cf1e3b055750bfc5b1
description:
  Convierte texto a nombre seguro para uso en paths. Remueve caracteres especiales, reemplaza espacios con _, trunca con hash suffix si excede max_length.

example:
  ```python
  sanitize_for_path("Hello World!")       # "Hello_World"
  sanitize_for_path("中文标题")             # "中文标题"
  sanitize_for_path("a" * 100)           # "aaa...aaa_<sha256[:8]>" (max 50 chars)
  sanitize_for_path("")                  # "section"
  sanitize_for_path("price: $100 @2024") # "price__100_2024"
  ```

tests:
  ["texto normal produce slug con underscores","caracteres especiales son removidos","texto muy largo trunca con hash suffix","texto vacio retorna section","texto CJK es preservado"]

notes:
  Funcion pura. Caracteres permitidos: `\w` (letras, digitos, _), rangos CJK `[\u4e00-\u9fff\u3040-\u30ff\uac00-\ud7af]`, espacios (convertidos a `_`) y guiones. El sufijo hash es `_` + primeros 8 caracteres del sha256 del texto original, lo que hace la truncacion determinista y evita colisiones entre textos largos similares. Si el texto es todo caracteres especiales y queda vacio tras limpiar, retorna `"section"` como fallback seguro.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"text","desc":"Texto a convertir en nombre seguro para filesystem"},{"name":"max_length","desc":"Longitud máxima permitida del resultado"}],"output":"String sanitizado apto para usar en paths y nombres de archivos"}


id: smart_split_content_py_core
name: smart_split_content
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def smart_split_content(content: str, max_tokens: int = 1024, max_chars: int = 8000) -> list[str]
tags: ["chunking","splitting","tokens","text","markdown","llm"]
uses_functions: ["estimate_token_count_py_core"]
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/parse_markdown_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e595ce1ddc0fbd0d2ccd1820f6ba05c174fbbb0d73a74706ccbed7da90a6c5ce
description:
  Divide contenido grande en partes respetando limites de tokens y caracteres. Divide por parrafos (doble newline). Si un parrafo individual excede el limite, lo corta por caracteres.

example:
  ```python
  # Short content — stays as one chunk
  parts = smart_split_content("Hello.\n\nWorld.", max_tokens=1024, max_chars=8000)
  # ["Hello.\n\nWorld."]

  # Long content — split at paragraph boundaries
  long_md = "\n\n".join(["paragraph text"] * 300)
  parts = smart_split_content(long_md, max_tokens=100, max_chars=8000)
  # [<chunk1>, <chunk2>, ...]
  ```

tests:
  ["contenido corto produce una sola parte","contenido largo divide en multiples partes","parrafo gigante que requiere forzar corte"]

notes:
  Funcion pura. Compone `estimate_token_count` para la estimacion de tokens. La estrategia de split es conservadora: acumula parrafos mientras quepan bajo AMBOS limites (tokens Y caracteres). Si un parrafo individual excede cualquier limite se fuerza corte byte a byte cada `max_chars` caracteres. Util para partir documentos grandes antes de enviarlos a LLMs con ventana de contexto limitada.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"content","desc":"Contenido grande a dividir"},{"name":"max_tokens","desc":"Límite máximo de tokens por chunk"},{"name":"max_chars","desc":"Límite máximo de caracteres por chunk"}],"output":"Lista de strings donde cada string es un chunk que respeta ambos límites"}


id: split_text_into_chunks_py_core
name: split_text_into_chunks
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def split_text_into_chunks(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]
tags: ["text","chunking","nlp","split","overlap","sentence","boundary","rag"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/split_text_into_chunks_test.py
file_path: python/functions/core/split_text_into_chunks.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f2cdc661bae12ca7d31d66a992efbfedb9e8dd14fc7c986c1f73d5911619f7d5
description:
  Divide texto en chunks de tamaño fijo con overlap, intentando cortar en límites de oración para no romper frases a mitad. Soporta separadores CJK (。！？) y occidentales (. ! ?).

example:
  ```python
  text = "First sentence is here. Second sentence follows. Third ends it."
  chunks = split_text_into_chunks(text, chunk_size=40, overlap=10)
  # ['First sentence is here.', 'is here. Second sentence follows.', ...]

  # Texto vacío
  split_text_into_chunks("")  # []

  # Texto corto (cabe en un chunk)
  split_text_into_chunks("Hello world.", chunk_size=500)  # ['Hello world.']
  ```

tests:
  ["texto corto cabe en 1 chunk","texto largo con oraciones corta en punto","texto cjk con separador ideografico","texto sin separadores corta en chunk size exacto","overlap funciona","texto vacio retorna lista vacia"]

notes:
  Algoritmo de sliding window con detección de límites de oración. Los separadores se evalúan en orden de prioridad:
  `。`, `！`, `？`, `.\n`, `!\n`, `?\n`, `\n\n`, `. `, `! `, `? `

  Un separador solo se acepta si el corte resultante está después del 30% del chunk (evita chunks muy cortos). Si no se encuentra ningún separador válido, el texto se corta en `chunk_size` exacto.

  El siguiente chunk empieza en `end - overlap`, garantizando solapamiento entre chunks consecutivos. Esto es útil para RAG (Retrieval-Augmented Generation) donde el contexto de límite de chunk puede ser crítico.

  Diferencia con `page_list_to_groups_py_core`: ese agrupa páginas por tokens con overlap configurable entre grupos de páginas. Este divide texto plano por caracteres con detección de oraciones.

code:
  """Split text into overlapping chunks with sentence-boundary awareness."""


  def split_text_into_chunks(
      text: str, chunk_size: int = 500, overlap: int = 50
  ) -> list[str]:
      """Divide texto en chunks de tamaño fijo con overlap, cortando en límites de oración.

      Args:
          text: Texto a dividir.
          chunk_size: Tamaño máximo de cada chunk en caracteres.
          overlap: Número de caracteres de solapamiento entre chunks consecutivos.

      Returns:
          Lista de chunks. Vacía si el texto es vacío.
      """
      if not text:
          return []

      if len(text) <= chunk_size:
          stripped = text.strip()
          return [stripped] if stripped else []

      # Separadores en orden de prioridad (más específicos primero)
      separators = ["。", "！", "？", ".\n", "!\n", "?\n", "\n\n", ". ", "! ", "? "]

      chunks: list[str] = []
      start = 0
      text_len = len(text)

      while start < text_len:
          end = start + chunk_size

          if end < text_len:
              # Buscar el último separador de oración dentro de text[start:end]
              # Solo aceptar si está después del 30% del chunk
              min_pos = start + int(chunk_size * 0.30)
              best_end = None

              for sep in separators:
                  sep_len = len(sep)
                  # Buscar la última ocurrencia del separador en text[start:end]
                  search_region = text[start:end]
                  pos = search_region.rfind(sep)
                  if pos == -1:
                      continue
                  abs_pos = start + pos + sep_len
                  if abs_pos > min_pos:
                      # Usar este separador solo si produce un corte más tarde que el mínimo
                      # y más temprano que chunk_size (ya garantizado por rfind en [start:end])
                      if best_end is None or abs_pos > best_end:
                          best_end = abs_pos

              if best_end is not None:
                  end = best_end

          chunk = text[start:end].strip()
          if chunk:
              chunks.append(chunk)

          start = end - overlap
          # Protección contra bucle infinito si overlap >= chunk_size o end no avanza
          if start >= end:
              start = end

      return chunks

params_schema:
  {"params":[{"name":"text","desc":"Texto a dividir en chunks"},{"name":"chunk_size","desc":"Tamaño objetivo de cada chunk en caracteres"},{"name":"overlap","desc":"Número de caracteres a solapar entre chunks consecutivos"}],"output":"Lista de strings donde cada uno es un chunk con solapamiento"}


id: strip_markdown_codeblock_py_core
name: strip_markdown_codeblock
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def strip_markdown_codeblock(text: str) -> str
tags: ["llm","text","cleaning","markdown","codeblock","json"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/strip_markdown_codeblock_test.py
file_path: python/functions/core/strip_markdown_codeblock.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 36e0897ec48d8417649bbdb885191882e7952c1522b04c2d79847f804266c23d
description:
  Remueve wrapping de code blocks markdown que modelos a veces anaden al responder JSON. Ej: ```json\n{...}\n``` → {...}

example:
  ```python
  wrapped = '```json\n{"key": "value"}\n```'
  result = strip_markdown_codeblock(wrapped)
  # '{"key": "value"}'

  plain = '{"key": "value"}'
  result = strip_markdown_codeblock(plain)
  # '{"key": "value"}'
  ```

tests:
  ["JSON sin codeblock no cambia","JSON con wrapper ```json es desenvuelto","JSON con wrapper ``` solo es desenvuelto","texto que no es JSON pasa sin cambio"]

notes:
  Funcion pura. Aplica cuatro pasos: strip inicial, remover prefijo ` ``` ` o ` ```json ` (case-insensitive), remover sufijo ` ``` `, strip final. Maneja tanto ` ```json ` como ` ``` ` generico. El flag IGNORECASE permite ` ```JSON ` o ` ```Json `.

code:
  """Remove markdown code block wrapping from LLM responses."""

  import re


  def strip_markdown_codeblock(text: str) -> str:
      """Remove markdown code block wrapping from LLM responses.

      LLMs sometimes wrap JSON (or other content) in markdown code fences like:
          ```json
          {"key": "value"}
          ```
      This function unwraps the content so it can be parsed directly.

      Args:
          text: Text that may be wrapped in a markdown code block.

      Returns:
          Unwrapped text with leading/trailing whitespace stripped.
      """
      text = text.strip()
      text = re.sub(r'^```(?:json)?\s*\n?', '', text, flags=re.IGNORECASE)
      text = re.sub(r'\n?```\s*$', '', text)
      return text.strip()

params_schema:
  {"params":[{"name":"text","desc":"String posiblemente envuelto en markdown code block"}],"output":"String con los delimitadores ```json/``` removidos si estaban presentes"}


id: strip_think_tags_py_core
name: strip_think_tags
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def strip_think_tags(text: str) -> str
tags: ["llm","text","cleaning","think","chain-of-thought","deepseek","minimax"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/strip_think_tags_test.py
file_path: python/functions/core/strip_think_tags.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 345039965dc09cbe89872386bb99e4069c961b695a0cf236f1d2abf483a0423a
description:
  Remueve tags <think>...</think> que algunos modelos (MiniMax, DeepSeek) incluyen en sus respuestas como chain of thought interno.

example:
  ```python
  raw = "<think>Let me reason step by step...</think>La respuesta es 42."
  result = strip_think_tags(raw)
  # "La respuesta es 42."

  multiline = "<think>\nPrimero analizo...\nLuego concluyo...\n</think>Respuesta final."
  result = strip_think_tags(multiline)
  # "Respuesta final."
  ```

tests:
  ["texto sin tags no cambia","texto con think tags los remueve","tags multilinea son removidos","multiples tags son removidos"]

notes:
  Funcion pura. Usa `re.sub` con el flag `[\s\S]*?` (lazy) para manejar contenido multilinea sin capturar de mas cuando hay multiples tags. El `.strip()` final elimina whitespace residual alrededor del texto util.

code:
  """Remove <think>...</think> tags from LLM responses."""

  import re


  def strip_think_tags(text: str) -> str:
      """Remove <think>...</think> tags from LLM responses.

      Some models (MiniMax, DeepSeek) include chain-of-thought blocks wrapped in
      <think> tags as part of their output. This function strips those blocks so
      that downstream code only sees the final answer.

      Args:
          text: Raw LLM response that may contain <think>...</think> blocks.

      Returns:
          Text with all <think>...</think> blocks removed and outer whitespace
          stripped.
      """
      return re.sub(r'<think>[\s\S]*?</think>', '', text).strip()

params_schema:
  {"params":[{"name":"text","desc":"String posiblemente con tags \u003cthink\u003e...\u003c/think\u003e"}],"output":"String con los think tags removidos"}


id: t_py_core
name: t
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
signature: def t(key: str, locale: str | None = None, **kwargs) -> str
tags: ["i18n","translation","locale","dot-path","interpolation"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["threading"]
test_file_path: python/functions/core/t_test.py
file_path: python/functions/core/t.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 544444150baa4e28a266e79675567b1e4cd51b1efa4f1f9369689ca7051b6f06
description:
  Traduce una clave con dot-path notation al idioma activo. Prioridad: parametro locale > thread-local > default. Soporta interpolacion {variable}. Fallback al locale default si la clave no existe; si tampoco existe, retorna la key.

example:
  ```python
  from t import t, _set_translations

  translations = {
      "en": {
          "report": {
              "sectionStart": "Generating section: {title}",
              "done": "Done"
          }
      },
      "es": {
          "report": {
              "done": "Listo"
          }
      }
  }
  _set_translations(translations, default_locale="en")

  t("report.sectionStart", locale="en", title="Introduction")
  # → "Generating section: Introduction"

  t("report.done", locale="es")
  # → "Listo"

  t("report.sectionStart", locale="es", title="Intro")
  # → "Generating section: Intro"  (fallback a en)

  t("nonexistent.key", locale="en")
  # → "nonexistent.key"
  ```

tests:
  ["key existente retorna traduccion","key inexistente retorna la key","interpolacion de variables","dot-path profundo","fallback a locale default"]

notes:
  Lee estado global (modulo-level `_translations` y `_locale_local` thread-local), por eso es impura. Configurar con `_set_translations()` al inicio de la aplicacion y `_set_locale()` por thread/request. La interpolacion usa `str.format(**kwargs)` — si hay un placeholder faltante, retorna el string sin interpolar para no romper el flujo. Inspirada conceptualmente en el modulo `locale.py` de MiroFish (AGPL-3.0); reimplementada desde cero.

code:
  """Traduccion de claves con dot-path notation e interpolacion de variables."""

  import threading

  _locale_local = threading.local()
  _translations: dict = {}
  _default_locale: str = "en"


  def _set_translations(translations: dict, default_locale: str = "en") -> None:
      """Configura el diccionario global de traducciones y el locale default.

      Llamar desde load_translations o al inicio de la aplicacion.
      """
      global _translations, _default_locale
      _translations = translations
      _default_locale = default_locale


  def _set_locale(locale: str) -> None:
      """Establece el locale para el thread actual."""
      _locale_local.locale = locale


  def _get_locale() -> str:
      """Retorna el locale del thread actual, o el default si no esta configurado."""
      return getattr(_locale_local, "locale", _default_locale)


  def _resolve_key(translations: dict, locale: str, key: str) -> str | None:
      """Navega dot-path en el diccionario de traducciones para un locale.

      Returns:
          El string de traduccion si existe, None si no.
      """
      locale_dict = translations.get(locale)
      if not locale_dict:
          return None

      parts = key.split(".")
      node = locale_dict
      for part in parts:
          if not isinstance(node, dict):
              return None
          node = node.get(part)
          if node is None:
              return None

      return node if isinstance(node, str) else None


  def t(key: str, locale: str | None = None, **kwargs) -> str:
      """Traduce una clave con dot-path notation al idioma actual.

      Determina el locale en orden de prioridad: parametro > thread-local > default.
      Soporta interpolacion de variables con {nombre} en el valor traducido.
      Si la clave no existe en el locale solicitado, intenta el locale default.
      Si tampoco existe, retorna la clave tal cual.

      Args:
          key: Clave de traduccion en dot-path notation (ej: "report.taskStarted").
          locale: Locale a usar. Si es None usa el locale del thread actual.
          **kwargs: Variables para interpolar en el string traducido.

      Returns:
          String traducido con variables interpoladas, o la key si no se encontro.

      Example:
          >>> # translations = {"en": {"report": {"sectionStart": "Section: {title}"}}}
          >>> t("report.sectionStart", locale="en", title="Introduction")
          'Section: Introduction'
          >>> t("nonexistent.key", locale="en")
          'nonexistent.key'
      """
      resolved_locale = locale if locale is not None else _get_locale()

      value = _resolve_key(_translations, resolved_locale, key)

      if value is None and resolved_locale != _default_locale:
          value = _resolve_key(_translations, _default_locale, key)

      if value is None:
          return key

      if kwargs:
          try:
              value = value.format(**kwargs)
          except (KeyError, ValueError):
              pass

      return value

params_schema:
  {"params":[{"name":"key","desc":"Clave de traducción con dot-path notation (ej: 'report.done')"},{"name":"locale","desc":"Locale a usar (default: usa locale thread-local)"},{"name":"kwargs","desc":"Variables para interpolación en el template"}],"output":"String traducido e interpolado, o la key si no se encuentra traducción"}


id: take_py_core
name: take
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def take(xs: list, n: int) -> list
description: Toma los primeros n elementos de una lista.
tags: ["list","functional","take","slice","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = take([1, 2, 3, 4, 5], 3)
  # [1, 2, 3]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Si n > len(xs), retorna toda la lista. No muta la original.
content_hash: 069d9e069f12f7eba8edacdf5f5537834fe9981ed4cf5123ad38906a97c77bac
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista de la que tomar elementos"},{"name":"n","desc":"Número de elementos a tomar desde el inicio"}],"output":"Lista con los primeros n elementos"}


id: task_manager_py_core
name: task_manager
kind: function
lang: py
domain: core
version: 1.0.0
purity: impure
tags: ["task","manager","singleton","thread-safe","background","async","polling","progress"]
uses_functions: []
uses_types: ["task_py_core","task_status_py_core"]
returns: []
error_type: error_go_core
imports: ["threading","uuid","datetime"]
tests: []
file_path: python/functions/core/task_manager.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a1a232a29013277dc2eec254bdb829488642ddd69c1506907c65d4d83eaf4ac8
params_schema: {"output":"Instancia singleton thread-safe para gestionar tareas de larga duración"}
signature:
  class TaskManager: create_task(task_type: str, metadata: dict | None = None) -> str; get_task(task_id: str) -> Task | None; update_task(task_id: str, ...) -> None; complete_task(task_id: str, result: dict) -> None; fail_task(task_id: str, error: str) -> None; list_tasks(task_type: str | None = None) -> list[dict]; cleanup_old_tasks(max_age_hours: int = 24) -> None

description:
  Gestor de tareas background en memoria, thread-safe, con patron singleton. Permite crear, actualizar, completar y limpiar tareas de larga duracion. Patron tipico: hilo background actualiza progreso via update_task(), frontend hace polling via get_task().

example:
  ```python
  manager = TaskManager()

  # Crear tarea
  task_id = manager.create_task("graph_build", metadata={"nodes": 500})

  # Hilo background actualiza progreso
  manager.update_task(task_id, status=TaskStatus.PROCESSING, progress=25, message="Cargando datos...")
  manager.update_task(task_id, progress=75, message="Construyendo grafo...")
  manager.complete_task(task_id, result={"graph_id": "abc-123"})

  # Frontend hace polling
  task = manager.get_task(task_id)
  print(task.to_dict())
  # {"task_id": "...", "status": "completed", "progress": 100, ...}

  # Listar todas las tareas de un tipo
  tasks = manager.list_tasks(task_type="graph_build")

  # Limpiar tareas antiguas (> 24h)
  manager.cleanup_old_tasks(max_age_hours=24)
  ```

notes:
  Singleton con double-checked locking: la primera instancia se crea una sola vez aunque multiples hilos llamen a `TaskManager()` simultaneamente.

  El lock `_tasks_lock` protege el diccionario interno `_tasks` en todas las operaciones de lectura y escritura.

  `update_task()` solo actualiza los campos que recibe como no-None, lo que permite actualizaciones parciales sin sobreescribir el estado existente.

  `cleanup_old_tasks()` solo elimina tareas en estado terminal (COMPLETED o FAILED). Las tareas PENDING o PROCESSING no se eliminan independientemente de su edad.

  El `error_type` referencia `error_go_core` como convencion del registry — los errores reales en Python se lanzan como excepciones, no se retornan.

code:
  """TaskManager — gestor de tareas background thread-safe con patron singleton."""

  import threading
  import uuid
  from datetime import datetime, timezone

  from task_status import TaskStatus
  from task import Task


  class TaskManager:
      """Gestor de tareas en memoria, thread-safe, con patron singleton.

      Uso tipico:
          manager = TaskManager()
          task_id = manager.create_task("graph_build", metadata={"nodes": 500})
          # en un hilo background:
          manager.update_task(task_id, progress=50, message="Procesando nodos...")
          manager.complete_task(task_id, result={"graph_id": "abc"})
          # desde el frontend (polling):
          task = manager.get_task(task_id)
      """

      _instance: "TaskManager | None" = None
      _lock: threading.Lock = threading.Lock()

      def __new__(cls) -> "TaskManager":
          if cls._instance is None:
              with cls._lock:
                  if cls._instance is None:
                      instance = super().__new__(cls)
                      instance._tasks: dict[str, Task] = {}
                      instance._tasks_lock = threading.Lock()
                      cls._instance = instance
          return cls._instance

      # ------------------------------------------------------------------
      # Operaciones publicas
      # ------------------------------------------------------------------

      def create_task(self, task_type: str, metadata: dict | None = None) -> str:
          """Crea una nueva tarea en estado PENDING y retorna su task_id.

          Args:
              task_type: Identificador libre del tipo de tarea ("graph_build", etc.).
              metadata:  Datos adicionales asociados a la tarea (parametros de entrada).

          Returns:
              task_id — UUID string de la tarea creada.
          """
          task_id = str(uuid.uuid4())
          now = datetime.now(timezone.utc)
          task = Task(
              task_id=task_id,
              task_type=task_type,
              status=TaskStatus.PENDING,
              created_at=now,
              updated_at=now,
              metadata=metadata or {},
          )
          with self._tasks_lock:
              self._tasks[task_id] = task
          return task_id

      def get_task(self, task_id: str) -> Task | None:
          """Retorna la tarea con el id dado, o None si no existe.

          Args:
              task_id: UUID de la tarea.

          Returns:
              Task o None.
          """
          with self._tasks_lock:
              return self._tasks.get(task_id)

      def update_task(
          self,
          task_id: str,
          status: TaskStatus | None = None,
          progress: int | None = None,
          message: str | None = None,
          result: dict | None = None,
          error: str | None = None,
          progress_detail: dict | None = None,
      ) -> None:
          """Actualiza campos de una tarea existente.

          Solo actualiza los campos que no sean None. Actualiza siempre updated_at.

          Args:
              task_id:         UUID de la tarea a actualizar.
              status:          Nuevo estado (TaskStatus), opcional.
              progress:        Nuevo porcentaje 0-100, opcional.
              message:         Nuevo mensaje de estado, opcional.
              result:          Resultado de la tarea, opcional.
              error:           Mensaje de error, opcional.
              progress_detail: Detalles granulares de progreso, opcional.
          """
          with self._tasks_lock:
              task = self._tasks.get(task_id)
              if task is None:
                  return
              if status is not None:
                  task.status = status
              if progress is not None:
                  task.progress = progress
              if message is not None:
                  task.message = message
              if result is not None:
                  task.result = result
              if error is not None:
                  task.error = error
              if progress_detail is not None:
                  task.progress_detail = progress_detail
              task.updated_at = datetime.now(timezone.utc)

      def complete_task(self, task_id: str, result: dict) -> None:
          """Marca la tarea como COMPLETED con el resultado dado.

          Args:
              task_id: UUID de la tarea.
              result:  Resultado serializable de la tarea completada.
          """
          self.update_task(
              task_id,
              status=TaskStatus.COMPLETED,
              progress=100,
              result=result,
          )

      def fail_task(self, task_id: str, error: str) -> None:
          """Marca la tarea como FAILED con el mensaje de error dado.

          Args:
              task_id: UUID de la tarea.
              error:   Descripcion del error.
          """
          self.update_task(
              task_id,
              status=TaskStatus.FAILED,
              error=error,
          )

      def list_tasks(self, task_type: str | None = None) -> list[dict]:
          """Lista todas las tareas, opcionalmente filtradas por tipo.

          Args:
              task_type: Si se especifica, solo retorna tareas de ese tipo.

          Returns:
              Lista de dicts serializados (via Task.to_dict()).
          """
          with self._tasks_lock:
              tasks = list(self._tasks.values())
          if task_type is not None:
              tasks = [t for t in tasks if t.task_type == task_type]
          return [t.to_dict() for t in tasks]

      def cleanup_old_tasks(self, max_age_hours: int = 24) -> None:
          """Elimina tareas terminadas (COMPLETED o FAILED) con mas de max_age_hours horas.

          Args:
              max_age_hours: Edad maxima en horas para tareas terminadas. Default: 24.
          """
          now = datetime.now(timezone.utc)
          terminal = {TaskStatus.COMPLETED, TaskStatus.FAILED}
          with self._tasks_lock:
              to_delete = [
                  tid
                  for tid, task in self._tasks.items()
                  if task.status in terminal
                  and (now - task.updated_at).total_seconds() > max_age_hours * 3600
              ]
              for tid in to_delete:
                  del self._tasks[tid]


id: to_csv_py_core
name: to_csv
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
tags: ["csv","serialization","export","tabular","format"]
uses_functions: []
uses_types: []
returns: []
imports: []
test_file_path: python/functions/core/to_csv_test.py
file_path: python/functions/core/to_csv.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a47505629d2e1127723fc259adb900d528901626f23b34d746bae86a999e35a7
signature:
  to_csv(rows: list[dict], columns: list[str] | None = None, delimiter: str = ',', include_header: bool = True) -> str

description:
  Serializa datos tabulares a CSV. Si columns es None, usa las keys de la primera fila. Escapa campos con comillas, newlines y delimiters correctamente (RFC 4180). Sin dependencias externas.

example:
  ```python
  rows = [{"nombre": "Ana", "edad": 30}, {"nombre": "Bob", "edad": 25}]
  csv = to_csv(rows)
  # "nombre,edad\r\nAna,30\r\nBob,25"

  # Con columnas especificas
  csv = to_csv(rows, columns=["edad", "nombre"])
  # "edad,nombre\r\n30,Ana\r\n25,Bob"

  # Sin header
  csv = to_csv(rows, include_header=False)
  # "Ana,30\r\nBob,25"
  ```

tests:
  ["lista simple csv con header","campos con comas comillas newlines","columns explicitas reordena y filtra","include header false","lista vacia retorna string vacio","valores none campo vacio"]

notes:
  Implementacion manual sin el modulo csv de stdlib para control total del escaping.
  Cumple RFC 4180: escapa con comillas dobles cualquier campo que contenga el delimiter, comillas o saltos de linea. Los valores None se serializan como campo vacio.

code:
  """Serializa datos tabulares a formato CSV (RFC 4180)."""


  def _escape_field(value: str, delimiter: str) -> str:
      """Escapa un campo segun RFC 4180 si contiene delimiter, comillas o newlines."""
      if any(c in value for c in (delimiter, '"', "\n", "\r")):
          return '"' + value.replace('"', '""') + '"'
      return value


  def to_csv(
      rows: list[dict],
      columns: list[str] | None = None,
      delimiter: str = ",",
      include_header: bool = True,
  ) -> str:
      """Serializa datos tabulares a CSV.

      Implementacion manual sin csv module para control total del escaping.
      Cumple RFC 4180: escapa campos con comillas, newlines y delimiters.

      Args:
          rows: Lista de dicts con los datos. Todas las filas deben tener
                las mismas keys (o un subconjunto de columns).
          columns: Columnas a incluir y su orden. Si es None, usa las keys
                   de la primera fila en orden de insercion.
          delimiter: Separador de campos. Por defecto coma.
          include_header: Si True, incluye la fila de encabezados.

      Returns:
          String CSV completo. String vacio si rows es vacio.
      """
      if not rows:
          return ""

      cols = columns if columns is not None else list(rows[0].keys())

      output_lines: list[str] = []

      if include_header:
          output_lines.append(delimiter.join(_escape_field(c, delimiter) for c in cols))

      for row in rows:
          fields = []
          for col in cols:
              raw = row.get(col)
              if raw is None:
                  fields.append("")
              else:
                  fields.append(_escape_field(str(raw), delimiter))
          output_lines.append(delimiter.join(fields))

      return "\r\n".join(output_lines)

params_schema:
  {"params":[{"name":"rows","desc":"Lista de diccionarios a serializar como CSV"},{"name":"columns","desc":"Lista opcional de nombres de columnas (orden y filtrado)"},{"name":"delimiter","desc":"Carácter delimitador para campos"},{"name":"include_header","desc":"Si True, incluye fila de encabezados"}],"output":"String con datos en formato CSV válido (RFC 4180)"}


id: to_jsonl_py_core
name: to_jsonl
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: to_jsonl(rows: list[dict]) -> str
tags: ["jsonl","json","serialization","export","streaming","format"]
uses_functions: []
uses_types: []
returns: []
imports: ["json"]
test_file_path: python/functions/core/to_jsonl_test.py
file_path: python/functions/core/to_jsonl.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 5ae54c6aabf7efa66f91b9ccb4a9860d84eec8a41affee153dd08e7cdbc9f4fe
description:
  Serializa a JSON Lines (newline-delimited JSON). Cada dict se serializa como una linea JSON independiente. Util para streaming, logging estructurado y formatos de intercambio.

example:
  ```python
  rows = [{"id": 1, "name": "Ana"}, {"id": 2, "name": "Bob"}]
  jsonl = to_jsonl(rows)
  # '{"id": 1, "name": "Ana"}\n{"id": 2, "name": "Bob"}'

  # Unicode preservado
  rows = [{"emoji": "🔥", "val": None}]
  jsonl = to_jsonl(rows)
  # '{"emoji": "🔥", "val": null}'
  ```

tests:
  ["lista de dicts jsonl","valores unicode none nested","lista vacia retorna string vacio","cada linea es json parseable"]

notes:
  Usa json.dumps con ensure_ascii=False para preservar Unicode. No incluye newline al final.
  Cada linea es un JSON valido independiente, parseable con json.loads.

code:
  """Serializa lista de dicts a formato JSON Lines (JSONL)."""

  import json


  def to_jsonl(rows: list[dict]) -> str:
      """Serializa a JSON Lines (newline-delimited JSON).

      Cada dict se serializa como una linea JSON independiente separada por \\n.
      Util para streaming, logging estructurado y formatos de intercambio de datos.

      Args:
          rows: Lista de dicts a serializar. Cada dict puede contener valores
                anidados, Unicode, None, numeros, etc.

      Returns:
          String con un JSON por linea. String vacio si rows es vacio.
          No incluye newline al final.
      """
      if not rows:
          return ""

      return "\n".join(json.dumps(row, ensure_ascii=False) for row in rows)

params_schema:
  {"params":[{"name":"rows","desc":"Lista de diccionarios a serializar"}],"output":"String en formato JSON Lines (newline-delimited JSON)"}


id: to_pascal_case_py_core
name: to_pascal_case
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def to_pascal_case(name: str) -> str
tags: ["string","naming","pascal-case","case-conversion","formatting","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/to_pascal_case_test.py
file_path: python/functions/core/to_pascal_case.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 6375eadc015917983a4d59867adae7824e264b517cc3726251561224a33ba69b
description:
  Convierte cualquier formato de nombre (snake_case, camelCase, kebab-case, mixto) a PascalCase. Retorna "Unknown" para entrada vacia.

example:
  ```python
  to_pascal_case("works_for")        # "WorksFor"
  to_pascal_case("camelCaseExample") # "CamelCaseExample"
  to_pascal_case("SCREAMING_SNAKE")  # "ScreamingSnake"
  to_pascal_case("kebab-case")       # "KebabCase"
  to_pascal_case("")                 # "Unknown"
  ```

tests:
  ["snake_case","camelCase","UPPER_SNAKE","kebab-case","pascal_case_idempotente","string_vacio","con_numeros"]

notes:
  Funcion pura, sin dependencias externas (solo `re` de stdlib).

  Algoritmo en dos pasos:
  1. Split por caracteres no alfanumericos: `re.split(r'[^a-zA-Z0-9]+', name)`
  2. Split adicional por boundary camelCase en cada parte: `re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_')`

  Cada word resultante se capitaliza (primera letra upper, resto lower) y se concatenan. Si no hay words validas, retorna `"Unknown"`.

code:
  """Convert any name format to PascalCase."""

  import re


  def to_pascal_case(name: str) -> str:
      """Convert any name format to PascalCase.

      Handles snake_case, camelCase, kebab-case, SCREAMING_SNAKE, and mixed formats.
      Splits on non-alphanumeric characters and camelCase boundaries, then capitalizes
      each word and joins them.

      Args:
          name: Input name in any format.

      Returns:
          str: Name converted to PascalCase. Returns "Unknown" for empty input.
      """
      parts = re.split(r'[^a-zA-Z0-9]+', name)
      words = []
      for part in parts:
          if not part:
              continue
          sub_parts = re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_')
          for w in sub_parts:
              if w:
                  words.append(w[0].upper() + w[1:].lower() if len(w) > 1 else w.upper())
      return ''.join(words) if words else 'Unknown'

params_schema:
  {"params":[{"name":"name","desc":"String en cualquier formato de naming (snake_case, camelCase, kebab-case, etc.)"}],"output":"String convertido a PascalCase, o 'Unknown' si está vacío"}


id: tree_to_flat_list_py_core
name: tree_to_flat_list
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def tree_to_flat_list(structure: Any) -> list[dict]
description: Convierte arbol jerarquico a lista plana en orden DFS. Mantiene nodos internos con sus campos.
tags: ["tree","flatten","dfs","hierarchy"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 8320935b3492bca41e64acbb6de1dc68d9f5ea55058fd5186b577323cd197a18
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
example:
  ```python
  tree = {"title": "Root", "nodes": [{"title": "Child", "nodes": []}]}
  tree_to_flat_list(tree)
  # [{"title": "Root", "nodes": [...]}, {"title": "Child", "nodes": []}]
  ```

notes:
  Funcion pura. A diferencia de flatten_tree, mantiene los nodos internos con su campo 'nodes' intacto (referencias, no copias).

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"structure","desc":"Árbol jerárquico con estructura anidada"}],"output":"Lista plana de nodos en orden DFS (depth-first search)"}


id: unique_py_core
name: unique
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def unique(xs: list) -> list
description: Elimina duplicados de una lista preservando el orden de aparicion.
tags: ["list","functional","unique","deduplicate","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = unique([1, 2, 2, 3, 1, 4])
  # [1, 2, 3, 4]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Preserva el orden de la primera aparicion. Requiere elementos hashables.
content_hash: 6ec7be71df965029b4a4916c98998085b760fbd8ece0ba5e92e8bd3e9d4c2d4e
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Lista potencialmente con elementos duplicados"}],"output":"Lista sin duplicados preservando orden de primera aparición"}


id: validate_git_ssh_uri_py_core
name: validate_git_ssh_uri
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def validate_git_ssh_uri(url: str) -> None
description: Valida el formato de una URI SSH de git (git@host:path). Lanza ValueError si la URI es invalida.
tags: ["git","ssh","validation","uri"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["URI valida","URI sin git@","URI sin colon","URI con path vacio"]
test_file_path: python/functions/core/parse_git_url_test.py
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f99b598639bcd61c5cba94d89b1ac3b8a4b8d8a0306afec89c6652cc88e09c00
example:
  ```python
  validate_git_ssh_uri("git@github.com:org/repo.git")
  # None (sin excepcion)

  validate_git_ssh_uri("https://github.com/org/repo")
  # ValueError: git SSH URI must start with 'git@', got: 'https://github.com/org/repo'

  validate_git_ssh_uri("git@github.com/org/repo")
  # ValueError: git SSH URI must contain ':'

  validate_git_ssh_uri("git@github.com:")
  # ValueError: git SSH URI must have a non-empty path after ':'
  ```

notes:
  Funcion pura. Valida solo la estructura sintatica del formato SSH de git: prefijo `git@`, separador `:`, y path no vacio. No verifica que el host sea alcanzable ni que el repo exista. Usar antes de `subprocess.run(["git", "clone", uri])` para dar mensajes de error claros al usuario.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"url","desc":"String con URI SSH de git a validar (formato git@host:path)"}],"output":"None si la URI es válida, lanza ValueError si es inválida"}


id: validate_json_schema_py_core
name: validate_json_schema
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def validate_json_schema(data: dict, schema: dict) -> tuple[bool, list[str]]
tags: ["validation","schema","json-schema","pure","core"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
test_file_path: python/functions/core/validate_json_schema_test.py
file_path: python/functions/core/validate_json_schema.py
created_at: 2026-04-05T11:22:55Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: e635587635f73b2cbd5f9699243f9531a591e80ee644a9e3e75ceda40657f6df
description:
  Valida un dict contra un subset practico de JSON Schema (draft 2020-12) sin dependencias externas. Soporta type, required, properties, items, minItems/maxItems, minimum/maximum, exclusiveMinimum/exclusiveMaximum, minLength/maxLength, pattern, enum. Acumula todos los errores con paths descriptivos.

example:
  ```python
  schema = {
      "type": "object",
      "required": ["name", "age"],
      "properties": {
          "name": {"type": "string"},
          "age": {"type": "integer", "minimum": 0},
          "email": {"type": "string", "pattern": r"^[^@]+@[^@]+$"},
      },
  }

  valid, errors = validate_json_schema({"name": "Alice", "age": 30}, schema)
  # valid = True, errors = []

  valid, errors = validate_json_schema({"name": 123, "age": -1}, schema)
  # valid = False
  # errors = [
  #   "$.name: expected string, got integer",
  #   "$.age: -1 < minimum 0",
  #   "$.email: required field missing or empty",  (si email en required)
  # ]
  ```

tests:
  ["schema simple con types y required","nested object validation","array con items schema","pattern validation en strings","numeric ranges","multiples errores acumulados con paths correctos","schema vacio acepta todo","data none contra required field"]

notes:
  Funcion pura. Solo usa `re` de la stdlib. No cubre el spec completo de JSON Schema — esta disenada para validacion practica de metadata de entities y resultados de queries. El path usa notacion `$.field[i]` para facilitar debugging. Schema vacio acepta cualquier valor (comportamiento estandar JSON Schema).

code:
  """Validacion de dicts contra un subset de JSON Schema (draft 2020-12) sin dependencias externas."""

  import re


  def validate_json_schema(data: dict, schema: dict) -> tuple[bool, list[str]]:
      """Valida un dict contra un schema JSON Schema (draft 2020-12).

      Soporta: type, required, properties, items, minItems/maxItems,
      minimum/maximum, exclusiveMinimum/exclusiveMaximum,
      minLength/maxLength, pattern, enum.

      Args:
          data: Datos a validar.
          schema: Schema JSON Schema (draft 2020-12) como dict.

      Returns:
          (valid, errors) — True y lista vacia si cumple, False y lista de
          errores descriptivos si no. Cada error incluye el path (ej: "$.address.zip").
      """
      errors: list[str] = []
      _validate(data, schema, "$", errors)
      return (len(errors) == 0, errors)


  _TYPE_CHECKS = {
      "string": lambda v: isinstance(v, str),
      "number": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
      "integer": lambda v: isinstance(v, int) and not isinstance(v, bool),
      "boolean": lambda v: isinstance(v, bool),
      "array": lambda v: isinstance(v, list),
      "object": lambda v: isinstance(v, dict),
      "null": lambda v: v is None,
  }

  _TYPE_NAMES = {
      str: "string",
      int: "integer",
      float: "number",
      bool: "boolean",
      list: "array",
      dict: "object",
      type(None): "null",
  }


  def _type_name(v: object) -> str:
      return _TYPE_NAMES.get(type(v), type(v).__name__)


  def _validate(data: object, schema: dict, path: str, errors: list[str]) -> None:
      if not schema:
          return

      # --- type ---
      expected_type = schema.get("type")
      if expected_type is not None:
          checker = _TYPE_CHECKS.get(expected_type)
          if checker and not checker(data):
              errors.append(
                  f"{path}: expected {expected_type}, got {_type_name(data)}"
              )
              # no seguir validando si el tipo base ya fallo
              return

      # --- enum ---
      if "enum" in schema:
          if data not in schema["enum"]:
              errors.append(
                  f"{path}: value {data!r} not in enum {schema['enum']}"
              )

      # --- object ---
      if isinstance(data, dict):
          required = schema.get("required", [])
          for field in required:
              if field not in data or data[field] is None or data[field] == "":
                  errors.append(f"{path}.{field}: required field missing or empty")

          properties = schema.get("properties", {})
          for field, sub_schema in properties.items():
              if field in data:
                  _validate(data[field], sub_schema, f"{path}.{field}", errors)

      # --- array ---
      elif isinstance(data, list):
          min_items = schema.get("minItems")
          max_items = schema.get("maxItems")
          if min_items is not None and len(data) < min_items:
              errors.append(
                  f"{path}: array length {len(data)} < minItems {min_items}"
              )
          if max_items is not None and len(data) > max_items:
              errors.append(
                  f"{path}: array length {len(data)} > maxItems {max_items}"
              )

          items_schema = schema.get("items")
          if items_schema:
              for i, item in enumerate(data):
                  _validate(item, items_schema, f"{path}[{i}]", errors)

      # --- string ---
      elif isinstance(data, str):
          min_len = schema.get("minLength")
          max_len = schema.get("maxLength")
          if min_len is not None and len(data) < min_len:
              errors.append(
                  f"{path}: string length {len(data)} < minLength {min_len}"
              )
          if max_len is not None and len(data) > max_len:
              errors.append(
                  f"{path}: string length {len(data)} > maxLength {max_len}"
              )
          pattern = schema.get("pattern")
          if pattern is not None and not re.search(pattern, data):
              errors.append(
                  f"{path}: value {data!r} does not match pattern {pattern!r}"
              )

      # --- number / integer ---
      elif isinstance(data, (int, float)) and not isinstance(data, bool):
          minimum = schema.get("minimum")
          maximum = schema.get("maximum")
          ex_min = schema.get("exclusiveMinimum")
          ex_max = schema.get("exclusiveMaximum")
          if minimum is not None and data < minimum:
              errors.append(f"{path}: {data} < minimum {minimum}")
          if maximum is not None and data > maximum:
              errors.append(f"{path}: {data} > maximum {maximum}")
          if ex_min is not None and data <= ex_min:
              errors.append(f"{path}: {data} <= exclusiveMinimum {ex_min}")
          if ex_max is not None and data >= ex_max:
              errors.append(f"{path}: {data} >= exclusiveMaximum {ex_max}")

params_schema:
  {"params":[{"name":"data","desc":"Diccionario a validar contra el schema"},{"name":"schema","desc":"Esquema JSON Schema (subset practico)"}],"output":"Tupla (válido: bool, errores: list[str]) con lista de errores si hay"}


id: write_node_ids_py_core
name: write_node_ids
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def write_node_ids(data: Any, node_id: int = 0) -> int
tags: ["tree","id","hierarchy","mutation"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 388c388bd905a288d5e642563af3c2ac374e227043df2c9a4699888ce7cde9fc
source_repo: https://github.com/VectifyAI/PageIndex
source_license: MIT
source_file: pageindex/utils.py
description:
  Asigna IDs secuenciales zero-padded (0001, 0002...) a todos los nodos de un arbol. Retorna siguiente contador.

example:
  ```python
  tree = [{"title": "A", "nodes": [{"title": "B", "nodes": []}]}]
  next_id = write_node_ids(tree)
  # tree[0]["node_id"] == "0000", tree[0]["nodes"][0]["node_id"] == "0001"
  # next_id == 2
  ```

notes:
  Funcion pura en el sentido de no tener I/O, pero muta el arbol de entrada in-place. IDs son strings de 4 digitos.

code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"data","desc":"Estructura árbol a la que añadir IDs"},{"name":"node_id","desc":"Contador inicial para los IDs (default: 0)"}],"output":"Próximo contador a usar después de asignar IDs"}


id: zip_with_py_core
name: zip_with
kind: function
lang: py
domain: core
version: 1.0.0
purity: pure
signature: def zip_with(xs: list, ys: list, fn: callable) -> list
description: Combina dos listas elemento a elemento con una funcion. Se detiene en la mas corta.
tags: ["list","functional","zip","combine","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  result = zip_with([1, 2, 3], [10, 20, 30], lambda a, b: a + b)
  # [11, 22, 33]
  ```

tests: []
file_path: python/functions/core/core.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura. Se detiene en la lista mas corta, como zip() de Python.
content_hash: de36b74fef535c15ad34e9feacd033853f630e46bcb33ec03ff38b9d384fce2b
code:
[cell repeats an earlier one in this file: 25395 characters]

params_schema:
  {"params":[{"name":"xs","desc":"Primera lista a combinar"},{"name":"ys","desc":"Segunda lista a combinar"},{"name":"fn","desc":"Función que combina elementos: fn(x, y)"}],"output":"Lista de resultados combinados, longitud igual a la lista más corta"}


id: detect_sql_injection_py_cybersecurity
name: detect_sql_injection
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def detect_sql_injection(input_str: str) -> tuple
tags: ["sqli","injection","detection","security","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 559cca8d964dbd534a83887182b5042d3bee818815bfc6b7f139131de4177e49
description:
  Detecta patrones de SQL injection en un string. Retorna (is_threat, pattern) con el nombre del patron detectado.

example:
  ```python
  detect_sql_injection("' OR '1'='1")
  # (True, "string_tautology")

  detect_sql_injection("; DROP TABLE users")
  # (True, "stacked_query")

  detect_sql_injection("hello world")
  # (False, "")
  ```

notes:
  Detecta 10 patrones: sql_keyword, tautology, stacked_query, comment_injection, string_tautology, union_select, hex_literal, char_function, concat_function, time_based. No reemplaza un WAF pero es util para logging y alertas tempranas.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"input_str","desc":"string a analizar en busca de patrones de SQL injection"}],"output":"tupla (is_threat, pattern) indicando si se detectó amenaza y el nombre del patrón"}


id: entropy_shannon_py_cybersecurity
name: entropy_shannon
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def entropy_shannon(data: bytes) -> float
tags: ["entropy","shannon","analysis","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["math","collections"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f74d9a99d61bd29572f136d680d8971d4e964779c382c7174899957fd1f381df
description:
  Calcula la entropia de Shannon de datos binarios (0-8 bits por byte). Util para detectar datos cifrados o comprimidos.

example:
  ```python
  # Datos aleatorios (alta entropia)
  entropy_shannon(bytes(range(256)))
  # ~8.0

  # Datos repetitivos (baja entropia)
  entropy_shannon(b"aaaaaaaaaa")
  # 0.0

  # Texto normal
  entropy_shannon(b"hello world")
  # ~2.84
  ```

notes:
  Entropia alta (>7.5) sugiere datos cifrados o comprimidos. Entropia baja (<3) sugiere datos estructurados o repetitivos. Retorna 0.0 para datos vacios.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"data","desc":"bytes cuya entropia de Shannon se desea calcular"}],"output":"valor float de entropia entre 0 y 8 bits por byte"}


id: envelope_decrypt_py_cybersecurity
name: envelope_decrypt
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: impure
signature: def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes
uses_functions: ["envelope_encrypt_py_cybersecurity"]
uses_types: []
returns: []
error_type: error_go_core
imports: ["cryptography","struct"]
test_file_path: python/functions/cybersecurity/envelope_encrypt_test.py
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: d7dc98b639867b817cdbac801b6a3b3dd601652fcec298478c9552f5d64301df
description:
  Descifra datos cifrados con envelope_encrypt. Si los datos no comienzan con el magic b'OVE1', los retorna sin modificar (passthrough). Soporta archivos que pueden o no estar cifrados sin necesidad de chequeo previo.

tags:
  ["decryption","aes","gcm","envelope-encryption","dek","kek","cryptography","cybersecurity","passthrough"]

example:
  ```python
  import secrets
  from cybersecurity import envelope_encrypt, envelope_decrypt

  master_key = secrets.token_bytes(32)

  # Caso 1: descifrar datos cifrados
  ciphertext = envelope_encrypt(b"datos secretos", master_key)
  plaintext = envelope_decrypt(ciphertext, master_key)
  # plaintext == b"datos secretos"

  # Caso 2: passthrough — datos no cifrados
  raw = b"archivo en plano"
  result = envelope_decrypt(raw, master_key)
  # result == b"archivo en plano"  (sin modificar)

  # Caso 3: key incorrecta — lanza InvalidTag
  wrong_key = secrets.token_bytes(32)
  # envelope_decrypt(ciphertext, wrong_key)  → cryptography.exceptions.InvalidTag
  ```

tests:
  ["decrypt de datos cifrados","decrypt de datos no cifrados passthrough","key incorrecta","envelope truncado","magic invalido"]

notes:
  Implementacion original inspirada en OpenViking `openviking/crypto/encryptor.py` (AGPL-3.0). Reimplementada desde cero.

  - **Passthrough**: si `ciphertext` no empieza con `b"OVE1"`, se retorna sin modificar. Permite usar la funcion indistintamente en archivos cifrados y no cifrados.
  - **Autenticacion GCM**: si la master_key es incorrecta o los datos fueron manipulados, `cryptography.exceptions.InvalidTag` es lanzado por la capa GCM — nunca se retorna texto corrupto.
  - **ValueError**: lanzado si el envelope tiene magic correcto pero estructura invalida (truncado o version no soportada).
  - `master_key` debe ser de exactamente 32 bytes para AES-256.
  - Requiere `cryptography` instalado: `uv add cryptography`.

code:
  """Cybersecurity pure functions: hashing, parsing, and security utilities."""

  import hashlib
  import math
  import re
  import base64
  import secrets
  import struct
  from collections import Counter
  from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
  from cryptography.hazmat.primitives.ciphers.aead import AESGCM


  def hash_sha256(data: bytes) -> str:
      """Calcula el hash SHA-256 de datos binarios. Retorna hex digest."""
      return hashlib.sha256(data).hexdigest()


  def hash_md5(data: bytes) -> str:
      """Calcula el hash MD5 de datos binarios. Retorna hex digest."""
      return hashlib.md5(data).hexdigest()


  def entropy_shannon(data: bytes) -> float:
      """Calcula la entropia de Shannon de datos binarios (0-8 bits por byte).

      Entropia alta (>7.5) sugiere datos cifrados o comprimidos.
      Entropia baja (<3) sugiere datos estructurados o repetitivos.
      """
      if not data:
          return 0.0
      length = len(data)
      counts = Counter(data)
      entropy = 0.0
      for count in counts.values():
          p = count / length
          if p > 0:
              entropy -= p * math.log2(p)
      return entropy


  _SQL_INJECTION_PATTERNS = [
      (r"('\s*OR\s+'[^']*'\s*=\s*'[^']*'?)", "string_tautology"),
      (r"('\s*(OR|AND)\s+'?\d+\s*=\s*\d+)", "tautology"),
      (r"(;\s*(DROP|DELETE|UPDATE|INSERT)\b)", "stacked_query"),
      (r"(UNION\s+(ALL\s+)?SELECT)", "union_select"),
      (r"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\b\s)", "sql_keyword"),
      (r"(--\s*$|/\*|\*/)", "comment_injection"),
      (r"(BENCHMARK\s*\(|SLEEP\s*\(|WAITFOR\s+DELAY)", "time_based"),
      (r"(CHAR\s*\(\s*\d+)", "char_function"),
      (r"(CONCAT\s*\()", "concat_function"),
      (r"(0x[0-9a-fA-F]{4,})", "hex_literal"),
  ]


  def detect_sql_injection(input_str: str) -> tuple:
      """Detecta patrones de SQL injection en un string.

      Retorna (is_threat, pattern) donde pattern es el nombre del patron
      detectado o cadena vacia si no hay amenaza.
      """
      for pattern, name in _SQL_INJECTION_PATTERNS:
          if re.search(pattern, input_str, re.IGNORECASE):
              return (True, name)
      return (False, "")


  _URL_REGEX = re.compile(
      r"https?://[^\s<>\"'\)\]]+",
      re.IGNORECASE,
  )


  def extract_urls(text: str) -> list:
      """Extrae todas las URLs (http/https) de un texto."""
      return _URL_REGEX.findall(text)


  def is_base64(s: str) -> bool:
      """Verifica si un string es base64 valido.

      Acepta base64 estandar y URL-safe. Requiere al menos 4 caracteres.
      """
      if len(s) < 4:
          return False
      b64_pattern = re.compile(r"^[A-Za-z0-9+/\-_]*={0,2}$")
      if not b64_pattern.match(s):
          return False
      try:
          decoded = base64.b64decode(s, validate=True)
          return len(decoded) > 0
      except Exception:
          try:
              decoded = base64.urlsafe_b64decode(s)
              return len(decoded) > 0
          except Exception:
              return False


  def is_hex(s: str) -> bool:
      """Verifica si un string es hexadecimal valido.

      Acepta con o sin prefijo 0x. Requiere al menos 2 caracteres (sin prefijo).
      """
      clean = s.strip()
      if clean.startswith(("0x", "0X")):
          clean = clean[2:]
      if len(clean) < 2:
          return False
      return bool(re.fullmatch(r"[0-9a-fA-F]+", clean))


  def levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de Levenshtein (edit distance) entre dos strings.

      Util para deteccion de typosquatting en dominios y fuzzy matching.
      """
      if len(a) < len(b):
          return levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)

      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(
                      curr_row[j] + 1,       # insert
                      prev_row[j + 1] + 1,   # delete
                      prev_row[j] + cost,     # substitute
                  )
              )
          prev_row = curr_row
      return prev_row[-1]


  def jaccard_similarity(a: list, b: list) -> float:
      """Calcula el coeficiente de similitud de Jaccard entre dos listas.

      J(A,B) = |A interseccion B| / |A union B|. Retorna 0.0 si ambas vacias.
      Util para comparar conjuntos de tokens, features, o IoCs.
      """
      set_a = set(a)
      set_b = set(b)
      if not set_a and not set_b:
          return 0.0
      intersection = set_a & set_b
      union = set_a | set_b
      return len(intersection) / len(union)


  def normalize_url(raw_url: str) -> str:
      """Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros.

      Util para deduplicacion de URLs y comparacion de IoCs.
      """
      parsed = urlparse(raw_url)
      scheme = parsed.scheme.lower() or "http"
      netloc = parsed.netloc.lower()
      path = parsed.path or "/"
      # Remove trailing slash except for root
      if path != "/" and path.endswith("/"):
          path = path.rstrip("/")
      # Sort query parameters
      params = parse_qs(parsed.query, keep_blank_values=True)
      sorted_query = urlencode(sorted(params.items()), doseq=True)
      # Drop fragment
      return urlunparse((scheme, netloc, path, parsed.params, sorted_query, ""))


  # --- Envelope Encryption (AES-256-GCM) ---

  _ENVELOPE_MAGIC = b"OVE1"
  _ENVELOPE_VERSION = 0x01
  _HEADER_SIZE = 12  # magic(4) + version(1) + reserved(1) + efk_len(2) + kiv_len(2) + div_len(2)


  def _build_envelope(
      encrypted_file_key: bytes,
      key_iv: bytes,
      data_iv: bytes,
      encrypted_content: bytes,
  ) -> bytes:
      """Construye el formato binario del envelope (helper puro interno).

      Header (12 bytes):
          Magic    (4B): b"OVE1"
          Version  (1B): 0x01
          Reserved (1B): 0x00
          EFK_len  (2B): longitud de encrypted_file_key (big-endian)
          KIV_len  (2B): longitud de key_iv (big-endian)
          DIV_len  (2B): longitud de data_iv (big-endian)
      Seguido de: encrypted_file_key + key_iv + data_iv + encrypted_content
      """
      header = (
          _ENVELOPE_MAGIC
          + struct.pack(">BBHHH", _ENVELOPE_VERSION, 0x00,
                        len(encrypted_file_key), len(key_iv), len(data_iv))
      )
      return header + encrypted_file_key + key_iv + data_iv + encrypted_content


  def _parse_envelope(ciphertext: bytes) -> tuple:
      """Parsea el envelope binario y retorna sus componentes (helper puro interno).

      Returns:
          (encrypted_file_key, key_iv, data_iv, encrypted_content)

      Raises:
          ValueError: si el envelope esta truncado o la version no es soportada.
      """
      if len(ciphertext) < _HEADER_SIZE:
          raise ValueError(
              f"Envelope truncado: se esperaban al menos {_HEADER_SIZE} bytes, "
              f"se recibieron {len(ciphertext)}"
          )

      magic = ciphertext[:4]
      if magic != _ENVELOPE_MAGIC:
          raise ValueError(f"Magic invalido: se esperaba {_ENVELOPE_MAGIC!r}, se obtuvo {magic!r}")

      version, _reserved, efk_len, kiv_len, div_len = struct.unpack(">BBHHH", ciphertext[4:12])

      if version != _ENVELOPE_VERSION:
          raise ValueError(f"Version de envelope no soportada: {version}")

      offset = _HEADER_SIZE
      encrypted_file_key = ciphertext[offset : offset + efk_len]
      offset += efk_len
      key_iv = ciphertext[offset : offset + kiv_len]
      offset += kiv_len
      data_iv = ciphertext[offset : offset + div_len]
      offset += div_len
      encrypted_content = ciphertext[offset:]

      if (
          len(encrypted_file_key) != efk_len
          or len(key_iv) != kiv_len
          or len(data_iv) != div_len
      ):
          raise ValueError("Envelope truncado: longitudes declaradas exceden los datos disponibles")

      return encrypted_file_key, key_iv, data_iv, encrypted_content


  def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes:
      """Cifra datos usando patron Envelope Encryption con AES-256-GCM.

      Genera una file key aleatoria de 32 bytes, cifra los datos con ella,
      luego cifra la file key con la master_key. El resultado es un envelope
      binario que contiene todo lo necesario para descifrar con la master_key.

      Args:
          plaintext: Datos a cifrar (puede ser vacio).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Envelope binario cifrado.

      Raises:
          Exception: Si ocurre un error en el cifrado (clave de longitud incorrecta, etc.).
      """
      # 1. Generar file_key aleatoria (DEK: Data Encryption Key)
      file_key = secrets.token_bytes(32)

      # 2. Cifrar contenido con la file_key
      data_iv = secrets.token_bytes(12)
      aesgcm_data = AESGCM(file_key)
      encrypted_content = aesgcm_data.encrypt(data_iv, plaintext, None)

      # 3. Cifrar file_key con la master_key (KEK: Key Encryption Key)
      key_iv = secrets.token_bytes(12)
      aesgcm_key = AESGCM(master_key)
      encrypted_file_key = aesgcm_key.encrypt(key_iv, file_key, None)

      # 4. Construir envelope
      return _build_envelope(encrypted_file_key, key_iv, data_iv, encrypted_content)


  def envelope_decrypt(ciphertext: bytes, master_key: bytes) -> bytes:
      """Descifra datos cifrados con envelope_encrypt.

      Si los datos no empiezan con el magic b"OVE1", se asume que no estan
      cifrados y se retornan tal cual (comportamiento passthrough). Esto
      permite usar la funcion en archivos que pueden o no estar cifrados.

      Args:
          ciphertext: Envelope cifrado (o datos en plano si no tienen magic).
          master_key: Clave maestra de 32 bytes (AES-256).

      Returns:
          Datos descifrados, o ciphertext sin modificar si no tiene magic.

      Raises:
          ValueError: Si el envelope esta corrupto o truncado.
          cryptography.exceptions.InvalidTag: Si la master_key es incorrecta
              o los datos fueron manipulados (falla de autenticacion GCM).
      """
      # Passthrough: si no comienza con magic, asumir que no esta cifrado
      if not ciphertext.startswith(_ENVELOPE_MAGIC):
          return ciphertext

      # Parsear envelope
      encrypted_file_key, key_iv, data_iv, encrypted_content = _parse_envelope(ciphertext)

      # Descifrar file_key con master_key
      aesgcm_key = AESGCM(master_key)
      file_key = aesgcm_key.decrypt(key_iv, encrypted_file_key, None)

      # Descifrar contenido con file_key
      aesgcm_data = AESGCM(file_key)
      return aesgcm_data.decrypt(data_iv, encrypted_content, None)

params_schema:
  {"params":[{"name":"ciphertext","desc":"datos cifrados con envelope_encrypt o datos sin cifrar (passthrough si no comienzan con magic OVE1)"},{"name":"master_key","desc":"clave maestra KEK de 32 bytes para descifrar la file key"}],"output":"bytes desencriptados, o bytes sin modificar si el magic no es OVE1"}


id: envelope_encrypt_py_cybersecurity
name: envelope_encrypt
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: impure
signature: def envelope_encrypt(plaintext: bytes, master_key: bytes) -> bytes
tags: ["encryption","aes","gcm","envelope-encryption","dek","kek","cryptography","cybersecurity"]
uses_functions: []
uses_types: []
returns: []
error_type: error_go_core
imports: ["cryptography","secrets","struct"]
test_file_path: python/functions/cybersecurity/envelope_encrypt_test.py
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 7e61accc062c67b606ea417b018a6c2d43a70e6c8c131d3504526469f60e84a3
description:
  Cifra datos usando patron Envelope Encryption con AES-256-GCM. Genera una file key aleatoria (DEK), cifra los datos con ella, luego cifra la file key con la master_key (KEK). Retorna un envelope binario con magic b'OVE1'.

example:
  ```python
  import secrets
  from cybersecurity import envelope_encrypt, envelope_decrypt

  master_key = secrets.token_bytes(32)  # 256-bit KEK
  plaintext = b"datos confidenciales"

  ciphertext = envelope_encrypt(plaintext, master_key)
  # ciphertext[:4] == b"OVE1"

  recovered = envelope_decrypt(ciphertext, master_key)
  # recovered == plaintext
  ```

tests:
  ["encrypt → decrypt roundtrip","datos vacios","datos grandes","ciphertext tiene magic correcto","ciphertext es distinto cada vez"]

notes:
  Implementacion original inspirada en OpenViking `openviking/crypto/encryptor.py` (AGPL-3.0). Reimplementada desde cero.

  - La file key (DEK) es de 32 bytes generados con `secrets.token_bytes` (CSPRNG).
  - Tanto el cifrado de datos como el de la file key usan AES-256-GCM con IVs de 12 bytes.
  - El GCM auth tag (16 bytes) garantiza autenticidad e integridad.
  - `master_key` debe ser de exactamente 32 bytes para AES-256.
  - Requiere `cryptography` instalado: `uv add cryptography`.

documentation:
  ## Formato del envelope

  ```
  Magic    (4B): b"OVE1"       identificador de formato
  Version  (1B): 0x01          version del protocolo
  Reserved (1B): 0x00          reservado para uso futuro
  EFK_len  (2B): big-endian    longitud de encrypted_file_key
  KIV_len  (2B): big-endian    longitud de key_iv
  DIV_len  (2B): big-endian    longitud de data_iv
  --- header: 12 bytes total ---
  Encrypted File Key (variable, incluye GCM auth tag de 16B)
  Key IV             (12B)
  Data IV            (12B)
  Encrypted Content  (variable, incluye GCM auth tag de 16B)
  ```

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"plaintext","desc":"datos a cifrar con AES-256-GCM"},{"name":"master_key","desc":"clave maestra KEK de 32 bytes para cifrar la file key"}],"output":"envelope binario con magic b'OVE1' conteniendo datos cifrados"}


id: extract_urls_py_cybersecurity
name: extract_urls
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def extract_urls(text: str) -> list
description: Extrae todas las URLs (http/https) de un texto. Util para analisis de IoCs y threat intelligence.
tags: ["url","extract","parsing","ioc","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: f67c7bf3f5e420dfe573229d1f7071387209520ca94456c451c75651c7444939
example:
  ```python
  extract_urls("Visit https://example.com and http://test.org/path?q=1")
  # ["https://example.com", "http://test.org/path?q=1"]

  extract_urls("no urls here")
  # []
  ```

notes:
  Usa regex para extraer URLs con esquema http/https. No valida que las URLs sean alcanzables. Util para extraer indicadores de compromiso (IoCs) de logs, emails o reportes de threat intelligence.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"text","desc":"string de texto del cual extraer URLs HTTP/HTTPS"}],"output":"lista de strings con todas las URLs encontradas"}


id: hash_md5_py_cybersecurity
name: hash_md5
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def hash_md5(data: bytes) -> str
description: Calcula el hash MD5 de datos binarios. Retorna hex digest.
tags: ["hash","md5","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["hashlib"]
example:
  ```python
  h = hash_md5(b"hello")
  # "5d41402abc4b2a76b9719d911017c592"
  ```

tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 63462af1382995eef2107d0865249b487cda23e7d6c51de3e2c125d5c8927187
notes:
  Usa hashlib de stdlib. MD5 no es seguro para propositos criptograficos pero es util para checksums, fingerprinting de archivos e identificacion rapida de IoCs.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"data","desc":"bytes a ser hasheado"}],"output":"string hexadecimal del hash MD5 (32 caracteres)"}


id: hash_sha256_py_cybersecurity
name: hash_sha256
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def hash_sha256(data: bytes) -> str
description: Calcula el hash SHA-256 de datos binarios. Retorna hex digest.
tags: ["hash","sha256","crypto","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["hashlib"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ceeb9f43a5181fcd5d6e2f6d2ba095589625439bdb91f949638bbf29cb56009b
example:
  ```python
  h = hash_sha256(b"hello")
  # "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
  ```

notes:
  Usa hashlib de stdlib. Funcion pura sin side effects. SHA-256 produce un digest de 64 caracteres hexadecimales (256 bits).

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"data","desc":"bytes a ser hasheado"}],"output":"string hexadecimal del hash SHA-256 (64 caracteres)"}


id: is_base64_py_cybersecurity
name: is_base64
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def is_base64(s: str) -> bool
tags: ["base64","validation","encoding","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re","base64"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ef44b434fe7e4394057ada22677d572da9ff67134609776c4eab600d1136ad61
description:
  Verifica si un string es base64 valido. Acepta base64 estandar y URL-safe. Requiere minimo 4 caracteres.

example:
  ```python
  is_base64("aGVsbG8=")
  # True

  is_base64("not!valid")
  # False

  is_base64("ab")
  # False (menos de 4 caracteres)
  ```

notes:
  Verifica tanto el formato (regex) como que el decode sea exitoso. Util para detectar datos codificados en payloads sospechosos, headers HTTP o parametros de URL.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"s","desc":"string a validar como base64"}],"output":"booleano indicando si el string es base64 valido (minimo 4 caracteres)"}


id: is_hex_py_cybersecurity
name: is_hex
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def is_hex(s: str) -> bool
tags: ["hex","validation","encoding","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["re"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 046c08bf7df38b84716681c2c5fe9c33591575c6923dc1e64e9df29b1350cd6a
description:
  Verifica si un string es hexadecimal valido. Acepta con o sin prefijo 0x. Requiere minimo 2 caracteres.

example:
  ```python
  is_hex("4a6f686e")
  # True

  is_hex("0x4a6f686e")
  # True

  is_hex("xyz")
  # False

  is_hex("a")
  # False (menos de 2 caracteres)
  ```

notes:
  Util para validar hashes, direcciones de memoria, shellcode y otros datos hexadecimales en contexto de seguridad.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"s","desc":"string a validar como hexadecimal"}],"output":"booleano indicando si el string es hexadecimal valido con o sin prefijo 0x (minimo 2 caracteres)"}


id: jaccard_similarity_py_cybersecurity
name: jaccard_similarity
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def jaccard_similarity(a: list, b: list) -> float
tags: ["jaccard","similarity","comparison","sets","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 3fbb86722a904baaea67e84a58e079b1974569b05c334ed67f872d34e38329c8
description:
  Calcula el coeficiente de similitud de Jaccard entre dos listas. J(A,B) = |A interseccion B| / |A union B|.

example:
  ```python
  jaccard_similarity(["a", "b", "c"], ["b", "c", "d"])
  # 0.5

  jaccard_similarity(["a", "b"], ["a", "b"])
  # 1.0

  jaccard_similarity([], [])
  # 0.0
  ```

notes:
  Convierte las listas a sets internamente. Retorna 0.0 si ambas listas son vacias. Util para comparar conjuntos de tokens, features de malware, IoCs compartidos entre muestras, o tags de vulnerabilidades.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"a","desc":"primera lista para comparacion"},{"name":"b","desc":"segunda lista para comparacion"}],"output":"coeficiente float de similitud de Jaccard entre 0.0 y 1.0"}


id: levenshtein_distance_py_cybersecurity
name: levenshtein_distance
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def levenshtein_distance(a: str, b: str) -> int
tags: ["levenshtein","distance","fuzzy","typosquatting","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 2e6ecac9eba12e49b1b70ffbd1f30d865f7119e1a5769764cf3b559881d73226
description:
  Calcula la distancia de Levenshtein (edit distance) entre dos strings. Util para deteccion de typosquatting en dominios.

example:
  ```python
  levenshtein_distance("google.com", "gooogle.com")
  # 1

  levenshtein_distance("paypal.com", "paypa1.com")
  # 1

  levenshtein_distance("abc", "abc")
  # 0
  ```

notes:
  Implementacion O(n*m) con optimizacion de espacio (dos filas). Sin dependencias externas. Util para detectar dominios de typosquatting comparando contra dominios legitimos conocidos.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"a","desc":"primer string para comparacion de distancia de edicion"},{"name":"b","desc":"segundo string para comparacion de distancia de edicion"}],"output":"entero con la distancia de Levenshtein (minimo numero de ediciones)"}


id: normalize_url_py_cybersecurity
name: normalize_url
kind: function
lang: py
domain: cybersecurity
version: 1.0.0
purity: pure
signature: def normalize_url(raw_url: str) -> str
tags: ["url","normalize","ioc","dedup","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["urllib.parse"]
tests: []
file_path: python/functions/cybersecurity/cybersecurity.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 99a36ba484bcf6398fc8f6e800cf79d6944cac72f186181d8fb1acf55e990ece
description:
  Normaliza una URL: lowercase del host, elimina fragmentos, ordena parametros. Util para deduplicacion de IoCs.

example:
  ```python
  normalize_url("HTTPS://Example.COM/path?b=2&a=1#frag")
  # "https://example.com/path?a=1&b=2"

  normalize_url("http://test.org/path/")
  # "http://test.org/path"
  ```

notes:
  Operaciones de normalizacion: lowercase de scheme y host, eliminacion de trailing slash (excepto root), ordenamiento alfabetico de query parameters, eliminacion de fragmentos. Usa urllib.parse de stdlib.

code:
[cell repeats an earlier one in this file: 10336 characters]

params_schema:
  {"params":[{"name":"raw_url","desc":"URL en formato string a normalizar"}],"output":"URL normalizada con host en minusculas, fragmentos eliminados y parametros ordenados"}


id: aggregate_by_group_py_datascience
name: aggregate_by_group
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def aggregate_by_group(rows: list[dict], group_by: list[str], aggs: dict[str, str]) -> list[dict]
tags: ["datascience","tabular","groupby","aggregate","transform","python"]
uses_functions: []
uses_types: []
returns: []
imports: ["collections"]
test_file_path: python/functions/datascience/aggregate_by_group_test.py
file_path: python/functions/datascience/aggregate_by_group.py
created_at: 2026-04-05T11:25:56Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: ef3fece0affeb2b2c57afdb5771f4cb130e89da3f0306a1e2297ea0f94a35376
description:
  GROUP BY + agregaciones sobre datos tabulares. aggs es un dict de columna a funcion (sum, mean, count, min, max, first, last, collect). collect acumula valores en lista. None se ignora en agregaciones numericas.

example:
  ```python
  rows = [
      {"dept": "eng", "salary": 100},
      {"dept": "eng", "salary": 120},
      {"dept": "sales", "salary": 80},
  ]
  aggregate_by_group(rows, group_by=["dept"], aggs={"salary": "mean"})
  # [{"dept": "eng", "salary": 110.0}, {"dept": "sales", "salary": 80.0}]
  ```

tests:
  ["Group by una columna con sum","Group by multiples columnas","Agregacion mean count min max","collect acumula en lista","Grupo con una sola fila","Campo con None se ignora en agregaciones numericas"]

notes:
  Funcion pura sin dependencias externas (solo collections.defaultdict de stdlib).
  Preserva el orden de primera aparicion de cada grupo.
  La funcion 'collect' no filtra None — acumula todos los valores incluyendo None.

code:
  """GROUP BY + agregaciones sobre datos tabulares list[dict]."""

  from collections import defaultdict


  def aggregate_by_group(
      rows: list[dict],
      group_by: list[str],
      aggs: dict[str, str],
  ) -> list[dict]:
      """Agrupa filas por una o varias columnas y aplica agregaciones.

      Equivalente a SQL GROUP BY con funciones de agregacion.
      La funcion 'collect' acumula todos los valores en una lista.
      Los valores None se ignoran en agregaciones numericas (sum, mean, min, max).

      Args:
          rows: Lista de dicts con los datos.
          group_by: Lista de columnas por las que agrupar.
          aggs: Dict de {columna: funcion}. Funciones: sum, mean, count,
                min, max, first, last, collect.

      Returns:
          Lista de dicts con las columnas de group_by mas los campos agregados.
          El orden de las filas sigue el orden de primera aparicion del grupo.
      """
      # Mantener orden de grupos con lista de claves
      group_keys: list[tuple] = []
      seen_groups: set[tuple] = set()
      buckets: dict[tuple, dict[str, list]] = defaultdict(lambda: defaultdict(list))

      for row in rows:
          gk = tuple(row.get(col) for col in group_by)
          if gk not in seen_groups:
              seen_groups.add(gk)
              group_keys.append(gk)
          for col in aggs:
              val = row.get(col)
              buckets[gk][col].append(val)

      def _aggregate(vals: list, func: str):
          if func == "collect":
              return vals
          if func == "count":
              return len(vals)
          if func == "first":
              return vals[0] if vals else None
          if func == "last":
              return vals[-1] if vals else None
          # Para sum, mean, min, max: ignorar None
          numeric = [v for v in vals if v is not None]
          if not numeric:
              return None
          if func == "sum":
              return sum(numeric)
          if func == "mean":
              return sum(numeric) / len(numeric)
          if func == "min":
              return min(numeric)
          if func == "max":
              return max(numeric)
          raise ValueError(f"Funcion de agregacion no soportada: {func}")

      result = []
      for gk in group_keys:
          record: dict = dict(zip(group_by, gk))
          for col, func in aggs.items():
              record[col] = _aggregate(buckets[gk][col], func)
          result.append(record)

      return result

params_schema:
  {"params":[{"name":"rows","desc":"lista de dicts donde cada dict representa una fila (ej: [{'dept': 'eng', 'salary': 100}, ...])"},{"name":"group_by","desc":"lista de nombres de columnas para agrupar (ej: ['dept']). Pueden ser multiples columnas."},{"name":"aggs","desc":"dict de columna -\u003e funcion de agregacion (ej: {'salary': 'mean'}). Soporta: sum, mean, count, min, max, first, last, collect"}],"output":"lista de dicts donde cada dict es un grupo con sus resultados de agregacion"}


id: autocorrelation_py_datascience
name: autocorrelation
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def autocorrelation(data: list, lag: int) -> float
description: Calcula la autocorrelacion de una serie temporal para un lag dado.
tags: ["statistics","timeseries","correlation","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  autocorrelation([1, 2, 3, 4, 5, 4, 3, 2, 1], 1)
  # ~0.489
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Autocorrelacion normalizada por la varianza. Retorna 0.0 si lag es invalido o la varianza es cero.
content_hash: b124fbfed9d8738af39c456cd34ac2675763020628dea39499b56fee49a4d4e5
code:
  """Pure datascience utilities — statistics and numerical functions.

  Uses only math stdlib. No external dependencies.
  """

  import math


  def pearson(xs: list, ys: list) -> float:
      """Pearson correlation coefficient between two lists of floats."""
      n = len(xs)
      if n != len(ys) or n == 0:
          return 0.0
      mean_x = sum(xs) / n
      mean_y = sum(ys) / n
      num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
      den_x = math.sqrt(sum((x - mean_x) ** 2 for x in xs))
      den_y = math.sqrt(sum((y - mean_y) ** 2 for y in ys))
      if den_x == 0.0 or den_y == 0.0:
          return 0.0
      return num / (den_x * den_y)


  def standardize(data: list) -> list:
      """Z-score standardization (mean=0, std=1)."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [0.0] * n
      return [(x - mean) / std for x in data]


  def min_max_scale(data: list) -> list:
      """Scale values to [0, 1] range."""
      if not data:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          return [0.0] * len(data)
      return [(x - lo) / (hi - lo) for x in data]


  def clip(data: list, lo: float, hi: float) -> list:
      """Clip values to [lo, hi]."""
      return [max(lo, min(hi, x)) for x in data]


  def detect_outliers(data: list, threshold: float) -> list:
      """Returns list of bools, True where |z-score| > threshold."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [False] * n
      return [abs((x - mean) / std) > threshold for x in data]


  def impute(data: list) -> list:
      """Replace None/NaN with mean of non-null values."""
      valid = [x for x in data if x is not None and not (isinstance(x, float) and math.isnan(x))]
      if not valid:
          return [0.0] * len(data)
      mean = sum(valid) / len(valid)
      return [
          mean if (x is None or (isinstance(x, float) and math.isnan(x))) else x
          for x in data
      ]


  def histogram(data: list, buckets: int) -> list:
      """Returns list of counts per bucket."""
      if not data or buckets <= 0:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          counts = [0] * buckets
          counts[0] = len(data)
          return counts
      width = (hi - lo) / buckets
      counts = [0] * buckets
      for x in data:
          idx = int((x - lo) / width)
          if idx >= buckets:
              idx = buckets - 1
          counts[idx] += 1
      return counts


  def rolling_window(xs: list, size: int) -> list:
      """Returns list of sublists (sliding windows of given size)."""
      if size <= 0 or size > len(xs):
          return []
      return [xs[i : i + size] for i in range(len(xs) - size + 1)]


  def autocorrelation(data: list, lag: int) -> float:
      """Autocorrelation at given lag."""
      n = len(data)
      if lag < 0 or lag >= n or n == 0:
          return 0.0
      mean = sum(data) / n
      var = sum((x - mean) ** 2 for x in data) / n
      if var == 0.0:
          return 0.0
      cov = sum((data[i] - mean) * (data[i + lag] - mean) for i in range(n - lag)) / n
      return cov / var


  def linspace(start: float, stop: float, num: int) -> list:
      """Generate evenly spaced values from start to stop (inclusive)."""
      if num <= 0:
          return []
      if num == 1:
          return [start]
      step = (stop - start) / (num - 1)
      return [start + i * step for i in range(num)]


  def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict:
      """Estima parámetros de un proceso Hawkes desde autocorrelación de arrivals.

      Ajusta exponencial a*exp(-b*lag) sobre la ACF.
      Retorna dict con alpha, beta, branching_ratio, acf.
      """
      import numpy as np
      from scipy.optimize import curve_fit

      arr = np.array(arrivals, dtype=float)
      mean_a = np.mean(arr)
      var_a = np.var(arr)
      if var_a == 0:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': [1.0]}

      acf = [1.0] + [
          float(np.mean((arr[lag:] - mean_a) * (arr[:-lag] - mean_a)) / var_a)
          for lag in range(1, max_lag)
      ]

      lags = np.arange(1, max_lag)
      acf_vals = np.array(acf[1:])

      if acf_vals[0] <= 0.01:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': acf}

      exp_decay = lambda x, a, b: a * np.exp(-b * x)
      try:
          popt, _ = curve_fit(exp_decay, lags, acf_vals, p0=[0.5, 0.5], maxfev=5000)
          alpha_est, beta_est = abs(popt[0]), abs(popt[1])
      except RuntimeError:
          alpha_est, beta_est = 0.0, 1.0

      branching = alpha_est / beta_est if beta_est > 0 else 0.0
      return {
          'alpha': round(alpha_est, 4),
          'beta': round(beta_est, 4),
          'branching_ratio': round(branching, 4),
          'acf': acf,
      }


  def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict:
      """Estima el exponente alpha de una distribución Pareto via MLE.

      α = n / Σ ln(xi / x_min) donde x_min es el percentil indicado.
      Alpha bajo = cola más pesada = más valores extremos.
      """
      import numpy as np

      arr = np.array([v for v in values if v > 0], dtype=float)
      if len(arr) < 10:
          return {'alpha': 0.0, 'x_min': 0.0, 'n_tail': 0}

      x_min = float(np.percentile(arr, x_min_percentile))
      tail = arr[arr >= x_min]

      if len(tail) < 2 or x_min <= 0:
          return {'alpha': 0.0, 'x_min': x_min, 'n_tail': len(tail)}

      alpha = float(len(tail) / np.sum(np.log(tail / x_min)))

      return {
          'alpha': round(alpha, 4),
          'x_min': round(x_min, 6),
          'n_tail': len(tail),
      }

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos de una serie temporal (ej: precios diarios, cantidades de eventos)"},{"name":"lag","desc":"numero de periodos para desplazar (ej: 1 para autocorrelacion con el valor anterior). Debe ser positivo."}],"output":"coeficiente de autocorrelacion normalizado en rango [-1, 1]. 1.0=correlacion perfecta, 0.0=sin correlacion"}


id: build_entity_schema_prompt_py_datascience
name: build_entity_schema_prompt
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def build_entity_schema_prompt(entity_presets: list[dict]) -> str
tags: ["prompt","llm","entity","schema","osint","graph","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["lista con varios presets","lista vacia retorna string vacio","preset sin metadata_fields"]
test_file_path: python/functions/datascience/build_entity_schema_prompt_test.py
file_path: python/functions/datascience/build_entity_schema_prompt.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 7b90e3fdfb41131299a13e16ffb96aff05cb84fa0ae3d3cd76b9f32d67e4b943
description:
  Genera la seccion del system prompt que describe los entity types disponibles para extraccion. Formatea los presets del registry en texto legible para el LLM.

example:
  ```python
  from build_entity_schema_prompt import build_entity_schema_prompt

  presets = [
      {
          "type_ref": "osint_person_go_cybersecurity",
          "label": "Person",
          "metadata_fields": ["full_name", "alias", "nationality", "dob", "risk_score"],
      },
      {
          "type_ref": "osint_organization_go_cybersecurity",
          "label": "Organization",
          "metadata_fields": ["legal_name", "country", "sector", "founded", "risk_score"],
      },
  ]

  prompt = build_entity_schema_prompt(presets)
  # Entity types available for extraction:
  #
  # 1. Person (type_ref: osint_person_go_cybersecurity)
  #    Attributes: full_name, alias, nationality, dob, risk_score
  #
  # 2. Organization (type_ref: osint_organization_go_cybersecurity)
  #    Attributes: legal_name, country, sector, founded, risk_score
  ```

notes:
  Funcion pura. No requiere dependencias externas.

  El formato de salida es deliberadamente sencillo para maximizar la comprension por el LLM: numero de orden, label humano, type_ref del registry y lista de atributos en una sola linea.

  Si un preset no tiene `metadata_fields` (o tiene lista vacia), se omite la linea de atributos.

  Pensada para componer con `build_relation_schema_prompt` al construir el system prompt completo de extraccion de grafos OSINT.

code:
  """Genera la seccion del system prompt que describe los entity types disponibles para extraccion."""


  def build_entity_schema_prompt(entity_presets: list[dict]) -> str:
      """Genera texto legible para el LLM describiendo los entity types disponibles.

      Formatea los presets del registry en una seccion del system prompt que indica
      al LLM que tipos de entidades puede extraer y que atributos tiene cada uno.

      Args:
          entity_presets: Lista de presets con campos 'label', 'type_ref' y
                          opcionalmente 'metadata_fields'. Ejemplo:
                          [{"type_ref": "osint_person_go_cybersecurity",
                            "label": "Person",
                            "metadata_fields": ["full_name", "alias"]}]

      Returns:
          String formateado con la seccion del prompt. Retorna string vacio si
          la lista de presets esta vacia.
      """
      if not entity_presets:
          return ""

      lines = ["Entity types available for extraction:", ""]

      for i, preset in enumerate(entity_presets, start=1):
          label = preset.get("label", "Unknown")
          type_ref = preset.get("type_ref", "")
          metadata_fields = preset.get("metadata_fields", [])

          lines.append(f"{i}. {label} (type_ref: {type_ref})")

          if metadata_fields:
              attrs = ", ".join(metadata_fields)
              lines.append(f"   Attributes: {attrs}")

          lines.append("")

      # Remove trailing blank line
      if lines and lines[-1] == "":
          lines.pop()

      return "\n".join(lines)

params_schema:
  {"params":[{"name":"entity_presets","desc":"lista de dicts describiendo tipos de entidad (ej: [{'type_ref': 'osint_person_go_cybersecurity', 'label': 'Person', 'metadata_fields': ['name', 'alias']}])"}],"output":"string con seccion del system prompt describiendo los entity types en formato legible para LLM"}


id: build_relation_schema_prompt_py_datascience
name: build_relation_schema_prompt
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def build_relation_schema_prompt(relation_types: list[str]) -> str
tags: ["prompt","llm","relation","schema","osint","graph","extraction"]
uses_functions: []
uses_types: []
returns: []
imports: []
tests: ["lista con varios tipos","lista vacia retorna string vacio","un solo tipo"]
test_file_path: python/functions/datascience/build_relation_schema_prompt_test.py
file_path: python/functions/datascience/build_relation_schema_prompt.py
created_at: 2026-04-05T11:23:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: 925a17dd2d241cf132753d0cc8e84278049fbc836e796d8bcb32b6fe829505fa
description:
  Genera la seccion del system prompt con los tipos de relacion permitidos para extraccion. Formatea la lista de tipos en texto legible para el LLM.

example:
  ```python
  from build_relation_schema_prompt import build_relation_schema_prompt

  types = ["funds", "employs", "communicates_with", "owns"]
  prompt = build_relation_schema_prompt(types)
  # Allowed relation types:
  # funds, employs, communicates_with, owns
  ```

notes:
  Funcion pura. No requiere dependencias externas.

  La salida es una sola linea con todos los tipos separados por coma, precedida por el encabezado. El formato es minimal para no consumir tokens innecesarios del contexto del LLM.

  Pensada para componer con `build_entity_schema_prompt` al construir el system prompt completo de extraccion de grafos OSINT.

code:
  """Genera la seccion del system prompt con los tipos de relacion permitidos."""


  def build_relation_schema_prompt(relation_types: list[str]) -> str:
      """Genera texto legible para el LLM describiendo los tipos de relacion permitidos.

      Formatea la lista de tipos de relacion en una seccion del system prompt que
      indica al LLM que relaciones puede extraer entre entidades.

      Args:
          relation_types: Lista de strings con los tipos de relacion permitidos.
                          Ejemplo: ["funds", "employs", "communicates_with"]

      Returns:
          String formateado con la seccion del prompt. Retorna string vacio si
          la lista esta vacia.
      """
      if not relation_types:
          return ""

      joined = ", ".join(relation_types)
      return f"Allowed relation types:\n{joined}"

params_schema:
  {"params":[{"name":"relation_types","desc":"lista de nombres de tipos de relacion permitidos (ej: ['funds', 'employs', 'owns', 'communicates_with'])"}],"output":"string con una linea describiendo los tipos de relacion permitidos en formato legible para LLM"}


id: clip_py_datascience
name: clip
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
signature: def clip(data: list, lo: float, hi: float) -> list
description: Recorta los valores de la lista al rango [lo, hi].
tags: ["clipping","bounds","python"]
uses_functions: []
uses_types: []
returns: []
imports: []
example:
  ```python
  clip([1, 5, 10, -3], 0, 7)
  # [1, 5, 7, 0]
  ```

tests: []
file_path: python/functions/datascience/datascience.py
created_at: 2026-04-05T11:22:32Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
notes: Funcion pura sin dependencias. Equivalente a numpy.clip pero sin numpy.
content_hash: 395e574ff4c402047c4a1dec4d0a82c5a6ee5f9f1629cb5f08b6ac691e8f5ffe
code:
  """Pure datascience utilities — statistics and numerical functions.

  Uses only math stdlib. No external dependencies.
  """

  import math


  def pearson(xs: list, ys: list) -> float:
      """Pearson correlation coefficient between two lists of floats."""
      n = len(xs)
      if n != len(ys) or n == 0:
          return 0.0
      mean_x = sum(xs) / n
      mean_y = sum(ys) / n
      num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
      den_x = math.sqrt(sum((x - mean_x) ** 2 for x in xs))
      den_y = math.sqrt(sum((y - mean_y) ** 2 for y in ys))
      if den_x == 0.0 or den_y == 0.0:
          return 0.0
      return num / (den_x * den_y)


  def standardize(data: list) -> list:
      """Z-score standardization (mean=0, std=1)."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [0.0] * n
      return [(x - mean) / std for x in data]


  def min_max_scale(data: list) -> list:
      """Scale values to [0, 1] range."""
      if not data:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          return [0.0] * len(data)
      return [(x - lo) / (hi - lo) for x in data]


  def clip(data: list, lo: float, hi: float) -> list:
      """Clip values to [lo, hi]."""
      return [max(lo, min(hi, x)) for x in data]


  def detect_outliers(data: list, threshold: float) -> list:
      """Returns list of bools, True where |z-score| > threshold."""
      n = len(data)
      if n == 0:
          return []
      mean = sum(data) / n
      std = math.sqrt(sum((x - mean) ** 2 for x in data) / n)
      if std == 0.0:
          return [False] * n
      return [abs((x - mean) / std) > threshold for x in data]


  def impute(data: list) -> list:
      """Replace None/NaN with mean of non-null values."""
      valid = [x for x in data if x is not None and not (isinstance(x, float) and math.isnan(x))]
      if not valid:
          return [0.0] * len(data)
      mean = sum(valid) / len(valid)
      return [
          mean if (x is None or (isinstance(x, float) and math.isnan(x))) else x
          for x in data
      ]


  def histogram(data: list, buckets: int) -> list:
      """Returns list of counts per bucket."""
      if not data or buckets <= 0:
          return []
      lo = min(data)
      hi = max(data)
      if hi == lo:
          counts = [0] * buckets
          counts[0] = len(data)
          return counts
      width = (hi - lo) / buckets
      counts = [0] * buckets
      for x in data:
          idx = int((x - lo) / width)
          if idx >= buckets:
              idx = buckets - 1
          counts[idx] += 1
      return counts


  def rolling_window(xs: list, size: int) -> list:
      """Returns list of sublists (sliding windows of given size)."""
      if size <= 0 or size > len(xs):
          return []
      return [xs[i : i + size] for i in range(len(xs) - size + 1)]


  def autocorrelation(data: list, lag: int) -> float:
      """Autocorrelation at given lag."""
      n = len(data)
      if lag < 0 or lag >= n or n == 0:
          return 0.0
      mean = sum(data) / n
      var = sum((x - mean) ** 2 for x in data) / n
      if var == 0.0:
          return 0.0
      cov = sum((data[i] - mean) * (data[i + lag] - mean) for i in range(n - lag)) / n
      return cov / var


  def linspace(start: float, stop: float, num: int) -> list:
      """Generate evenly spaced values from start to stop (inclusive)."""
      if num <= 0:
          return []
      if num == 1:
          return [start]
      step = (stop - start) / (num - 1)
      return [start + i * step for i in range(num)]


  def estimate_hawkes(arrivals: list[int], max_lag: int = 30) -> dict:
      """Estima parámetros de un proceso Hawkes desde autocorrelación de arrivals.

      Ajusta exponencial a*exp(-b*lag) sobre la ACF.
      Retorna dict con alpha, beta, branching_ratio, acf.
      """
      import numpy as np
      from scipy.optimize import curve_fit

      arr = np.array(arrivals, dtype=float)
      mean_a = np.mean(arr)
      var_a = np.var(arr)
      if var_a == 0:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': [1.0]}

      acf = [1.0] + [
          float(np.mean((arr[lag:] - mean_a) * (arr[:-lag] - mean_a)) / var_a)
          for lag in range(1, max_lag)
      ]

      lags = np.arange(1, max_lag)
      acf_vals = np.array(acf[1:])

      if acf_vals[0] <= 0.01:
          return {'alpha': 0.0, 'beta': 1.0, 'branching_ratio': 0.0, 'acf': acf}

      exp_decay = lambda x, a, b: a * np.exp(-b * x)
      try:
          popt, _ = curve_fit(exp_decay, lags, acf_vals, p0=[0.5, 0.5], maxfev=5000)
          alpha_est, beta_est = abs(popt[0]), abs(popt[1])
      except RuntimeError:
          alpha_est, beta_est = 0.0, 1.0

      branching = alpha_est / beta_est if beta_est > 0 else 0.0
      return {
          'alpha': round(alpha_est, 4),
          'beta': round(beta_est, 4),
          'branching_ratio': round(branching, 4),
          'acf': acf,
      }


  def estimate_pareto_alpha(values: list[float], x_min_percentile: float = 90.0) -> dict:
      """Estima el exponente alpha de una distribución Pareto via MLE.

      α = n / Σ ln(xi / x_min) donde x_min es el percentil indicado.
      Alpha bajo = cola más pesada = más valores extremos.
      """
      import numpy as np

      arr = np.array([v for v in values if v > 0], dtype=float)
      if len(arr) < 10:
          return {'alpha': 0.0, 'x_min': 0.0, 'n_tail': 0}

      x_min = float(np.percentile(arr, x_min_percentile))
      tail = arr[arr >= x_min]

      if len(tail) < 2 or x_min <= 0:
          return {'alpha': 0.0, 'x_min': x_min, 'n_tail': len(tail)}

      alpha = float(len(tail) / np.sum(np.log(tail / x_min)))

      return {
          'alpha': round(alpha, 4),
          'x_min': round(x_min, 6),
          'n_tail': len(tail),
      }

params_schema:
  {"params":[{"name":"data","desc":"lista de valores numericos a recortar"},{"name":"lo","desc":"limite inferior (minimo permitido). Valores menores seran reemplazados por lo."},{"name":"hi","desc":"limite superior (maximo permitido). Valores mayores seran reemplazados por hi."}],"output":"lista de valores con todos los elementos recortados al rango [lo, hi]"}


id: deduplicate_entities_py_datascience
name: deduplicate_entities
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
uses_functions: ["normalize_entity_name_py_core","merge_entity_attributes_py_core"]
uses_types: ["entity_candidate_py_datascience","deduplication_result_py_datascience"]
returns: ["deduplication_result_py_datascience"]
imports: ["uuid"]
test_file_path: python/functions/datascience/deduplicate_entities_test.py
file_path: python/functions/datascience/deduplicate_entities.py
created_at: 2026-04-05T11:24:52Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: dbbaff336f28c177533e8da1fe7ddc34331e6524303840a4e1fa5de8653b0119
signature:
  def deduplicate_entities(candidates: list[EntityCandidate], name_threshold: float = 0.85, same_type_only: bool = True) -> DeduplicationResult

description:
  Agrupa entidades candidatas que refieren a la misma entidad real usando fuzzy matching de nombres (Levenshtein + Jaccard) y Union-Find para clusters transitivos. Retorna entidades mergeadas con mapas de resolucion de IDs y log de merges.

tags:
  ["deduplication","entity","fuzzy","levenshtein","jaccard","union-find","knowledge-graph","nlp","fuzzygraph","datascience"]

example:
  ```python
  from python.types.datascience.entity_candidate import EntityCandidate
  from python.functions.datascience.deduplicate_entities import deduplicate_entities

  candidates = [
      EntityCandidate(name="John Smith", type_ref="person", confidence=0.9),
      EntityCandidate(name="Smith, John", type_ref="person", confidence=0.85),
      EntityCandidate(name="Google", type_ref="organization", confidence=0.95),
      EntityCandidate(name="Google LLC", type_ref="organization", confidence=0.88),
  ]

  result = deduplicate_entities(candidates, name_threshold=0.85, same_type_only=True)
  # result.total_before = 4
  # result.total_after = 2
  # result.merge_log = [
  #   {"canonical": "John Smith", "merged": ["Smith, John"], "score": 0.91, "reason": "fuzzy_name"},
  #   {"canonical": "Google", "merged": ["Google LLC"], "score": 0.89, "reason": "fuzzy_name"},
  # ]
  ```

tests:
  ["John Smith y Smith, John se mergean","Google y Google LLC se mergean","192.168.1.1 y 192.168.1.1 se mergean por matching exacto","John Smith (person) y John Smith (organization) NO se mergean","Clusters transitivos: A~B, B~C -\u003e {A, B, C} en un solo cluster","Entidades sin duplicados pasan sin modificacion","Confidence toma el max del cluster; atributos se fusionan","Lista vacia retorna resultado vacio","name_to_id contiene todos los nombres originales del cluster"]

notes:
  Funcion pura. Implementa Levenshtein y Jaccard internamente para evitar dependencias externas a este modulo. Las funciones del registry `levenshtein_distance_py_cybersecurity` y `jaccard_similarity_py_cybersecurity` son equivalentes pero requieren imports adicionales — la implementacion inline mantiene la funcion sin dependencias de stdlib.

  El `name_to_id` del resultado es el mapa de resolucion principal para la fase de deduplicacion de relaciones: permite resolver cualquier variante de nombre de una entidad a su ID canonico.

documentation:
  ## Algoritmo

  1. **Normalizar nombres** usando `normalize_entity_name()` sobre cada candidato segun su `type_ref`
  2. **Comparacion pairwise** dentro del mismo tipo (si `same_type_only=True`):
     - Para tipos tecnicos (ip, email, domain, crypto_wallet, phone): matching exacto normalizado
     - Para el resto: `score = max(levenshtein_sim, jaccard_sim)` + bonus por contencion (+0.3) y acronimos (+0.3)
  3. **Union-Find** para clusters transitivos: si A~B y B~C, entonces {A, B, C} forman un cluster
  4. **Merge por cluster:**
     - Nombre canonico: candidato con mayor `confidence`
     - Atributos: `merge_entity_attributes()` sobre todos los candidatos del cluster
     - Confidence: `max` del cluster
     - Source chunks: union de todos los candidatos
     - `merged_from`: union de todos los nombres originales

  ## Heuristicas de similitud de nombres

  | Heuristica | Efecto |
  |---|---|
  | Levenshtein | `1 - (edit_distance / max_len)` |
  | Jaccard sobre tokens | `\|A ∩ B\| / \|A ∪ B\|` |
  | Score base | `max(lev_sim, jaccard_sim)` |
  | Contencion (a in b o b in a) | `+0.3` hasta max 1.0 |
  | Acronimo ("FBI" ~ "Federal Bureau of Investigation") | `+0.3` hasta max 1.0 |
  | Tipos exactos (ip/email/domain) | solo matching exacto, ignora umbral |

  ## Complejidad

  - Pairwise: O(N^2) — aceptable para <1000 entidades (tipico por documento)
  - Union-Find con path compression: O(α(N)) amortizado por operacion
  - Para escalar a >1000: pre-filtrar por primera letra o n-gram index antes de comparar

code:
  """Deduplica entidades candidatas usando fuzzy matching de nombres."""

  from __future__ import annotations

  import sys
  import os
  import uuid

  sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", ".."))

  from python.types.datascience.entity_candidate import EntityCandidate
  from python.types.datascience.deduplication_result import DeduplicationResult
  from python.functions.core.normalize_entity_name import normalize_entity_name
  from python.functions.core.merge_entity_attributes import merge_entity_attributes


  # ── Similitud helpers ──────────────────────────────────────────────────────────

  def _levenshtein(a: str, b: str) -> int:
      """Distancia de edicion Levenshtein entre dos strings."""
      if a == b:
          return 0
      if not a:
          return len(b)
      if not b:
          return len(a)
      prev = list(range(len(b) + 1))
      for i, ca in enumerate(a, 1):
          curr = [i]
          for j, cb in enumerate(b, 1):
              cost = 0 if ca == cb else 1
              curr.append(min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + cost))
          prev = curr
      return prev[-1]


  def _jaccard(tokens_a: list[str], tokens_b: list[str]) -> float:
      """Similitud de Jaccard entre dos conjuntos de tokens."""
      set_a = set(tokens_a)
      set_b = set(tokens_b)
      if not set_a and not set_b:
          return 1.0
      inter = len(set_a & set_b)
      union = len(set_a | set_b)
      return inter / union if union else 0.0


  def _name_similarity(a: str, b: str) -> float:
      """Score de similitud entre dos nombres normalizados.

      Combina similitud de Levenshtein y Jaccard sobre tokens.
      Aplica bonus de contencion (+0.3) y deteccion de acronimos.
      """
      if a == b:
          return 1.0

      # Similitud Levenshtein
      max_len = max(len(a), len(b))
      lev_sim = 1.0 - (_levenshtein(a, b) / max_len) if max_len else 1.0

      # Similitud Jaccard sobre tokens
      tokens_a = a.split()
      tokens_b = b.split()
      jac_sim = _jaccard(tokens_a, tokens_b)

      score = max(lev_sim, jac_sim)

      # Bonus de contencion: un nombre contiene al otro
      if a in b or b in a:
          score = min(1.0, score + 0.3)

      # Deteccion de acronimo: "FBI" ~ "Federal Bureau of Investigation"
      if _is_acronym_of(a, tokens_b) or _is_acronym_of(b, tokens_a):
          score = min(1.0, score + 0.3)

      return score


  def _is_acronym_of(candidate: str, tokens: list[str]) -> bool:
      """Comprueba si candidate es un acronimo formado por las iniciales de tokens."""
      if not candidate or not tokens:
          return False
      initials = "".join(t[0] for t in tokens if t).upper()
      return candidate.upper() == initials


  _EXACT_TYPES = {"ip", "email", "domain", "crypto_wallet", "phone"}


  def _is_exact_type(entity_type: str) -> bool:
      """Tipos tecnicos donde solo se acepta matching exacto."""
      return entity_type.lower() in _EXACT_TYPES


  # ── Union-Find ─────────────────────────────────────────────────────────────────

  class _UnionFind:
      def __init__(self, n: int) -> None:
          self._parent = list(range(n))
          self._rank = [0] * n

      def find(self, x: int) -> int:
          while self._parent[x] != x:
              self._parent[x] = self._parent[self._parent[x]]
              x = self._parent[x]
          return x

      def union(self, x: int, y: int) -> None:
          rx, ry = self.find(x), self.find(y)
          if rx == ry:
              return
          if self._rank[rx] < self._rank[ry]:
              rx, ry = ry, rx
          self._parent[ry] = rx
          if self._rank[rx] == self._rank[ry]:
              self._rank[rx] += 1


  # ── Implementacion principal ────────────────────────────────────────────────────

  def deduplicate_entities(
      candidates: list[EntityCandidate],
      name_threshold: float = 0.85,
      same_type_only: bool = True,
  ) -> DeduplicationResult:
      """Agrupa entidades candidatas que refieren a la misma entidad real.

      Usa fuzzy matching de nombres (Levenshtein + Jaccard) y Union-Find para
      detectar clusters transitivos. Por cada cluster genera una entidad canonica
      mergeando atributos de todos sus miembros.

      Para tipos tecnicos (ip, email, domain, crypto_wallet, phone) solo se
      acepta matching exacto normalizado, ignorando el umbral de nombre.

      Args:
          candidates: lista de EntityCandidate a deduplicar.
          name_threshold: score minimo para considerar dos nombres iguales (0-1).
          same_type_only: si True, solo compara entidades del mismo type_ref.

      Returns:
          DeduplicationResult con entidades deduplicadas, mapas de resolucion
          e historial de merges.
      """
      if not candidates:
          return DeduplicationResult(
              entities=[],
              entity_id_map={},
              name_to_id={},
              merge_log=[],
              total_before=0,
              total_after=0,
          )

      n = len(candidates)

      # Paso 1: normalizar nombres
      normalized: list[str] = []
      for c in candidates:
          norm = normalize_entity_name(c.name, c.type_ref)
          normalized.append(norm)

      # Paso 2: Union-Find sobre todos los indices
      uf = _UnionFind(n)

      # Paso 3: comparacion pairwise (con agrupacion por tipo si same_type_only)
      merge_pairs: list[tuple[int, int, float]] = []

      for i in range(n):
          for j in range(i + 1, n):
              if same_type_only and candidates[i].type_ref != candidates[j].type_ref:
                  continue

              ni, nj = normalized[i], normalized[j]
              et = candidates[i].type_ref.lower()

              if _is_exact_type(et):
                  if ni == nj:
                      uf.union(i, j)
                      merge_pairs.append((i, j, 1.0))
                  continue

              score = _name_similarity(ni, nj)
              if score >= name_threshold:
                  uf.union(i, j)
                  merge_pairs.append((i, j, score))

      # Paso 4: agrupar indices por raiz del Union-Find
      clusters: dict[int, list[int]] = {}
      for i in range(n):
          root = uf.find(i)
          clusters.setdefault(root, []).append(i)

      # Paso 5: merge por cluster
      merged_entities: list[EntityCandidate] = []
      entity_id_map: dict[str, str] = {}
      name_to_id: dict[str, str] = {}
      merge_log: list[dict] = []

      # Pares mergeados para construir el log
      merged_pairs_by_root: dict[int, list[tuple[int, int, float]]] = {}
      for i, j, score in merge_pairs:
          root = uf.find(i)
          merged_pairs_by_root.setdefault(root, []).append((i, j, score))

      for root, indices in clusters.items():
          cluster_candidates = [candidates[idx] for idx in indices]

          if len(cluster_candidates) == 1:
              c = cluster_candidates[0]
              canonical_name = c.name
              canonical_norm = normalized[indices[0]]
              merged_attrs = c.attributes
              merged_confidence = c.confidence
              merged_chunks = list(c.source_chunk_indices)
              merged_from = list(c.merged_from) if c.merged_from else [c.name]
          else:
              # Candidato con mayor confidence es el canonico
              best = max(cluster_candidates, key=lambda c: c.confidence)
              canonical_name = best.name
              canonical_norm = normalize_entity_name(best.name, best.type_ref)

              merged_attrs = merge_entity_attributes(
                  [c.attributes for c in cluster_candidates]
              )
              merged_confidence = max(c.confidence for c in cluster_candidates)

              merged_chunks: list[int] = []
              seen_chunks: set[int] = set()
              for c in cluster_candidates:
                  for idx in c.source_chunk_indices:
                      if idx not in seen_chunks:
                          merged_chunks.append(idx)
                          seen_chunks.add(idx)

              merged_from: list[str] = []
              seen_names: set[str] = set()
              for c in cluster_candidates:
                  names_to_add = c.merged_from if c.merged_from else [c.name]
                  for nm in names_to_add:
                      if nm not in seen_names:
                          merged_from.append(nm)
                          seen_names.add(nm)

              # Log de merge
              other_names = [c.name for c in cluster_candidates if c is not best]
              pairs = merged_pairs_by_root.get(root, [])
              max_score = max((s for _, _, s in pairs), default=1.0)
              merge_log.append(
                  {
                      "canonical": canonical_name,
                      "merged": other_names,
                      "score": round(max_score, 4),
                      "reason": "fuzzy_name",
                  }
              )

          ent_id = str(uuid.uuid4())
          entity = EntityCandidate(
              name=canonical_name,
              name_normalized=canonical_norm,
              type_ref=cluster_candidates[0].type_ref,
              type_label=cluster_candidates[0].type_label,
              attributes=merged_attrs,
              confidence=merged_confidence,
              source_chunk_indices=merged_chunks,
              merged_from=merged_from,
          )
          merged_entities.append(entity)

          # Poblar mapas de resolucion
          entity_id_map[canonical_norm] = ent_id
          for orig_name in merged_from:
              name_to_id[orig_name] = ent_id
          name_to_id[canonical_norm] = ent_id

      return DeduplicationResult(
          entities=merged_entities,
          entity_id_map=entity_id_map,
          name_to_id=name_to_id,
          merge_log=merge_log,
          total_before=n,
          total_after=len(merged_entities),
      )

params_schema:
  {"params":[{"name":"candidates","desc":"lista de EntityCandidate a deduplicar. Cada uno tiene name, type_ref, confidence."},{"name":"name_threshold","desc":"umbral de similitud para fuzzy matching de nombres en rango [0, 1] (tipico: 0.85). Mayor = mas estricto."},{"name":"same_type_only","desc":"si True, solo mergea candidatos con el mismo type_ref. Si False, mergea por similitud ignoring tipo."}],"output":"DeduplicationResult con entidades mergeadas, mapas de resolucion de IDs (name_to_id) y log de merges realizados"}


id: deduplicate_relations_py_datascience
name: deduplicate_relations
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["datascience","extraction","knowledge-graph","nlp","deduplication","fuzzy-match","fuzzygraph"]
uses_functions: ["levenshtein_distance_py_cybersecurity"]
uses_types: ["relation_candidate_py_datascience"]
returns: []
imports: []
test_file_path: python/functions/datascience/deduplicate_relations_test.py
file_path: python/functions/datascience/deduplicate_relations.py
created_at: 2026-04-05T11:24:52Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: fbd08ccf219db9d19d0960a9925fef047b2b5ee4ea669321e76b2ff8dc3eefa1
signature:
  def deduplicate_relations(relations: list[RelationCandidate], entity_id_map: dict[str, str]) -> list[RelationCandidate]

description:
  Deduplica relaciones candidatas resolviendo from_name/to_name a entity IDs finales via entity_id_map. Descarta self-loops y relaciones sin match. Mergea duplicados (mismo from_id, to_id, relation_type) concatenando descripciones unicas y tomando max confidence.

example:
  ```python
  from python.types.datascience.relation_candidate import RelationCandidate
  from python.functions.datascience.deduplicate_relations import deduplicate_relations

  # entity_id_map producido por deduplicate_entities
  entity_id_map = {
      "john smith": "entity_001",
      "smith, john": "entity_001",  # alias mergeado
      "acme corp": "entity_002",
  }

  relations = [
      RelationCandidate(from_name="John Smith", to_name="Acme Corp",
                        relation_type="works_at", description="John es CEO",
                        confidence=0.9, source_chunk_index=0),
      RelationCandidate(from_name="Smith, John", to_name="Acme Corp",
                        relation_type="works_at", description="CEO de Acme",
                        confidence=0.7, source_chunk_index=2),
  ]

  result = deduplicate_relations(relations, entity_id_map)
  # → 1 RelationCandidate con from_id="entity_001", to_id="entity_002",
  #   confidence=0.9, description="John es CEO; CEO de Acme"
  ```

tests:
  ["dos relaciones identicas se colapsan en una","relacion con nombre mergeado se resuelve al id correcto","self loop se descarta","nombre no mapeado sin fuzzy match se descarta","relaciones distintas se mantienen","merge descripcion concatena unicas","lista vacia retorna lista vacia","fuzzy match resuelve nombre cercano"]

notes:
  La funcion es pura: no hace I/O, no tiene efectos secundarios. El logging es
  de nivel DEBUG/WARNING — en produccion configurar el logger de la aplicacion.

  **Resolucion de nombres:**
  - Lookup exacto primero (lowercase strip del nombre contra las claves del mapa).
  - Si no hay match exacto, fuzzy match con Levenshtein (threshold=3 ediciones).
  - Si sigue sin match, la relacion se descarta con `logger.warning`.

  **Self-loops:** relaciones donde `from_id == to_id` siempre se descartan.

  **Merge:** cuando varias relaciones comparten `(from_id, to_id, relation_type)`:
  - `confidence`: max del grupo.
  - `description`: union de descripciones unicas (no duplicadas), separadas por `'; '`.
  - `from_name` / `to_name` / `source_chunk_index`: del primer candidato del grupo.

  **Integracion con fuzzygraph:**
  Esta funcion es el paso 4 del pipeline de extraccion. Recibe el output de
  `extract_relations_llm` (relaciones crudas con nombres de texto) y el
  `entity_id_map` producido por `deduplicate_entities`. Produce la lista final
  de relaciones para `ExtractionResult`.

code:
  """Deduplica RelationCandidate resolviendo nombres a IDs y colapsando duplicados."""

  import logging
  import os
  import sys

  logger = logging.getLogger(__name__)

  # --- Importar levenshtein_distance desde cybersecurity ---
  # Soporta dos contextos:
  #   1. Ejecutado desde python/functions/datascience/ (pytest local)
  #   2. Ejecutado desde la raiz del registry (fn run)
  def _levenshtein_distance(a: str, b: str) -> int:
      """Calcula la distancia de edicion de Levenshtein entre dos strings."""
      if len(a) < len(b):
          return _levenshtein_distance(b, a)
      if len(b) == 0:
          return len(a)
      prev_row = list(range(len(b) + 1))
      for i, ca in enumerate(a):
          curr_row = [i + 1]
          for j, cb in enumerate(b):
              cost = 0 if ca == cb else 1
              curr_row.append(
                  min(curr_row[j] + 1, prev_row[j + 1] + 1, prev_row[j] + cost)
              )
          prev_row = curr_row
      return prev_row[-1]


  try:
      _here = os.path.dirname(os.path.abspath(__file__))
      _cyber_path = os.path.join(_here, "..", "cybersecurity")
      if _cyber_path not in sys.path:
          sys.path.insert(0, _cyber_path)
      from cybersecurity import levenshtein_distance as _lev
  except ImportError:
      _lev = None  # type: ignore

  levenshtein_distance = _lev if _lev is not None else _levenshtein_distance


  def _fuzzy_resolve(name: str, entity_id_map: dict[str, str], threshold: int = 3) -> str:
      """Intenta resolver un nombre contra las claves del mapa por fuzzy match.

      Recorre todas las claves de entity_id_map y busca la mas cercana segun
      distancia de Levenshtein. Retorna el entity_id si la distancia es <=
      threshold, o '' si no hay match aceptable.

      Args:
          name: nombre a resolver (ya en lowercase strip).
          entity_id_map: mapa nombre_normalizado -> entity_id.
          threshold: distancia maxima de edicion para considerar match (default 3).

      Returns:
          entity_id del mejor match o '' si no hay match.
      """
      best_id = ""
      best_dist = threshold + 1
      for key, entity_id in entity_id_map.items():
          dist = levenshtein_distance(name, key)
          if dist < best_dist:
              best_dist = dist
              best_id = entity_id
      return best_id if best_dist <= threshold else ""


  def deduplicate_relations(
      relations: list,
      entity_id_map: dict[str, str],
  ) -> list:
      """Deduplica relaciones candidatas resolviendo nombres a IDs de entidad finales.

      Algoritmo:
      1. Para cada RelationCandidate, intentar resolver from_name y to_name al
         entity_id via entity_id_map (lookup exacto primero, ignorando mayusculas).
         Si no hay match exacto, intentar fuzzy match con levenshtein_distance.
         Si sigue sin match, descartar la relacion con warning.
      2. Descartar self-loops (from_id == to_id).
      3. Deduplicar por (from_id, to_id, relation_type):
         - description: concatenar descripciones unicas separadas por '; '
         - confidence: max del grupo
      4. Retornar lista limpia de RelationCandidate con from_id y to_id resueltos.

      Args:
          relations: lista de RelationCandidate con from_name/to_name originales.
          entity_id_map: mapa nombre_normalizado -> entity_id (output de
              deduplicate_entities). Permite resolver nombres que fueron mergeados.

      Returns:
          Lista deduplicada de RelationCandidate con from_id y to_id resueltos.
      """
      # Importar tipo — funciona tanto desde datascience/ como desde raiz del registry
      try:
          _types_path = os.path.join(
              os.path.dirname(os.path.abspath(__file__)),
              "..", "..", "..", "python", "types", "datascience",
          )
          if _types_path not in sys.path:
              sys.path.insert(0, _types_path)
          from relation_candidate import RelationCandidate
      except ImportError:
          from python.types.datascience.relation_candidate import RelationCandidate  # type: ignore

      resolved: list = []

      for rel in relations:
          # --- Resolver from_name ---
          from_key = rel.from_name.lower().strip()
          from_id = entity_id_map.get(from_key, "")
          if not from_id:
              from_id = _fuzzy_resolve(from_key, entity_id_map)
              if not from_id:
                  logger.warning(
                      "deduplicate_relations: no se pudo resolver from_name=%r — descartando",
                      rel.from_name,
                  )
                  continue

          # --- Resolver to_name ---
          to_key = rel.to_name.lower().strip()
          to_id = entity_id_map.get(to_key, "")
          if not to_id:
              to_id = _fuzzy_resolve(to_key, entity_id_map)
              if not to_id:
                  logger.warning(
                      "deduplicate_relations: no se pudo resolver to_name=%r — descartando",
                      rel.to_name,
                  )
                  continue

          # --- Descartar self-loops ---
          if from_id == to_id:
              logger.debug(
                  "deduplicate_relations: self-loop descartado (from=%r, to=%r, type=%r)",
                  rel.from_name,
                  rel.to_name,
                  rel.relation_type,
              )
              continue

          resolved.append(
              RelationCandidate(
                  from_name=rel.from_name,
                  to_name=rel.to_name,
                  from_id=from_id,
                  to_id=to_id,
                  relation_type=rel.relation_type,
                  description=rel.description,
                  confidence=rel.confidence,
                  source_chunk_index=rel.source_chunk_index,
              )
          )

      # --- Deduplicar por (from_id, to_id, relation_type) ---
      groups: dict[tuple, list] = {}
      for rel in resolved:
          key = (rel.from_id, rel.to_id, rel.relation_type)
          groups.setdefault(key, []).append(rel)

      result: list = []
      for (from_id, to_id, rel_type), group in groups.items():
          if len(group) == 1:
              result.append(group[0])
              continue

          # Mergear: max confidence + union de descripciones unicas
          best_confidence = max(r.confidence for r in group)
          seen_desc: set[str] = set()
          descriptions: list[str] = []
          for r in group:
              if r.description and r.description not in seen_desc:
                  descriptions.append(r.description)
                  seen_desc.add(r.description)

          result.append(
              RelationCandidate(
                  from_name=group[0].from_name,
                  to_name=group[0].to_name,
                  from_id=from_id,
                  to_id=to_id,
                  relation_type=rel_type,
                  description="; ".join(descriptions),
                  confidence=best_confidence,
                  source_chunk_index=group[0].source_chunk_index,
              )
          )

      return result

params_schema:
  {"params":[{"name":"relations","desc":"lista de RelationCandidate con from_name, to_name, relation_type, description, confidence"},{"name":"entity_id_map","desc":"dict producido por deduplicate_entities mapando nombres mergeados a IDs canonicos (ej: {'john smith': 'entity_001'})"}],"output":"lista de RelationCandidate deduplicadas, con from_id y to_id resueltos al entity_id_map, sin self-loops"}


id: detect_drift_py_datascience
name: detect_drift
kind: function
lang: py
domain: datascience
version: 1.0.0
purity: pure
tags: ["drift","statistics","z-score","monitoring","executions","operations","datascience"]
uses_functions: []
uses_types: []
returns: []
imports: ["math"]
test_file_path: python/functions/datascience/detect_drift_test.py
file_path: python/functions/datascience/detect_drift.py
created_at: 2026-04-05T11:25:06Z
updated_at: 2026-04-05T16:40:39Z
props: []
emits: []
variant: []
content_hash: a7a17e4855a17c9dd5b3d7a709ceede9d8bd314f0f733d7b44b8673bd09d78af
signature:
  def detect_drift(history: list[dict], current: dict, fields: list[str], threshold: float = 2.0) -> list[dict]

description:
  Detecta drift estadistico comparando metricas de la ejecucion actual contra el historial usando z-score. Si |z| > threshold, el campo ha drifteado. Util para monitorizar executions en operations.db.

example:
  ```python
  history = [
      {"records_out": 100, "duration_ms": 500},
      {"records_out": 105, "duration_ms": 480},
      {"records_out": 98,  "duration_ms": 510},
  ]
  current = {"records_out": 50, "duration_ms": 2000}

  results = detect_drift(history, current, ["records_out", "duration_ms"])
  # [
  #   {"field": "records_out", "current": 50, "mean": 101.0, "std": 3.6, "z_score": -14.2, "drifted": True},
  #   {"field": "duration_ms", "current": 2000, "mean": 496.7, "std": 15.3, "z_score": 98.3, "drifted": True},
  # ]
  ```

tests:
  ["campo con drift claro (z \u003e threshold)","campo estable (z \u003c threshold)","historial con un solo punto → std=0, no puede calcular → drifted=False con nota","historial vacio → todos drifted=False","threshold custom"]

notes:
  Funcion pura. Solo stdlib (`math`).

  El z-score usa desviacion estandar poblacional (dividir por N, no N-1) para ser consistente con historial de cualquier tamanio.

  Casos especiales:
  - **Historial vacio**: z_score=0.0, drifted=False para todos los campos.
  - **Un solo punto en historial**: std=0.0, z_score=0.0, drifted=False. No hay suficiente historia para calcular variabilidad.
  - **Std=0 con N>=2**: todos los valores historicos identicos. z_score=0.0, drifted=False (cualquier desviacion seria tecnicamente infinita, pero se asume que el sistema es muy estable).

  Pensado para el paso ANALIZAR del bucle reactivo: comparar `metrics` de la ejecucion actual con executions historicas de `operations.db`.

code:
  """detect_drift — detecta drift estadistico por z-score comparando metricas contra historial."""

  import math


  def detect_drift(
      history: list[dict],
      current: dict,
      fields: list[str],
      threshold: float = 2.0,
  ) -> list[dict]:
      """Detecta drift estadistico comparando metricas actuales contra el historial.

      Usa z-score: si |z| > threshold, el campo ha drifteado. Pensado para
      comparar metrics de executions sucesivas en operations.db.

      Args:
          history: Lista de dicts con metricas historicas. Cada dict puede
                   contener cualquier combinacion de los campos indicados.
          current: Dict con las metricas de la ejecucion actual.
          fields: Lista de campos numericos a analizar.
          threshold: Umbral de z-score para considerar drift. Default 2.0.

      Returns:
          Lista de dicts con: field, current, mean, std, z_score, drifted.
          Si el historial tiene 0 o 1 punto, z_score=0.0 y drifted=False
          porque no hay suficiente informacion estadistica.
      """
      results = []

      for field in fields:
          values = [
              float(h[field])
              for h in history
              if field in h and h[field] is not None
          ]

          current_val = float(current.get(field, 0))

          if len(values) == 0:
              results.append({
                  "field": field,
                  "current": current_val,
                  "mean": 0.0,
                  "std": 0.0,
                  "z_score": 0.0,
                  "drifted": False,
              })
              continue

          n = len(values)
          mean = sum(values) / n

          if n < 2:
              # Un solo punto: no hay std, no podemos calcular z-score
              results.append({
                  "field": field,
                  "current": current_val,
                  "mean": mean,
                  "std": 0.0,
                  "z_score": 0.0,
                  "drifted": False,
              })
              continue

          variance = sum((v - mean) ** 2 for v in values) / n
          std = math.sqrt(variance)

          if std == 0.0:
              # Todos los valores identicos: z_score indeterminado, no drift
              z_score = 0.0
              drifted = False
          else:
              z_score = (current_val - mean) / std
              drifted = abs(z_score) > threshold

          results.append({
              "field": field,
              "current": current_val,
              "mean": mean,
              "std": std,
              "z_score": z_score,
              "drifted": drifted,
          })

      return results

params_schema:
  {"params":[{"name":"history","desc":"lista de dicts con metricas historicas (ej: [{'records_out': 100, 'duration_ms': 500}, ...])"},{"name":"current","desc":"dict con metricas actuales a comparar (ej: {'records_out': 50, 'duration_ms': 2000})"},{"name":"fields","desc":"lista de nombres de campos a monitorizar en current vs history (ej: ['records_out', 'duration_ms'])"},{"name":"threshold","desc":"umbral de z-score para declarar drift (tipico: 2.0-3.0). Mayor = menos sensible."}],"output":"lista de dicts con {field, current, mean, std, z_score, drifted} para cada campo monitoreado"}


