egutierrez
dff0c0d2b7
Extractores nuevos en python/functions/cybersecurity/: - extract_ip_addresses (IPv4 + IPv6 con validacion ipaddress) - extract_emails (RFC 5322 simplificado) - extract_domains (FQDNs con TLD valido, lista estatica) - extract_file_hashes (MD5/SHA1/SHA256/SHA512, algoritmo por longitud) - extract_crypto_wallets (BTC legacy + bech32, ETH 0x+40hex) - extract_cve_ids (CVE-YYYY-NNNN+) - extract_mac_addresses (xx:xx:xx + xx-xx-xx, separador uniforme) - extract_phone_numbers (E.164 + ES local 9 digitos) Pipeline: - extract_iocs corre todos, deduplica spans contenidos. Mantiene purity:pure (kind:function con uses_functions no vacio) porque la regla del registry exige que los pipelines sean impuros. Todas devuelven list[dict] con value/start/end/type para que el caller (issues 0038-0040) pueda reconciliar offsets con spans NER sin reparsing. Refs #0037 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
60 lines
2.6 KiB
Markdown
60 lines
2.6 KiB
Markdown
---
|
|
name: extract_iocs
|
|
kind: function
|
|
lang: py
|
|
domain: cybersecurity
|
|
version: "1.0.0"
|
|
purity: pure
|
|
signature: "def extract_iocs(text: str, types: list[str] | None = None) -> list[dict]"
|
|
description: "Pipeline puro que corre todos los extractores de IoC (IP, email, dominio, hash, wallet, CVE, MAC, telefono) y devuelve lista unificada con `type`. Deduplica spans contenidos. Si types se pasa, filtra los tipos a ejecutar."
|
|
tags: [ioc, pipeline, regex, extract, cybersecurity, python]
|
|
uses_functions:
|
|
- extract_ip_addresses_py_cybersecurity
|
|
- extract_emails_py_cybersecurity
|
|
- extract_domains_py_cybersecurity
|
|
- extract_file_hashes_py_cybersecurity
|
|
- extract_crypto_wallets_py_cybersecurity
|
|
- extract_cve_ids_py_cybersecurity
|
|
- extract_mac_addresses_py_cybersecurity
|
|
- extract_phone_numbers_py_cybersecurity
|
|
uses_types: []
|
|
returns: []
|
|
returns_optional: false
|
|
error_type: ""
|
|
imports: []
|
|
params:
|
|
- name: text
|
|
desc: "string de texto del que extraer IoCs"
|
|
- name: types
|
|
desc: "lista opcional de tipos a extraer (email, ip_address, domain, file_hash, crypto_wallet, cve_id, mac_address, phone_number). None = todos."
|
|
output: "lista de dicts {value, start, end, type, ...} ordenada por offset, sin spans contenidos"
|
|
tested: true
|
|
tests:
|
|
- "Pipeline corre todos los extractores"
|
|
- "Filtro por types subset"
|
|
- "Deduplica spans contenidos (dominio dentro de email)"
|
|
- "Tipos desconocidos se ignoran"
|
|
test_file_path: "python/functions/cybersecurity/tests/test_extract_iocs.py"
|
|
file_path: "python/functions/cybersecurity/extract_iocs.py"
|
|
---
|
|
|
|
## Ejemplo
|
|
|
|
```python
|
|
extract_iocs("Reach alice@example.com from 10.0.0.5; CVE-2023-1234")
|
|
# [{"value": "alice@example.com", "start": 6, "end": 23, "type": "email"},
|
|
# {"value": "10.0.0.5", "start": 29, "end": 37, "type": "ip_address"},
|
|
# {"value": "CVE-2023-1234", "start": 39, "end": 52, "type": "cve_id"}]
|
|
|
|
extract_iocs("Only IPs: 8.8.8.8 here", types=["ip_address"])
|
|
# [{"value": "8.8.8.8", ..., "type": "ip_address"}]
|
|
```
|
|
|
|
## Notas
|
|
|
|
Es **funcion** y no `kind: pipeline` porque la regla del registry exige que pipelines sean impuros — esta no lo es: solo compone funciones puras y deduplica. Mantiene `purity: pure` con `uses_functions` no vacio.
|
|
|
|
Deduplicacion: un match completamente contenido en otro (ej. `example.com` dentro de `alice@example.com`) se descarta. Empate exacto de span: gana el primero segun el orden de `_EXTRACTORS` en el modulo (email > ip > crypto_wallet > cve > mac > file_hash > phone > domain). Reordenar el dict cambia la prioridad si tienes overlaps habituales.
|
|
|
|
Bench informal: ~50-80 ms por MB de texto sobre CPU moderna (depende del numero de matches).
|