egutierrez
55dcdd1164
Extractores nuevos en python/functions/cybersecurity/: - extract_ip_addresses (IPv4 + IPv6 con validacion ipaddress) - extract_emails (RFC 5322 simplificado) - extract_domains (FQDNs con TLD valido, lista estatica) - extract_file_hashes (MD5/SHA1/SHA256/SHA512, algoritmo por longitud) - extract_crypto_wallets (BTC legacy + bech32, ETH 0x+40hex) - extract_cve_ids (CVE-YYYY-NNNN+) - extract_mac_addresses (xx:xx:xx + xx-xx-xx, separador uniforme) - extract_phone_numbers (E.164 + ES local 9 digitos) Pipeline: - extract_iocs corre todos, deduplica spans contenidos. Mantiene purity:pure (kind:function con uses_functions no vacio) porque la regla del registry exige que los pipelines sean impuros. Todas devuelven list[dict] con value/start/end/type para que el caller (issues 0038-0040) pueda reconciliar offsets con spans NER sin reparsing. Refs #0037 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2.6 KiB
2.6 KiB
name, kind, lang, domain, version, purity, signature, description, tags, uses_functions, uses_types, returns, returns_optional, error_type, imports, params, output, tested, tests, test_file_path, file_path
| name | kind | lang | domain | version | purity | signature | description | tags | uses_functions | uses_types | returns | returns_optional | error_type | imports | params | output | tested | tests | test_file_path | file_path | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| extract_iocs | function | py | cybersecurity | 1.0.0 | pure | def extract_iocs(text: str, types: list[str] | None = None) -> list[dict] | Pipeline puro que corre todos los extractores de IoC (IP, email, dominio, hash, wallet, CVE, MAC, telefono) y devuelve lista unificada con `type`. Deduplica spans contenidos. Si types se pasa, filtra los tipos a ejecutar. |
|
|
false |
|
lista de dicts {value, start, end, type, ...} ordenada por offset, sin spans contenidos | true |
|
python/functions/cybersecurity/tests/test_extract_iocs.py | python/functions/cybersecurity/extract_iocs.py |
Ejemplo
extract_iocs("Reach alice@example.com from 10.0.0.5; CVE-2023-1234")
# [{"value": "alice@example.com", "start": 6, "end": 23, "type": "email"},
# {"value": "10.0.0.5", "start": 29, "end": 37, "type": "ip_address"},
# {"value": "CVE-2023-1234", "start": 39, "end": 52, "type": "cve_id"}]
extract_iocs("Only IPs: 8.8.8.8 here", types=["ip_address"])
# [{"value": "8.8.8.8", ..., "type": "ip_address"}]
Notas
Es funcion y no kind: pipeline porque la regla del registry exige que pipelines sean impuros — esta no lo es: solo compone funciones puras y deduplica. Mantiene purity: pure con uses_functions no vacio.
Deduplicacion: un match completamente contenido en otro (ej. example.com dentro de alice@example.com) se descarta. Empate exacto de span: gana el primero segun el orden de _EXTRACTORS en el modulo (email > ip > crypto_wallet > cve > mac > file_hash > phone > domain). Reordenar el dict cambia la prioridad si tienes overlaps habituales.
Bench informal: ~50-80 ms por MB de texto sobre CPU moderna (depende del numero de matches).