dataforge/fn_registry
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

chore: auto-commit (43 archivos)

Browse Source 
- .mcp.json
- bash/functions/infra/write_mcp_jupyter_config.md
- bash/functions/infra/write_mcp_jupyter_config.sh
- cpp/CMakeLists.txt
- cpp/apps/chart_demo
- cpp/apps/shaders_lab
- cpp/functions/gfx/gl_framebuffer.cpp
- cpp/functions/gfx/gl_framebuffer.h
- cpp/functions/gfx/gl_framebuffer.md
- cpp/functions/gfx/mesh_gpu.md
- ...

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Egutierrez
2026-05-30 17:28:47 +02:00
parent a2efdcf003
commit fd5787c55f
44 changed files with 3924 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/functions/pipelines/extract_hls_from_cdp_tab.md
+91
View File
@@ -0,0 +1,91 @@
---
name: extract_hls_from_cdp_tab
kind: function
lang: py
domain: pipelines
version: "1.0.0"
purity: impure
signature: "def extract_hls_from_cdp_tab(debug_port: int = 9222, url_substring: str | None = None, live_capture_s: float = 6.0, timeout_s: float = 15.0) -> dict"
description: "Extrae URLs de manifiestos HLS (master.m3u8, index*.m3u8) de todas las pestañas e iframes de Chrome via CDP. Combina performance.getEntriesByType('resource') con escucha de eventos Network en vivo para capturar manifests ya cargados y los que se piden tras conectar."
tags: [navegator, cdp, hls, m3u8, scraping]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: [json, re, sys, os, time, urllib.request, threading, websocket]
params:
- name: debug_port
desc: "Puerto de remote debugging de Chrome (default 9222). Chrome debe lanzarse con --remote-debugging-port=9222."
- name: url_substring
desc: "Si se especifica, solo inspecciona targets (pestañas/iframes) cuyo URL contiene este substring. Ej: 'luluvdo.com'. Si None, inspecciona todos."
- name: live_capture_s
desc: "Segundos de escucha de eventos Network.requestWillBeSent/responseReceived en vivo por target (default 6.0). Caza manifests que se piden despues de conectar."
- name: timeout_s
desc: "Timeout en segundos para conexion WebSocket y operaciones recv por target (default 15.0)."
output: "dict {status: 'ok'|'error', targets: [{url, masters, variants}], all_m3u8: [lista plana deduplicada], error: str}. Si no hay m3u8: status ok, listas vacias."
tested: false
tests: []
test_file_path: ""
file_path: "python/functions/pipelines/extract_hls_from_cdp_tab.py"
---
## Ejemplo
```python
import sys
sys.path.insert(0, "python/functions")
from pipelines.extract_hls_from_cdp_tab import extract_hls_from_cdp_tab
# Chrome lanzado con --remote-debugging-port=9222 --remote-allow-origins=*
# reproduciendo un video en luluvdo.com (o cualquier player hls.js)
r = extract_hls_from_cdp_tab(debug_port=9222, url_substring="luluvdo.com")
print(r["all_m3u8"])
# ['https://cdn.luluvdo.com/.../urlset/master.m3u8?e=28800&t=abc123']
# Sin filtro: escanea todas las pestanas e iframes
r = extract_hls_from_cdp_tab(debug_port=9222)
for target in r["targets"]:
if target["masters"]:
print(f"Tab: {target['url']}")
print(f" Master: {target['masters']}")
print(f" Variantes: {target['variants']}")
```
Lanzar Chrome con las flags necesarias:
```bash
# Desde WSL (Chrome Windows)
/mnt/c/Program\ Files/Google/Chrome/Application/chrome.exe \
--remote-debugging-port=9222 \
--remote-allow-origins="*" \
--user-data-dir=/tmp/chrome_cdp_profile
```
Con proxy NordVPN (para sitios bloqueados por ISP):
```bash
# 1. Levantar bridge
source bash/functions/infra/start_nordvpn_socks_bridge.sh
start_nordvpn_socks_bridge --port 8889
# 2. Lanzar Chrome con proxy
chrome.exe --proxy-server=http://127.0.0.1:8889 --remote-debugging-port=9222 --remote-allow-origins="*"
# 3. Extraer HLS
r = extract_hls_from_cdp_tab(9222)
# 4. Descargar con el mismo proxy
# yt-dlp --proxy http://127.0.0.1:8889 --referer https://luluvdo.com/ r["all_m3u8"][0]
```
## Cuando usarla
Cuando un video se esta reproduciendo en Chrome controlado por CDP y quieres la URL del HLS (master.m3u8) sin reversear el player. Funciona con luluvdo.com, streamwish, filemoon y cualquier player hls.js porque lee lo que el navegador YA pidio al CDN, sin tocar codigo ofuscado. El player ya descifro/cargo el manifest — tu solo lo recoges.
Combina con `start_nordvpn_socks_bridge_bash_infra` cuando el sitio esta bloqueado por ISP: Chrome con --proxy-server apunta al bridge local.
## Gotchas
- **`--remote-allow-origins=*` OBLIGATORIO**: Chrome rechaza conexiones CDP con 403 si falta este flag (o el origin concreto). El header `Origin: http://localhost` en `create_connection` tambien es necesario por el mismo motivo.
- **El `<video>` tiene src `blob:`** (hls.js demuxa en memoria) — NO sirve para obtener el manifest. El URL real solo aparece en performance entries o en eventos Network.
- **El player suele vivir en un iframe**, no en la pestana top — la funcion escanea TODOS los targets para no perderlo.
- **El token del master.m3u8 caduca** (ej. `e=28800` = ~8h) y suele estar ligado a la IP de salida. Descargar desde la MISMA IP y con el mismo proxy. Añadir el Referer del host del player: `yt-dlp --proxy <proxy> --referer https://<host>/ <master.m3u8>`.
- **websocket-client** debe estar en el venv: `uv pip install websocket-client`. Si falta, el error indica el comando exacto.
- **live_capture_s=0** desactiva la escucha en vivo (solo lee performance entries). Util si el video ya esta cargado y quieres rapidez.
- Si `url_substring` es demasiado especifico y el iframe tiene otro dominio (CDN), no pasara el filtro. Usar None para ver todos los targets y luego ajustar.
python/functions/pipelines/extract_hls_from_cdp_tab.py
+258
View File
@@ -0,0 +1,258 @@
"""Extrae URLs de manifiestos HLS (master.m3u8, index*.m3u8) de tabs Chrome via CDP."""
import json
import re
import sys
import os
import time
import urllib.request
import urllib.error
import threading
HLS_PATTERN = re.compile(r'\.m3u8', re.IGNORECASE)
DATA_URL_PATTERN = re.compile(r'^data:', re.IGNORECASE)
def _get_targets(debug_port: int) -> list:
"""Obtiene la lista de targets via /json/list."""
url = f"http://127.0.0.1:{debug_port}/json/list"
try:
req = urllib.request.Request(url)
with urllib.request.urlopen(req, timeout=5) as resp:
return json.loads(resp.read())
except Exception as e:
raise RuntimeError(f"No se pudo conectar a Chrome en {debug_port}: {e}. "
"Asegurate de que Chrome esta corriendo con --remote-debugging-port.") from e
def _send_recv(ws, msg_id: int, method: str, params: dict, timeout: float = 10.0) -> dict:
"""Envía un mensaje CDP y espera la respuesta con el mismo id, ignorando eventos."""
import websocket as ws_module
payload = json.dumps({"id": msg_id, "method": method, "params": params})
ws.send(payload)
deadline = time.time() + timeout
while time.time() < deadline:
remaining = deadline - time.time()
if remaining <= 0:
break
ws.sock.settimeout(min(remaining, 1.0))
try:
raw = ws.recv()
except ws_module.WebSocketTimeoutException:
continue
except Exception:
break
try:
msg = json.loads(raw)
except Exception:
continue
if msg.get("id") == msg_id:
return msg
return {}
def _collect_from_target(ws_url: str, timeout_s: float, live_capture_s: float) -> dict:
"""
Conecta a un target CDP y extrae URLs m3u8 via:
- Method A: performance.getEntriesByType('resource')
- Method B: Network events en vivo durante live_capture_s segundos
"""
try:
import websocket as ws_module
except ImportError as e:
raise ImportError(
"websocket-client no esta instalado. "
"Instalar con: uv pip install websocket-client"
) from e
masters: list[str] = []
variants: list[str] = []
seen: set[str] = set()
def classify(url: str) -> None:
if not url or DATA_URL_PATTERN.match(url):
return
if not HLS_PATTERN.search(url):
return
if url in seen:
return
seen.add(url)
# "master" en el path o sin segmento de variante = master manifest
if re.search(r'master', url, re.IGNORECASE):
masters.append(url)
else:
variants.append(url)
try:
ws = ws_module.create_connection(
ws_url,
timeout=timeout_s,
header=["Origin: http://localhost"],
)
except Exception as e:
return {"error": str(e), "masters": [], "variants": []}
try:
msg_id = 1
# Method A: performance entries ya acumuladas
result = _send_recv(ws, msg_id, "Runtime.evaluate", {
"expression": (
"JSON.stringify("
" performance.getEntriesByType('resource')"
" .map(e => e.name)"
" .filter(n => /\\.m3u8/i.test(n))"
")"
),
"returnByValue": True,
}, timeout=timeout_s)
msg_id += 1
if result and not result.get("error"):
try:
val = result.get("result", {}).get("result", {}).get("value", "[]")
for url in json.loads(val):
classify(url)
except Exception:
pass
# Method B: escucha Network events en vivo
_send_recv(ws, msg_id, "Network.enable", {}, timeout=timeout_s)
msg_id += 1
deadline = time.time() + live_capture_s
while time.time() < deadline:
remaining = deadline - time.time()
if remaining <= 0:
break
ws.sock.settimeout(min(remaining, 0.5))
try:
raw = ws.recv()
except ws_module.WebSocketTimeoutException:
continue
except Exception:
break
try:
msg = json.loads(raw)
except Exception:
continue
method = msg.get("method", "")
params = msg.get("params", {})
if method == "Network.requestWillBeSent":
classify(params.get("request", {}).get("url", ""))
elif method == "Network.responseReceived":
classify(params.get("response", {}).get("url", ""))
finally:
try:
ws.close()
except Exception:
pass
return {"error": "", "masters": masters, "variants": variants}
def extract_hls_from_cdp_tab(
debug_port: int = 9222,
url_substring: str | None = None,
live_capture_s: float = 6.0,
timeout_s: float = 15.0,
) -> dict:
"""
Extrae URLs de manifiestos HLS (.m3u8) de todas las pestañas e iframes
de un Chrome con remote debugging activo.
Args:
debug_port: Puerto de remote debugging de Chrome (default 9222).
url_substring: Si se especifica, solo inspecciona targets cuyo URL contiene este substring.
live_capture_s: Segundos de escucha de eventos Network en vivo por target.
timeout_s: Timeout de conexion websocket y recv por target.
Returns:
dict con status, targets, all_m3u8 (lista plana deduplicada), error.
"""
# Verificar que websocket-client esta disponible antes de empezar
try:
import websocket # noqa: F401
except ImportError as e:
return {
"status": "error",
"targets": [],
"all_m3u8": [],
"error": (
"websocket-client no esta instalado. "
"Instalar con: uv pip install websocket-client"
),
}
# 1. Obtener lista de targets
try:
all_targets = _get_targets(debug_port)
except RuntimeError as e:
return {
"status": "error",
"targets": [],
"all_m3u8": [],
"error": str(e),
}
# 2. Filtrar targets validos (page o iframe con wsUrl)
candidates = []
for t in all_targets:
t_type = t.get("type", "")
ws_url = t.get("webSocketDebuggerUrl", "")
t_url = t.get("url", "")
if t_type not in ("page", "iframe"):
continue
if not ws_url:
continue
if url_substring and url_substring not in t_url:
continue
candidates.append(t)
# 3. Inspeccionar cada target
results = []
all_m3u8_set: set[str] = set()
for t in candidates:
ws_url = t["webSocketDebuggerUrl"]
t_url = t.get("url", "")
data = _collect_from_target(ws_url, timeout_s=timeout_s, live_capture_s=live_capture_s)
for u in data["masters"] + data["variants"]:
all_m3u8_set.add(u)
results.append({
"url": t_url,
"masters": data["masters"],
"variants": data["variants"],
})
return {
"status": "ok",
"targets": results,
"all_m3u8": sorted(all_m3u8_set),
"error": "",
}
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Extrae URLs HLS de Chrome via CDP")
parser.add_argument("--debug-port", type=int, default=9222)
parser.add_argument("--url-substring", type=str, default=None)
parser.add_argument("--live-capture-s", type=float, default=6.0)
parser.add_argument("--timeout-s", type=float, default=15.0)
args = parser.parse_args()
result = extract_hls_from_cdp_tab(
debug_port=args.debug_port,
url_substring=args.url_substring,
live_capture_s=args.live_capture_s,
timeout_s=args.timeout_s,
)
print(json.dumps(result, indent=2))