dataforge/web_scraping
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: auto-commit (4 archivos)

Browse Source 
- .gitignore
- CAPABILITIES_TODO.md
- CHROMIUM_SYSTEM.md
- hoppscotch/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Egutierrez
2026-06-11 00:16:47 +02:00
parent 0b3a1313c2
commit 65ca2b3d43
12 changed files with 745 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hoppscotch/README.md
+145
View File
@@ -0,0 +1,145 @@
# Hoppscotch — consola de APIs + colecciones versionadas
GUI para explorar/probar APIs a mano (Hoppscotch Desktop) con las consultas guardadas
como JSON versionado en este sub-repo, y un puente bidireccional con el motor de replay
del registry para **lanzar y automatizar** esas mismas peticiones desde terminal o scripts.
Grupo de capacidad: `hoppscotch`. Página madre: `docs/capabilities/hoppscotch.md`.
## El ciclo
```
GUI Hoppscotch ──export .json──▶ parse_hoppscotch_collection ──call specs──▶ run_hoppscotch_collection ──▶ http_replay_sequence
▲ (automatiza, headless)
└──── build_hoppscotch_collection ◀── call specs (de HAR destilado, o a mano) ────────────────────────────┘
```
- **GUI → automatización**: editas/creas requests en la app, exportas la colección a
`collections/<algo>.json`, y `run_hoppscotch_collection` las ejecuta sin tocar la GUI.
- **Automatización → GUI**: tienes call specs (p.ej. de un flujo grabado con HAR y destilado
con `har_extract_calls`), `build_hoppscotch_collection` te da un `.json` importable en la app
para inspeccionarlas visualmente.
## Estructura
```
hoppscotch/
collections/ # Colecciones .json VERSIONADAS (sin secretos)
registry_api.json # ejemplo: 4 endpoints GET del registry_api
environments/ # Variables de entorno Hoppscotch
.gitignore # ignora *.json (secretos) salvo *.example.json
registry_api.example.json # plantilla VERSIONADA (valores placeholder)
registry_api.json # real, GITIGNORED (puede llevar tokens)
README.md
```
Regla de secretos: las **colecciones** se versionan; los **environments reales** no (llevan
tokens/basicAuth). Solo la plantilla `*.example.json` viaja en git. En las requests, los valores
sensibles van como variable de environment con la sintaxis Hoppscotch `<<nombre>>`, nunca
hardcodeados.
## Lanzar la app Desktop
Binario instalado por el `.deb`: `/usr/bin/hoppscotch-desktop` (menú de aplicaciones: "Hoppscotch").
Lanzado desde una sesión gráfica normal o, de forma aislada:
```bash
systemd-run --user --unit=hoppscotch-gui --setenv=DISPLAY=:0 /usr/bin/hoppscotch-desktop
```
### Importar una colección en la GUI
1. Panel **Collections** (izquierda) → menú **Import / Export**.
2. **Import** → **Hoppscotch** → **Import from File**.
3. Selecciona `collections/registry_api.json`.
4. (Opcional) Importa también el environment: panel **Environments** → Import →
`environments/registry_api.example.json` y rellena los valores reales (no se versionan).
5. Selecciona el environment activo arriba a la derecha y pulsa **Send** en cualquier request.
### Exportar / ver el JSON que genera la GUI
Collections → menú → **Export** → **Hoppscotch** → guarda en `collections/`. Así se ve el
esquema nativo de la versión instalada (la app migra al esquema más reciente al importar/exportar).
`parse_hoppscotch_collection` lee cualquier versión v1..v12 por campos estables.
## Automatizar desde el registry
Lanzar toda una colección o un subconjunto, headless, con sustitución de variables:
```python
import sys, os
sys.path.insert(0, os.path.join(os.path.expanduser("~/fn_registry"), "python", "functions"))
from infra.run_hoppscotch_collection import run_hoppscotch_collection
res = run_hoppscotch_collection(
"projects/web_scraping/hoppscotch/collections/registry_api.json",
environment_path="projects/web_scraping/hoppscotch/environments/registry_api.json",
params={"baseURL": "https://registry.organic-machine.com"}, # pisa el environment
only=["status", "locations"], # filtra por nombre de request; None = todas
)
print(res["status"])
for s in res["steps"]:
print(s["method"], s["url"], "->", s["status_code"], "ok" if s["ok"] else "FAIL")
```
`run_hoppscotch_collection` convierte la sintaxis Hoppscotch `<<var>>` a la del motor de replay
`{{var}}`, mergea environment + params (params del caller ganan), filtra por `only`, y reproduce
las peticiones sobre una sesión HTTP compartida (cookie jar entre pasos).
## Añadir peticiones rápido
Sin editar el `.json` a mano, con `add_hoppscotch_request` (reusa el mapeo de `build`, no
destruye el archivo si el JSON es inválido):
```python
from infra.add_hoppscotch_request import add_hoppscotch_request
r = add_hoppscotch_request(
"projects/web_scraping/hoppscotch/collections/registry_api.json",
"GET", "<<baseURL>>/api/search?q=metabase&kind=function",
name="search metabase", headers={"Accept": "application/json"})
print(r["status"], r["total_requests"]) # ok 8
```
Tras añadir, re-importa la colección en la app Desktop (no recarga el `.json` en caliente) o
córrela headless con `run_hoppscotch_collection` / `hopp test`.
## Validar la colección
Dos motores independientes, ambos verde sobre `registry_api.json` (7 requests → 7×200):
```bash
# Motor nativo Hoppscotch (corre también preRequestScript/testScript JS):
hopp test collections/registry_api.json -e environments/registry_api.cli.json
# Motor del registry (integrado, telemetría, params/extract):
python3 -c 'from infra.run_hoppscotch_collection import run_hoppscotch_collection as r; \
print(r("collections/registry_api.json", environment_path="environments/registry_api.json")["status"])'
```
## Funciones del registry
| ID | Qué hace |
|---|---|
| `build_hoppscotch_collection_py_infra` | call specs → dict de colección Hoppscotch (importable en la GUI) |
| `parse_hoppscotch_collection_py_infra` | dict de colección Hoppscotch (v1..v12) → call specs |
| `run_hoppscotch_collection_py_infra` | lee colección + environment .json y ejecuta vía `http_replay_sequence` |
| `add_hoppscotch_request_py_infra` | añade una petición a una colección `.json` existente (reusa `build`, no destructivo) |
| `http_replay_sequence_py_infra` | motor de replay HTTP (secuencia con sesión, `{{param}}`, extract) |
| `har_extract_calls_py_cybersecurity` | HAR destilado → call specs (origen del lado izquierdo del ciclo) |
## Gotchas
- **Sintaxis de variables**: la GUI usa `<<var>>`; el motor de replay usa `{{var}}`.
`run_hoppscotch_collection` traduce automáticamente. Si compones a mano con
`http_replay_sequence`, traduce tú (`<<x>>` → `{{x}}`).
- **Endpoints con efecto** (POST/DELETE destructivos): filtra con `only` para no ejecutar a
ciegas toda la colección. El replay reproduce lo que haya; la responsabilidad de no disparar
acciones irreversibles es del caller.
- **Esquema de versión**: `build_*` genera el formato canónico estable v1/v2 que la app migra al
importar. `parse_*` lee cualquier versión por campos. No fijamos v12/v17 a mano para no romper
con cambios de esquema upstream.
- **multipart/form-data**: `parse_*` lo marca `body_type="raw"` y no reconstruye el cuerpo binario.
- **Formato dual de environment**: la GUI Desktop importa el array `[{v,name,variables:[{key,value,secret}]}]`
(`registry_api.json` / `.example.json`); el CLI `hopp` v0.31.2 quiere un **objeto plano**
`{name,variables:[{key,value}]}` SIN `v` ni `secret` (`registry_api.cli.json`). Pasar el array a
`hopp test -e` da `MALFORMED_ENV_FILE`. `run_hoppscotch_collection` (registry) usa el array.
hoppscotch/collections/licenseplatedata.json
+97
View File
@@ -0,0 +1,97 @@
{
"v": 1,
"name": "licenseplatedata",
"folders": [],
"requests": [
{
"v": "2",
"endpoint": "<<lpd_baseURL>>/plate-to-vin?plate=<<plate>>&state=<<state>>",
"name": "plate-to-vin",
"params": [],
"headers": [
{
"key": "X-API-Key",
"value": "<<lpd_api_key>>",
"active": true
},
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<lpd_baseURL>>/vin/<<vin>>",
"name": "decode VIN",
"params": [],
"headers": [
{
"key": "X-API-Key",
"value": "<<lpd_api_key>>",
"active": true
},
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<lpd_baseURL>>/vehicle-images?vin=<<vin>>",
"name": "vehicle-images",
"params": [],
"headers": [
{
"key": "X-API-Key",
"value": "<<lpd_api_key>>",
"active": true
},
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
}
]
}
hoppscotch/collections/plate_to_vin.json
+112
View File
@@ -0,0 +1,112 @@
{
"v": 1,
"name": "Plate to VIN (free APIs)",
"folders": [],
"requests": [
{
"v": "2",
"endpoint": "https://api.auto.dev/plate/<<state>>/<<plate>>",
"name": "autodev_plate_to_vin",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
},
{
"key": "Authorization",
"value": "Bearer <<auto_dev_key>>",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "https://api.carsxe.com/v2/platedecoder?key=<<carsxe_key>>&plate=<<plate>>&state=<<state>>",
"name": "carsxe_plate_decoder",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVinValues/<<vin>>?format=json",
"name": "nhtsa_decode_vin_values",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "https://vpic.nhtsa.dot.gov/api/vehicles/DecodeVin/<<vin>>?format=json",
"name": "nhtsa_decode_vin",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
}
]
}
hoppscotch/collections/registry_api.json
+182
View File
@@ -0,0 +1,182 @@
{
"v": 1,
"name": "registry_api",
"folders": [],
"requests": [
{
"v": "2",
"endpoint": "<<baseURL>>/api/status",
"name": "status",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/search?q=slice",
"name": "search basico",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/search?q=http&lang=py&kind=function",
"name": "search filtrado (py/function)",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/locations",
"name": "locations (todos)",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/locations/<<pc>>",
"name": "locations por PC",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/functions/filter_slice_go_core",
"name": "show funcion",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
},
{
"v": "2",
"endpoint": "<<baseURL>>/api/search?q=metabase&kind=function",
"name": "search metabase",
"params": [],
"headers": [
{
"key": "Accept",
"value": "application/json",
"active": true
}
],
"method": "GET",
"auth": {
"authType": "none",
"authActive": true
},
"preRequestScript": "",
"testScript": "",
"body": {
"contentType": null,
"body": null
},
"requestVariables": []
}
]
}
hoppscotch/environments/.gitignore
+4
View File
@@ -0,0 +1,4 @@
# Environments de Hoppscotch llevan secretos (tokens, basicAuth). NUNCA se versionan.
# Solo se versiona la plantilla *.example.json (valores con <<placeholder>>).
*.json
!*.example.json
hoppscotch/environments/registry_api.example.json
+22
View File
@@ -0,0 +1,22 @@
[
{
"v": 1,
"name": "registry_api",
"variables": [
{
"key": "baseURL",
"value": "https://registry.organic-machine.com",
"secret": false
},
{
"key": "pc",
"value": "lucas-linux",
"secret": false
},
{
"key": "registryToken",
"secret": true
}
]
}
]
hoppscotch/selfhost/.env.example
+14
View File
@@ -0,0 +1,14 @@
# Generado 2026-06-10T20:27:47Z. Secretos reales — NO versionar (ver .gitignore).
DATABASE_URL=postgresql://postgres:<password>@db:5432/hoppscotch?connect_timeout=300
POSTGRES_PASSWORD=<genera-con-openssl-rand-hex-16>
DATA_ENCRYPTION_KEY=<32-chars-openssl-rand-hex-16>
WHITELISTED_ORIGINS=http://localhost:3170,http://localhost:3009,http://localhost:3100,app://localhost_3200,app://hoppscotch
PROXY_APP_URL=https://proxy.hoppscotch.io
TRUST_PROXY=false
VITE_BASE_URL=http://localhost:3009
VITE_SHORTCODE_BASE_URL=http://localhost:3009
VITE_ADMIN_URL=http://localhost:3100
VITE_BACKEND_GQL_URL=http://localhost:3170/graphql
VITE_BACKEND_WS_URL=ws://localhost:3170/graphql
VITE_BACKEND_API_URL=http://localhost:3170/v1
ENABLE_SUBPATH_BASED_ACCESS=false
hoppscotch/selfhost/.gitignore
+7
View File
@@ -0,0 +1,7 @@
# El .env lleva secretos (DATA_ENCRYPTION_KEY, POSTGRES_PASSWORD) → nunca se versiona.
# Solo viaja la plantilla .env.example.
.env
# Archivos oficiales descargados como referencia (no son nuestra config).
.env.example.upstream
docker-compose.upstream.yml
hoppscotch/selfhost/docker-compose.yml
+60
View File
@@ -0,0 +1,60 @@
# Hoppscotch self-hosted (Community Edition) — stack local para trabajar en conjunto.
#
# Arquitectura: imagen all-in-one pre-built (app 3000 + admin 3100 + backend GraphQL 3170
# + bundle 3200) + Postgres 15 + Mailpit (captura el magic link del login, sin OAuth externo
# ni SMTP real). Todo escucha solo en 127.0.0.1 (no se expone a la red).
#
# El puerto 3000 del host está ocupado por otro proceso, así que la app se publica en 3009.
# Las VITE_* del .env quedan coherentes con ese remapeo (el AIO las inyecta en runtime).
#
# Levantar: docker compose up -d
# Logs: docker compose logs -f hopp_aio
# Parar: docker compose down (los datos persisten en el volumen hopp_pgdata)
# Reset total: docker compose down -v (BORRA la base de datos)
services:
db:
image: postgres:15
container_name: hopp_db
restart: unless-stopped
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: hoppscotch
volumes:
- hopp_pgdata:/var/lib/postgresql/data
ports:
- "127.0.0.1:5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d hoppscotch"]
interval: 5s
timeout: 5s
retries: 20
# Mailpit: servidor SMTP de pruebas. Recibe el correo del magic link en el SMTP interno
# (mail:1025) y lo expone en una API/UI web (127.0.0.1:8025) desde donde se lee el enlace.
mail:
image: axllent/mailpit:latest
container_name: hopp_mail
restart: unless-stopped
ports:
- "127.0.0.1:8025:8025" # UI + API REST de Mailpit
# SMTP en el puerto interno 1025, accesible por la red de docker como mail:1025
aio:
image: hoppscotch/hoppscotch:2026.5.0
container_name: hopp_aio
restart: unless-stopped
env_file:
- ./.env
depends_on:
db:
condition: service_healthy
ports:
- "127.0.0.1:3009:3000" # app web (3000 del host está ocupado)
- "127.0.0.1:3100:3100" # admin dashboard
- "127.0.0.1:3170:3170" # backend: GraphQL /graphql, REST /v1
- "127.0.0.1:3200:3200" # bundle server
volumes:
hopp_pgdata: