egutierrez
5c7871dc86
- Include viz/data_table.h; add g_dt_collectors + g_dt_results State members - ##collectors (3 cols): flat cells array -> data_table::render, Text renderers - ##results (4 cols): flat cells array -> data_table::render, Badge on Kind col (function=#3b82f6, pipeline=#8b5cf6, component=#f59e0b) - CMakeLists.txt: add fn_table_viz guard link block - app.md: populate uses_functions with Wave-1 stack IDs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
75 lines
3.4 KiB
Markdown
75 lines
3.4 KiB
Markdown
---
|
|
name: odr_console
|
|
lang: cpp
|
|
domain: tools
|
|
description: "Lanzador GUI de funciones del registry para recolectar datos online. Panel de busqueda FTS5, jobs queue async (workers concurrentes), pipeline builder DAG, browser DuckDB, assertions/proposals. Aplica bucle reactivo de 5 pasos sobre operations.db propia."
|
|
tags: [imgui, data-collection, scraping, duckdb, jobs, cdp]
|
|
uses_functions:
|
|
- data_table_cpp_viz
|
|
- viz_render_cpp_viz
|
|
- compute_stage_cpp_core
|
|
- tql_to_sql_cpp_core
|
|
- llm_anthropic_cpp_core
|
|
uses_types: []
|
|
framework: "imgui"
|
|
entry_point: "main.cpp"
|
|
dir_path: "projects/online_data_recopilation/apps/odr_console"
|
|
repo_url: ""
|
|
---
|
|
|
|
## Notas
|
|
|
|
App C++ ImGui que orquesta:
|
|
|
|
1. **Launcher panel** — busqueda FTS5 sobre `registry.db`. Lanza cualquier funcion/pipeline con form auto-generado desde `params_schema`.
|
|
2. **Pipeline builder** — DAG visual con `imgui_node_editor`. Compone collectors validando composabilidad (`returns` ↔ `uses_types`). Persiste en `operations.relations` con status `designed`.
|
|
3. **Jobs queue** — Pool de N workers (default 4). Cada job = subprocess Python collector. Live progress por panel (`PROGRESS:` en stderr). Reusa `jobs_pool_cpp_core` (extraido de osint_graph en issue 0065).
|
|
4. **Datasets browser** — Panel DuckDB embebido. Query editor + tabla preview + ImPlot para charts. Lee parquet de `vaults/odr_data/`.
|
|
5. **Entities + assertions** — Vista de `operations.entities` por dataset. Editor SQL para assertions. Boton "Eval --react" lanza paso 4 del bucle.
|
|
6. **Proposals inbox** — Lista pending de `registry.proposals` originadas por assertions fallidas.
|
|
|
|
### Estructura
|
|
|
|
```
|
|
odr_console/
|
|
main.cpp # fn::run_app + render() + paneles
|
|
views_launcher.cpp # Panel 1
|
|
views_pipelines.cpp # Panel 2
|
|
views_jobs.cpp # Panel 3
|
|
views_datasets.cpp # Panel 4
|
|
views_assertions.cpp # Panel 5
|
|
views_proposals.cpp # Panel 6
|
|
data_registry.cpp # Lee registry.db (FTS5, funcs, types)
|
|
data_operations.cpp # CRUD operations.db (relations/executions/entities/assertions)
|
|
data_duck.cpp # DuckDB connector + ingest
|
|
collectors/ # Mismo schema que enrichers de graph_explorer
|
|
api_hn_top/ # MVP: HackerNews top stories via API
|
|
manifest.yaml
|
|
run.py
|
|
migrations/ # operations.db migrations (esquema 5-pasos)
|
|
CMakeLists.txt
|
|
```
|
|
|
|
### Local files (regla cpp_apps §7)
|
|
|
|
- `local_files/odr_console.ini` — settings persistidas
|
|
- `local_files/imgui.ini` — layout
|
|
- `local_files/odr.duckdb` — DuckDB embebido (datos crudos pequeños)
|
|
- `local_files/cache/<sha[0:2]>/<sha>.{html,json,parquet}` — cache addressable
|
|
- `operations.db` queda en el dir del exe (consultable por `fn ops`)
|
|
|
|
### Decisiones tomadas
|
|
|
|
| Tema | Decision |
|
|
|---|---|
|
|
| Workers default | 4 (mas que graph_explorer porque crawls esperan red) |
|
|
| operations.db | Una unica por la app |
|
|
| DuckDB | Embebido (linkar libduckdb), no subprocess |
|
|
| Collectors lang inicial | Python (espejo graph_explorer enrichers) |
|
|
| Browser | CDP via `cdp-cli` Go (issue 0038) cuando aplica |
|
|
|
|
### Decisiones pendientes
|
|
|
|
- Refactor jobs system: en paralelo a MVP. Ver issue 0065.
|
|
- Schema operations.db: requiere migracion 001 con relations/executions/entities/types_snapshot/assertions/assertion_results (ver `fn_operations/migrations/`).
|