chore: auto-commit (286 archivos)
- .claude/agents/fn-orquestador/SKILL.md - .claude/commands/fn_claude.md - .claude/rules/INDEX.md - .claude/rules/cpp_apps.md - .claude/rules/ids_naming.md - CHANGELOG.md - apps/dag_engine/README.md - apps/dag_engine/api.go - apps/dag_engine/dags_migrated/example.yaml - apps/dag_engine/dags_migrated/example_lineage_tracking.yaml - ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,81 @@
|
||||
# Sink — Funciones que escriben datos a destinos externos
|
||||
|
||||
Tag: `sink`. Grupo de funciones que **escriben** datos a un destino externo: BD, archivo, dashboard, alerta, email. Input: datos. Output: efecto observable. Ultimo eslabon del flujo en `data_factory` (analogia Factorio = rocket silo).
|
||||
|
||||
Filtro MCP: `mcp__registry__fn_search query="" tag="sink"`.
|
||||
|
||||
## Funciones del grupo
|
||||
|
||||
| ID | Lang | Destino |
|
||||
|---|---|---|
|
||||
| [bq_insert_rows_py_infra](../../python/functions/infra/bq_insert_rows.md) | py | BigQuery streaming insert |
|
||||
| [bq_load_from_file_py_infra](../../python/functions/infra/bq_load_from_file.md) | py | BigQuery batch load (file) |
|
||||
| [bq_load_from_gcs_py_infra](../../python/functions/infra/bq_load_from_gcs.md) | py | BigQuery batch load (GCS) |
|
||||
| [bq_export_to_gcs_py_infra](../../python/functions/infra/bq_export_to_gcs.md) | py | BigQuery -> GCS export |
|
||||
| [metabase_create_card_py_infra](../../python/functions/infra/metabase_create_card.md) | py | Metabase saved question |
|
||||
| [metabase_export_card_py_infra](../../python/functions/infra/metabase_export_card.md) | py | Metabase card export (CSV/Excel) |
|
||||
| [metabase_create_dashboard_subscription_py_infra](../../python/functions/infra/metabase_create_dashboard_subscription.md) | py | Dashboard email subscription |
|
||||
| [metabase_create_card_alert_py_infra](../../python/functions/infra/metabase_create_card_alert.md) | py | Card alert (threshold/schedule) |
|
||||
| [http_post_json_py_infra](../../python/functions/infra/http_post_json.md) | py | HTTP JSON POST |
|
||||
| [http_post_json_go_infra](../../functions/infra/http_post_json.md) | go | HTTP JSON POST |
|
||||
| [db_insert_row_go_infra](../../functions/infra/db_insert_row.md) | go | SQL row insert |
|
||||
|
||||
## Ejemplo canonico
|
||||
|
||||
Persistir resultados a BigQuery + crear card Metabase + alerta.
|
||||
|
||||
```python
|
||||
from infra import bq_insert_rows, metabase_create_card, metabase_create_card_alert
|
||||
|
||||
# 1. Insertar filas a BQ
|
||||
bq_insert_rows(
|
||||
project_id="my-gcp-project",
|
||||
dataset_id="analytics",
|
||||
table_id="daily_kpis",
|
||||
rows=results,
|
||||
)
|
||||
|
||||
# 2. Card en Metabase apuntando a esa tabla
|
||||
card = metabase_create_card(
|
||||
name="Daily KPIs",
|
||||
sql="SELECT * FROM `my-gcp-project.analytics.daily_kpis` WHERE date >= CURRENT_DATE() - 7",
|
||||
database_id=3,
|
||||
)
|
||||
|
||||
# 3. Alerta si la metrica clave cae
|
||||
metabase_create_card_alert(
|
||||
card_id=card["id"],
|
||||
threshold={"goal": 100, "above_goal": False},
|
||||
schedule="hourly",
|
||||
)
|
||||
```
|
||||
|
||||
## Fronteras del grupo
|
||||
|
||||
NO cubre:
|
||||
- **Extract** (leer de fuente externa) -> [[extractor]].
|
||||
- **Transform** (modificar datos sin efectos externos) -> [[transformer]].
|
||||
- Escritura a la BD interna del registry (`registry.db`, `operations.db`) — tag `registry` o equivalente.
|
||||
- Deploy de codigo (eso es `deploy`).
|
||||
- Logs / telemetria propia (van a `call_monitor.db`, no son sinks de pipeline).
|
||||
|
||||
## Cuando NO usar `sink`
|
||||
|
||||
- Si la funcion escribe pero los datos NO salen del proceso actual (cache en memoria) -> no es sink.
|
||||
- Si la funcion devuelve algo al caller sin efecto externo -> es transformer.
|
||||
|
||||
## Side effects observables
|
||||
|
||||
Por definicion, todos los sinks son **impuros** y tienen `error_type` definido. La salida del flujo data_factory es siempre visible: una fila nueva en BD, un email enviado, un dashboard actualizado.
|
||||
|
||||
## Consumidores
|
||||
|
||||
- `data_factory` — tab Sinks.
|
||||
- `dag_engine` DAG steps con `function: <sink_id>`.
|
||||
- Pipelines de produccion (con secrets de credenciales fuera del registry).
|
||||
|
||||
## Notas
|
||||
|
||||
- BQ load/insert/export todos como sink — escriben a recurso externo.
|
||||
- Metabase notifications (subscription + alert) son sinks "puros" en el sentido de que producen efectos sin cambiar datos.
|
||||
- HTTP POST cuenta como sink solo cuando se usa para **enviar** datos (webhook, API target), no para "obtener via POST".
|
||||
Reference in New Issue
Block a user