egutierrez
a03675113a
- .claude/agents/fn-orquestador/SKILL.md - .claude/commands/fn_claude.md - .claude/rules/INDEX.md - .claude/rules/cpp_apps.md - .claude/rules/ids_naming.md - CHANGELOG.md - apps/dag_engine/README.md - apps/dag_engine/api.go - apps/dag_engine/dags_migrated/example.yaml - apps/dag_engine/dags_migrated/example_lineage_tracking.yaml - ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
82 lines
3.7 KiB
Markdown
82 lines
3.7 KiB
Markdown
# Sink — Funciones que escriben datos a destinos externos
|
|
|
|
Tag: `sink`. Grupo de funciones que **escriben** datos a un destino externo: BD, archivo, dashboard, alerta, email. Input: datos. Output: efecto observable. Ultimo eslabon del flujo en `data_factory` (analogia Factorio = rocket silo).
|
|
|
|
Filtro MCP: `mcp__registry__fn_search query="" tag="sink"`.
|
|
|
|
## Funciones del grupo
|
|
|
|
| ID | Lang | Destino |
|
|
|---|---|---|
|
|
| [bq_insert_rows_py_infra](../../python/functions/infra/bq_insert_rows.md) | py | BigQuery streaming insert |
|
|
| [bq_load_from_file_py_infra](../../python/functions/infra/bq_load_from_file.md) | py | BigQuery batch load (file) |
|
|
| [bq_load_from_gcs_py_infra](../../python/functions/infra/bq_load_from_gcs.md) | py | BigQuery batch load (GCS) |
|
|
| [bq_export_to_gcs_py_infra](../../python/functions/infra/bq_export_to_gcs.md) | py | BigQuery -> GCS export |
|
|
| [metabase_create_card_py_infra](../../python/functions/infra/metabase_create_card.md) | py | Metabase saved question |
|
|
| [metabase_export_card_py_infra](../../python/functions/infra/metabase_export_card.md) | py | Metabase card export (CSV/Excel) |
|
|
| [metabase_create_dashboard_subscription_py_infra](../../python/functions/infra/metabase_create_dashboard_subscription.md) | py | Dashboard email subscription |
|
|
| [metabase_create_card_alert_py_infra](../../python/functions/infra/metabase_create_card_alert.md) | py | Card alert (threshold/schedule) |
|
|
| [http_post_json_py_infra](../../python/functions/infra/http_post_json.md) | py | HTTP JSON POST |
|
|
| [http_post_json_go_infra](../../functions/infra/http_post_json.md) | go | HTTP JSON POST |
|
|
| [db_insert_row_go_infra](../../functions/infra/db_insert_row.md) | go | SQL row insert |
|
|
|
|
## Ejemplo canonico
|
|
|
|
Persistir resultados a BigQuery + crear card Metabase + alerta.
|
|
|
|
```python
|
|
from infra import bq_insert_rows, metabase_create_card, metabase_create_card_alert
|
|
|
|
# 1. Insertar filas a BQ
|
|
bq_insert_rows(
|
|
project_id="my-gcp-project",
|
|
dataset_id="analytics",
|
|
table_id="daily_kpis",
|
|
rows=results,
|
|
)
|
|
|
|
# 2. Card en Metabase apuntando a esa tabla
|
|
card = metabase_create_card(
|
|
name="Daily KPIs",
|
|
sql="SELECT * FROM `my-gcp-project.analytics.daily_kpis` WHERE date >= CURRENT_DATE() - 7",
|
|
database_id=3,
|
|
)
|
|
|
|
# 3. Alerta si la metrica clave cae
|
|
metabase_create_card_alert(
|
|
card_id=card["id"],
|
|
threshold={"goal": 100, "above_goal": False},
|
|
schedule="hourly",
|
|
)
|
|
```
|
|
|
|
## Fronteras del grupo
|
|
|
|
NO cubre:
|
|
- **Extract** (leer de fuente externa) -> [[extractor]].
|
|
- **Transform** (modificar datos sin efectos externos) -> [[transformer]].
|
|
- Escritura a la BD interna del registry (`registry.db`, `operations.db`) — tag `registry` o equivalente.
|
|
- Deploy de codigo (eso es `deploy`).
|
|
- Logs / telemetria propia (van a `call_monitor.db`, no son sinks de pipeline).
|
|
|
|
## Cuando NO usar `sink`
|
|
|
|
- Si la funcion escribe pero los datos NO salen del proceso actual (cache en memoria) -> no es sink.
|
|
- Si la funcion devuelve algo al caller sin efecto externo -> es transformer.
|
|
|
|
## Side effects observables
|
|
|
|
Por definicion, todos los sinks son **impuros** y tienen `error_type` definido. La salida del flujo data_factory es siempre visible: una fila nueva en BD, un email enviado, un dashboard actualizado.
|
|
|
|
## Consumidores
|
|
|
|
- `data_factory` — tab Sinks.
|
|
- `dag_engine` DAG steps con `function: <sink_id>`.
|
|
- Pipelines de produccion (con secrets de credenciales fuera del registry).
|
|
|
|
## Notas
|
|
|
|
- BQ load/insert/export todos como sink — escriben a recurso externo.
|
|
- Metabase notifications (subscription + alert) son sinks "puros" en el sentido de que producen efectos sin cambiar datos.
|
|
- HTTP POST cuenta como sink solo cuando se usa para **enviar** datos (webhook, API target), no para "obtener via POST".
|