dataforge/fn_registry
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(pipelines): auto-commit con 4 cambios

Browse Source 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Egutierrez
2026-05-14 02:14:09 +02:00
parent ca1bf5a59b
commit b5a56bb5ff
4 changed files with 312 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bash/functions/pipelines/clone_project_subrepos.md
+79
View File
@@ -0,0 +1,79 @@
---
name: clone_project_subrepos
kind: function
lang: bash
domain: pipelines
version: "1.0.0"
purity: impure
signature: "clone_project_subrepos <project_id> [--owner OWNER] [--dry-run]"
description: "Clona todos los sub-repos Gitea de los artefactos (apps + analysis, NO vaults) de un project. Util en PC recien clonado para traer un project entero sin clonar uno a uno."
tags: [git, gitea, project, launcher]
uses_functions: []
uses_types: []
returns: []
returns_optional: false
error_type: "error_go_core"
imports: []
params:
- name: "project_id"
desc: "ID del project en tabla projects (ej. fn_monitoring, osint_graph)"
- name: "--owner"
desc: "Dueno Gitea de los repos, default dataforge"
- name: "--dry-run"
desc: "Modo simulacion: imprime las acciones sin ejecutarlas"
output: "Tabla de resultados por sub-repo: kind | name | status (cloned/skip/failed). Exit 0 si todo cloned o skip, 1 si algun failed."
tested: false
tests: []
test_file_path: ""
file_path: "bash/functions/pipelines/clone_project_subrepos.sh"
---
## Ejemplo
```bash
# Desde fn_registry recien clonado en PC nuevo
./fn run clone_project_subrepos fn_monitoring
# Resultado esperado:
# KIND NAME STATUS
# ---------- ----------------------------------- -------
# app call_monitor [cloned]
# app registry_dashboard [cloned]
# app sqlite_api [cloned]
# analysis domain_coverage_gaps [cloned]
#
# Siguiente paso sugerido:
# cd /home/lucas/fn_registry && CGO_ENABLED=1 ./fn index && ./fn sync
# Con owner alternativo
./fn run clone_project_subrepos aurgi --owner miorg
# Dry-run para ver que clonaria sin ejecutar
./fn run clone_project_subrepos osint_graph --dry-run
```
## Cuando usarla
Cuando llegas a un PC nuevo con solo fn_registry clonado y quieres trabajar en un project concreto sin clonar cada sub-repo a mano. Tambien tras `fn sync` cuando los `pc_locations` indican que faltan artefactos del project en este PC.
## Flujo interno
1. Valida que `registry.db` existe y que `project_id` esta en tabla `projects`.
2. Query SQL: `SELECT 'app'... FROM apps WHERE project_id=? UNION ALL SELECT 'analysis'... FROM analysis WHERE project_id=?` — vaults excluidos deliberadamente.
3. Por cada artefacto: si `dir_path/.git` existe → `[skip]`; si `repo_url` esta vacio en BD → construye `${GITEA_URL}/<owner>/<basename>`; ejecuta `git clone --branch master`.
4. Imprime tabla resumen. Sugiere `fn index && fn sync` al terminar.
## Variables de entorno
- `FN_REGISTRY_ROOT` — raiz del registry; default `/home/lucas/fn_registry`
- `GITEA_URL` — URL base de Gitea; default `https://gitea-dgg044oo04woo4ggcsws4gk0.organic-machine.com`
- Auth git/ssh: el pipeline confía en la config local del usuario (SSH key, credential helper)
## Gotchas
- **Vaults NO se clonan.** Son symlinks a datos externos; el usuario los crea aparte (`mkdir -p ~/vaults/... && ln -s ...`).
- **Si `repo_url` esta vacio en BD**, se construye como `${GITEA_URL}/<owner>/<basename(dir_path)>`. Si tu convencion de nombres Gitea difiere, rellena `repo_url` en los `app.md`/`analysis.md` antes de ejecutar y re-indexa.
- **Auth:** el script no inyecta credenciales. Si clone falla por auth, configura SSH key o credential helper (`git config --global credential.helper ...`) y relanza.
- **El project debe existir en tabla `projects`** y los artefactos llevar `project_id` correcto en su `.md`. Si no, el query no devuelve filas. Verificar con: `sqlite3 registry.db "SELECT id FROM projects;"`.
- **Branch `master` asumido.** Todos los repos del ecosistema usan `master` (ADR 0002). Si un repo usa otra branch, clonar manualmente.
- **Artefactos sin `dir_path`** quedan como `[skip: dir_path vacio en BD]`. Rellenar el campo en el `.md` y re-indexar.
bash/functions/pipelines/clone_project_subrepos.sh
+162
View File
@@ -0,0 +1,162 @@
#!/usr/bin/env bash
# Pipeline: clone_project_subrepos — Clona todos los sub-repos Gitea de un project
# (apps + analysis, NO vaults) en el PC actual. Util tras llegar a un PC nuevo.
set -euo pipefail
clone_project_subrepos() {
local project_id=""
local owner="dataforge"
local dry_run=0
# --- Parseo de argumentos ---
while [[ $# -gt 0 ]]; do
case "$1" in
--owner)
owner="$2"
shift 2
;;
--dry-run)
dry_run=1
shift
;;
-*)
echo "[error] Opcion desconocida: $1" >&2
echo "Uso: clone_project_subrepos <project_id> [--owner OWNER] [--dry-run]" >&2
return 1
;;
*)
project_id="$1"
shift
;;
esac
done
if [[ -z "$project_id" ]]; then
echo "[error] project_id requerido." >&2
echo "Uso: clone_project_subrepos <project_id> [--owner OWNER] [--dry-run]" >&2
return 1
fi
# --- Resolver paths ---
local registry_root="${FN_REGISTRY_ROOT:-/home/lucas/fn_registry}"
local db="$registry_root/registry.db"
local gitea_url="${GITEA_URL:-https://gitea-dgg044oo04woo4ggcsws4gk0.organic-machine.com}"
# --- Validar registry.db ---
if [[ ! -f "$db" ]]; then
echo "[error] registry.db no encontrado en $db" >&2
echo " Asegurate de que FN_REGISTRY_ROOT apunta a la raiz del registry." >&2
return 1
fi
if ! command -v sqlite3 >/dev/null 2>&1; then
echo "[error] sqlite3 no esta en PATH." >&2
return 1
fi
# --- Validar que el project existe ---
local project_check
project_check=$(sqlite3 "$db" "SELECT COUNT(*) FROM projects WHERE id = '$project_id';" 2>/dev/null || echo "0")
if [[ "$project_check" == "0" ]]; then
echo "[error] project_id '$project_id' no encontrado en tabla projects." >&2
echo " Proyectos disponibles:" >&2
sqlite3 "$db" "SELECT ' ' || id || ' — ' || COALESCE(description,'') FROM projects;" 2>/dev/null >&2 || true
return 1
fi
echo "=== clone_project_subrepos: project=$project_id ===" >&2
[[ "$dry_run" -eq 1 ]] && echo " [DRY-RUN: ninguna accion se ejecutara]" >&2
echo " owner=$owner gitea=$gitea_url" >&2
echo "" >&2
# --- Query: apps + analysis del project (NO vaults) ---
local rows
rows=$(sqlite3 "$db" \
"SELECT 'app', id, name, COALESCE(dir_path,''), COALESCE(repo_url,'') FROM apps WHERE project_id = '$project_id'
UNION ALL
SELECT 'analysis', id, name, COALESCE(dir_path,''), COALESCE(repo_url,'') FROM analysis WHERE project_id = '$project_id';" \
2>/dev/null || true)
if [[ -z "$rows" ]]; then
echo "[warn] No se encontraron apps ni analysis para project '$project_id'." >&2
echo " Verifica que los artefactos tienen project_id='$project_id' en sus .md y que se ha ejecutado fn index." >&2
return 0
fi
# --- Procesar cada fila ---
local summary_lines=()
local had_failures=0
while IFS='|' read -r kind id name dir_path repo_url; do
[[ -z "$dir_path" ]] && {
summary_lines+=("$(printf '%-10s %-35s %s' "$kind" "$name" "[skip: dir_path vacio en BD]")")
continue
}
local target_dir="$registry_root/$dir_path"
local basename_dir
basename_dir="$(basename "$dir_path")"
# Construir repo_url si no esta en BD
if [[ -z "$repo_url" ]]; then
repo_url="${gitea_url}/${owner}/${basename_dir}"
fi
# Comprobar si ya esta clonado
if [[ -d "$target_dir/.git" ]]; then
summary_lines+=("$(printf '%-10s %-35s %s' "$kind" "$name" "[skip] ya clonado")")
continue
fi
# Dry-run: solo imprimir
if [[ "$dry_run" -eq 1 ]]; then
summary_lines+=("$(printf '%-10s %-35s %s' "$kind" "$name" "[dry-run] git clone $repo_url $target_dir")")
continue
fi
# Crear directorio padre si no existe
mkdir -p "$(dirname "$target_dir")"
# Clonar
local clone_out
if clone_out=$(git clone --branch master "$repo_url" "$target_dir" 2>&1); then
summary_lines+=("$(printf '%-10s %-35s %s' "$kind" "$name" "[cloned]")")
else
# Detectar error de auth
local status_msg
if echo "$clone_out" | grep -qi "authentication\|permission denied\|could not read\|403\|401"; then
status_msg="[failed] auth (configura SSH o credential helper)"
else
status_msg="[failed] $(echo "$clone_out" | tail -1)"
fi
summary_lines+=("$(printf '%-10s %-35s %s' "$kind" "$name" "$status_msg")")
had_failures=1
echo " [error] $name: $clone_out" >&2
fi
done <<< "$rows"
# --- Tabla resumen ---
echo ""
printf '%-10s %-35s %s\n' "KIND" "NAME" "STATUS"
printf '%-10s %-35s %s\n' "----------" "-----------------------------------" "-------"
for line in "${summary_lines[@]}"; do
echo "$line"
done
echo ""
if [[ "$dry_run" -eq 0 ]]; then
echo "Siguiente paso sugerido:"
echo " cd $registry_root && CGO_ENABLED=1 ./fn index && ./fn sync"
echo ""
echo " fn index — actualiza registry.db con los nuevos artefactos"
echo " fn sync — sincroniza pc_locations para este PC"
fi
if [[ "$had_failures" -eq 1 ]]; then
return 1
fi
return 0
}
clone_project_subrepos "$@"
docs/capabilities/INDEX.md
+1
View File
@@ -22,6 +22,7 @@ Indice de grupos de capacidades del registry. Cada grupo agrupa >=3 funciones qu
| [doctor](doctor.md) | 11 | Diagnostico read-only del registry: artefactos, servicios, drift, funciones huerfanas |
| [notebook](notebook.md) | 5 | Operar Jupyter Lab colaborativo (discover/read/exec/write/kernel) |
| [cpp-windows](cpp-windows.md) | 7 | Compilar, desplegar, lanzar y verificar apps C++ en Windows desde WSL2 |
| [git](git.md) | 19 | Operaciones git y Gitea: clonar, commit, push/pull, hooks, TBD, webhooks, sync entre PCs |
## Como anadir grupo
docs/capabilities/git.md
+70
View File
@@ -0,0 +1,70 @@
# Capability: git
Operaciones git y Gitea: descubrir repos, clonar, commit, push/pull, hooks, gestión de ramas TBD, webhooks y sincronización entre PCs. Cubre tanto el lado local (git CLI) como el lado remoto (Gitea API REST).
## Funciones
| ID | Firma | Que hace |
|---|---|---|
| `discover_git_repos_bash_infra` | `discover_git_repos(root_dir: string) -> string` | Lista todos los repos git bajo un directorio raiz (find -name .git). Base de full_git_push/pull. |
| `git_pull_with_stash_bash_infra` | `git_pull_with_stash(repo_dir: string) -> string` | Stashea cambios dirty, hace pull --ff-only, recupera stash. Retorna estado: ok/already-up-to-date/diverged/stash-conflict. |
| `git_push_if_ahead_bash_infra` | `git_push_if_ahead(repo_dir: string) -> string` | Push solo si hay commits locales adelantados al remoto. No hace nada si ya esta sincronizado. |
| `git_auto_commit_dirty_bash_infra` | `git_auto_commit_dirty(repo_dir: string, message: string) -> string` | Commitea todos los cambios dirty con mensaje automatico. Retorna el subject del commit o vacío si no habia cambios. |
| `ensure_repo_synced_bash_infra` | `ensure_repo_synced(dir: string, owner: string, repo: string, branch: string, msg: string) -> void` | Inicializa git en un directorio, crea repo en Gitea si no existe, commitea y pushea. Idempotente. |
| `gitea_create_repo_bash_infra` | `gitea_create_repo(owner: string, repo: string) -> json` | Crea repo en Gitea via API REST. Requiere GITEA_URL y GITEA_TOKEN. |
| `gitea_push_directory_bash_infra` | `gitea_push_directory(dir: string, owner: string, repo: string, branch: string, msg: string) -> void` | Push de un directorio local a Gitea. Inicializa git si hace falta. |
| `gitea_list_repos_bash_infra` | `gitea_list_repos(owner: string) -> json` | Lista repos de un usuario/org en Gitea via API. |
| `gitea_add_collaborator_bash_infra` | `gitea_add_collaborator(owner: string, repo: string, user: string, permission: string) -> void` | Añade colaborador a un repo Gitea con permiso dado (read/write/admin). |
| `gitea_create_webhook_bash_infra` | `gitea_create_webhook(owner: string, repo: string, url: string, secret: string) -> json` | Crea webhook push en repo Gitea apuntando a una URL (ej. deploy_server). |
| `tbd_branch_create_bash_infra` | `tbd_branch_create(issue_num: string, slug: string) -> void` | Crea rama TBD `issue/<N>-<slug>` desde master actualizado. |
| `tbd_branch_finish_bash_infra` | `tbd_branch_finish(branch: string) -> void` | Merge --no-ff de rama TBD a master, push y elimina la rama. |
| `git_hook_audit_app_drift_bash_infra` | `git_hook_audit_app_drift() -> void` | Hook pre-commit que detecta drift entre imports reales y uses_functions del app.md. |
| `pre_commit_hook_install_bash_infra` | `pre_commit_hook_install(repo_dir: string) -> void` | Instala hooks pre-commit en un repo git local. |
| `scan_secrets_in_dirty_bash_cybersecurity` | `scan_secrets_in_dirty(repo_dir: string) -> string` | Escanea archivos dirty de un repo buscando patrones de secrets antes de commitear. |
| `full_git_pull_bash_pipelines` | `full_git_pull() -> stdout` | Pull de fn_registry + todos los sub-repos + submodulos + fn sync. |
| `full_git_push_bash_pipelines` | `full_git_push([commit_message]) -> stdout` | Commit + push de fn_registry + todos los sub-repos + fn sync. |
| `gitea_init_app_bash_pipelines` | `gitea_init_app(app_dir: string, owner: string, repo: string) -> void` | Inicializa repo Gitea para una app nueva: git init, crea repo remoto, primer commit, push. |
| `clone_project_subrepos_bash_pipelines` | `clone_project_subrepos <project_id> [--owner OWNER] [--dry-run]` | Clona todos los sub-repos (apps + analysis) de un project en el PC actual. |
## Ejemplo canonico
```bash
# --- Flujo 1: PC nuevo — traer un project completo ---
# Solo tienes fn_registry clonado. Quieres trabajar en fn_monitoring.
./fn run clone_project_subrepos fn_monitoring
# KIND NAME STATUS
# ---------- ----------------------------------- -------
# app call_monitor [cloned]
# app registry_dashboard [cloned]
# app sqlite_api [cloned]
# analysis domain_coverage_gaps [cloned]
#
# Siguiente paso sugerido:
# cd /home/lucas/fn_registry && CGO_ENABLED=1 ./fn index && ./fn sync
CGO_ENABLED=1 ./fn index && ./fn sync
# --- Flujo 2: Push diario de todos los repos ---
./fn run full_git_push "feat: nueva funcion X"
# Descubre repos, escanea secrets, commitea dirty trees, pushea adelantados, fn sync
# --- Flujo 3: Crear app nueva con sub-repo Gitea ---
./fn run gitea_init_app apps/mi_app dataforge mi_app
# --- Flujo 4: Rama TBD para una issue ---
# (desde el directorio de la app)
source bash/functions/infra/tbd_branch_create.sh
tbd_branch_create 0123 nueva-feature
# ... trabajo ...
source bash/functions/infra/tbd_branch_finish.sh
tbd_branch_finish issue/0123-nueva-feature
```
## Fronteras
- **No gestiona credenciales**: el grupo asume SSH key o credential helper configurados. Para secrets usa `pass_get_bash_infra` (grupo separado).
- **No hace rebase interactivo**: TBD usa merge --no-ff, nunca rebase.
- **No clona vaults**: `clone_project_subrepos` omite vaults deliberadamente (son symlinks a datos externos).
- **No hace deploy**: push a remoto != deploy a VPS. Para deploy ver grupo `docker` / `deploy_server`.
- **No gestiona submodulos de forma independiente**: los submodulos se actualizan dentro de `full_git_pull`, no hay función atómica de submodulo separada.