{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tick Data y Estimación de Parámetros\n",
    "\n",
    "**Problema inverso:** Dada la salida de tick data del simulador (como si fuera data real), estimar los parámetros que generaron ese mercado.\n",
    "\n",
    "## Estructura\n",
    "1. Exportar simulación en formato estándar (estilo Tardis.dev + LOBSTER)\n",
    "2. Desde esos datos, estimar: σ, λ, spread, Hawkes α/β, cola de tamaños\n",
    "3. Ventanas deslizantes para ver cómo cambian en el tiempo\n",
    "4. Comparar estimaciones vs parámetros reales de la simulación\n",
    "\n",
    "## Formato de datos\n",
    "\n",
    "Inspirado en los proveedores más usados:\n",
    "- **Tardis.dev** (cripto): trades normalizados con `exchange, symbol, timestamp, side, price, amount`\n",
    "- **LOBSTER** (equities): message file + orderbook file pareados por evento\n",
    "- **Databento** (multi-asset): acciones como `add/cancel/modify/trade` con fixed-point prices\n",
    "\n",
    "Nuestro formato combina lo mejor: trades + snapshots del book en cada tick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tipos definidos: Side, OrderType, OrderStatus, Order, Trade\n",
      "OrderBook definido\n",
      "MatchingEngineFIFO definido\n",
      "✓ test_basic_match\n",
      "✓ test_partial_fill\n",
      "✓ test_fifo_priority\n",
      "✓ test_price_priority\n",
      "✓ test_no_match_spread\n",
      "✓ test_market_order\n",
      "✓ test_cancel\n",
      "\n",
      "=== Todos los tests pasaron ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderBook(best_bid=99.85, best_ask=100.25, spread=0.4000000000000057, bids_top3=[(99.85, 4.8), (99.72, 2.8), (99.48, 22.799999999999997)], asks_top3=[(100.25, 5.4), (100.29, 6.3), (100.41, 8.2)])\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 51756 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data",
     "transient": {}
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== ANTES ===\n",
      "Best ask: 100.25\n",
      "Best bid: 99.85\n",
      "Spread: 0.4000\n",
      "Midprice: 100.0500\n",
      "\n",
      "Ask depth (5 niveles): [(100.25, 5.4), (100.29, 6.3), (100.41, 8.2), (100.43, 14.9), (100.5, 18.9)]\n",
      "\n",
      "=== MARKET BUY 50 ===\n",
      "Trades ejecutados: 5\n",
      "  5.4 @ 100.25\n",
      "  6.3 @ 100.29\n",
      "  8.2 @ 100.41\n",
      "  14.9 @ 100.43\n",
      "  15.2 @ 100.50\n",
      "\n",
      "Precio promedio ponderado: 100.4109\n",
      "Slippage vs best ask: 0.1609\n",
      "\n",
      "=== DESPUÉS ===\n",
      "Best ask: 100.5\n",
      "Best bid: 99.85\n",
      "Spread: 0.6500000000000057\n",
      "OrderBook(best_bid=99.85, best_ask=100.5, spread=0.6500000000000057, bids_top3=[(99.85, 4.8), (99.72, 2.8), (99.48, 22.799999999999997)], asks_top3=[(100.5, 3.699999999999994), (100.53, 6.5), (100.54, 1.6)])\n",
      "Tipos definidos: Side, OrderType, OrderStatus, Order, Trade\n",
      "OrderBook definido\n",
      "MatchingEngineFIFO definido\n",
      "✓ test_basic_match\n",
      "✓ test_partial_fill\n",
      "✓ test_fifo_priority\n",
      "✓ test_price_priority\n",
      "✓ test_no_match_spread\n",
      "✓ test_market_order\n",
      "✓ test_cancel\n",
      "\n",
      "=== Todos los tests pasaron ===\n",
      "OrderBook(best_bid=99.85, best_ask=100.25, spread=0.4000000000000057, bids_top3=[(99.85, 4.8), (99.72, 2.8), (99.48, 22.799999999999997)], asks_top3=[(100.25, 5.4), (100.29, 6.3), (100.41, 8.2)])\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 51756 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data",
     "transient": {}
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== ANTES ===\n",
      "Best ask: 100.25\n",
      "Best bid: 99.85\n",
      "Spread: 0.4000\n",
      "Midprice: 100.0500\n",
      "\n",
      "Ask depth (5 niveles): [(100.25, 5.4), (100.29, 6.3), (100.41, 8.2), (100.43, 14.9), (100.5, 18.9)]\n",
      "\n",
      "=== MARKET BUY 50 ===\n",
      "Trades ejecutados: 5\n",
      "  5.4 @ 100.25\n",
      "  6.3 @ 100.29\n",
      "  8.2 @ 100.41\n",
      "  14.9 @ 100.43\n",
      "  15.2 @ 100.50\n",
      "\n",
      "Precio promedio ponderado: 100.4109\n",
      "Slippage vs best ask: 0.1609\n",
      "\n",
      "=== DESPUÉS ===\n",
      "Best ask: 100.5\n",
      "Best bid: 99.85\n",
      "Spread: 0.6500000000000057\n",
      "OrderBook(best_bid=99.85, best_ask=100.5, spread=0.6500000000000057, bids_top3=[(99.85, 4.8), (99.72, 2.8), (99.48, 22.799999999999997)], asks_top3=[(100.5, 3.699999999999994), (100.53, 6.5), (100.54, 1.6)])\n",
      "Engine cargado desde notebook 01\n",
      "Parámetros cargados\n",
      "  Precio inicial: 100.0\n",
      "  Volatilidad: 0.02\n",
      "  Makers: 5 (spread=0.5, γ=0.1)\n",
      "  Takers λ: 2.0 (Hawkes α=0.5)\n",
      "  Ticks: 500\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 56404 characters]",
      "text/plain": [
       "<Figure size 1200x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data",
     "transient": {}
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Takers por tick: min=0, max=8, mean=2.0\n",
      "Agentes definidos: MakerState, maker_quotes (Avellaneda-Stoikov), taker_order (power-law)\n",
      "run_simulation() definida\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total trades: 11740\n",
      "Spread promedio: 0.5285\n",
      "Trades/tick promedio: 23.5\n",
      "\n",
      "Estado final de makers:\n",
      "  maker_0: inventario=153.3, PnL=1807.18\n",
      "  maker_1: inventario=77.3, PnL=1399.79\n",
      "  maker_2: inventario=195.5, PnL=77.56\n",
      "  maker_3: inventario=90.9, PnL=-4273.37\n",
      "  maker_4: inventario=-23.7, PnL=-6305.50\n"
     ]
    }
   ],
   "source": [
    "# Cargar engine y simulador\n",
    "%run /home/lucas/fn_registry/analysis/estudio_mercados/notebooks/01_matching_engine_fifo.ipynb\n",
    "%run /home/lucas/fn_registry/analysis/estudio_mercados/notebooks/02_simulacion_mercado.ipynb\n",
    "import polars as pl\n",
    "print(\"Todo cargado\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exportar simulación como tick data\n",
    "\n",
    "Dos DataFrames:\n",
    "- **trades_df**: cada trade (estilo Tardis.dev)\n",
    "- **book_df**: snapshot del book en cada tick (estilo LOBSTER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simulation_to_tick_data(p: SimParams, book_depth: int = 5):\n",
    "    \"\"\"Ejecuta simulación y exporta como tick data normalizado.\n",
    "\n",
    "    Retorna (trades_df, book_df, true_params) donde:\n",
    "    - trades_df: cada trade con timestamp, price, qty, side, buyer_id, seller_id\n",
    "    - book_df: snapshot del book en cada tick con N niveles de profundidad\n",
    "    - true_params: los parámetros reales para comparar después\n",
    "    \"\"\"\n",
    "    rng = np.random.default_rng(p.seed)\n",
    "    fund_prices = generate_fundamental_prices(p)\n",
    "\n",
    "    engine = MatchingEngineFIFO()\n",
    "    makers = [MakerState(maker_id=f\"maker_{i}\") for i in range(p.n_makers)]\n",
    "\n",
    "    # Acumuladores de datos\n",
    "    trade_records = []\n",
    "    book_records = []\n",
    "    hawkes_excitation = 0.0\n",
    "    trade_seq = 0\n",
    "\n",
    "    for t in range(p.n_ticks):\n",
    "        mid = fund_prices[t]\n",
    "\n",
    "        # Makers requote\n",
    "        for maker in makers:\n",
    "            for oid in maker.active_order_ids:\n",
    "                engine.cancel(oid)\n",
    "            maker.active_order_ids = []\n",
    "            quotes = maker_quotes(maker, mid, p, t, rng)\n",
    "            for q in quotes:\n",
    "                engine.submit(q)\n",
    "                maker.active_order_ids.append(q.order_id)\n",
    "\n",
    "        # Takers\n",
    "        lam = max(0.1, p.n_takers_lambda + hawkes_excitation)\n",
    "        n_takers = rng.poisson(lam)\n",
    "        tick_trades = 0\n",
    "\n",
    "        for taker_i in range(n_takers):\n",
    "            order = taker_order(mid, p, rng)\n",
    "            trades = engine.submit(order)\n",
    "            tick_trades += len(trades)\n",
    "\n",
    "            for tr in trades:\n",
    "                trade_seq += 1\n",
    "                trade_records.append({\n",
    "                    \"sequence\": trade_seq,\n",
    "                    \"tick\": t,\n",
    "                    \"timestamp_ns\": t * 1_000_000_000 + taker_i * 1_000_000 + trade_seq,\n",
    "                    \"price\": round(tr.price, 6),\n",
    "                    \"qty\": round(tr.qty, 6),\n",
    "                    \"side\": \"buy\" if order.side == Side.BUY else \"sell\",  # taker side\n",
    "                    \"buyer_order_id\": tr.buyer_order_id[:8],\n",
    "                    \"seller_order_id\": tr.seller_order_id[:8],\n",
    "                    \"trade_id\": tr.trade_id[:8],\n",
    "                })\n",
    "\n",
    "                # Update maker inventory\n",
    "                for maker in makers:\n",
    "                    if tr.buyer_order_id in maker.active_order_ids:\n",
    "                        maker.inventory += tr.qty\n",
    "                    elif tr.seller_order_id in maker.active_order_ids:\n",
    "                        maker.inventory -= tr.qty\n",
    "\n",
    "        hawkes_excitation *= np.exp(-p.hawkes_beta)\n",
    "        hawkes_excitation += p.hawkes_alpha * tick_trades\n",
    "\n",
    "        # Book snapshot (estilo LOBSTER: estado completo tras el tick)\n",
    "        bids = engine.book.depth(Side.BUY, book_depth)\n",
    "        asks = engine.book.depth(Side.SELL, book_depth)\n",
    "\n",
    "        record = {\n",
    "            \"tick\": t,\n",
    "            \"timestamp_ns\": (t + 1) * 1_000_000_000 - 1,  # fin del tick\n",
    "            \"fundamental_price\": round(fund_prices[t], 6),\n",
    "            \"n_trades\": tick_trades,\n",
    "        }\n",
    "        for i in range(book_depth):\n",
    "            record[f\"bid_price_{i}\"] = round(bids[i][0], 6) if i < len(bids) else None\n",
    "            record[f\"bid_size_{i}\"] = round(bids[i][1], 6) if i < len(bids) else 0.0\n",
    "            record[f\"ask_price_{i}\"] = round(asks[i][0], 6) if i < len(asks) else None\n",
    "            record[f\"ask_size_{i}\"] = round(asks[i][1], 6) if i < len(asks) else 0.0\n",
    "\n",
    "        book_records.append(record)\n",
    "\n",
    "    trades_df = pl.DataFrame(trade_records)\n",
    "    book_df = pl.DataFrame(book_records)\n",
    "\n",
    "    return trades_df, book_df\n",
    "\n",
    "\n",
    "print(\"simulation_to_tick_data() definida\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generar datos con parámetros conocidos\n",
    "TRUE_PARAMS = SimParams(\n",
    "    initial_price=100.0,\n",
    "    sigma=0.02,\n",
    "    mu=0.0,\n",
    "    jump_intensity=0.03,\n",
    "    jump_size_std=0.05,\n",
    "    n_makers=5,\n",
    "    maker_spread=0.5,\n",
    "    gamma=0.1,\n",
    "    n_takers_lambda=3.0,\n",
    "    taker_size_alpha=2.0,\n",
    "    hawkes_alpha=0.6,\n",
    "    hawkes_beta=1.0,\n",
    "    n_ticks=1000,\n",
    "    seed=123,\n",
    ")\n",
    "\n",
    "trades_df, book_df = simulation_to_tick_data(TRUE_PARAMS)\n",
    "\n",
    "print(f\"Trades: {trades_df.shape[0]} filas\")\n",
    "print(trades_df.head(5))\n",
    "print(f\"\\nBook snapshots: {book_df.shape[0]} filas\")\n",
    "print(book_df.head(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Guardar como CSV (como lo haría Tardis.dev o Kaiko)\n",
    "trades_df.write_csv(\"../data/trades.csv\")\n",
    "book_df.write_csv(\"../data/book_snapshots.csv\")\n",
    "print(\"Exportado a data/trades.csv y data/book_snapshots.csv\")\n",
    "print(f\"  trades.csv: {trades_df.estimated_size('mb'):.2f} MB\")\n",
    "print(f\"  book_snapshots.csv: {book_df.estimated_size('mb'):.2f} MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Estimación de parámetros desde tick data\n",
    "\n",
    "Ahora fingimos que **no conocemos los parámetros** — solo tenemos los CSV.\n",
    "\n",
    "### Qué podemos medir directamente:\n",
    "| Observable | Parámetro que revela |\n",
    "|---|---|\n",
    "| Retornos log entre ticks | **sigma** (volatilidad) |\n",
    "| Trades por tick | **lambda** (arrival rate) |\n",
    "| Autocorrelación de arrivals | **hawkes_alpha, hawkes_beta** (contagio) |\n",
    "| Spread bid-ask | **maker_spread** + **gamma** |\n",
    "| Distribución de tamaños | **taker_size_alpha** (cola power-law) |\n",
    "| Outliers en retornos | **jump_intensity, jump_size_std** |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cargar como si fuera data externa\n",
    "trades = pl.read_csv(\"../data/trades.csv\")\n",
    "book = pl.read_csv(\"../data/book_snapshots.csv\")\n",
    "print(f\"Cargados: {trades.shape[0]} trades, {book.shape[0]} snapshots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Estimar sigma (volatilidad)\n",
    "\n",
    "Calculamos retornos logarítmicos del midprice tick a tick.\n",
    "La desviación estándar de esos retornos es nuestra estimación de σ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimate_sigma(book: pl.DataFrame) -> dict:\n",
    "    \"\"\"Estima volatilidad desde midprices del book.\"\"\"\n",
    "    mid = ((book[\"bid_price_0\"] + book[\"ask_price_0\"]) / 2).alias(\"mid\")\n",
    "    df = book.with_columns(mid)\n",
    "    df = df.with_columns(\n",
    "        (pl.col(\"mid\").log() - pl.col(\"mid\").shift(1).log()).alias(\"log_return\")\n",
    "    ).drop_nulls(\"log_return\")\n",
    "\n",
    "    returns = df[\"log_return\"].to_numpy()\n",
    "    sigma_est = np.std(returns)\n",
    "\n",
    "    return {\n",
    "        \"sigma_estimated\": round(sigma_est, 6),\n",
    "        \"sigma_true\": TRUE_PARAMS.sigma,\n",
    "        \"returns\": returns,\n",
    "    }\n",
    "\n",
    "\n",
    "sigma_result = estimate_sigma(book)\n",
    "print(f\"σ estimada: {sigma_result['sigma_estimated']}\")\n",
    "print(f\"σ real:     {sigma_result['sigma_true']}\")\n",
    "print(f\"Error:      {abs(sigma_result['sigma_estimated'] - sigma_result['sigma_true']):.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Estimar lambda (arrival rate de takers)\n",
    "\n",
    "Contamos trades por tick. La media es nuestra λ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimate_lambda(book: pl.DataFrame) -> dict:\n",
    "    \"\"\"Estima arrival rate desde trades por tick.\"\"\"\n",
    "    arrivals = book[\"n_trades\"].to_numpy()\n",
    "    lambda_est = np.mean(arrivals)\n",
    "\n",
    "    return {\n",
    "        \"lambda_estimated\": round(lambda_est, 4),\n",
    "        \"lambda_true\": TRUE_PARAMS.n_takers_lambda,\n",
    "        \"arrivals\": arrivals,\n",
    "    }\n",
    "\n",
    "\n",
    "lambda_result = estimate_lambda(book)\n",
    "print(f\"λ estimada: {lambda_result['lambda_estimated']}\")\n",
    "print(f\"λ real:     {lambda_result['lambda_true']}\")\n",
    "print(f\"(λ estimada > λ real porque Hawkes amplifica la media)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Estimar Hawkes α y β (contagio)\n",
    "\n",
    "La autocorrelación de arrivals revela el clustering:\n",
    "- Hawkes α = 0 → autocorrelación cae a 0 inmediatamente (Poisson)\n",
    "- Hawkes α > 0 → autocorrelación decae exponencialmente (contagio)\n",
    "\n",
    "Ajustamos una exponencial `a * exp(-b * lag)` a la autocorrelación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import curve_fit\n",
    "\n",
    "\n",
    "def estimate_hawkes(arrivals: np.ndarray, max_lag: int = 30) -> dict:\n",
    "    \"\"\"Estima parámetros Hawkes desde autocorrelación de arrivals.\"\"\"\n",
    "    # Calcular autocorrelación\n",
    "    mean_arr = np.mean(arrivals)\n",
    "    var_arr = np.var(arrivals)\n",
    "    if var_arr == 0:\n",
    "        return {\"hawkes_alpha_est\": 0, \"hawkes_beta_est\": 1}\n",
    "\n",
    "    acf = np.array([\n",
    "        np.mean((arrivals[lag:] - mean_arr) * (arrivals[:-lag] - mean_arr)) / var_arr\n",
    "        if lag > 0 else 1.0\n",
    "        for lag in range(max_lag)\n",
    "    ])\n",
    "\n",
    "    # Ajustar exponencial a lags 1..max_lag\n",
    "    lags = np.arange(1, max_lag)\n",
    "    acf_values = acf[1:]\n",
    "\n",
    "    # Solo ajustar si hay autocorrelación positiva\n",
    "    if acf_values[0] <= 0.01:\n",
    "        return {\n",
    "            \"hawkes_alpha_est\": 0.0,\n",
    "            \"hawkes_beta_est\": 1.0,\n",
    "            \"acf\": acf,\n",
    "            \"branching_ratio\": 0.0,\n",
    "        }\n",
    "\n",
    "    exp_decay = lambda x, a, b: a * np.exp(-b * x)\n",
    "    try:\n",
    "        popt, _ = curve_fit(exp_decay, lags, acf_values, p0=[0.5, 0.5], maxfev=5000)\n",
    "        a_est, b_est = popt\n",
    "    except RuntimeError:\n",
    "        a_est, b_est = 0.0, 1.0\n",
    "\n",
    "    # Branching ratio η = α/β (debe ser < 1 para estacionariedad)\n",
    "    branching = a_est / b_est if b_est > 0 else 0\n",
    "\n",
    "    return {\n",
    "        \"hawkes_alpha_est\": round(abs(a_est), 4),\n",
    "        \"hawkes_beta_est\": round(abs(b_est), 4),\n",
    "        \"hawkes_alpha_true\": TRUE_PARAMS.hawkes_alpha,\n",
    "        \"hawkes_beta_true\": TRUE_PARAMS.hawkes_beta,\n",
    "        \"branching_ratio\": round(branching, 4),\n",
    "        \"acf\": acf,\n",
    "    }\n",
    "\n",
    "\n",
    "hawkes_result = estimate_hawkes(lambda_result[\"arrivals\"])\n",
    "print(f\"Hawkes α estimada: {hawkes_result['hawkes_alpha_est']}  (real: {hawkes_result['hawkes_alpha_true']})\")\n",
    "print(f\"Hawkes β estimada: {hawkes_result['hawkes_beta_est']}  (real: {hawkes_result['hawkes_beta_true']})\")\n",
    "print(f\"Branching ratio η = α/β: {hawkes_result['branching_ratio']}  (< 1 = estacionario)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Estimar taker_size_alpha (cola de tamaños)\n",
    "\n",
    "Ajustamos una distribución power-law (Pareto) a los tamaños de trades.\n",
    "El estimador MLE del exponente α de Pareto es: `α = n / Σ ln(xᵢ / x_min)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimate_pareto_alpha(trades: pl.DataFrame) -> dict:\n",
    "    \"\"\"Estima exponente power-law de tamaños de trades via MLE.\"\"\"\n",
    "    sizes = trades[\"qty\"].to_numpy()\n",
    "    sizes = sizes[sizes > 0]\n",
    "    x_min = np.min(sizes)\n",
    "\n",
    "    # MLE para Pareto: α = n / Σ ln(xi / xmin)\n",
    "    n = len(sizes)\n",
    "    alpha_est = n / np.sum(np.log(sizes / x_min))\n",
    "\n",
    "    return {\n",
    "        \"alpha_estimated\": round(alpha_est, 4),\n",
    "        \"alpha_true\": TRUE_PARAMS.taker_size_alpha,\n",
    "        \"sizes\": sizes,\n",
    "        \"x_min\": x_min,\n",
    "    }\n",
    "\n",
    "\n",
    "pareto_result = estimate_pareto_alpha(trades)\n",
    "print(f\"α (tamaños) estimada: {pareto_result['alpha_estimated']}\")\n",
    "print(f\"α (tamaños) real:     {pareto_result['alpha_true']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Estimar jump intensity (frecuencia de saltos)\n",
    "\n",
    "Los jumps producen retornos anormalmente grandes.\n",
    "Contamos cuántos retornos exceden 3σ — esos son probables jumps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimate_jumps(returns: np.ndarray, threshold_sigma: float = 3.0) -> dict:\n",
    "    \"\"\"Estima intensidad y tamaño de jumps desde retornos.\"\"\"\n",
    "    sigma = np.std(returns)\n",
    "    threshold = threshold_sigma * sigma\n",
    "\n",
    "    jump_mask = np.abs(returns) > threshold\n",
    "    n_jumps = np.sum(jump_mask)\n",
    "    jump_intensity_est = n_jumps / len(returns)\n",
    "\n",
    "    jump_sizes = np.abs(returns[jump_mask])\n",
    "    jump_size_std_est = np.std(jump_sizes) if len(jump_sizes) > 1 else 0.0\n",
    "\n",
    "    return {\n",
    "        \"jump_intensity_est\": round(jump_intensity_est, 4),\n",
    "        \"jump_intensity_true\": TRUE_PARAMS.jump_intensity,\n",
    "        \"jump_size_std_est\": round(jump_size_std_est, 4),\n",
    "        \"jump_size_std_true\": TRUE_PARAMS.jump_size_std,\n",
    "        \"n_jumps_detected\": int(n_jumps),\n",
    "        \"jump_mask\": jump_mask,\n",
    "    }\n",
    "\n",
    "\n",
    "jump_result = estimate_jumps(sigma_result[\"returns\"])\n",
    "print(f\"Jump intensity estimada: {jump_result['jump_intensity_est']}  (real: {jump_result['jump_intensity_true']})\")\n",
    "print(f\"Jump size std estimada:  {jump_result['jump_size_std_est']}  (real: {jump_result['jump_size_std_true']})\")\n",
    "print(f\"Jumps detectados: {jump_result['n_jumps_detected']} de {len(sigma_result['returns'])} ticks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Resumen: estimaciones vs realidad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary = pl.DataFrame({\n",
    "    \"Parámetro\": [\"sigma\", \"lambda\", \"hawkes_alpha\", \"hawkes_beta\", \"taker_size_alpha\", \"jump_intensity\", \"jump_size_std\"],\n",
    "    \"Real\": [\n",
    "        TRUE_PARAMS.sigma, TRUE_PARAMS.n_takers_lambda, TRUE_PARAMS.hawkes_alpha,\n",
    "        TRUE_PARAMS.hawkes_beta, TRUE_PARAMS.taker_size_alpha,\n",
    "        TRUE_PARAMS.jump_intensity, TRUE_PARAMS.jump_size_std,\n",
    "    ],\n",
    "    \"Estimado\": [\n",
    "        sigma_result[\"sigma_estimated\"], lambda_result[\"lambda_estimated\"],\n",
    "        hawkes_result[\"hawkes_alpha_est\"], hawkes_result[\"hawkes_beta_est\"],\n",
    "        pareto_result[\"alpha_estimated\"],\n",
    "        jump_result[\"jump_intensity_est\"], jump_result[\"jump_size_std_est\"],\n",
    "    ],\n",
    "})\n",
    "summary = summary.with_columns(\n",
    "    ((pl.col(\"Estimado\") - pl.col(\"Real\")).abs() / pl.col(\"Real\") * 100).round(1).alias(\"Error %\")\n",
    ")\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Ventanas deslizantes: cómo cambian los parámetros en el tiempo\n",
    "\n",
    "Los parámetros **no son constantes**. Un mercado real tiene:\n",
    "- Periodos de calma (σ baja, λ baja)\n",
    "- Periodos de pánico (σ alta, λ alta, Hawkes fuerte)\n",
    "- Cambios de régimen (el spread se ensancha cuando los makers se asustan)\n",
    "\n",
    "Estimamos cada parámetro en ventanas de W ticks y vemos su evolución."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rolling_estimates(book: pl.DataFrame, trades: pl.DataFrame, window: int = 100, step: int = 10):\n",
    "    \"\"\"Calcula estimaciones de parámetros en ventanas deslizantes.\"\"\"\n",
    "    n = book.shape[0]\n",
    "    results = []\n",
    "\n",
    "    # Pre-calcular midprice y retornos\n",
    "    mid = ((book[\"bid_price_0\"] + book[\"ask_price_0\"]) / 2).to_numpy()\n",
    "    log_returns = np.diff(np.log(mid))\n",
    "    arrivals_full = book[\"n_trades\"].to_numpy()\n",
    "    spreads_full = (book[\"ask_price_0\"] - book[\"bid_price_0\"]).to_numpy()\n",
    "\n",
    "    # Pre-calcular tamaños por tick\n",
    "    sizes_by_tick = trades.group_by(\"tick\").agg(pl.col(\"qty\")).sort(\"tick\")\n",
    "\n",
    "    for start in range(0, n - window, step):\n",
    "        end = start + window\n",
    "        center = (start + end) // 2\n",
    "\n",
    "        # Sigma: std de retornos en la ventana\n",
    "        ret_window = log_returns[start:end-1]\n",
    "        sigma_w = np.std(ret_window) if len(ret_window) > 1 else 0\n",
    "\n",
    "        # Lambda: media de arrivals\n",
    "        arr_window = arrivals_full[start:end]\n",
    "        lambda_w = np.mean(arr_window)\n",
    "\n",
    "        # Spread: media\n",
    "        spread_w = np.mean(spreads_full[start:end])\n",
    "\n",
    "        # Varianza/media ratio (> 1 indica clustering, Hawkes)\n",
    "        var_mean_ratio = np.var(arr_window) / np.mean(arr_window) if np.mean(arr_window) > 0 else 1.0\n",
    "\n",
    "        # Jump detection: fraction de retornos > 3σ\n",
    "        if sigma_w > 0:\n",
    "            jump_frac = np.mean(np.abs(ret_window) > 3 * sigma_w)\n",
    "        else:\n",
    "            jump_frac = 0.0\n",
    "\n",
    "        results.append({\n",
    "            \"tick\": center,\n",
    "            \"sigma\": round(sigma_w, 6),\n",
    "            \"lambda\": round(lambda_w, 4),\n",
    "            \"spread\": round(spread_w, 6),\n",
    "            \"var_mean_ratio\": round(var_mean_ratio, 4),\n",
    "            \"jump_fraction\": round(jump_frac, 4),\n",
    "            \"midprice\": round(mid[center], 4),\n",
    "        })\n",
    "\n",
    "    return pl.DataFrame(results)\n",
    "\n",
    "\n",
    "rolling = rolling_estimates(book, trades, window=100, step=5)\n",
    "print(f\"Ventanas calculadas: {rolling.shape[0]}\")\n",
    "print(rolling.head(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_rolling_estimates(rolling: pl.DataFrame, true_params: SimParams):\n",
    "    \"\"\"Dashboard de parámetros estimados en ventanas deslizantes.\"\"\"\n",
    "    ticks = rolling[\"tick\"].to_numpy()\n",
    "\n",
    "    fig, axes = plt.subplots(6, 1, figsize=(14, 16), gridspec_kw={'height_ratios': [2, 1, 1, 1, 1, 1]})\n",
    "\n",
    "    # Panel 1: Midprice\n",
    "    ax = axes[0]\n",
    "    ax.plot(ticks, rolling[\"midprice\"].to_numpy(), color='#3498db', linewidth=0.8)\n",
    "    ax.set_ylabel('Midprice')\n",
    "    ax.set_title('Evolución temporal de parámetros estimados (ventana = 100 ticks)')\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    # Panel 2: Sigma\n",
    "    ax = axes[1]\n",
    "    ax.fill_between(ticks, rolling[\"sigma\"].to_numpy(), color='#e74c3c', alpha=0.5)\n",
    "    ax.axhline(y=true_params.sigma, color='black', linestyle='--', linewidth=0.8, label=f'Real σ={true_params.sigma}')\n",
    "    ax.set_ylabel('σ (volatilidad)')\n",
    "    ax.legend(fontsize=8)\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    # Panel 3: Lambda\n",
    "    ax = axes[2]\n",
    "    ax.fill_between(ticks, rolling[\"lambda\"].to_numpy(), color='#e67e22', alpha=0.5)\n",
    "    ax.axhline(y=true_params.n_takers_lambda, color='black', linestyle='--', linewidth=0.8, label=f'Real λ={true_params.n_takers_lambda}')\n",
    "    ax.set_ylabel('λ (trades/tick)')\n",
    "    ax.legend(fontsize=8)\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    # Panel 4: Spread\n",
    "    ax = axes[3]\n",
    "    ax.fill_between(ticks, rolling[\"spread\"].to_numpy(), color='#9b59b6', alpha=0.5)\n",
    "    ax.axhline(y=true_params.maker_spread, color='black', linestyle='--', linewidth=0.8, label=f'Real spread={true_params.maker_spread}')\n",
    "    ax.set_ylabel('Spread')\n",
    "    ax.legend(fontsize=8)\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    # Panel 5: Var/Mean ratio (proxy de Hawkes)\n",
    "    ax = axes[4]\n",
    "    ax.fill_between(ticks, rolling[\"var_mean_ratio\"].to_numpy(), color='#2ecc71', alpha=0.5)\n",
    "    ax.axhline(y=1.0, color='black', linestyle='--', linewidth=0.8, label='Poisson (ratio=1)')\n",
    "    ax.set_ylabel('Var/Mean ratio')\n",
    "    ax.legend(fontsize=8)\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    # Panel 6: Jump fraction\n",
    "    ax = axes[5]\n",
    "    ax.bar(ticks, rolling[\"jump_fraction\"].to_numpy(), color='#e74c3c', alpha=0.6, width=5)\n",
    "    ax.axhline(y=true_params.jump_intensity, color='black', linestyle='--', linewidth=0.8, label=f'Real jump_int={true_params.jump_intensity}')\n",
    "    ax.set_ylabel('Jump fraction')\n",
    "    ax.set_xlabel('Tick')\n",
    "    ax.legend(fontsize=8)\n",
    "    ax.grid(True, alpha=0.3)\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_rolling_estimates(rolling, TRUE_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Interpretación: qué revelan las ventanas\n",
    "\n",
    "Lo que deberías ver en los gráficos:\n",
    "\n",
    "- **σ (volatilidad)**: Fluctúa alrededor del valor real. Los picos coinciden con jumps o ráfagas de Hawkes. Esto es **clustering de volatilidad** — la volatilidad alta tiende a persistir.\n",
    "\n",
    "- **λ (arrival rate)**: La media está por encima del λ real porque Hawkes amplifica la tasa efectiva. Los picos son las ráfagas de contagio.\n",
    "\n",
    "- **Spread**: Se ensancha cuando σ sube — los makers se protegen subiendo su spread cuando el mercado es más volátil. Esto es el **efecto Avellaneda-Stoikov** en acción.\n",
    "\n",
    "- **Var/Mean ratio**: Si fuera Poisson puro, sería ~1. Valores > 1 indican clustering (Hawkes). Cuanto más alto, más contagio.\n",
    "\n",
    "- **Jump fraction**: Spikes esporádicos donde se detectan retornos anormales."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Distribuciones: comparar datos vs modelo teórico"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n",
    "\n",
    "# --- Distribución de retornos ---\n",
    "ax = axes[0][0]\n",
    "returns = sigma_result[\"returns\"]\n",
    "ax.hist(returns, bins=80, density=True, color='#3498db', alpha=0.6, label='Observado')\n",
    "# Superponer gaussiana teórica\n",
    "x = np.linspace(returns.min(), returns.max(), 200)\n",
    "from scipy.stats import norm\n",
    "ax.plot(x, norm.pdf(x, 0, sigma_result['sigma_estimated']), 'r-', linewidth=1.5, label=f'Normal(0, {sigma_result[\"sigma_estimated\"]:.4f})')\n",
    "ax.set_title('Distribución de retornos')\n",
    "ax.set_xlabel('Log-return')\n",
    "ax.legend(fontsize=8)\n",
    "ax.grid(True, alpha=0.3)\n",
    "\n",
    "# --- QQ plot de retornos ---\n",
    "ax = axes[0][1]\n",
    "from scipy.stats import probplot\n",
    "probplot(returns, dist=\"norm\", plot=ax)\n",
    "ax.set_title('QQ-Plot: retornos vs Normal')\n",
    "ax.grid(True, alpha=0.3)\n",
    "\n",
    "# --- Distribución de tamaños (log-log) ---\n",
    "ax = axes[1][0]\n",
    "sizes = pareto_result[\"sizes\"]\n",
    "sizes_sorted = np.sort(sizes)[::-1]\n",
    "ranks = np.arange(1, len(sizes_sorted) + 1)\n",
    "ax.loglog(sizes_sorted, ranks / len(sizes_sorted), '.', markersize=1, alpha=0.5, color='#2ecc71')\n",
    "# Superponer Pareto teórica\n",
    "x_fit = np.logspace(np.log10(sizes.min()), np.log10(sizes.max()), 100)\n",
    "ax.loglog(x_fit, (x_fit / sizes.min()) ** (-pareto_result['alpha_estimated']), 'r-',\n",
    "          label=f'Pareto α={pareto_result[\"alpha_estimated\"]:.2f}')\n",
    "ax.set_title('Distribución de tamaños (CCDF)')\n",
    "ax.set_xlabel('Tamaño')\n",
    "ax.set_ylabel('P(X > x)')\n",
    "ax.legend(fontsize=8)\n",
    "ax.grid(True, alpha=0.3)\n",
    "\n",
    "# --- Autocorrelación de arrivals ---\n",
    "ax = axes[1][1]\n",
    "acf = hawkes_result[\"acf\"]\n",
    "lags = np.arange(len(acf))\n",
    "ax.bar(lags, acf, color='#e67e22', alpha=0.6)\n",
    "# Superponer decaimiento exponencial ajustado\n",
    "if hawkes_result['hawkes_alpha_est'] > 0:\n",
    "    x_fit = np.arange(1, len(acf))\n",
    "    ax.plot(x_fit, hawkes_result['hawkes_alpha_est'] * np.exp(-hawkes_result['hawkes_beta_est'] * x_fit),\n",
    "            'r-', linewidth=2, label=f'Exp fit: α={hawkes_result[\"hawkes_alpha_est\"]:.2f}, β={hawkes_result[\"hawkes_beta_est\"]:.2f}')\n",
    "ax.set_title('Autocorrelación de trades/tick')\n",
    "ax.set_xlabel('Lag')\n",
    "ax.set_ylabel('ACF')\n",
    "ax.legend(fontsize=8)\n",
    "ax.grid(True, alpha=0.3)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "print(\"Notas:\")\n",
    "print(\"- QQ-Plot: colas pesadas = los extremos se desvían de la línea (por jumps)\")\n",
    "print(\"- CCDF log-log: línea recta = power-law confirmada\")\n",
    "print(\"- ACF: decaimiento lento = contagio Hawkes presente\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
