{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PRESS criterion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Predicted Residual Error Sum of Squares (PRESS), called also the ordinary cross-validation, is\n",
    "based on the basic leave-one-out cross-validation, which is proposed by <cite data-footcite-t=Allen1974-bu>D. M. Allen, 1974</cite>.<br>\n",
    "Let $\\mathbf{x}_\\lambda^{(l)}$ be the solution in which the $l\\text{-th}$ observation is omitted.\n",
    "The PRESS criterion's argument is that if $\\lambda$ is a good choice, then the $l\\text{-th}$ component $\\left(\\mathbf{T}\\mathbf{x}_\\lambda^{(l)}\\right)_l$ should be a good predictor of $b_l$.\n",
    "Therefore, the PRESS criterion leads to choosing $\\lambda$ as the minimizer of the PRESS function $\\mathcal{P}(\\lambda)$, defined by\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{P}(\\lambda)\n",
    "\\equiv\n",
    "\\sum_{l=1}^{M}\n",
    "\\left[\n",
    "    \\left(\n",
    "        \\mathbf{T}\\mathbf{x}_\\lambda^{(l)}\n",
    "    \\right)_l\n",
    "    -\n",
    "    b_l\n",
    "\\right]^2.\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "It can be rewritten by Sherman-Morrison-Woodbury formula:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{P}(\\lambda)\n",
    "=\n",
    "\\|\n",
    "\\mathbf{B}_\\lambda\n",
    "\\left(\n",
    "\\mathbf{I} -\\mathbf{A}_\\lambda\n",
    "\\right)\n",
    "\\mathbf{b}\n",
    "\\|_2^2,\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\mathbf{A}_\\lambda\n",
    "&\\equiv\n",
    "\\mathbf{T}\\left(\\mathbf{T}^\\mathsf{T}\\mathbf{T} + \\lambda\\mathbf{H}\\right)^{-1}\\mathbf{T}^\\mathsf{T},\\\\\n",
    "\\mathbf{B}_\\lambda\n",
    "&\\equiv\n",
    "\\text{diag}\n",
    "\\left(\n",
    "    \\cdots,\\frac{1}{1 - a_{\\lambda, ii}},\\cdots\n",
    "\\right),\\\\\n",
    "a_{\\lambda, ii}\n",
    "&\\equiv\n",
    "\\left(\\mathbf{A}_\\lambda\\right)_{ii}.\n",
    "\\end{align*}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using [series-expansion form of the solution](inversion.ipynb#Series-expansion-of-the-solution), $\\mathcal{P}(\\lambda)$ can be written as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{P}(\\lambda)\n",
    "=\n",
    "\\text{Comming soon...}.\n",
    "\\label{eq:PRESS_series}\n",
    "\\end{equation}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deriviation of \\eqref{eq:PRESS_series}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using decomposed solution form, We have\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\mathbf{A}_\\lambda \\mathbf{b}\n",
    "&=\n",
    "\\mathbf{T} \\mathbf{x}_\\lambda\\\\\n",
    "&=\n",
    "\\mathbf{T}\\tilde{\\mathbf{V}}\\mathbf{F}_\\lambda\\mathbf{S}^{-1}\\mathbf{U}^\\mathsf{T}\\mathbf{b}\\\\\n",
    "&=\n",
    "\\mathbf{U}\\mathbf{S}\\mathbf{F}_\\lambda\\mathbf{S}^{-1}\\mathbf{U}^\\mathsf{T}\\mathbf{b}\\quad(\\because \\mathbf{T}\\tilde{\\mathbf{V}} = \\mathbf{U}\\mathbf{S})\\\\\n",
    "&=\n",
    "\\mathbf{U}\\mathbf{F}_\\lambda\\mathbf{U}^\\mathsf{T}\\mathbf{b}.\\\\\n",
    "\\therefore\n",
    "\\mathbf{A}_\\lambda &= \\mathbf{U}\\mathbf{F}_\\lambda\\mathbf{U}^\\mathsf{T}.\n",
    "\\end{align*}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limitation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comming soon..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. footbibliography::"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}