Mathursday: PAC with Hoeffding-Bernstein

PAC results with Hoeffding-Bernstein’s inequality are the bread and butter of machine learning theory. In this post, we’ll see how to use Hoeffding’s inequality to derive agnostic PAC bounds with an error rate that decreases as Image may be NSFW.
Clik here to view. $O\left(\sqrt{\frac{1}{n}}\right)$ in number of training samples Image may be NSFW.
Clik here to view.. Then, we’ll see that in the realizable setting we can use Bernstein’s inequality to improve this to Image may be NSFW.
Clik here to view. $O\left(\frac{1}{n}\right)$ .

Supervised Learning

Consider the task of regression with input space Image may be NSFW.
Clik here to view. $\mathcal{X}$ and output space Image may be NSFW.
Clik here to view. $\mathcal{Y}$ . We have access to a model class (also known as hypothesis family) Image may be NSFW.
Clik here to view. $\mathcal{F} = \left\{ f: \mathcal{X} \rightarrow \mathcal{Y} \right\}$ . We will assume Image may be NSFW.
Clik here to view. $\mathcal{F}$ to be finite for simplicity. There exists some unknown data distribution Image may be NSFW.
Clik here to view. $D \in \Delta\left(\mathcal{X} \times \mathcal{Y}\right)$ that is generating examples. Of course, we don’t have access to this data distribution but we have access to some samples from it. Let Image may be NSFW.
Clik here to view. $S = \{(x_i, y_i)\}_{i=1}^n$ be Image may be NSFW.
Clik here to view. independent identically distributed (i.i.d) samples from the data distribution i.e., Image may be NSFW.
Clik here to view. $(x_i, y_i) \sim D(.,.)$ and probability of observing a sample Image may be NSFW.
Clik here to view. is given by Image may be NSFW.
Clik here to view. $P(S) = \prod_{i=1}^n D(x_i, y_i)$ .

In order, to evaluate the performance of a model we will need a loss function. Let Image may be NSFW.
Clik here to view. $\ell: \mathcal{Y} \times \mathcal{Y} \rightarrow [0, 1]$ be a loss function. Given a datapoint Image may be NSFW.
Clik here to view. (x, y) and a model Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ , the loss associated with our prediction is given by Image may be NSFW.
Clik here to view. $\ell(y, f(x))$ . For now, we will not make any assumption about Image may be NSFW.
Clik here to view. $D, \mathcal{F}, \ell$ .

Our aim is to find a function Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ that is as good as possible. What is a good function? Well one that has the least loss function across the input space i.e., its generalization error Image may be NSFW.
Clik here to view. L_D(f) is as low as possible: Image may be NSFW.
Clik here to view. $L_D(f) = \mathbb{E}_{(x, y) \sim D}\left[\ell(y, f(x)) \right]$ . The only signal we have for finding such a Image may be NSFW.
Clik here to view. is the training data Image may be NSFW.
Clik here to view.. We will define the training error Image may be NSFW.
Clik here to view. $L_S(f) = \frac{1}{n} \sum_{i=1}^n \ell(y_i, f(x_i))$ . The aim of machine learning theory is to (i) allow us to find a “good” Image may be NSFW.
Clik here to view., and (ii) bound the performance of the predicted model in terms of Image may be NSFW.
Clik here to view. $S, \mathcal{F}$ and other relevant parameters. The first result we will look at is the Occam’s Razor Bound and then we will see how to improve this result.

Occam’s Razor Bound

One of the first result that is taught in a machine learning theory class is the Occam’s Razor bound. The formal result is given in the theorem below. Occam’s Razor bound suggests using the model that minimizes the error on the observed sample Image may be NSFW.
Clik here to view. i.e., the solution of the following optimization: Image may be NSFW.
Clik here to view. $\hat{f}_S := \inf_{f \in \mathcal{F}} L_S(f)$ . This solution is called the empirical risk minimizer (ERM) or the ERM solution. We will drop the Image may be NSFW.
Clik here to view. from the notation when it is clear from the context.

Theorem: Let Image may be NSFW.
Clik here to view. $(\mathcal{X}, \mathcal{Y}, D, \ell)$ be a supervised learning problem with input space Image may be NSFW.
Clik here to view. $\mathcal{X}$ , output space Image may be NSFW.
Clik here to view. $\mathcal{Y}$ , data distribution Image may be NSFW.
Clik here to view. $D \in \Delta(\mathcal{X} \times \mathcal{Y})$ , and Image may be NSFW.
Clik here to view. $\ell: \mathcal{Y} \times \mathcal{Y} \rightarrow [0, 1]$ be the loss function. We are given Image may be NSFW.
Clik here to view. i.i.d samples Image may be NSFW.
Clik here to view. $\{(x_i, y_i\}_{i=1}^n$ from the data distribution Image may be NSFW.
Clik here to view.. Let Image may be NSFW.
Clik here to view. $\mathcal{F}$ be a finite collection of models from Image may be NSFW.
Clik here to view. $\mathcal{X}$ to Image may be NSFW.
Clik here to view. $\mathcal{Y}$ . Fix any Image may be NSFW.
Clik here to view. $\delta \in (0, 1)$ then with probability at least Image may be NSFW.
Clik here to view. $1 - \delta$ over samples Image may be NSFW.
Clik here to view. drawn i.i.d. from Image may be NSFW.
Clik here to view., we have:

Image may be NSFW.
Clik here to view. $\forall f \in \mathcal{F}, \,\, \left|L_D(f) - L_S(f)\right| < \sqrt{\frac{1}{2n} \log\left(\frac{2|\mathcal{F}|}{\delta}\right)}$ , and

Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < \inf_{f\in \mathcal{F}} L_D(f) + \sqrt{\frac{2}{n} \log\left(\frac{2|\mathcal{F}|}{\delta}\right)}$ .

Proof: Fix any model in the model family Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ . Let Image may be NSFW.
Clik here to view. $Z_i = \ell(y_i, f(x_i))$ . Then observe Image may be NSFW.
Clik here to view. $Z_i \in [0, 1]$ and Image may be NSFW.
Clik here to view. $\{Z_i\}_{i=1}^n$ are i.i.d random variables. They are random variables with respect to the sample Image may be NSFW.
Clik here to view. which is randomly chosen. Further, Image may be NSFW.
Clik here to view. $L_S(f) = \frac{1}{n}\sum_{i=1}^n Z_i$ and

Image may be NSFW.
Clik here to view. $\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n Z_i\right] = \frac{1}{n} \sum_{i=1}^n \mathbb{E}\left[Z_i\right] = \frac{1}{n} \sum_{i=1}^n L_D(f) = L_D(f)$ ,

where the first equality follows from linearity of expectation and the second inequality follows from iid assumption. Then using Hoeffding’s inequality:

Image may be NSFW.
Clik here to view. $P( \left| L_D(f) - L_S(f) \right| \ge t) \le 2\exp^{-2nt^2}$ ,

Setting Image may be NSFW.
Clik here to view. $\delta = 2\exp^{-2nt^2}$ , we get: Image may be NSFW.
Clik here to view. $\left|L_D(f) - L_S(f) \right| < \sqrt{\frac{1}{2n}\log\left(\frac{2}{\delta}\right)}$ with probability of at least Image may be NSFW.
Clik here to view. $1 - \delta$ . Let Image may be NSFW.
Clik here to view. Q_f be result derived in the above equation then we have Image may be NSFW.
Clik here to view. $P(Q_f) \ge 1 - \delta$ or in other words Image may be NSFW.
Clik here to view. $P(\overline{Q_f}) < \delta$ (Image may be NSFW.
Clik here to view. $\overline{X}$ represents the complement of measurable set Image may be NSFW.
Clik here to view.). Even though we derived this result for a fixed Image may be NSFW.
Clik here to view., we never used any property of Image may be NSFW.
Clik here to view. and therefore, it holds for all Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ . Therefore, we can bound the probability of Image may be NSFW.
Clik here to view. $\cap_{f \in \mathcal{F}} Q_f$ using union bound:

Image may be NSFW.
Clik here to view. $P(Q_{\hat{f}}) = 1 - P(\overline{Q_{\hat{f}}}) \ge 1 - \sum_{f \in \mathcal{F}} P(\overline{Q_f}) \ge 1 - \sum_{f \in \mathcal{F}} \delta = 1 - \delta |\mathcal{F}|$ .

This means with probability of at least Image may be NSFW.
Clik here to view. $1 - \delta |\mathcal{F}|$ we have:

Image may be NSFW.
Clik here to view. $\forall f \in \mathcal{F}, \,\,\,\,\,\, \left| L_D(f) - L_S(f) \right| < \sqrt{\frac{1}{2n} \log\left(\frac{2}{\delta}\right)}$ .

Replacing Image may be NSFW.
Clik here to view. $\delta$ by Image may be NSFW.
Clik here to view. $\frac{\delta}{|\mathcal{F}|}$ proves the first result. Let Image may be NSFW.
Clik here to view. $\hat{f}$ be the ERM solution and fix Image may be NSFW.
Clik here to view. in Image may be NSFW.
Clik here to view. $\mathcal{F}$ . Then:

Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < L_S(\hat{f}) + \sqrt{\frac{1}{2n} \log\left(\frac{2|\mathcal{F}|}{\delta}\right)}$ (from Hoeffidng’s inequality)
Image may be NSFW.
Clik here to view. $L_S(\hat{f}) \le L_S(f)$ (from the definition of ERM)
Image may be NSFW.
Clik here to view. $L_S(f) < L_D(f) + \sqrt{\frac{1}{2n} \log\left(\frac{2|\mathcal{F}|}{\delta}\right)}$ (from Hoeffidng’s inequality)

This implies, for any Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ , with probability at least Image may be NSFW.
Clik here to view. $1 - \delta$ : Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < L_D(f) + \sqrt{\frac{2}{n} \log\left(\frac{2|\mathcal{F}|}{\delta}\right)}$ . As this holds for any Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ , therefore, we can pick the Image may be NSFW.
Clik here to view. with smallest value of Image may be NSFW.
Clik here to view. L_D(f) . This proves the second result.

There are couple of things that stand out in this result:

More training data helps: Formally, the bound improves as Image may be NSFW.
Clik here to view. $O\left(\sqrt{\frac{1}{n}}\right)$ in sample size Image may be NSFW.
Clik here to view.. Informally, this means as we have more training data we will learn a model whose generalized error has a better upper bound.
Complex model class can be undesirable: The bound becomes weaker as the model family Image may be NSFW.
Clik here to view. $\mathcal{F}$ becomes larger. This is an important lesson in model regularization: if we have two model class Image may be NSFW.
Clik here to view. $\mathcal{F}_1, \mathcal{F}_2$ that have the same training error on a sample Image may be NSFW.
Clik here to view., then if Image may be NSFW.
Clik here to view. is more complex class than Image may be NSFW.
Clik here to view. i.e., Image may be NSFW.
Clik here to view. $|\mathcal{F}_1| < |\mathcal{F}_2|$ , then the bound on the generalization error for the later will be weaker. Note that the result does not say that the larger model class is always bad. We can get better bounds with a more complex model class provided we can appropriately improve the training error.

Improving Occam’s Razor Bound with Bernstein’s Inequality

Bernstein’s inequality allows us to incorporate the variance of the random variables in the bound thereby allowing us to get tighter bound for low variance settings.

Theorem (Bernstein’s Inequality): Let Image may be NSFW.
Clik here to view. $\left\{X_i\right\}_{i=1}^n$ be Image may be NSFW.
Clik here to view. i.i.d random variables with mean Image may be NSFW.
Clik here to view. $\mu$ and variance Image may be NSFW.
Clik here to view. $\sigma^2$ . Further, let Image may be NSFW.
Clik here to view. $\left|X_i - \mu \right| \le M$ almost surely, for all Image may be NSFW.
Clik here to view.. Then:

Image may be NSFW.
Clik here to view. $P\left(\left| \frac{1}{n}\sum_{i=1}^n X_i - \mu \right| > t \right) \le 2\exp\left\{- \frac{3nt^2}{6\sigma^2 + 2Mt} \right\}$

If the variance was negligible Image may be NSFW.
Clik here to view. $\sigma^2 \approx 0$ then we will get Image may be NSFW.
Clik here to view. $P\left(\left| \frac{1}{n}\sum_{i=1}^n X_i - \mu \right| > t \right) \le 2\exp\left\{-\frac{3nt}{2M}\right\}$ . This bound, if we repeat what we did for Occam’s Razor Bound, will give us Image may be NSFW.
Clik here to view. $O\left( \frac{1}{n}\right)$ bound.

Before we proceed to derive a Image may be NSFW.
Clik here to view. $O\left( \frac{1}{n}\right)$ in realizable setting, we will first state the Bernstein’s inequality in the more familiar form. Setting the right hand side to Image may be NSFW.
Clik here to view. $\delta$ and solving the quadratic equation gives us:

With probability at least Image may be NSFW.
Clik here to view. $1-\delta$ we have:

Image may be NSFW.
Clik here to view. $\left| \frac{1}{n}\sum_{i=1}^n X_i - \mu \right| \le \frac{M}{3n} \log\left( \frac{2}{\delta} \right) + \sqrt{\frac{M^2}{9n^2} \log^2\left(\frac{2}{\delta}\right) + \frac{2\sigma^2}{n}\log\left(\frac{2}{\delta}\right)}$

We now apply the inequality Image may be NSFW.
Clik here to view. $\forall a, b \in [0, \infty), \sqrt{a + b} \le \sqrt{a} + \sqrt{b}$ to get:

Image may be NSFW.
Clik here to view. $\left| \frac{1}{n}\sum_{i=1}^n X_i - \mu \right| \le \frac{2M}{3n} \log\left( \frac{2}{\delta} \right) + \sqrt{\frac{2\sigma^2}{n}\log\left(\frac{2}{\delta}\right)}$ .

Theorem: Let Image may be NSFW.
Clik here to view. $(\mathcal{X}, \mathcal{Y}, D, \ell)$ be a supervised learning problem with input space Image may be NSFW.
Clik here to view. $\mathcal{X}$ , output space Image may be NSFW.
Clik here to view. $\mathcal{Y}$ , data distribution Image may be NSFW.
Clik here to view. $D \in \Delta(\mathcal{X} \times \mathcal{Y})$ , and loss function Image may be NSFW.
Clik here to view. $\ell \in [0, 1]$ . Let there be Image may be NSFW.
Clik here to view. $f^\star \in \mathcal{F}$ such that Image may be NSFW.
Clik here to view. $L_D(f^\star) = 0$ . We are given Image may be NSFW.
Clik here to view. i.i.d samples Image may be NSFW.
Clik here to view. $\{(x_i, y_i\}_{i=1}^n$ from the data distribution Image may be NSFW.
Clik here to view.. Let Image may be NSFW.
Clik here to view. $\mathcal{F}$ be a finite collection of models from Image may be NSFW.
Clik here to view. $\mathcal{X}$ to Image may be NSFW.
Clik here to view. $\mathcal{Y}$ . Fix any Image may be NSFW.
Clik here to view. $\delta \in (0, 1)$ then with probability at least Image may be NSFW.
Clik here to view. $1 - \delta$ we have:

Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < \frac{20}{3n} \log\left( \frac{2|\mathcal{F}|}{\delta} \right)$ .

Proof: Fix a Image may be NSFW.
Clik here to view. $f \in \mathcal{F}$ and define Image may be NSFW.
Clik here to view. $\{Z_1, \cdots, Z_n\}$ as before. This time we will apply Bernstein’s inequality. We have Image may be NSFW.
Clik here to view. $|Z_i - \mu| < 1$ as the loss function is bounded in Image may be NSFW.
Clik here to view. [0,1] . Bernstein’s inequality gives us:

Image may be NSFW.
Clik here to view. $\left|L_D(f) - L_S(f) \right| \le \frac{2}{3n} \log\left( \frac{2}{\delta} \right) + \sqrt{\frac{2\sigma^2}{n}\log\left(\frac{2}{\delta}\right)}$ ,

with probability of at least Image may be NSFW.
Clik here to view. $1 - \delta$ . All we need to do is bound the variance Image may be NSFW.
Clik here to view. $\sigma^2 = \mathbb{E}_{(x, y) \sim D}[ (\ell(y, f(x)) - L_D(f) )^2] = \mathbb{E}_{(x, y) \sim D}[\ell(y, f(x))^2] - L_D(f)^2$ . As the loss function is bounded in Image may be NSFW.
Clik here to view. [0, 1] therefore, Image may be NSFW.
Clik here to view. $\mathbb{E}_{(x, y) \sim D}[\ell(y, f(x))^2] \le \mathbb{E}_{(x, y) \sim D}[1. \ell(y, f(x))] = L_D(f)$ . This gives us Image may be NSFW.
Clik here to view. $\sigma^2 \le L_D(f)$ . Plugging this in we get:

Image may be NSFW.
Clik here to view. $\left|L_D(f) - L_S(f) \right| \le \frac{2}{3n} \log\left( \frac{2}{\delta} \right) + \sqrt{\frac{2L_D(f)}{n}\log\left(\frac{2}{\delta}\right)}$ .

Applying AM-GM inequality, we have Image may be NSFW.
Clik here to view. $\sqrt{\frac{2L_D(f)}{n}\log\left(\frac{2}{\delta}\right)} < \frac{L_D(f)}{2} + \frac{1}{n}\log\left(\frac{2}{\delta}\right)$ giving us,

Image may be NSFW.
Clik here to view. $\left|L_D(f) - L_S(f) \right| \le \frac{L_D(f)}{2} + \frac{5}{3n} \log\left( \frac{2}{\delta} \right)$ ,

with probability Image may be NSFW.
Clik here to view. $1-\delta$ . Applying union bound on all hypothesis we get:

Image may be NSFW.
Clik here to view. $\forall f \in \mathcal{F}, \,\,\,\, \left|L_D(f) - L_S(f) \right| \le \frac{L_D(f)}{2} + \frac{5}{3n} \log\left( \frac{2|\mathcal{F}|}{\delta} \right)$ ,

with probability at least Image may be NSFW.
Clik here to view. $1-\delta$ . Let Image may be NSFW.
Clik here to view. $\hat{f}$ be the ERM solution then:

Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < L_S(\hat{f}) + \frac{L_D(\hat{f})}{2} + \frac{5}{3n} \log\left( \frac{2|\mathcal{F}|}{\delta} \right)$ (Bernstein’s inequality)
Image may be NSFW.
Clik here to view. $L_S(\hat{f}) < L_S(f^\star)$ (ERM solution)
Image may be NSFW.
Clik here to view. $L_S(f^\star) < \frac{3}{2}L_D(f^\star) + \frac{5}{3n} \log\left( \frac{2|\mathcal{F}|}{\delta} \right)$ (Bernstein’s inequality)
Image may be NSFW.
Clik here to view. $L_D(f^\star) = 0$ (realizability condition)

Combining these we get Image may be NSFW.
Clik here to view. $L_D(\hat{f}) < \frac{20}{3n} \log\left( \frac{2|\mathcal{F}|}{\delta} \right)$ with probability at least Image may be NSFW.
Clik here to view. $1-\delta$ . Hence, proved.

So what is happening? We are making a realizability assumption i.e., there is a function Image may be NSFW.
Clik here to view. $f^\star \in \mathcal{F}$ such that Image may be NSFW.
Clik here to view. $L_D(f^\star) = 0$ . As the loss is lower bounded by 0, therefore, the variance of the samples of its loss are 0. Now as our ERM solver is becoming better with more data, it is converging towards this Image may be NSFW.
Clik here to view. $f^\star$ whose variance is 0. Therefore, the variance of the ERM (which is bounded by its mean Image may be NSFW.
Clik here to view. $\mu$ ) is becoming smaller. This helps in bounding the rate of convergence.

Summary: We saw how Hoeffding’s inequality gives us a Image may be NSFW.
Clik here to view. $O(\sqrt{\frac{1}{n}})$ bound in number of samples Image may be NSFW.
Clik here to view.. When we are in the realizable case, we can get Image may be NSFW.
Clik here to view. $O(\frac{1}{n})$ using Bernstein’s inequality.

Mathursday: PAC with Hoeffding-Bernstein

Supervised Learning

Occam’s Razor Bound

Improving Occam’s Razor Bound with Bernstein’s Inequality

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

Ominde Commission Report and Recommendations – Ominde Report of 1964

Bureau of Internal Revenue: Regional Offices (Directory)

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

Mp3 Download: Mdu - Kunjenjenjena

How the kill the job , when DTP request running for long hours.

Microsoft Intune から展開しているアプリのアップデートについて

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

Car crash in Dunton Bassett leaves driver in critical condition

Macky 2, Two Others In Road Accident

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

Detroit mafia: D’Anna Brothers agree to plea deal

Delivery block field greyed out using VA02

Muloraki Au

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

FIAT 500 B0111 B0112