Ordinary Least Square Method

11 Key excerpts on "Ordinary Least Square Method"

• 

  eBook - ePub

  Econometrics

  Econometrics Unleashed, Mastering Data-Driven Economics

  • Fouad Sabry(Author)
  • 2023(Publication Date)
  • One Billion Knowledgeable

    (Publisher)

  Chapter 8: Ordinary least squares

  For selecting the unknown parameters of a linear regression model (with fixed level-one effects of a linear function of a set of explanatory variables), ordinary least squares (OLS) is a linear least squares approach in statistics. minimizes the sum of squares of deviations between the observed value of the dependent variable and the result of the (linear) function of the independent variable in the input data set.

  This is represented geometrically as the sum of squared differences between each data point in the set and its corresponding point on the regression surface, with smaller differences indicating a better fit to the data. In the case of simple linear regression, where there is only one regressor on the right side of the regression equation, the resulting estimator can be stated by a straightforward formula.

  The OLS estimator is optimal in the class of linear unbiased estimators when the errors are homoscedastic and serially uncorrelated, consistent for the level-one fixed effects when the regressors are exogenous and form perfect colinearity (rank condition), and consistent for the variance estimate of the residuals when the regressors have finite fourth moments. When the error variances are finite, OLS gives minimum-variance mean-unbiased estimate. If we further assume that the errors follow a normal distribution with zero mean, then OLS is the best non-linear unbiased estimator because it maximizes the likelihood of the data.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  The Multivariate Social Scientist

  Introductory Statistics Using Generalized Linear Models

  • Graeme D Hutcheson, Nick Sofroniou(Authors)
  • 1999(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  Chapter 3 Ordinary Least-Squares Regression  

  Ordinary least-squares (OLS) regression is one of the most popular statistical techniques used in the social sciences. It is used to predict values of a continuous response variable using one or more explanatory variables and can also identify the strength of the relationships between these variables (these two goals of regression are often referred to as prediction and explanation).

  OLS regression assumes that all variables entered into the analysis are continuous and the regression procedure attaches importance to actual values. Response variables, i.e., those variables which are being modelled, must be continuous and be recorded on at least an interval scale if they are to be modelled using OLS regression1 . Response variables which cannot be assumed to be continuous may be more appropriately analysed using other generalized linear modelling techniques discussed in this book, such as logistic regression (for variables which are dichotomous) or loglinear analysis (for categorical variables in the form of frequency counts). Though explanatory variables are also required to be continuous, dichotomous data can legitimately be used in a regression model. This is particularly useful as it makes it possible to include multi-category ordered and unordered categorical explanatory variables in a regression model provided that they are appropriately coded into a number of dichotomous ‘dummy’ categories.

  OLS regression is a generalized linear modelling technique, which, as the name suggests, models linear relationships. The three components of generalized linear models for OLS regression are a random component for the response variable, which is assumed to be Normally distributed, a systematic component representing the fixed values of the explanatory variables in terms of a linear function, and finally, a link function which maps the systematic component onto the random component. In OLS regression, this is simply an identity link which means that the fitted value of the response variable is the same as the linear predictor arising from the systematic component. This might appear to be quite restrictive since a number of the relationships one might wish to model are likely to be non-linear. It is possible, however, to model non-linear relationships using OLS regression if appropriate transformations are applied to one or more of the variables which render the relationships linear.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  A Guide to Modern Econometrics

  • Marno Verbeek(Author)
  • 2017(Publication Date)
  • Wiley

    (Publisher)

  2 An Introduction to Linear Regression The linear regression model in combination with the method of ordinary least squares (OLS) is one of the cornerstones of econometrics. In the first part of this book we shall review the linear regression model with its assumptions, how it can be estimated, evaluated and interpreted and how it can be used for generating predictions and for testing economic hypotheses. This chapter starts by introducing the ordinary least squares method as an algebraic tool, rather than a statistical one. This is because OLS has the attractive property of providing a best linear approximation, irrespective of the way in which the data are generated, or any assumptions imposed. The linear regression model is then introduced in Section 2.2, while Section 2.3 discusses the properties of the OLS estimator in this model under the so-called Gauss–Markov assumptions. Section 2.4 discusses goodness-of-fit measures for the linear model, and hypothesis testing is treated in Section 2.5. In Section 2.6, we move to cases where the Gauss–Markov conditions are not necessarily satisfied and the small sample properties of the OLS estimator are unknown. In such cases, the limiting behaviour of the OLS estimator when – hypothetically – the sample size becomes infinitely large is commonly used to approximate its small sample properties. An empirical example concerning the capital asset pricing model (CAPM) is provided in Section 2.7. Sections 2.8 and 2.9 discuss data problems related to multicollinearity, outliers and missing observations, while Section 2.10 pays attention to prediction using a linear regression model. Throughout, an empirical example concerning individual wages is used to illustrate the main issues. Additional discussion on how to interpret the coefficients in the linear model, how to test some of the model’s assumptions and how to compare alternative models is provided in Chapter 3, which also contains three extensive empirical illustrations.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Cross-Over Experiments

  Design, Analysis and Application

  • David Ratkowsky, Richard Alldredge, Marc A. Evans(Authors)
  • 2020(Publication Date)
  • CRC Press

    (Publisher)

  8

  Ordinary Least SquaresEstimation Versus OtherCriteria of Estimation:Justification for Using theMethodology Presented inThis Book

  The methodology associated with the statistical analysis of cross-over designs presented in Section 1.1 rests wholly upon the assumption that least squares estimation, also called ordinary least squares and abbreviated OLS, is valid. In this chapter we examine the assumptions that have to be made to justify the use of this widely used criterion of estimation in regression analysis and the analysis of variance of designed experiments. Cross-over designs differ from many other designs in use in applied science by virtue of the fact that the experimental units (the subjects) are used more than once, thus making it unlikely that the measurements at different time periods for the same subject are independent of each other. It is the form in which the dependence occurs which determines whether or not the OLS estimator will be appropriate. Thus, we examine several possible error structures and see how closely actual data sets meet the assumptions needed to justify the use of OLS.

  8.1.   The Assumptions Underlying OLS

  It is sometimes stated that the use of ordinary least squares rests upon the assumption of independent errors. Although the independent error case is a sufficient condition for the validity of OLS, provided of course that these errors are also identically normally distributed, it is not a necessary condition. It is also sometimes stated that a more general condition of correlated errors, known as “uniform covariance” or “compound symmetry”, is required, but this too is a sufficient, not a necessary, condition to justify OLS. The most general condition, which is both sufficient and necessary as a justification for the use of OLS estimation, is that the errors obey a covariance structure called the Type H structure studied by Huynh and Feldt (1970). In this section, we will describe these various assumptions, and in Section 8.2

  Sign up to read

  Learn more about book
• 

  No longer available |Learn more

  Python: Advanced Predictive Analytics

  • Joseph Babcock, Ashish Kumar(Authors)
  • 2017(Publication Date)
  • Packt Publishing

    (Publisher)

  Several complexities complicate this analysis in practice. First, the relationships we fit usually involve not one, but several inputs. We can no longer draw a two dimensional line to represent this multi-variate relationship, and so must increasingly rely on more advanced computational methods to calculate this trend in a high-dimensional space. Secondly, the trend we are trying to calculate may not even be a straight line – it could be a curve, a wave, or even more complex patterns. We may also have more variables than we need, and need to decide which, if any, are relevant for the problem at hand. Finally, we need to determine not just the trend that best fits the data we have, but also generalizes best to new data.

  In this chapter we will learn:

  • How to prepare data for a regression problem
  • How to choose between linear and nonlinear methods for a given problem
  • How to perform variable selection and assess over-fitting

  Linear regression

  Ordinary Least Squares (OLS ).

  We will start with the simplest model of linear regression, where we will simply try to fit the best straight line through the data points we have available. Recall that the formula for linear regression is:

  Where y is a vector of n responses we are trying to predict, X is a vector of our input variable also of length n, and β is the slope response (how much the response y increases for each 1-unit increase in the value of X). However, we rarely have only a single input; rather, X will represent a set of input variables, and the response y is a linear combination of these inputs. In this case, known as multiple linear regression, X is a matrix of n rows (observations) and m columns (features), and β is a vector set of slopes or coefficients which, when multiplied by the features, gives the output. In essence, it is just the trend line incorporating many inputs, but will also allow us to compare the magnitude effect of different inputs on the outcome. When we are trying to fit a model using multiple linear regression, we also assume that the response incorporates a white noise error term ε, which is a normal distribution with mean 0 and a constant variance for all data points.

  To solve for the coefficients β in this model, we can perform the following calculations:

  The value of β is known the ordinary least squares estimate of the coefficients. The result will be a vector of coefficients β for the input variables. We make the following assumptions about the data:

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  A Guide to Modern Econometrics

  • Marno Verbeek(Author)
  • 2014(Publication Date)
  • Wiley

    (Publisher)

  2 An Introduction to Linear Regression Two of the cornerstones of econometrics are the so-called linear regression model and the ordinary least squares (OLS) estimation method. In the first part of this book we shall review the linear regression model with its assumptions, how it can be estimated, how it can be used for generating predictions and for testing economic hypotheses. Unlike many textbooks, I do not start with the statistical regression model with the standard, Gauss–Markov, assumptions. In my view the role of the assumptions underlying the linear regression model is best appreciated by first treating the most important technique in econometrics, ordinary least squares, as an algebraic tool rather than a statistical one. This is the topic of Section 2.1. The linear regression model is then introduced in Section 2.2, while Section 2.3 discusses the properties of the OLS estimator in this model under the so-called Gauss–Markov assumptions. Section 2.4 discusses goodness-of-fit measures for the linear model, and hypothesis testing is treated in Section 2.5. In Section 2.6, we move to cases where the Gauss–Markov conditions are not necessarily satisfied and the small sample properties of the OLS estimator are unknown. In such cases, the limiting behaviour of the OLS estimator when – hypothetically – the sample size becomes infinitely large is commonly used to approximate its small sample properties. An empirical example concerning the capital asset pricing model (CAPM) is provided in Section 2.7. Sections 2.8 and 2.9 discuss data problems related to multicollinearity, outliers and missing observations, while Section 2.10 pays attention to prediction using a linear regression model. Throughout, an empirical example concerning individual wages is used to illustrate the main issues.

  Sign up to read

  Learn more about book
• 

  Available until 4 Dec |Learn more

  Applied Statistics for the Social and Health Sciences

  • Rachel A. Gordon(Author)
  • 2012(Publication Date)
  • Routledge

    (Publisher)

  Fundamentally, regression modeling involves the algebra and geometry of a function, starting with a straight line. Understanding this basic math aids interpretation of our results. Returning to this basic math will be quite helpful in understanding more complicated models as we move forward, and we will repeatedly revisit the strategies for interpretation that we introduce in this chapter. Whereas a mathematical straight line function is deterministic, statistical models contain a systematic (straight line) and probabilistic component. Because of this, in regression, there are three basic parameters of the model: the intercept and slope of the straight line and the conditional variance of the distribution of values around that straight line. We will see that this is true in both the population and the sample. To test social science research questions, we need a reliable strategy for estimating the parameters of the model (the intercept, slope, and conditional variance). Ordinary least squares (OLS) is the most commonly used approach. We look in detail in this chapter at the least squares estimators and their standard errors and how to answer our research questions and evaluate our hypotheses using these estimates. We then discuss a number of strategies for evaluating the substantive size of statistically signi fi cant effects (see Box 8.1 ). In Part 4 of the book we will consider an alternative estimation strategy for the model parameters.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Econometrics

  • Fumio Hayashi(Author)
  • 2011(Publication Date)
  • Princeton University Press

    (Publisher)

  CHAPTER 1

  Finite-Sample Properties of OLS

  ABSTRACT

  The Ordinary Least Squares (OLS) estimator is the most basic estimation procedure in econometrics. This chapter covers the finite- or small-sample properties of the OLS estimator, that is, the statistical properties of the OLS estimator that are valid for any given sample size. The materials covered in this chapter are entirely standard. The exposition here differs from that of most other textbooks in its emphasis on the role played by the assumption that the regressors are “strictly exogenous.”

  In the final section, we apply the finite-sample theory to the estimation of the cost function using cross-section data on individual firms. The question posed in Nerlove’s (1963) study is of great practical importance: are there increasing returns to scale in electricity supply? If yes, microeconomics tells us that the industry should be regulated. Besides providing you with a hands-on experience of using the techniques to test interesting hypotheses, Nerlove’s paper has a careful discussion of why the OLS is an appropriate estimation procedure in this particular application.

  1.1 The Classical Linear Regression Model

  In this section we present the assumptions that comprise the classical linear regression model. In the model, the variable in question (called the dependent variable , the regressand , or more generically the left-hand [-side] variable ) is related to several other variables (called the regressors , the explanatory variables , or the right-hand [-side] variables ). Suppose we observe n values for those variables. Let

  yi

  be the i -th observation of the dependent variable in question and let (x

  i 1

  , x

  i 2

  , . . . ,

  xiK

  ) be the i -th observation of the K regressors. The sample or data is a collection of those n

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Applied Regression Analysis

  • Norman R. Draper, Harry Smith(Authors)
  • 2014(Publication Date)
  • Wiley-Interscience

    (Publisher)

  CHAPTER 1 Fitting a Straight Line by Least Squares 1.0. INTRODUCTION: THE NEED FOR STATISTICAL ANALYSIS In today's industrial processes, there is no shortage of "information." No matter how small or how straightforward a process may be, measuring instruments abound. They tell us such things as input temperature, concentration of reactant, percent catalyst, steam temperature, consumption rate, pressure, and so on, depending on the character- istics of the process being studied. Some of these readings are available at regular intervals, every five minutes perhaps or every half hour; others are observed continu- ously. Still other readings are available with a little extra time and effort. Samples of the end product may be taken at intervals and, after analysis, may provide measurements of such things as purity, percent yield, glossiness, breaking strength, color, or whatever other properties of the end product are important to the manufacturer or user. In research laboratories, experiments are being performed daily. These are usually small, carefully planned studies and result in sets of data of modest size. The objective is often a quick yet accurate analysis, enabling the experimenter to move on to "better" experimental conditions, which will produce a product with desirable characteristics. Additional data can easily be obtained if needed, however. if the decision is ini- tially unclear. A Ph.D. researcher may travel into an African jungle for a one-year period of intensive data-gathering on plants or animals. She will return with the raw material for her thesis and will put much effort into analyzing the data she has, searching for the messages that they contain. It will not be easy to obtain more data once her trip is completed, so she must carefully analyze every aspect of what data she has. Regression analysis is a technique that can be used in any of these situations.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Introductory Regression Analysis

  with Computer Application for Business and Economics

  • Allen Webster(Author)
  • 2013(Publication Date)
  • Routledge

    (Publisher)

  Table 2.4 .

  The values sum to zero because the residuals are positive when the model under-estimates the actual value of Y i (Y i > Ŷ ). However, the residuals are negative if the model over-estimates the true value for exports as ( Y i < Ŷ ).

  For example, Figure 2.4 shows the actual exports to Canada were Y c = 14.2 thousand. The model estimates the level of exports to be

  The residual is (14.2 − 10.063) = 4.14.

  ■ Figure 2.4 Arco's Regression Line

  Conversely, while the actual exports to Russia were Y R = 7.5 thousand, the estimated exports were

  The residual is (7.5 − 9.89) = −2.39. If these calculations are made for all observations in the data set, the positive and negative residuals cancel out and the residuals sum to zero.

  But if the residuals are squared, as shown in Equation (2.3.2) , the negative values disappear and no longer cancel out the positive values. The sum of the residuals squared is, however, minimized. Thus, the sum of the squared residuals, Σ(Y i − Ŷ )2 , will be minimized. That is, using the OLS procedure described here, the sum of the squared residuals will be smaller than that produced by any line other than the one OLS provides. Hence, the term “ordinary least squares.” It is the sum of the squared residuals that is “least.” From Table 2.4 we see the sum of the residuals squared is 105.073.

  That the squared residuals sum to the least possible amount testifies to the soundness of the OLS process. If you can construct a model so that the residuals sum to zero and the residuals squared are minimized it would bear witness to the model's reliability.

  The sum of the residuals for any and all models will always be zero because of the cancelation process. The sum of the residuals squared will vary among models but it will always be less than that reported by any regression model developed by any other procedure.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Advanced Statistics with Applications in R

  • Eugene Demidenko(Author)
  • 2019(Publication Date)
  • Wiley

    (Publisher)

  Condition (8.3) means that the column vectors of matrix X are linearly indepen- dent: no column can be expressed as a linear combination of others. If condition (8.3) does not hold, the coefficients cannot be uniquely estimated. Linear model (8.2) can be succinctly written as y ∼ (Xβ 2 I) (8.4) meaning that (y) = Xβ and cov(y) =  2 I Note that (8.4) specifies only the first two moments of y without specifying the distribution. In a sense, (8.4) may be referred to as a semiparametric statistical model. If y has a multivariate normal distribution, we write y ∼ N (Xβ 2 I) (8.5) According to model (8.4),   and   do not correlate for  6=  and, accord- ing to model (8.5), they are mutually independent. Since matrix X is fixed, cov(ε) = cov(y) The errors have the same variance (we call them homoscedas- tic), and under model (8.5), they are normally distributed. 8.1. Basic definitions and linear least squares 629 x1 x2 Y Figure 8.1: Observation-space geometry of the OLS (m = 2). The least squares plane minimizes the sum of squared vertical distances between observation Y  and the projected point (fitted value). Function olsg() saves 360 *.jpg files in the folder olsg for viewing at all angles. Linear model can be expressed in terms of vector columns of matrix X as y = 1 x 1 +  2 x 2 +  +   x  + ε where X is composed of the -dimensional vectors {x    = 1  } or in other notation X = [x 1  x 2   x  ] and, in the presence of the intercept, x 1 is a vector of ones. Similarly to simple linear regression from Section 6.7, the least squares is used to estimate the beta coefficients as the solution of an optimization problem: min  ky − Xβk 2  (8.6) The ordinary least squares (OLS) defined by this criterion finds the plane in  +1 that minimizes the sum of squared vertical distances (along the -axis); see Figure 8.1. This geometric illustration will be referred to as the observation- space geometry.

  Sign up to read

  Learn more about book