12 Generalized Linear Models

GLM in Oracle Data Mining

GLM is a parametric modeling technique. Parametric models make assumptions about the distribution of the data. When the assumptions are met, parametric models can be more efficient than non-parametric models.

The challenge in developing models of this type involves assessing the extent to which the assumptions are met. For this reason, quality diagnostics are key to developing quality parametric models.

Ridge Regression

The best regression models are those in which the predictors correlate highly with the target, but there is very little correlation between the predictors themselves. Multicollinearity is the term used to describe multivariate regression with correlated predictors.

Ridge regression is a technique that compensates for multicollinearity. Oracle Data Mining supports ridge regression for both regression and classification mining functions. The algorithm automatically uses ridge if it detects singularity (exact multicollinearity) in the data.

Information about singularity is returned in the global model details. See "Global Model Statistics for Linear Regression" and "Global Model Statistics for Logistic Regression".

Build Settings

You can use build settings to specify:

• Coefficient confidence — The GLMS_CONF_LEVEL setting indicates the degree of certainty that the true coefficient lies within the confidence bounds computed by the model. The default confidence is .95.

• Row weights — The ODMS_ROW_WEIGHT_COLUMN_NAME setting identifies a column that contains a weighting factor for the rows.

• Row diagnostics — The GLMS_DIAGNOSTICS_TABLE_NAME setting identifies a table to contain row-level diagnostics.

Additional build settings are available to:

• Control the use of ridge regression, as described in "Ridge Regression".

• Specify the handling of missing values in the training data, as described in "Data Preparation for GLM".

• Specify the target value to be used as a reference in a logistic regression model, as described in "Logistic Regression" .

Diagnostics

GLM models generate many metrics to help you evaluate the quality of the model.

Data Preparation for Linear Regression

When ADP is enabled, the build data are standardized using a widely used correlation transformation (Netter, et. al, 1990). The data are first centered by subtracting the attribute means from the attribute values for each observation. Then the data are scaled by dividing each attribute value in an observation by the square root of the sum of squares per attribute across all observations. This transformation is applied to both numeric and categorical attributes.

Prior to standardization, categorical attributes are exploded into N-1 columns where N is the attribute cardinality. The most frequent value (mode) is omitted during the explosion transformation. In the case of highest frequency ties, the attribute values are sorted alpha-numerically in ascending order, and the first value on the list is omitted during the explosion. This explosion transformation occurs whether or not ADP is enabled.

In the case of high cardinality categorical attributes, the described transformations (explosion followed by standardization) can increase the build data size because the resulting data representation is dense. To reduce memory, disk space, and processing requirements, an alternative approach needs to be used. For large datasets where the estimated internal dense representation would require more than 1Gb of disk space, categorical attributes are not standardized. Under these circumstances, the VIF statistic should be used with caution.

Data Preparation for Logistic Regression

Categorical attributes are exploded into N-1 columns where N is the attribute cardinality. The most frequent value (mode) is omitted during the explosion transformation. In the case of highest frequency ties, the attribute values are sorted alpha-numerically in ascending order and the first value on the list is omitted during the explosion. This explosion transformation occurs whether or not ADP is enabled.

When ADP is enabled, numerical attributes are standardized by scaling the attribute values by a measure of attribute variability. This measure of variability is computed as the standard deviation per attribute with respect to the origin (not the mean) (Marquardt, 1980).

Missing Values

When building or applying a model, Oracle Data Mining automatically replaces missing values of numerical attributes with the mean and missing values of categorical attributes with the mode.

You can configure a GLM model to override the default treatment of missing values. With the ODMS_MISSING_VALUE_TREATMENT setting, you can cause the algorithm to delete rows in the training data that have missing values instead of replacing them with the mean or the mode. However, when the model is applied, Oracle Data Mining will perform the usual mean/mode missing value replacement. As a result, statistics generated from scoring may not match the statistics generated from building the model.

If you want to delete rows with missing values in the scoring the model, you must perform the transformation explicitly. To make build and apply statistics match, you must remove the rows with NULLs from the scoring data before performing the apply operation. You can do this by creating a view.

CREATE VIEW viewname AS SELECT * from tablename 
     WHERE column_name1 is NOT NULL 
     AND   column_name2 is NOT NULL 
     AND   column_name3 is NOT NULL ..... 

Coefficient Statistics for Linear Regression

GLM regression models generate the following coefficient statistics:

• Linear coefficient estimate

• Standard error of the coefficient estimate

• t-value of the coefficient estimate

• Probability of the t-value

• Variance Inflation Factor (VIF)

• Standardized estimate of the coefficient

• Lower and upper confidence bounds of the coefficient

Global Model Statistics for Linear Regression

GLM regression models generate the following statistics that describe the model as a whole:

• Model degrees of freedom

• Model sum of squares

• Model mean square

• Model F statistic

• Model F value probability

• Error degrees of freedom

• Error sum of squares

• Error mean square

• Corrected total degrees of freedom

• Corrected total sum of squares

• Root mean square error

• Dependent mean

• Coefficient of variation

• R-Square

• Adjusted R-Square

• Akaike's information criterion

• Schwarz's Baysian information criterion

• Estimated mean square error of the prediction

• Hocking Sp statistic

• JP statistic (the final prediction error)

• Number of parameters (the number of coefficients, including the intercept)

• Number of rows

• Whether or not the model converged

• Whether or not a covariance matrix was computed

Row Diagnostics for Linear Regression

For linear regression, the diagnostics table has the columns described in Table 12-1. All the columns are NUMBER, except the CASE_ID column, which preserves the type from the training data.

Reference Class

You can use the build setting GLMS_REFERENCE_CLASS_NAME to specify the target value to be used as a reference in a binary logistic regression model. Probabilities will be produced for the other (non-reference) class. By default, the algorithm chooses the value with the highest prevalence. If there are ties, the attributes are sorted alpha-numerically in ascending order.

Coefficient Statistics for Logistic Regression

GLM classification models generate the following coefficient statistics:

• Name of the predictor

• Coefficient estimate

• Standard error of the coefficient estimate

• Wald chi-square value of the coefficient estimate

• Probability of the Wald chi-square value

• Standardized estimate of the coefficient

• Lower and upper confidence bounds of the coefficient

• Exponentiated coefficient

• Exponentiated coefficient for the upper and lower confidence bounds of the coefficient

Global Model Statistics for Logistic Regression

GLM classification models generate the following statistics that describe the model as a whole:

• Akaike's criterion for the fit of the intercept only model

• Akaike's criterion for the fit of the intercept and the covariates (predictors) model

• Schwarz's criterion for the fit of the intercept only model

• Schwarz's criterion for the fit of the intercept and the covariates (predictors) model

• -2 log likelihood of the intercept only model

• -2 log likelihood of the model

• Likelihood ratio degrees of freedom

• Likelihood ratio chi-square probability value

• Pseudo R-square Cox an Snell

• Pseudo R-square Nagelkerke

• Dependent mean

• Percent of correct predictions

• Percent of incorrect predictions

• Percent of ties (probability for two cases is the same)

• Number of parameters (the number of coefficients, including the intercept)

• Number of rows

• Whether or not the model converged

• Whether or not a covariance matrix was computed.

Row Diagnostics for Logistic Regression

For logistic regression, the diagnostics table has the columns described in Table 12-2. All the columns are NUMBER, except the CASE_ID and TARGET_VALUE columns, which preserve the type from the training data.