A Beginner Template for Descriptive Statistics in R

Why is it important to understand your data?¶

Descriptive statistics will provide you with one important thing: understanding your data.

Understanding the data is the only way to make clever choices for the next steps of the analysis process. Indeed knowing minimal information about the data will allow you - for instance - to correctly prepare the preprocessing phase:

• What are the columns to (re)encode (e.g. from string factors to numerical values),
• What data can I throw away because they bring no added value,
• What is the distribution that rules the data, the skewness, and so forth.

Descriptive statistics brings a lot of answers to these questions.

Take away?¶

After reading this article, the reader will be provided with:

• a well documented template to obtain a runnable R program that calculates essential metrics described in section [ What Are Descriptive Statistics?],
• a solid understanding of all the calculated metrics (e.g. what, why, how),

Preliminaries¶

It may seem dummy to insist on that step; but believe me when I say that it is very easy to forget it: display a sample or a subset of your raw data:

1
2
3

ncol(dataset)
nrow(dataset)
head(dataset)

When the dataset points to the Ionosphere [Rep89] data, then the result is something like:

1
2
3
4
5
6
7
8
9
10
11
12
13

35
  351

    V1 V2      V3       V4       V5       V6       V7       V8      V9      V10
  1  1  0 0.99539 -0.05889  0.85243  0.02306  0.83398 -0.37708 1.00000  0.03760
  2  1  0 1.00000 -0.18829  0.93035 -0.36156 -0.10868 -0.93597 1.00000 -0.04549
  3  1  0 1.00000 -0.03365  1.00000  0.00485  1.00000 -0.12062 0.88965  0.01198
  ...
        V27      V28      V29      V30      V31      V32      V33      V34 Class
  1  0.41078 -0.46168  0.21266 -0.34090  0.42267 -0.54487  0.18641 -0.45300  good
  2 -0.20468 -0.18401 -0.19040 -0.11593 -0.16626 -0.06288 -0.13738 -0.02447   bad
  3  0.58984 -0.22145  0.43100 -0.17365  0.60436 -0.24180  0.56045 -0.38238  good
  ...

On the console, everything looks fine with the data; but... not so good. Since we are doing Data Science and not pure Statistics, we are constrained by the computer’s laws. It means that, one should - first of all - discover the data types of each columns, features, or variables. Because, “What You See (on the console) Is Not What You Get (in real life)!”:

1

sapply(dataset, levels)

The command produces:

1
2
3
4
5
6
7
8

$V1
[1] "0" "1"

$V2
[1] "0"
...
$Class
[1] "bad"  "good"

The previous results illustrates a frequent trap easy to avoid. Indeed, by simply displaying the data, one can reasonably think that the column V1 is numeric... but no it is not! It is called a factor column in R speaking: a set of string labels; exactly like the Class column. In addition, this command reveals the real nature of V2: a constant column with a single value for all lines (e.g. observations). Since V2 is a constant column, under the machine learning perspective it brings no added value at all, and should be removed to save space.

Thus one can apply now some early processing actions:

• transform column V1 from strings to numerics, and
• drop column V2 which is useless

1
2

dataset$V1 <- as.numeric(as.character(dataset$V1))
dataset <- dataset[,-2] # col index of V2 is 2

Descriptive Statistics¶

The Descriptive Statistics [Yau] of a dataset can be defined as the first set of figures used to represent that dataset. These figures SHOULD at least contain the following metrics.

1
2

quantile_vector <- c(.1, .25, .5, .75) # 10%, 25%, 50%, 75%
sapply(dataset, percentile, quantile_vector=quantile_vector)

Libraries¶

The first actions consist to load the required libraries. It is difficult to guess a-priori how many of them will be required. Thus, if you don’t know which ones to import, then leave it empty for now and come back later to import the right ones.

1
2
3
4
5
6
7
8
9

# a) Load the required libraries

    # Example:
    # library(mlbench)

# b) set seed

    # Example:
    # seed <- 1234

In the previous listing, the mlbench [FL12] package is imported as example. Indeed, this package is a good data provider among many other things. The reader is invidted to read the internal documentation about this package. Finally, the seed global variable is set to guarantee the reproductibility of the program when the same conditions hold.

Functions¶

Functions can be defined here. They are optional ; but they can increase the code reuse level.

Data loading¶

Now it is time to load the dataset and place it into a data frame object.

1
2
3
4
5
6

# Example: the Ionosphere dataset from mlbench package
# data(Ionosphere)
# dataset <- Ionosphere

# Example: from CSV file
# dataset <- read.table(...)

Here, the developer can update the parameters lib.loc or package to provide more information about the location of the data. Please once again confer to the internal help system for more information about the data() function.

Data summary¶

This part is where you effectively implement the descriptive statistics steps already described and illustrated with examples.