Exploratory data analysis (EDA) is used to analyze and investigate data sets and summarize their main characteristics numerically as well as visually.

The primary aim of exploratory data analysis is to examine the data for distribution, outliers and anomalies to direct specific testing of your hypothesis. It helps look at the data before making any assumptions. It can help identify obvious errors, as well as better understand patterns within the data and find interesting relations among the variables.

10 Quick Steps in Data Exploration and Preprocessing

1. Identification of variables and data types: This step helps in understanding whether the variable is numeric or categorical. Within Numeric we can check if its Discrete (results from counting for example – number of employees in a line of business) or Continuous (the number which can take any value such as daily expenses of a household). Within Categorical we can check if the variable is Ordinal (if the categories can be ordered logically for example – rating from customer on product satisfaction) or Nominal (this has levels that cannot be ordered such as Gender-Male, Female).

• Analyzing the basic metrics: This includes understanding the data and what are the possible measurement strategies that could be employed to measure those variables.

• Non-Graphical Univariate Analysis: This process numerically explores the data by looking at summary statistics of each variable. Summary statistics provide various measures such as minimum, maximum, mean, 25 percentile score, 50^th percentile score, 75^th percentile score, count etc.

• Graphical Univariate Analysis: This includes detail study of each variable used in the analysis. The variables are graphically explored using histogram, box plot etc. to understand the data distribution.

• Bivariate Analysis: This includes taking 2 variables at a time and assessing their correlation. It also includes mapping of each independent variable against dependent variable to see if it can influence the dependent variable significantly.

• Variable transformations: Transformation is a mathematical operation that changes the measurement scale of a variable. This is usually done to make dataset useable with a particular statistical test or method. Many statistical methods require data that follow a particular kind of distribution, usually a normal distribution.

• Missing value treatment: Missing data in your data set can reduce the power / fit of a model or can lead to a biased model. It can lead to wrong prediction or classification. Missing values can be imputed using various methods or algorithms. Basic imputation includes replacing missing values with mean, median or mode depending upon the data type and distribution.

• Outlier treatment: Outliers are extremely low/high values in your data set. It usually is calculated using box plot and values outside the range of < Q1-1.5*IQR or >Q3+ 1.5*IQR are considered as outliers commonly. Having said this, outliers are very sensitive and must be carefully excluded, included or imputed. This becomes an easy task if you have a strong domain knowledge and know the metrics used for analysis.

• Correlation Analysis: Correlation analysis is used to quantify the degree to which two variables are related. Through the correlation analysis, you evaluate correlation coefficient that tells you how much one variable change when the other one does. Correlation analysis provides you with a linear relationship between two variables.

• Dimensionality Reduction: Dimensionality reduction is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension.There are many ways to reduce the dimension. One most popular way is by application of Principal Component Analysis (PCA).