Data preprocessing: Difference between revisions

Revision as of 17:04, 15 June 2012

Revision as of 08:20, 6 July 2012

Line 1:

'''Data pre-processing''' is an often neglected buta important step in the data mining process. The phrase [[GIGO|"garbage in, garbage out"]] is particularly applicable to [[data mining]] and [[machine learning]] projects. Data-gathering methods are often loosely controlled, resulting in out-of-range values (e.g., Income: −100), impossible data combinations (e.g., Gender: Male, Pregnant: Yes), [[missing values]], etc. Analyzing data that has not been carefully screened for such problems can produce misleading results. Thus, the representation and quality of data is first and foremost before running an analysis.<ref>Pyle, D., 1999. ''Data Preparation for Data Mining.'' Morgan Kaufmann Publishers, [[Los Altos, California]].</ref>

If there is much irrelevant and redundant information present or noisy and unreliable data, then [[knowledge discovery]] during the training phase is more difficult. Data preparation and filtering steps can take considerable amount of processing time. Data pre-processing includes [[Data cleaning|cleaning]], normalization, transformation, [[feature extraction]] and selection, etc. The product of data pre-processing is the final [[training set]]. Kotsiantis et al. (2006) present a well-known algorithm for each step of data pre-processing.<ref>S. Kotsiantis, D. Kanellopoulos, P. Pintelas, "Data Preprocessing for Supervised Leaning", ''International Journal of Computer Science'', 2006, Vol 1 N. 2, pp 111–117.</ref>