Data preprocessing: Difference between revisions

Revision as of 12:20, 20 December 2012

Revision as of 12:20, 20 December 2012

Line 1:

'''Data pre-processing''' is a important step in the [[data mining]] process. The phrase [[GIGO|"garbage in, garbage out"]] is particularly applicable to data mining and [[machine learning]] projects. Data-gathering methods are often loosely controlled, resulting in [[range error|out-of-range]] values (e.g., Income: −100), impossible data combinations (e.g., Gender: Male, Pregnant: Yes), [[missing values]], etc. Analyzing data that has not been carefully screened for such problems can produce misleading results. Thus, the representation and [[data quality|quality of data]] is first and foremost before running an analysis.<ref>Pyle, D., 1999. ''Data Preparation for Data Mining.'' Morgan Kaufmann Publishers, [[Los Altos, California]].</ref>

the training phase is more difficult. Data preparation and filtering steps can take considerable amount of processing time. Data pre-processing includes [[Data cleaning|cleaning]], [[data normalization|normalization]], [[data transformation|transformation]], [[feature extraction]] and selection, etc. The product of data pre-processing is the final [[training set]]. Kotsiantis et al. (2006) present a well-known algorithm for each step of data pre-processing.<ref>S. Kotsiantis, D. Kanellopoulos, P. Pintelas, "Data Preprocessing for Supervised Leaning", ''International Journal of Computer Science'', 2006, Vol 1 N. 2, pp 111–117.</ref>