Data preprocessing: Difference between revisions

Revision as of 20:44, 16 January 2009

Revision as of 16:06, 23 July 2009

Line 1:

{{context}}

ManyData factorspre-processing affect(i.e., preparation / [[Data_cleaning|cleaning]]) is an often neglected but important step in the successdata ofmining process. The phrase [[GIGO|"Garbage In, Garbage Out"]] is particularly applicable to [[data mining]] and [[Machinemachine learning]] projects. Data gathering methods are often loosely controlled, resulting in out-of-range values (MLe.g., Income: -100), onimpossible adata givencombinations task(e.g., TheGender: Male, Pregnant: Yes), [[missing values]], etc. Analyzing data that has not been carefully screened for such problems can produce misleading results. Thus, the representation and quality of the instance [[data]] is first and foremost before running an analysis.<ref>Pyle, D., 1999. ''Data Preparation for [[Data mining|Data Mining]].'' Morgan Kaufmann Publishers, [[Los Altos]], CA.</ref>

If there is much irrelevant and redundant information present or noisy and unreliable data, then knowledge discovery during the training phase is more difficult. It is well known that data preparation and filtering steps take considerable amount of processing time in ML problems. Data [[Preprocessing|pre-processing]] includes data cleaning, normalization, transformation, [[feature extraction]] and selection, etc. The product of data pre-processing is the final [[training set]]. Kotsiantis et al. (2006) present a well know algorithm for each step of data pre-processing.<ref>S. Kotsiantis, D. Kanellopoulos, P. Pintelas, Data Preprocessing for Supervised Leaning, International Journal of [[Computer science|Computer Science]], 2006, Vol 1 N. 2, pp 111-117.</ref>

==References==

{{reflist}}