Data preprocessing: Difference between revisions

Revision as of 11:26, 7 August 2023

Revision as of 11:31, 7 August 2023

Line 16:

==Data mining==

{{Cleanup section|date=August 2023|reason=This section requires grammar and capitalisation fixes}}

The origins of data preprocessing are located in [[data mining]].{{cn|date=March 2021}} The idea is to aggregate existing information and search in the content. Later it was recognized, that for machine learning and neural networks a data preprocessing step is needed too. So it has become to a universal technique which is used in computing in general.

Data preprocessing allows for the removal of unwanted data with the use of data cleaning, this allows the user to have a dataset to contain more valuable information after the preprocessing stage for data manipulation later in the data mining process. Editing such dataset to either correct data corruption or human error is a crucial step to get accurate quantifiers like true positives, true negatives, [[False positives and false negatives]] found in a [[Confusionconfusion matrix]] that are commonly used for a medical diagnosis. Users are able to join data files together and use preprocessing to filter any unnecessary noise from the data which can allow for higher accuracy. Users use Python programming scripts accompanied by the pandas library which gives them the ability to import data from a [[Commacomma-separated values]] as a data-frame. The data-frame is then used to manipulate data that can be challenging otherwise to do in Excel. [[pandas (software)]] which is a powerful tool that allows for data analysis and manipulation; which makes data visualizations, statistical operations and much more, a lot easier. Many also use the [[R (programming language)|R programming language]] to do such tasks as well.

The reason why a user transforms existing files into a new one is because of many reasons. Data preprocessing has the objective to add missing values, aggregate information, label data with categories ([[Data binning]]) and smooth a trajectory.{{cn|date=March 2021}} More advanced techniques like principal component analysis and [[feature selection]] are working with statistical formulas and are applied to complex datasets which are recorded by GPS trackers and motion capture devices.