Data preprocessing: Difference between revisions

Revision as of 22:00, 25 September 2023

Revision as of 21:32, 8 December 2023

Line 20:

===Data mining===

{{Cleanup section|date=August 2023|reason=This section requires grammar and capitalisation fixes}}

The origins of data preprocessing are located in [[data mining]].{{cn|date=March 2021}} The idea is to aggregate existing information and search in the content. Later it was recognized, that for machine learning and neural networks a data preprocessing step is needed too. So it has become to a universal technique which is used in computing in general.

Data preprocessing allows for the removal of unwanted data with the use of data cleaning, this allows the user to have a dataset to contain more valuable information after the preprocessing stage for data manipulation later in the data mining process. Editing such dataset to either correct data corruption or human error is a crucial step to get accurate quantifiers like true positives, true negatives, [[false positives and false negatives]] found in a [[confusion matrix]] that are commonly used for a medical diagnosis. Users are able to join data files together and use preprocessing to filter any unnecessary noise from the data which can allow for higher accuracy. Users use Python programming scripts accompanied by the pandas library which gives them the ability to import data from a [[comma-separated values]] as a data-frame. The data-frame is then used to manipulate data that can be challenging otherwise to do in Excel. [[pandas (software)]] which is a powerful tool that allows for data analysis and manipulation; which makes data visualizations, statistical operations and much more, a lot easier. Many also use the [[R (programming language)|R programming language]] to do such tasks as well.