Data preprocessing: Difference between revisions

Revision as of 00:00, 7 March 2024

Revision as of 08:09, 7 April 2024

Line 20:

===Data mining===

{{Cleanup section|date=August 2023|reason=This section requires grammar and capitalisation fixes}}

Data preprocessing allowsalows for the removal of unwanted data with the use of data cleaning, this allows the user to have a dataset to contain more valuable information after the preprocessing stage for data manipulation later in the data mining process. Editing such dataset to either correct data corruption or human error is a crucial step to get accurate quantifiers like true positives, true negatives, [[false positives and false negatives]] found in a [[confusion matrix]] that are commonly used for a medical diagnosis. Users are able to join data files together and use preprocessing to filter any unnecessary noise from the data which can allow for higher accuracy. Users use Python programming scripts accompanied by the pandas library which gives them the ability to import data from a [[comma-separated values]] as a data-frame. The data-frame is then used to manipulate data that can be challenging otherwise to do in Excel. [[Pandas (software)]] which is a powerful tool that allows for data analysis and manipulation; which makes data visualizations, statistical operations and much more, a lot easier. Many also use the [[R (programming language)|R programming language]] to do such tasks as well.

The reason why a user transforms existing files into a new one is because of many reasons. Aspects of data preprocessing may include imputing missing values, aggregating numerical quantities and transforming continuous data into categories ([[data binning]]).<ref>{{Cite book |last1=Hastie |first1=Trevor |url=https://books.google.com/books?id=eBSgoAEACAAJ |title=The Elements of Statistical Learning: Data Mining, Inference, and Prediction |last2=Tibshirani |first2=Robert |last3=Friedman |first3=Jerome H. |date=2009 |publisher=Springer |isbn=978-0-387-84884-6 |language=en}}</ref> More advanced techniques like principal component analysis and [[feature selection]] are working with statistical formulas and are applied to complex datasets which are recorded by GPS trackers and motion capture devices.