Lai, CS orcid.org/0000-0002-4169-4438, Tao, Y, Xu, F et al. (7 more authors) (2019) A robust correlation analysis framework for imbalanced and dichotomous data with uncertainty. Information Sciences, 470. pp. 58-77. ISSN 0020-0255
Abstract
Correlation analysis is one of the fundamental mathematical tools for identifying dependence between classes. However, the accuracy of the analysis could be jeopardized due to variance error in the data set. This paper provides a mathematical analysis of the impact of imbalanced data concerning Pearson Product Moment Correlation (PPMC) analysis. To alleviate this issue, the novel framework Robust Correlation Analysis Framework (RCAF) is proposed to improve the correlation analysis accuracy. A review of the issues due to imbalanced data and data uncertainty in machine learning is given. The proposed framework is tested with in-depth analysis of real-life solar irradiance and weather condition data from Johannesburg, South Africa. Additionally, comparisons of correlation analysis with prominent sampling techniques, i.e., Synthetic Minority Over-Sampling Technique (SMOTE) and Adaptive Synthetic (ADASYN) sampling techniques are conducted. Finally, K-Means and Wards Agglomerative hierarchical clustering are performed to study the correlation results. Compared to the traditional PPMC, RCAF can reduce the standard deviation of the correlation coefficient under imbalanced data in the range of 32.5%–93.02%.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2018 Elsevier Inc. This is an author produced version of a paper published in Information Sciences. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Pearson product-moment correlation; imbalanced data; clearness index; dichotomous variable |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Civil Engineering (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 20 Aug 2018 10:44 |
Last Modified: | 16 Aug 2019 00:43 |
Status: | Published |
Publisher: | Elsevier |
Identification Number: | 10.1016/j.ins.2018.08.017 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:134706 |