arXiv preprint ·
Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
In brief
Audit First, Analyze Better
The authors build a C++17 pipeline that audits variable coverage before moving through anomaly detection, PCA, clustering, and machine learning. Applied to 421,937 LoRaWAN frames collected over two years, it shows that temperature and humidity—not all four proposed variables—are the viable pair for joint analysis.
What they found: The coverage audit found CO2 at 0% and pressure at 4.24%, so the planned four-variable analysis was not supported by the data. Temperature and humidity were jointly available in 205,005 frames, with a reported correlation of r = -0.662; the first PCA component captured 83.1% of variance. Isolation Forest flagged 2.12% of records as multivariate anomalies, compared with none detected by the univariate Z-score, while calibrated DBSCAN marked 0.17% as noise across 13 clusters.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Long-term environmental IoT/LPWAN deployments often aggregate heterogeneous data streams from diverse devices without systematic auditing of each variable's actual coverage prior to analysis-an increased risk when multiple sensor types coexist within the same network. This work proposes a C++17 processing pipeline applied to the Tour Perret LoRaWAN dataset (421 937 frames, June 2021--June 2023, 5 devices). A systematic coverage audit integrated into the extraction phase reveals that CO2 is completely missing (0% coverage) and pressure is reported only by an isolated base module (Wyres, 4.24% of the corpus), invalidating the initial hypothesis of a joint four-variable analysis. The analysis is refocused on the Temperature--Humidity pair, jointly available across 205{,}005 frames (95.76%) from 4 Elsys EMS sensors. The pipeline sequentially performs anomaly detection (modified Z-score, Isolation Forest), bivariate Principal Component Analysis, clustering (K-means, DBSCAN), and a multilayer perceptron predicting humidity from temperature and time, whose residuals serve as a detector for slow drift. On real-world data: Temperature--Humidity correlation r=-0.662, two-component PCA (PC1 =83.1% of variance), Isolation Forest detects 2.12% multivariate anomalies while univariate Z-score detects none, calibrated DBSCAN isolates 0.17% noise across 13 clusters, and the MLP achieves $R^2=0.672$ (RMSE =10.54%) in cross-validation. This work demonstrates that per-variable coverage auditing prior to modeling is a necessary methodological step for heterogeneous multi-sensor IoT datasets.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Covariance-Informed Ensemble Score Filtering: Embedding Localized Covariance Geometry into Score-Based Transport
Satellite-derived bathymetry of Arabian Gulf ports under leakage-free validation: a multi-site benchmark of empirical, machine-learning and deep-learning models against 285,000 multibeam soundings, with an outlook for UAE waters
GNN-VIEW: Visual Inspection and Explanation of Graph-based Earth-System Models
Huayu-Global: Skillful real-time quasi-global precipitation retrieval
FASST: A Compact Stateless Atmosphere Model for Long-Term Ocean Coupling
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.