Changjun Zhao,∗ Weiru Niu,∗ Qiming Li,∗ Songling Du,∗ and Jianhua Chen∗
Internet of Things (IoT), big data, distributed file system, energy- saving task scheduling ∗ Marketing Division of State Grid Gansu Electric Power Company, Lanzhou, 730000, China; e-mail: {zcj, nwr, lqm, dusongling}@gs.sgcc.com.cn; [email protected] Corresponding author: Jianhua Chen Full name Abbreviation Central Processing Unit CPU Input/Output IO Distributed File System HDFS Sensor Files Storag SensorFS istribmed Memory File System DMFS Yet Another Resource Negotiator, YARN Dy
Efficient storage and management of internet answer data in power marketing are crucial for the development of power enterprises. To achieve this, the paper establishes a distributed storage system called Sensor Files Storage based on the Hadoop Distributed File System extension. This system ensures high scalability and data security, enabling scientific retrieval and backup processing of data for power marketing enterprises. Meanwhile, an energy- saving scheduling framework for stream tasks using Yet Another Resource Negotiator is established. Combined with server dynamic voltage and frequency scaling, a reasonable allocation of tasks and servers is achieved, and two batch task scheduling algorithms are given. Compared with the hierarchical clustering method, the average reduction ratio of the time overhead of the sensor clustering algorithm in schemes 1–5 was about 9900%, 9700%, 9600%, and 9500%, respectively. For different task types, the error value of the actual and estimated power consumption was within 10 W, and the error rate was below 5%. This paper aims to provide new ideas for the optimisation of big data management theory and system optimisation methods.
Important Links:
Go Back
