DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction

Machine learning models often converge slowly and are unstable due to the significant variance of random data when using a sample estimate gradient in SGD. To increase the speed of convergence and improve stability, a distributed SGD algorithm based on variance reduction, named DisSAGD, is proposed in this study. DisSAGD corrects the gradient estimate for each iteration by using the gradient variance of historical iterations without full gradient computation or additional storage, i.e., it reduces the mean variance of historical gradients in order to reduce the error in updating parameters. We implemented DisSAGD in distributed clusters in order to train a machine learning model by sharing parameters among nodes using an asynchronous communication protocol. We also propose an adaptive learning rate strategy, as well as a sampling strategy, to address the update lag of the overall parameter distribution, which helps to improve the convergence speed when the parameters deviate from the optimal value—when one working node is faster than another, this node will have more time to compute the local gradient and sample more samples for the next iteration. Our experiments demonstrate that DisSAGD significantly reduces waiting times during loop iterations and improves convergence speed when compared to traditional methods, and that our method can achieve speed increases for distributed clusters.

Evenly Distributed Clusters over a Target Area of Wireless Sensor Network

Wireless sensor networks(WSNs) finds wide applications in variousfields. The most important problem faced by these networksis low lifetime. These are generally battery powered devices withability to communicate with each other. Networks should be designedso that load is equally distributed among the nodes. In WSNs maximum load is on nodes being cluster heads, so for proper load distribution various nodes should get chance of becoming cluster head. Further entire network should have proper connectivity that is clusters should be evenly distributed throughout the network. To achieve this paper discusses algorithm to get dominating sets in a fully connected network.Dominating sets ensure that either a node is a cluster head or is adjacent to a cluster head. This leads to even distribution whichmay increase the lifespan of entire network. Not much attention has been given to even distribution of clusters.WSNs consists ofspatially distributed nodes over a target area with sensing and communication facility. Purpose of thesenodes is to study the entire area and communicate their observation to the central base station. This work presents an idea to form evenly distributed clusters. Even distribution in necessary for proper load sharing and prolonging the life of network. It needs much more emphasis than given to it. Further Ranking methodology has been discussed to rank the dominating sets based on certain parameters. This ranking methodology is used to determine which dominating set should become cluster heads ensuring even distribution.Ranking methodology comes into play if more than one dominating sets are obtained. These may be used where it is difficult for humans to physically visit on a regular basis. High lifetime of network ensures less physical presence of humans.