Stochastic optimal controller design for medium access constrained networked control systems with unknown dynamics

This paper proposes a stochastic optimal controller for networked control systems (NCS) with unknown dynamics and medium access constraints. The medium access constraint of NCS is modelled as a Markov Decision Process (MDP) that switches modes depending the channel access to the actuators. We then show that using the MDP assumption, the NCS with medium access constraint can be modelled as a Markovian jump linear system. Then a stochastic optimal controller is proposed that minimizes the quadratic cost function using Q-learning algorithm. The resulting control algorithm simultaneously optimizes the quadratic cost function and also allocates the network bandwidth judiciously by designing a scheduler. Two compensation strategies transmit zero and zero-order hold for control inputs that fail to get an access to channel are studied. The proposed controller and scheduler are illustrated using experiments on networks and simulations on an industrial four-tank system. The advantage of the proposed approach is that the optimal controller and scheduler can be designed forward-in-time for NCS with unknown dynamics. This is a departure from traditional dynamic programming based approaches that assume complete knowledge of the NCS dynamics and network constraints beforehand to solve the optimal controller problem backward-in-time.