TL;DR: A survey of the state of the art in protocol specification and verification can be found in this paper, with a focus on state exploration, symbolic execution, structural induction, and program proof.
Abstract: Interest in more rigorous definition and analysis of communication protocols is increasing. This paper surveys the current state of the art in protocol specification and verification. Methods for specification such as abstract machines, Petri nets, formal languages, abstract data types, and programs are described and compared. Verification methods including state exploration, symbolic execution, structural induction, and program proof are discussed. Work is progressing rapidly in many of these areas, and no clearly superior method has emerged yet. At least in the area of specification, some of these methods are ready for use by a wider community of protocol designers and users.