Discovering the connectivity patterns of directed networks is a crucial step towards understanding complex systems such as human brains and financial markets. Network inference approaches aim at estimating the hidden topology given nodal observations. Existing approaches relying on structural equation models (SEMs) require full knowledge of exogenous inputs, which may be unrealistic in certain applications. Recent tensor-based alternatives advocate reformulation of SEMs as a three-way tensor decomposition task that only requires second-order statistics of exogenous inputs for identifying the hidden topology. However, the tensor-based methods are computationally expensive, and is hard to incorporate prior information of the network structure (e.g., sparsity and local smoothness), but prior information is often important for enhancing performance. The present work puts forth a joint diagonalizaition (JD)-based formulation for directed network inference. JD can be viewed as a variant of tensor decomposition, but features more efficient algorithms and can readily incorporate prior information of network topology. New topology identification guarantees that do not rely on knowledge of exogenous inputs are established. Judiciously designed simulations are presented to showcase the effectiveness of the proposed approach.