A threat model is an view of a system architecture that identifies security related information. The purpose of a threat model is to proactively identify potential vulnerabilities.
Threat modeling is best done during the design process while there is still time to make design as well as implementation changes. Threat modeling is typically done by a cross-functional team that includes software designers, security experts, software developers, quality engineers, and possibly others. The output of a threat model is an annotated architecture that defines the trust boundaries with an identification of information flows across them, and a list of potential vulnerabilities.
The Microsoft SDL (Secure Development Lifecycle) identifies threat modeling as foundational and Microsoft puts their money where their mouth is with lots of excellent resources here: Microsoft Threat Modeling. Threat modeling is neither easy nor cheap (with the expensive human resources brought to bear), but there really isn’t any alternative if you aim to have a comprehensive understanding of the security of your system rather than just respond to issues as they arise.