Analyzing Communications and Software Systems Security

We rely on various communications and software systems where security is critical. Many of these systems have transformed drastically over time with the addition of new features and technologies to accommodate our increasing needs. Unfortunately, such a transformation can introduce new security threats and weaknesses. This dissertation studies security threats and weaknesses in systems that continue to evolve with legacy and modern software components and paradigms. In this dissertation, we study four different types of information systems: desktop, mobile communications, cloud, and hardware. Our analysis mainly involved building attacks to exploit the vulnerabilities to demonstrate the practicality of our research findings. We uncovered various security issues in each of the systems analyzed. Also, we present various defense and mitigation solutions to address the security issues we found. We discussed our research findings with a wide range of audiences through peer-reviewed publications, responsible disclosure efforts, and by giving talks at various conferences. The summary of the results is as follows. First, we found insecure use of local communication channels in desktop applications. Second, we discovered several security issues in commercial VPN clients that a network adversary can exploit. Third, we studied mobile communication systems and uncovered security weaknesses of signaling protocols. Also, we present a conceptual framework to model the threats and attacks to mobile networks. Fourth, we demonstrate how adversaries can conduct cross-site scripting attacks by exploiting third-party add-ons of cloud application suites. Finally, we also conduct a human factor analysis to identify usability and security pitfalls faced by software developers when using trusted platform module library APIs. In summary, the contributions of this dissertation include a novel adversary model to study local communication inside a computer, a conceptual framework to study mobile communication systems, the discovery of several new types of security vulnerabilities, and insights into developers' struggles while using security technologies.