Security Failures in Modern Software

Security vulnerabilities are a major concern for software developers. Some vulnerabilities are simple software bugs, while others result from fundamental changes in the software architecture and the underlying technologies. This dissertation studies the security concerns that arise from these ongoing architectural developments in modern software. A major change in software systems over time has been the shift towards more distributed architectures. This development takes place on several levels. One of the most prominent changes has been the transformation of cloud applications towards a microservice architecture, in which loosely-coupled software modules communicate over the network through well-defined APIs. This architecture enables each module to be developed and operated independently. Moreover, the APIs can be opened for third parties to build add-on features. A similar architectural transformation can also be seen in desktop applications. Instead of running as a single computer program, many follow the client-server architecture and have separate frontend and backend components. The components run on the same computer and connect to each other through inter-process communication (IPC). There have also been changes to the underlying networking technologies. In enterprise and data-center networks, the traditional network paradigm is gradually replaced with software-defined networking (SDN) for more flexibility and control. Regular users, on the other hand, have adopted virtual private networks (VPN), which were initially developed for corporate networking, as a solution for enhanced security and privacy in the distributed software world. The contributions of this dissertation include discovery of several new types of security failures in modern software, and empirical analysis of these vulnerabilities in deployed software products. We study the security of third-party add-ons in cloud applications and explain how they can bring cross-site scripting vulnerabilities to the applications. We show that such vulnerabilities appear widely in the wild. We also study the security of IPC between software components inside the computer and show that desktop application developers have overlooked critical security issues. We find IPC in many applications, including password managers, security tokens, and cryptocurrency wallets, to be vulnerable to impersonation and man-in-the-middle attacks mounted by local attackers. Furthermore, we study the security of SDN with focus on topology poisoning attacks by compromised network elements. We also examine commercial VPN services and identify several configuration flaws in the VPN clients. Finally, we analyze the potential solutions of each type of vulnerability.