Cloud service providers are often prohibited from accessing the content of tenant VMs, yet current techniques for monitoring attacks and unauthorized activities rely on virtual machine introspection (VMI). While the introspections are useful for narrowing down the semantic gap between the status observed at the hypervisor-level and that seen in a VM, they potentially reveal the sensitive information of a tenant stored in the machine. In this paper, we aim to infer specific program activities in a VM without VMI methods, where our approach has to solve the strong semantic gap problem. We introduce Infermatic, a system that utilizes only hypervisor-level features and supervised machine learning methods to infer program behaviors in a VM. Using the classifiers trained by Infermatic, we can also bridge the strong semantic gap by systematically identifying the semantic links between our hypervisor features and selected program behaviors. In evaluations, we demonstrate that the hypervisor features are effective in isolating program activities and do so with an average accuracy of 0.875 (AUC) for the 24 behaviors that we have identified. In addition, our statistical models (or trained classifiers) can identify the hypervisor features that accurately characterize selected program behaviors when they involve lower-level operations. We further extend Infermatic's ability to detect program behaviors to other security applications—we present a malicious VM detector for the cloud that achieves an average detection of 0.817 (AUC). Our detector shows the hypervisor features are resilient against evasion attacks even when an attacker can reduce the number of available features to the system. Moreover, we present that the detector can operate in a scalable manner by identifying a malicious VM even when the VM under inspection is collocated with other VM's operating under workloads.