Nanospacecraft could enable detailed characterization of many asteroids in a small timeframe by launching multiple spacecraft simultaneously to visit a large set of targets. To be able to characterize as large of a set of asteroids as possible, however, visits to individual asteroids would be limited to flybys, which would have to be autonomous. As an additional challenge, due to the limitations of nanospacecraft and a massively parallel architecture, Earth-based localization and communication infrastructure, such as the Deep Space Network, cannot be relied upon. We have developed an optical instrument prototype and an image simulation system for autonomous asteroid flybys and 3D multispectral mapping using nanospacecraft. The final system is targeted to fit into a single CubeSat unit (a 10 cm cube) and to have the mass of less than 1 kg. The images are used for structure from motion algorithms to determine the quality of 3D reconstruction to be expected from this mission. We are presenting the instrument design, the simulation system, and the simultaneous localization and mapping system developed for nanospacecraft asteroid flybys.