Ideal photodiodes can detect all incoming photons independently of the wavelength, angle or intensity of the incident light. Present-day photodiodes notably suffer from optical losses and generated charge carriers are often lost via recombination. Here, we demonstrate a device with an external quantum efficiency above 96% over the wavelength range 250–950 nm. Instead of a conventional p–n junction, we use negatively charged alumina to form an inversion layer that generates a collecting junction extending to a depth of 30 µm in n-type silicon with bulk resistivity larger than 10 kΩ cm. We enhance the collection efficiency further by nanostructuring the photodiode surface, which results in higher effective charge density and increased charge-carrier concentration in the inversion layer. Additionally, nanostructuring and efficient surface passivation allow for a reliable device response with incident angles up to 70°. We expect the considered device to improve data quality, reduce the area of photodiodes as well as decrease the cost per pixel.