Control of light, light-matter interactions, frequency conversion: in our Perspective just published in ACS Photonics we discuss the state of the art, emerging challenges and new frontiers for ultrafast all-optical metasurfaces.