We propose critical interactivity as a concept to study and design the dynamic and transitory aspects of data visualizations. Theoretically, interactivity is often described as the means to support analytical tasks, while in practice, it encompasses the techniques that alter visual representations. These notions are a useful starting point to study the role of interactivity in critical engagements with data visualizations. At the core of critical interactivity is the negotiation of authority and agency: authority as authors provide structure and context, and agency as viewers navigate and interpret the data on their own terms. This raises the critical question: who has the power to control the visualization? Drawing from four case studies in science communication, art history, anthropology, and climate advocacy, we examine how critical interactivity links exploration and narration. We reflect on the effort involved in preparing data, and propose design strategies for implementing critical interactivity in data visualization.