How to Evaluate Phone Policies in Schools

The Toolkit for Assessing Phones in Schools Release, Part 2

Source. Shutterstock

Note: If you’re an educator, school administrator, or a policymaker interested in phone-free school policies, this post is for you! If you’re a parent or other stakeholder who is interested in helping your school implement and/or measure a new phone policy, we invite you to share this post and the Toolkit for Assessing Phones in Schools website with your school’s leaders.

Over the past few years, momentum to restrict smartphone use in schools has grown rapidly. Notably, 40 U.S. states and around one quarter of countries in the world have now adopted or proposed policies aimed at reducing phone use during the school day. With growing public and institutional interest in phone policies, schools, school districts, and governments need an easy, research-validated way to help assess their effectiveness and impact.

That’s why, last fall, we launched the Toolkit for Assessing Phones in Schools (TAPS), a free, ready-to-use, research-informed toolkit that helps stakeholders measure the effects of a new or modified school phone policy. The TAPS includes four core surveys for students, teachers, administrators, and parents that are available in two user-friendly formats (Google Forms and printable PDF files), accompanying methodologies, and a detailed User Guide. The TAPS empowers you, as an educator, administrator, or school leader, to assess how your school’s phone policy affects your students directly.

And now, we added some new resources to the TAPS that make it even easier to measure changes in your school’s phone policy, including:

• A new manual scoring guide that walks you step-by-step through how to analyze the responses for each survey based on question type, as well as how to create easy-to-understand visuals of the data.

• An add-to-cart survey design experience that allows you to choose “a la carte” from our library of measures, for researchers who want to customize their own surveys.

• Coming Soon: A new Data Dashboard, an automated scoring app that does the analysis work for you, immediately transforming Google Form results into graphs, averages, and other useful statistics. (Sign up on our website to get updates on the Dashboard release.)

We’re also offering an exciting opportunity for schools or legislators to partner with the Stanford Social Media Lab on your survey implementation and measurement. If you choose to partner with the Stanford Lab, the team will help you design your policy study, implement your pre- and post-policy surveys via the Qualtrics platform, and conduct a comprehensive statistical analysis. If you’re interested in partnering, please fill out our contact form.

Even if you don’t partner directly with the Stanford Lab, you can share your findings with us after implementing the TAPS. In either case, you will be contributing to a large body of research on the efficacy of phone policies.

Together, these TAPS enhancements are designed to lower barriers to evaluation and produce consistent, policy-relevant evidence on what happens when schools go phone-free. As more and more schools adopt the TAPS, we will be able to draw stronger, evidence-backed conclusions about phone policies, ultimately helping leaders and educators make decisions that prioritize student well-being, attention, and learning.

You can learn more about the new features below, jump right into the TAPS website, or fill out our contact form if you’re interested in partnering with us!

Our latest release of the TAPS introduces new tools designed to support two primary use cases.

The first: states, research organizations, and large districts seeking to conduct rigorous, multi-school evaluations — with the option of collaborating with the Stanford Social Media Lab. These efforts typically involve tracking outcomes across many schools and, in some cases, over multiple years.

The second: individual schools that want to better understand the efficacy of their phone policies internally, without launching a full-scale research study.

Deep Dive on the New TAPS Features

1. Qualtrics-Ready Surveys

Best for: Organizations interested in partnering with the Stanford Social Media lab to design and execute robust studies, evaluate phone policies across multiple schools, and conduct advanced statistical analyses will benefit from these expanded components of the TAPS.

The Student, Teacher, Administrator, and Parent surveys are now available in Qualtrics format. Qualtrics is a secure, widely used survey platform that supports advanced data collection, branching logic, and statistical analysis. The Qualtrics versions are better suited for large-scale studies and longitudinal data collection (versus the more basic Google Forms versions).

Image 1. Online and mobile versions of the Teacher Survey using the Qualtrics interface

2. Expanded Research Collaboration Opportunities

Best for: Organizations interested in partnering with the Stanford Social Media lab to design and execute robust studies, evaluate phone policies across multiple schools, and conduct advanced statistical analyses will benefit from these expanded components of the TAPS.

Organizations that partner with the Stanford Social Media Lab can design more comprehensive evaluation strategies, including longitudinal tracking, careful measurement of confounding variables, and tailored survey configurations. Additionally, researchers interested in executing their own study can request access to the Qualtrics survey. These collaborations make it possible to move beyond descriptive reporting toward stronger causal inference about the effects of school phone policies.

If your school district or organization is interested in collaborating with the Stanford Lab, please fill out our contact form on the TAPS website and indicate “Research Collaboration” in the “Subject” field.

3. Clear Scoring and Interpretation Guidance

Best for: Individual schools, school leaders, and school districts that want to conduct their own analysis.

To support practical decision-making, TAPS now includes a manual scoring guide that helps schools interpret results without advanced statistical training. In addition, we have developed an automated Data Dashboard that will make this process even easier (coming out soon).

• The manual scoring guide provides clear instructions for individuals who want to conduct the data analysis themselves.

• The TAPS Data Dashboard (coming soon) is a tool that automatically generates statistics, plots, and a compilation of feedback from respondent data. To use it, simply download a CSV file of data collected through the TAPS Google Forms surveys and upload the data to the Dashboard. The Dashboard can support both an analysis of data from a single timepoint and a longitudinal analysis of data from two timepoints (e.g., before and after a new phone policy was implemented).

These guides are designed to answer common, applied questions: Are students reporting fewer distractions? Do teachers perceive changes in classroom management? Are there differences across grade levels?

4. Customizable Shopping Cart Survey Builder

Best for: Individual schools, school districts, states, and research organizations who want to fully customize their phone policy surveys.

The Custom Shopping Cart survey option gives researchers the flexibility to individualize the TAPS surveys to their own interests, and work with schools to develop a relevant and sound study of their phone policies. Within the Measures Library, users can explore over 80 validated measures covering a large diversity of school outcomes, some of which are included in the six core TAPS surveys and some which are not. Then users can bookmark the measures most relevant to their goals and automatically generate a PDF file of all selected measures and questions. This document can then be used to adapt the existing TAPS surveys or create a new survey altogether.

Cautions and Caveats

While TAPS is a valuable tool for assessing and tracking changes in school environments, it is not designed to definitively establish causality with high confidence. Like most observational tools, the TAPS cannot fully control for all confounding variables without a randomized control trial.

That said, the TAPS makes it easy for individual schools to track changes within their school, and for larger entities such multi-school districts or states, as well as research teams, to conduct studies in which they can compare the change scores, over time, between schools that changed their phone policy and schools that did not. Such studies can offer meaningful evidence to help schools decide whether to continue, adjust, or discontinue a policy based on observed outcomes and stakeholder feedback.

For guidance on interpreting results and understanding the limits of causal claims, see the TAPS: User Guide.

Try It Out

We care about phones in schools because we care about the education of our students. We developed these additions and features in direct response to your feedback after our initial launch of the TAPS. If you’re introducing a new phone policy in your school, district, or state — or if you want to evaluate an already established policy — the TAPS will make it easy for you to measure the changes. Take a look!

If you’re interested in collaborating with the Stanford Lab, please fill out our contact form.