Stop Letting Educational Tech Grade Itself

During a recent nationwide gathering of district technology leaders, one of the attendees asked a simple question: Are we measuring the impact on student learning, or are we simply buying into the sales pitch?

The simple answer should have been, “yes, of course,” but the truth is that no one answered with conviction.

That lack of response tells a compelling story of how schools buy technology. Every district is prepared to deliver the documents on the spot: the board resolution, the signed contract, the license count, and the rollout plan. Almost none of them is ready to show data on whether the purchase actually helped students 18 months later.

Adoption is easy because tools are cheap to pilot, fast to implement, and often included in the bundles of products districts are already paying for. Accountability is hard because it requires breaking two comfortable habits: using physical observation as a substitute for proof, and letting the software grade its own homework.

The Clipboard and the Black Box

If you ask a superintendent or a district administrator how they can be sure about the effectiveness of a new instructional tool, they will share stories about the walkthroughs in classrooms. Students were looking engaged on their screens. Teachers seemed supported. Everything felt fine.

Visible compliance is not evidence of learning.

Being able to see students in class relieves you of the need to define the objective. “Working” becomes a surface impression that never turns into a testable claim. You cannot audit an impression. You cannot budget for it. And you cannot submit it to a school board on renewal day.

The twin sister to the clipboard problem is the vendor echo chamber. Districts invest in the product. Then they rely on its dashboard as the main source of information about the progress of learning. If the app says that the reading level of the students grew two tiers, but there is no external assessment or authentic student work to verify that gain, then you do not have evidence. There is a self-reporting algorithm protecting the contract.

Using technology to assess its own impact is as flawed as relying on screen time as proof of understanding.

No Hallway

I serve as Chief Academic Officer of a public charter network in Southern California that educates about 2,500 students, transitional kindergarten through 12th grade, using a non-classroom-based model. Our students learn at home on flexible schedules, communicating with their credentialed teachers and completing assignments virtually. Some are top-notch athletes or artists, some are medically fragile, and some left a traditional campus because the standard mold stopped fitting them.

Our model has no hallways and no classroom doorframes to stand in.

If a student struggles in our network, nobody sees them slumped at a desk. We see it in our operational data: widening gaps between weekly submissions, declining quality of work, skipped check-ins, or a change in how a student writes to an advisor.

For a long time, after 20 years spent working in traditional schools, I considered this lack of physical space a disadvantage of our model. But I have changed my mind. Being forced to write down and measure what brick-and-mortar systems keep informal helped us create an accountable model.

Four Questions to Ask Before You Buy

Whenever we evaluate new technology, or when colleagues ask how to cut through a vendor pitch, I work through four planning questions before anyone signs a purchase order:

1. What is it for? Name the specific human decision or instructional bottleneck the tool needs to address. Avoid vague goals like “increasing engagement.” Think about concrete utility: currently it takes two weeks for our advisors to understand if a student in our independent study program is struggling with math, and this tool must decrease that timeframe to three days.
2. What does it strengthen? Specify the exact human capacity that is going to be developed. Will it give a teacher deeper diagnostic insight during a check-in, or will it help a student develop self-regulation? If you cannot name a human capacity, then you have digital busywork.
3. What does it replace? Every tool displaces something else. If you do not specify what product or manual process this tool will replace, then you create additional cognitive load for an overworked staff. But most importantly, think about unintended displacement. If an early literacy app leads teachers to stop listening to students read aloud in real time, then this efficiency has cost you a vital human practice.
4. What does it make easy to do poorly? Every platform reduces the friction of doing something undesirable. Will it make it easy for students to show shallow compliance without understanding? Will it make it easy for teachers to replace personal feedback with a generic prompt? If you cannot anticipate how the tool will fail, then you cannot design safeguards around it.

How to Run an Honest Pilot

After you answer the four questions above, the operational rules become straightforward:

• Capture the baseline first. A pilot launched without a performance baseline produces nothing but testimonials. Testimonials are marketing, not data. If you have no systems that can measure the issue at hand, then fix the data gap before buying software.
• Evaluate the outcomes independently of vendor telemetry. Companies report the number of logins, prompt counts, and internal badges because the metrics are easy to retrieve from servers. These data show you that the platform was opened, not that it taught something durable. Verify the results with independent assessments and student work from outside the software environment.
• Schedule the review on the purchase day: Fix the date and name the person responsible for the evaluation on the day the contract is signed. Reviewing software during renewal season fails because canceling the platform then feels like a public admission of a mistake.
• Establish walk-away criteria in advance: If there is no specific failure that will cause you to cancel the use of the software, then you are not running a pilot. You are running a rollout under a gentler name.

Why Campus Districts Have the Harder Problem

Schools with physical buildings have a harder version of this problem, not an easier one.

The walkthrough provides confidence without data, and unearned confidence is what kills inquiry. A traditional district can run for years on the impressions gathered in hallways, only to realize at renewal time that nobody ever defined what success looked like.

The ground is shifting anyway. Independent study programs, credit recovery, hybrid models, and online dual enrollment move parts of the learning experience beyond the physical classroom every semester. The percentage of the school day a principal can personally observe keeps getting smaller. Every district is moving toward the conditions we deal with daily, whether they planned for it or not.

Who’s Going to Check?

Not the teachers. They will tell you honestly what they think of a tool, but they did not choose it, and auditing it is not a fair thing to add to a full caseload.

Not the vendor. A product team optimizes for platform usage across all of its customers, which means your students exist as nothing but aggregated activity logs. The pitch deck exists to close a sale.

That leaves the people who signed the purchase order, asked teachers to change their instructional routines, and told families the investment would help their children thrive.

Eighteen months from now, someone needs to show whether the purchase improved students’ skills, and to point at something other than a classroom walkthrough to prove it. In my organization, there is no building to walk into and no room to read.

We find the evidence anyway.