Please note: This transcript was automatically generated. We have reviewed it to ensure it reflects the original conversation, but we may not have caught every transcription error.

Christopher Saldaña: Hi, everyone. I'm Chris Saldaña, and this is the National Education Policy Center's Talks Education podcast. In this month's podcast, we spoke with Susanna Loeb, who's a professor at Stanford Graduate School of Education and faculty director of the SCALE Initiative, where she also serves as founder and executive director of the National Student Support Accelerator, an effort to expand access to relation-based, high-impact tutoring.

In this month's podcast, Professor Loeb discusses the recent release of Getting Down to Facts 3, a sweeping research project organized by Stanford's SCALE Initiative that involved 112 researchers producing 55 technical reports, 22 research briefs, and a 40-page summary offering a 360-degree look at California's public education system from preschool through high school.

So Getting Down to Facts began nearly 20 years ago. Can you take us back to the beginning? What was happening in California pre-K-12 education that made this kind of comprehensive independent research effort necessary, and how did the original vision shape what Getting Down to Facts has become? 

Susanna Loeb: Yeah.

So it is really wild that it started nearly 20 years ago, and at that time, California's education system was really struggling. So we had low achievement, we had inequitable funding, really illogical funding system, a governance structure that nobody could fully explain, and lots and lots of rules.

We had a really big education code, and districts didn't even know what was in there, so that they would just do what the district next to them did because they wanted to make sure they were following the rules, but they couldn't comprehend because there were so many of them. Everybody knew that something was wrong, but nobody really agreed on what that was.

The finance people said it was money. The governance people said it was structure. The curriculum people said there was a standards problem, and they were all partly right, which made it really hard to act and to know what to do. And so what Getting Down to Facts tried to do was to really step back and look at the system at the whole -- as a whole at once, not just one lever, but how all those pieces were connected.

And when we did that, we started to see how all those different separate problems weren't really separate. Inequitable funding was tied to governance dysfunction, which was tied to accountability gaps, which was tied to good intentions at the state level, but those not really reliably reaching the classroom And that first effort really helped to build a shared factual foundation, and it ended up informing some of California's biggest reforms, including their local control funding formula, which really fundamentally changed and I think made so much better how states fund schools.

And then the second round came after that, and we were used to stepping back and looking at the whole thing. But I think that's really the important part that stuck with us, is that it's good to take a look at the system as a whole because some of the specific things that are wrong with what's going on could be tied back to something where you don't really see those connections until you step back.

Christopher Saldaña: That's a great segue to my next question, which is when you look back, how did Getting Down to Facts research make its way into policy and practice? And more specifically, what did it take for the body of research like this to move from technical reports into real decisions made in Sacramento and in districts?

Susanna Loeb: Yeah. So that's a great question. And the honest answer is that research never really moves into policy on its own. It needs a lot of help. What we learned -- I think we learned a few things. First, we learned that timing matters enormously. California was at a moment where policymakers were genuinely looking for a new framework.

They weren't just looking for ammunition for a position they'd already taken. They were trying to understand what needed to change. But timing matters also because when we put out the first Getting Down to Facts 1, then the recession hit. And so even though there was interest in making change, it wasn't really a good time for that.

And the reports, because they were comprehensive and really weren't very time-specific in that way, they were able to be really effective once we came out of that recession and had the opportunity to make big changes. So I think timing matters a lot. Being very strategic and thinking about how you get the research into policy is important, but relationships are also important.

We weren't producing reports in isolation and handing them over. Researchers were then in conversation with policymakers and advocates throughout the process, so we produced information that set the stage. It didn't give direct recommendations, but it put us all on the same place, knowing what was going on and what the issues were.

But the researchers, Mike Kirst is a terrific example of this, were then very much part of the process of turning that into policy and moving it forward. So I guess the other thing, again, it's what I said the first time, but I think the coherence of the report was also really important for moving it forward because it allowed people to think about multiple aspects at once and really helped in that coordination process.

Christopher Saldaña: In that work that happens afterwards, when you have the researchers and other individuals trying to communicate this message and sustain whatever progress is able to happen with, given a context, how do you think about translating and communicating findings so they get used? 

Susanna Loeb: I think the most common failure is thinking that the research is good enough that people will find it and use it, and they won't.

So rigor is necessary, but it's nowhere near sufficient, and we spend a lot of time making sure that the research really speaks to the questions that the decision-makers are grappling with, and then is in a form that they can use. So we start the whole Getting Down to Facts project, Getting Down to Facts 3, with a listening tour around the state, asking policymakers, advocacy groups, educators, district leaders, families, what questions they have, what insights they have, where they want research to help inform those decisions.

So we start at a base where we're answering relevant questions. Then I think the second failure mode is producing these findings on their own without synthesis, because individual studies tend to answer very specific research questions. So we're good at answering something really narrow, and then if you have 55 studies, which we have in Getting Down to Facts 3, you can take those 55 studies and pull them together so that they can answer a whole set of questions as a synthesis that they wouldn't answer on their own.

So you have to really think carefully about how you take what research can tell you and put it in a form or in a synthesis. You synthesize it so that the, so that you're answering the relevant questions. And then I think the third is putting it in actual forms that people can use. So you've gotten this narrow research, you've synthesized it, but then you wanna have materials that speak to policymakers who are interested in one topic or they're interested in another topic, or you wanna speak to district leaders.

You, you different decision makers, some will like a model policy, like what would you do with the policy? Others just want some kinds of information and a checklist for things they should think about when they're putting something together, and you really wanna think about that kind of format.

Some people like videos. I can't watch videos. I don't know why I wanna watch a video of research results, but I have learned that lots of people like to watch videos, and that's -- so we try to do it in all sorts of different forms. 

Christopher Saldaña: How does the framework that you're outlining change when you run into things like a financial crisis, Great Recession, or a COVID-19 pandemic?

Does it change? 

Susanna Loeb: Yeah, that's a good question. Certainly it does because you wanna think about the context, and often the context and seeing what's happened is really informative about what the needs are. So you could, dor example, in the COVID pandemic, we can look and see how California responded to that, and that gives us a lot of insights about their ability to respond to change.

And I think what we saw across the country is some states really are able to respond to changing conditions, whether it's the COVID pandemic or the introduction of artificial intelligence into our system, which is creating lots of need for change as well. And what we wanted out of Getting Down to Facts 3 is to think about what California could do so it could be, could not only provide excellent education now, but could be responsive to those kinds of changes, leveraging the good parts about them, but also then adjusting where it needs to so that it, we can continue to really provide excellent opportunities for students.

Christopher Saldaña: So I want to move to ask you about Getting Down to Facts 3, which is due, I think, to be released just prior to the release of this podcast. What kinds of questions are you asking in Getting Down to Facts 3, and how do these questions compare to earlier rounds? 

Susanna Loeb: So the underlying question really has shifted.

The first round was about what's broken. We really knew -- you could take a look at our school finance system, which gave funding to districts based on what their funding was in 1977, and was just like little bits added onto that. There was no --Two districts that looked very much the same would get very different amounts of money, and one district might have much greater needs and costs than another district and get the exact same amount.

So there was no logic to it. Things were broken. The second round was more about now that we fixed these clear broken things, why aren't those fixes enough to meaningfully improve student learning? And this round is more asking, "Oh, we've improved student learning in these different ways, but we don't, we can't seem to sustain it or scale it."

And so we're not really being able to scale or to be flexible to change. Like we didn't respond as well to the pandemic as we might have been. So it's much more about how do you create this learning system that just gets better over time and is responsive. And it sounds like a subtle difference, but it leads you to different places.

You start paying attention to things like coherence, whether the goals and guidance and supports are actually aligned, and to where the capacity is, and whether the people and institutions who are responsible for implementation have what they need to do the work. So that's really the question behind getting down to FACTS 3.0.

Christopher Saldaña: Is there anything, whether it was part of the listening tour or part of the process of the research that you have found particularly surprising or that you're paying particularly close attention to? 

Susanna Loeb: So I think that either the extent of fragmentation in the state systems, particularly the state systems of support for improvement, or the amount of administrative burden facing educators, how much time they were spending on paperwork and responding to these compliance requirements.

We knew these requirements were significant, but I think we found that really striking because that's time not spent on instruction. It's spent on filling out forms. And it's not concentrated in a few places. It shows up across the system. But I think the other striking feature is how much really good stuff was going on in schools. I spend a lot of time thinking about tutoring and providing students individualized attention, and a lot of that was going on.

There was a lot of work on making high schools more engaging and supportive for students, and that's going on. So we have these pockets of really good things going on in the state that we can build from. 

Christopher Saldaña: What do you see as the most pressing opportunities or challenges that Getting Down to Facts 3 is surfacing for California's pre-K-12 system?

Susanna Loeb: I think what we found is that California has really bold, ambitious goals for students that go beyond test scores to career and college readiness, as well as well-being and civic participation goals. But those goals don't always lead, as we were saying, to consistent supports and excellent educational experiences across the state.

And that is a result of what I think are the ABCs of what we need to do in California. And the first is really, the A is about alignment and accountability. You can think about a well-functioning system where you know that as a district, there are clear goals that to be working towards. You have some goals that are developed locally, but there are also state goals, and there is some system that's around you that can help you work towards those goals, and you know what that system is, and it's consistent.

We don't have that kind of alignment and accountability. We asked many local leaders who they were accountable to, and they just didn't really know. And it wasn't a sense that there was a really shared direction that the state was going in. So I think the A of alignment and accountability is really important to get right.

And the second is B, which is a balance between state and local control and really state guidance and local flexibility. We have 1,000 districts in the state. It's really important to have flexibility. They go from Los Angeles to Alpine County, very big and urban to very rural and small, and they face different opportunities to really support their students well.

And so we have to have that flexibility. But at the same time, California's turned over so many decisions to the local level. For example, they can use so many different math curriculum that each district, each of our districts has to review all the possible ones that the state says meet a level.

And so they spend a lot of time doing that kind of review, which is costly for so many people in the state to be doing it. And it also means that the systems of support have to be good for every possible one of those curricula that pass a relatively low bar. Whereas if we could have the state provide greater guidance with still choice, and then really provide supports around it, we could get a better balance between kind of the state guidance and local flexibility.

So we have A, accountability and alignment, B, really about this balance, and the third, and I think probably the most important, is about capacity. We have some great capacity needs in the state. The educator workforce, while we have of course wonderful educators, wonderful teachers and school leaders, and the state has made progress in increasing the supply of teachers through a number of grants they've put in.

We still have a relatively low supply. Many districts are looking for teachers that they can't hire. And there was a sense, for example, in elementary math, that the educators didn't have necessarily the skills and supports to take on these new math frameworks that the state's really been pushing.

So there's a real need for educator capacity, and many states have put in systems of support that helps districts build that capacity, but California's is particularly fractured. So we rely a lot on counties to do things, and some counties do it well, and some counties don't provide that well, and the districts are going in all sorts of different places to find this.

And so we found a need for capacity building, increased capacity at the district level and supports for that capacity building. And so I think those ABCs are a really important part of what our last finding is, which is that we have these great instances and proof points of success. And we have these great goals that we want for students, and it's such an innovative state, California, that we can do all these great things, but we really need this strong ABC system, and then an investment in innovation to, to address things like climate change, to leverage the new technologies, to address political stresses when they come in to maybe do with immigration, which is clearly one right now that California is facing.

So I think that's the final finding that I find very inspirational in some ways that we've got all these proof points. If we have this strong system and then invest in this innovation and learning from it, we really could go great places as a state. 

Christopher Saldaña: What do you see as some of the immediate opportunities that exist in terms of the findings that you have in Getting Down to Facts 3?

Susanna Loeb: As in the other Getting Down to Facts, some of these changes can take a while. It's not just six months later and everything will be there. But I think there are lots of opportunities to build alignment and accountability. We've seen already action in the state that tries to make it so that the governor has more oversight of the Department of Education, so that the governor is willing, in many ways, to put some money in to the Department of Education and into other state organizations that could help with this, with building capacity so that we could have more alignment.

Some of the policy proposals out there are about alignment in Sacramento. We looked at across some of the large states in the country, and we're the only one that actually outsources the accountability part. So while lots of states outsource capacity building, which you need to because it's a very big state and you need to have people all over the state doing the support, the setting of goals and the oversight of how schools and districts are doing is generally held at the state level and not given to all these different places.

And so I think that's an, a place to, to rethink our accountability. So I think there are lots of things to do in that area, and I think we can move forward there. The balance, I think, is really about policymakers realizing that compliance requirements are costly, and reducing thinking about the compliance requirements when they're putting in the policies.

When they give a lot of flexibility, that's good, but choice is also costly, and having guidance is really helpful. And I think that can -- just that kind of information can help in each of the policies that come out. And then I really do hope that there is investment with the money that's, we are a state that is that has these ups and downs because we are not as reliant on the property tax as some other states are.

But that also means that we sometimes have opportunities to do this kind of investment in learning, and if we could be really systematic about that, I think there are great opportunities that artificial intelligence offers us that can really streamline some of the systems and really help teachers.

It can coordinate -- Special education, for example, is very, very difficult to coordinate. We could have so much better coordination with really some thoughtful implementation there. I think we have a lot of opportunities and that my hope is that the state will move forward in trying to make use of these opportunities in a more systematic way than it has done in the past.

It has done this. Now it's a question of coordinating it. 

Christopher Saldaña: I wanna go back to ask you about the process that happens after the results have been published- 

Susanna Loeb: Yeah ... 

Christopher Saldaña: And the change process that happens. You mentioned the districts in California range from Los Angeles to Alpine County. And that reflects some of the politics that exists across the districts, but we know too that there will be politics within. How do you think about the different constituency groups that exist within the state and how these results can help them make sense and improve systems, but also how those sort of constituency groups might create tensions in trying to make this work be useful and be implemented?

Susanna Loeb: Yeah. It's a really good question, and it's a very interesting political world we work in is thinking about education. I think one of the powers of Getting Down to Facts is that we really do try to provide the facts. We talk to many, many superintendents and principals and teachers across the state, really trying to get representative samples as we try to figure out what's going on in the districts across the state.

We look at a lot of administrative data. We talk with a lot of families, and so we really try to get a picture that represents the full complexity of the state, and that we try as much as possible to present this information as this is what's here, and here are a number of opportunities, but that we don't give direct recommendations because we do understand that the actual policy development and the decisions in districts and state offices in schools is really a combination of what the facts and opportunities are and the goals and the contexts that you're in.

And what we're hope to do is to have people come together at a starting space and then have the discussion really be about how can we reach our goals and what our goals are, and to work those things out together. And that's why we're really as careful as we are about not trying to be too directive because we know that there is a lot of values that go into these decisions as there should be.

And I think it allows us to be useful not just this week or this month, but over the next eight years or whatever it is until we step back again. And that's the goal of Getting Down to Facts. 

Christopher Saldaña: That's awesome. I know that there are states that wish they had a Getting Down to Facts infrastructure, and for leaders in other states who want something like Getting Down to Facts, what does it take to pull this off?

The funding, the relationships you've mentioned, what sort of approach do you use and what are some of the necessary political conditions? 

Susanna Loeb: So I think you have to keep some things in mind when you're trying to put something like this together, and I do think that really any state could do this, that you just have to decide, yes, we wanna get this body of evidence and we have some funds to put into it.

We give the researchers a little bit of funds, but not that much. Researchers in a state want to contribute to these things, and they need some money for research assistants or whatever to do interviews, but it doesn't have to be huge quantities of money that goes into it. But I think you do need to have kind of a perspective on it and one thing is, it takes a while.

It's a two-year process to get to where we are right now, and each of the studies is not a perfect empirical study on its own. It really summarizes what we know about a topic and then adds on in a relatively short period of time what we can-- how much further we can go in that area. So just realize the kind of what is possible.

I think it's really important for the research to be independent. It only works if kind of people across the political spectrum trust the work. So you want to make sure that, again, that what we were just talking about, that it's not too political, that it really is about getting the information down there.

You want to think about the scope. I think it's both tempting to focus on one or two issues and lose that coherence, and also to think that you can cover everything, and you can't cover everything 'cause there are so many things. Civic education, I wish we could have covered civic education. We did not.

There are all these things that we'd love to cover in this and you have to decide what level you want to cover it at, but it is really good to approach those things that you want to cover from multiple angles. And then we talked about this as well, but I think getting everybody on board is really important.

That listening tour and understanding what people are interested in, keeping them involved. We have advisory groups as we go. We're doing all these pre-releases before we release the data in order to make sure that we have everybody's perspective in there. All of that work is a really important part of it.

But I think if you just stay curious, that you treat it as something that is imperfect but valuable, it can really help in a state to get everybody on the same page and realize what the issues are and what the opportu- like what the options are for what you might do. And so I really think it's doable.

Not that I want to do it this -- certainly not soon but it's possible for people to do this anywhere. 

Christopher Saldaña: Thank you, Professor Loeb, for being on this month's podcast. I'm sad to say that this will be the last podcast that I will host for the NEPC Talks Education podcast. But remember that we always hope you're safe and healthy, and for the latest analysis on education policy, you should subscribe to the NEPC newsletter at nepc.colorado.edu.