Questions to Ask a CS Content Provider

Lately, I’ve had my head down getting the CSforAll Consortium off the ground. One of the first things we did with our members was to do a needs assessment, asking 250 members to complete a form question, and interviewing 150 to get more information.

Larger results coming before too long, but there were many requests for a “dating app” for CS Education – how do schools and other members find each other in order create worthwhile partnerships? Some members suggested a “star” rating for curriculum providers, but I’ve pushed back against that option. First, I don’t want to be an arbiter of quality (or a moderator of comments). Secondly, I think stars are a bad idea. To extend the dating app analogy, what makes someone a good match is a complex set of questions – I’m not sure I would want a dating app that just gave folks stars. Instead you fill out a complex survey to help do a preliminary sort.

I’ve been thinking a lot about what are the right questions that might make up such a questionnaire for CS education. In addition to our current filtering options, I’m considering a standards (all, not just CS) mapping tool, some general pedagogical approaches encouraged by the curriculum, and some technological requirements.

Edsurge put out an article about what questions to ask an EdTech company before purchasing anything. From the article:

School leaders can better understand whether a flashy product matches their needs by asking its creators questions about what research guided the conceptualization of the tool, what data informed its specific attributes, and what quality and volume of research has been conducted regarding its effectiveness. I recommend beginning with six key questions to separate the promises from the proven solutions.

I like the questions and think they apply to CS education too – especially #6 if you are considering purchasing a curriculum. “What training and support will you offer to ensure we implement the solution properly and get the most out of it?”

I especially like the focus on appropriate implementation – too often new or unfamiliar curriculum and pedagogy comes with a key ingredient that makes it successful. For example, the Exploring Computer Science curriculum believes strongly in their culturally relevant approach and the professional development they offer contains pedagogical richness that you cannot get just downloading the curriculum documents. As we move past the early adopter phase of CSforAll, we need to be clear with our partners about what makes for quality CS from the beginning.

April 24th, 2017, posted by ldelyser

A Clear Definition of Computational Thinking and Related Topics

One of the most frequently asked questions in meetings about CS is “What is Computational Thinking? What does it mean for schools?” People often cite Jeannette Wing saying it is “ the thought processes involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information processing agent”.

Unfortunately, while that definition translates well to computer scientists, teachers, curriculum writers and administrators are often left unsure of what that looks like in a classroom, how to prepare students to engage in that kind of thinking, and what it means for assessment.

Recently, Computing at School, the UK movement for CS Education, released Computational Thinking: A guide for teachers. It is the best translation of computational thinking for K12 educators I have yet seen.  Is it perfect? no. But provides enough definitions – both of CT itself and the surrounding vocabulary, to help clarify in conversations. I highly recommend adding it to your reading queue if you work in or around schools.

February 12th, 2016, posted by ldelyser

Reductio Ad Absurdum: The danger of lean in philosophies of higher education

There has been a lot of great discussion around “great teaching” and the responsibility of college faculty for supporting students, not with outside interventions, but inside the classroom through good pedagogical approaches.

In some cases, it was suggested that students who were unable to self-learn beyond what professors could offer could perhaps find a better fit at institutions that were not as focused on research. The debate was lively, full of personal anecdotes by current faculty (who most likely were outliers among their students to continue studies and even earn a PhD), and pleas to think of what was “best for the students”.

The nuances to this debate are many. Faculty and educators who engage in the debate may choose their words carefully (or not so much) and understand often the intent behind statements that walk the line between micro-agressions and pragmatism. Administrators and the general public are not always able to read between the lines of our discourse.

Today in my news feed an article appeared about Mt. Saint Mary’s University in Maryland. The president of the University is trying to boost his retention numbers by early identification of students who struggle with the transition to college. His language is absurd, and the comments he makes are ridiculous. However, it is a stark reminder of the danger in fixed mindset and “lean-in” philosophies in higher education.

From the president:

Instead of thinking of students as cuddly bunnies, you just have to drown the bunnies … put a Glock to their heads.

This was his attempt to classify the “weed out” mentality he hoped his faculty would pursue. Inflammatory language – absolutely. What worries me more? The idea that students are pre-determined to succeed or fail based upon their inability to adjust quickly, lean in, and “take” their education.

February 11th, 2016, posted by ldelyser

References and Resources

This morning the SIGCSE list shifted to a positive light with folks thinking about how to reflect on practice. I offered the following as some references and resources that I thought would be great to share here. From my email:

Thanks for shifting this to a positive frame! I’d also like to contribute some resources to folks who are thinking about questioning their practice and looking for references.

While we know about some CS specific pedagogical approaches (through SIGCSE publications, ICER, and Mark Guzdial’s blog) there are also some more general resources people may not be aware of.

The Institute for Education Sciences (IES), which is the US DOE research division, has a couple of practice guides that may be of interest to this community.

Organizing Instruction and Study to Improve Student Learning

Encouraging Girls in Math and Science

In addition to those two guides, you may also be interested in applying some of the research-backed recommendations from these guides as well:

Using Student Achievement Data to Support Instructional Decision Making

Improving Mathematical Problem Solving Grades 4-8

Teaching Strategies for Improving Algebra Knowledge for Middle and High School Students

Although the last two are specific to mathematics, the struggles students have with abstraction and algebraic representation are mirrored in novices attempts to learn to program.  Substitute code problems and worked examples for the algebraic ones discussed and you have some interesting pedagogical practices that may impact your students.  Much of the research indicates the more students are able to discuss with each other (similar to the Peer Instruction work in CS, or the Think-Pair-Share protocol) the more they will learn and retain.

A quick note: even though some recommendations are marked with “low” evidence, I can assure you that does not mean “low” in the way you are thinking. Low indicates quasi-experimental design with results for a few studies. Few SIGCSE studies would meet IES’s standard for review, let alone be qualified as any level of evidence (as most lack a control group). This statement is not made to judge SIGCSE work, but instead communicate the rigor IES is using for evaluation.

I would also recommend the Teach Like a Champion series.  It is a great collection of pedagogical approaches that help clarify the power moves a teacher can make in real-time in a classroom to engage students in deeper learning. Not all of the suggestions are for everyone, but choosing a few can start you thinking about pedagogy in your class in a way that doesn’t feel like “more work” or taking time away from your research (except for the initial reading).

Without the option to have a faculty specific methods course for all – these resources (and recommending them to grad students and new faculty) can also be impactful. At CMU I taught a course – Introduction to CS Education which involved opportunities for microteaching, as well as readings from the practice guides above.  Grad students who took the course shared with me that they wrote better teaching statements, and made better research presentations after completing the course.

Thank you all for hanging in through this long message.

February 9th, 2016, posted by ldelyser

Tolerating “poor” teaching

I participate in a few list-servs around CS Education, and over the past 24 hours a thread on one of them has asked the question “Faculty who are poor teachers – why do we tolerate them?”

I responded to one of the posts, and I’d like to share that response here as well. Its going to start with a quote from a message to the list.

… my friends and relatives and neighbors who have
children in high school who are considering where to go to college often ask
me for my advice.  My answer is that one has to know one’s child.  Poor
students who need hand-holding should go to colleges (and some universities)
where teaching is a central focus and graduate and research programs are
either small or non-existent.  That is where they will find “teachers.”

Good students will learn despite everything that may get in their way.
Those students will do well at research universities even as undergraduates
if they realize that education will not be *given* to them at the
university, but it is there for them to *take*.  Such students get involved
with the professors, work in their labs, and learn as much or more from
other students as they do from any professors.  What do you get at name
schools such Harvard and MIT and Stanford and Carnegie Mellon and the like?
Great teaching by the professors?  Sometimes.  Great teaching by fellow
students?  Virtually always.

My response:

I have to admit I’m struggling with how to respond to this.

Over the past four years I have been working with the amazing teachers and students in NYC to help launch CSNYC and the CS4All movement. I have also been a faculty member at Carnegie Mellon University.  I taught high school for 8 years in one of the most competitive districts in the nation (constantly in US News top 50) and have taught CS at the local community college. I began my teaching career in a drop-out factory in South Florida, teaching 16 year old students who could not sound out the word “who” in 9th grade. I have experienced the broad range of students and classroom environments in my career.

Many of the statements in the above quote troubled me, and my struggles stem purely from a belief that the author does not see the micro-aggressions embedded in the statements made.

Please take a second and recognize your defenses probably just jumped up.

The first reaction I had was to the phrase “poor students who need hand holding”. I am shocked that in 2016 good pedagogy could still be referred to as ‘hand holding’, let alone be relegated to “poor” students.

Secondly, the implication that students who don’t succeed in poor pedagogical environments are just not working “hard enough” to seek out extra learning experiences reminds me of the Lean In argument. In my own experience, I had to work through undergrad – my father was unemployed, and I took a placement by Work Study as a way to help pay my tuition. I could not have worked in someone’s lab, and actually turned down offers, because I would have lost a significant part of my financial aid package.

If students are struggling to pay tuition, or placing themselves in lifelong debt to attend our institutions, and have cleared the hurdles we put in place for them to get there, should it not be our responsibility to TEACH them? I’m not asking you to change your assessments. Give the same homework, the same tests, just change your delivery.

If two students get the same score on the final exam, do they not have the same knowledge? Does it matter that one teacher used peer instruction? Perhaps engaged in some active learning techniques during lecture? Encouraged students to annotate in their notes and asked them to self explain code?

I hope this message opens a dialog. The poor students who have earned the right to attend institutions like CMU, MIT, and others should not have to forgo those experiences because we “tolerate” poor teachers.

February 2nd, 2016, posted by ldelyser

The definition of rigor

I got an email this morning from Ria Galanos asking about the definition of rigor.  I loved the question because it made me consider a term we use quite frequently.  I’m going to share my answer to her below.


Rigor is the intersection of the amount of content covered and the depth to which students understand and are able to apply that content.

A particular time frame of instruction is considered rigorous for three reasons:
1) An appropriate amount of content is covered for the time period.
You could spend an entire semester covering decision structures, students would get a LOT of practice, and be an expert in boolean expressions – but the time period is too long for the general community to accept that outcome as ‘rigorous’.
2) High expectations are communicated and held.
Students must demonstrate mastery of the stated course objectives to achieve decent scores.  A rigorous course includes assessment that measures individual student learning and holds students accountable for stated course objectives.
3) Students develop fluid knowledge.
Students in a rigorous course are expected to do more than memorize and regurgitate.  They should be able to apply the knowledge to new problems and situations, and discuss benefits and tradeoffs between solutions in open ended problems.  This aligns with higher levels of bloom.

What do you think? Does this align with your definition of rigor?

October 27th, 2015, posted by ldelyser

Do Different Teaching Methods Change Rigor?

Consider this:

Imagine I could create a ‘perfect’ assessment for whatever content knowledge you care about. Two students took the test and both achieved the same score under the same conditions. If the assessment is not flawed, is could one student be more ‘rigorous’ than the other?

When we talk about the increase in diversity in educational settings, we often jump to the conclusion that a lack of rigor is what opens the door for new populations to attain graduation, entrance, and prior success that differs from historical data.

I can tell you that here in NYC, some of the ways the numbers are changing is not from a decrease in rigor, but an improvement in instruction.  Teachers are using more active learning techniques, and as we understand the mechanisms of student learning and cognition better, they are using teaching strategies that take advantage of student strengths, as opposed to drilling them to correct for weakness.

I’ve heard college faculty discussing how they are afraid they will have to “water down” or reduce the rigor of courses to prepare for this flood of diverse students.

Mark Guzdial recently has been discussing the reform of teaching practices at the college level. Is a class with reformed teaching practices less rigorous?

Maybe, but maybe not for the rigor of the content we actually care about. If you are measuring whether students can learn on their own, or absorb material from a lecture – then yes, traditional college instructional practice is more rigorous.  But in terms of the CS content?

We need to be careful that our self reflection of our learning process does not conflate good (r bad) instructional practice with rigor.

Inspired by: “Professors Shouldn’t Teach to Younger Versions of Themselves

October 26th, 2015, posted by ldelyser

Assumptions and Best Intentions

The computer science education landscape is changing.

When discussing the need for the Running on Empty report, I used to say that CS Education was like champagne. A number of little bubbles of excellence, but those bubbles were not coming together to form a coherent movement.  As late as 2012, the most widely implemented form of computer science in the K12 space was the AP Computer Science A courses. And even today, it is the metric we use to define the pervasiveness of computer science at the HS level. (Despite the fact that by definition AP is supposed to represent post-secondary level work.)

But that is changing.

Programs are now growing beyond NSF funded pilots in a few schools or cities, to national operations with many schools and many more students.  Two great examples of this are the Bootstrap program and Exploring Computer Science. (By no means are they the only programs, but just two I’ve worked closely with over the last few years)

And yet, I am still astounded by the research funding and efforts put into developing “new” curricula and implementations to be used with high school students.  I’m focusing specifically in HS because I believe our MS and Elementary school space is still emerging and growing (there are some great programs out there for both, just not as well adopted yet).

Our problem is no longer a lack of curriculum.

On the contrary, there are a number of curriculum available, with materials, tools, and even lessons pre-created for teachers.  Yet, I still hear “I have a new idea for a program” far too often.

One of the most exciting things I get to work on is #CS4All.  While curriculum is a question for the project, we are no longer in a desert of options.  Instead what I need are pedagogy and methods to help reach the diverse learners in the schools.  Yes – I am aware of the body of research that includes peer instruction, active learning, and the wonderful resource that is CS Teaching Tips. But we need more – and we need a feedback loop between those pedagogical findings and the programs/curriculum that are available.

What prompted this post was a report about NYC’s department of education entitled “Redesigning the District Operating System”. The report was describing how bringing together teachers, designers, parents and students for the GapApp Challenge yielded a redefinition of the needs of teachers.

In the NYCDOE GapApp Challenge for middle school math, the central office assumption was that the most pressing classroom problems were curricular. When we interviewed and held design sessions with teachers, however, none of them mentioned curriculum. What surfaced for all of them was the difficulty in dealing with the range of abilities within a single classroom, something that was not even on the radar at the central Office of Teaching and Learning.”

Sometimes I feel like the CSED community falls prey to the same problem as the ‘central office’ did.  Assuming teachers’ problems are curricular (content) and not pedagogical.  Let’s think as a community about how we can explore pedagogy within our curricular implementations and partner with content providers to explore the implications of those interventions

October 19th, 2015, posted by ldelyser

Who will teach #CS4All?

Today I responded on Facebook to a post in the CS Education Discussion Forum about who are the teachers in the #CS4All initiative in NY. Just in case you are a first time visitor and have not heard, #CS4All will bring computer science to all of NYC public schools in a 10 year period.

The post on Facebook questions who will teach this subject? and offers some challenges for implementation.  #CS4All is set to prepare 5,000 teachers over the next 10 years (along with other things) for the challenge of teaching computer science in K-12. The blog post that was shared contains a number of links to other articles and highlights some of the challenges in NYC.  My response is here:

Thanks Nathaniel for commenting! Adam, I work with CSNYC so perhaps I can help answer your question. Yes, we are retraining a large number of excellent teachers to ALSO teach computer science in addition to their primary subject of certification. No, a majority of CS teachers in the city will not have a full degree in CS, and we will not be actively recruiting a large number of professionals to change careers to teaching (although a few of our programs rely on volunteers as a part of the model to supplement or improve teacher knowledge).

We are in a chicken and egg situation. The announcement came out and there are not a lot of details, except in our 2 years of experience with specific programs (although the announcement does not explicitly state those programs will be used in the new initiative, we are absolutely including them in our considerations). Now that the announcement has been made, and $$ committed – we can begin the hard work of implementation, which of course starts with planning. Planning for curriculum, planning for PD, planning for preservice (to help fill the gaps as teacher retirement and attrition happens), planning for evaluation, and planning for research.

Over the past two years we have worked with programs who have supported teachers of all subjects (Math, History, Art, Science, Technology) as they take the courageous leap to add new skills to their practice, and open themselves to additional topics that will be used to evaluate their performance. Almost all of our programs see the initial PD as a starting point, and almost all of our teachers engage with numerous ongoing opportunities to deepen their knowledge and enrich their understanding of CS. With no direct pipeline of teachers we need to work with both inservice and preservice folks to get the initiative off the ground. But if you look at our programs you will see that all of them believe in ongoing development to build the professionals with deep content knowledge necessary to teach.

I wanted to also post here since CSNYC is getting similar questions from the larger community. Yes, we are planning to primarily train existing teachers. No, that is not a long term sustainable plan which is why we are also working with pre-service programs and will have more news about programs before too long. Yes, we feel its important that teachers get a depth of content knowledge. No, we can’t do that all before their first day of class. Yes, we will continue to encourage programs that provide ongoing support, as well as host some of our own such as the Education and Pedagogy Meetups. (Next two in October – sign up!)

Rest assured the ducks analogy is applying here in NY right now.  Feet paddling furiously under water – much more to come as it gets vetted and approved by all collaborating partners. Sign up to “Get Involved” at the bottom of the #CS4All page to keep informed of news as it happens.

September 24th, 2015, posted by ldelyser

Why #CS4All Won’t Be Your Flavor of CS: It will be more

Amazing and exciting to be doing CS Education in NYC right now. In case you are living in a news-free zone, Mayor DeBlasio announced a 10 year initiative to bring CS to every school in NYC, giving every student access to experiences that will engage, teach, and hopefully inspire them. (#CS4All on twitter)

Because this was a policy announcement, there have been lots of questions and some concern. Here’s some of the bare details: CSNYC (the foundation I work for) will be an ongoing partner in this work throughout the 10-year period and is leveraging private sector funding to match the Mayor’s public funding of $40m, bringing the 10 year total to $80m. (See Fred Wilson’s post)

There are lots of people publicly on board, and even more behind the scenes ready to dig in and make the whole thing go. Lots more structural information coming out over the next few months, but this involves the expansion of experts at the city/district level, local support staff for teachers spread around the city, massive teacher professional development initiatives, and a lot of thoughtful curricular and pedagogical work. In addition, there is money set aside for external, independent evaluation to make sure that we do a good job with everyone’s money.

A question I’m getting a lot, and wanted to take some space (not on FB or Twitter) to answer was – What are you teaching them?

Underneath that question is another – what does CS mean for NYC public schools? Theres lots of folks expressing concerns in every direction – responding to key words in the policy address (or previous press) including “coding”, “computational thinking”, “problem solving”, “Java”, “scratch”, “robots”, and “skills”. My answer – Yes. Yes to all of the above, and at the same time, no. No to all of the above.

I know, not helpful (and maybe a little crazy).

But here’s the thing. NYC is not a one size fits all town, and CS doesn’t need to have only one implementation in education to be rigorous.

Let me also set the record straight, we are not simply spreading CS1 out over 13 years of school. (nor CS1 and 2)

College CS1 is exactly that – a college entry level course for students at that institution. K12 computer science should not seek to emulate that. We are the prequel. The algebra and arithmetic to it’s calculus. And that is a good thing.

We get to explore the underlying concepts of computer science in great detail, while also helping kids without daily access to technology catch up. We get to show them how colors work on a computer, how numbers can be used to encode things, why a computer game is no fun without decisions or boolean expressions, and why some things on a computer take longer than others. We want to inspire them to think creatively (and many of our programs explicitly teach design thinking) and prototype (yes sometimes even on paper)! We want them to plan, to debug, to take problems apart and put them back together. We want them to build, to modify, to replicate.

Yes, skills are in there too. They need to be able to read and write code in some fashion. They need to produce digital artifacts that are interactive and not static. And they need to talk or write about those artifacts to communicate why they are a milestone in their learning. Check out the flavors of CS that CSNYC already supports for a starting point. More details to come.

I say that #CS4All won’t be your flavor of CS – although you might find your flavor in some of the programs that eventually fall under the umbrella – but odds are you will find much more.

September 18th, 2015, posted by ldelyser