Assessing policy questions

Week 5

Learning objectives:

  • Understand basic elements of policy analysis
  • Understand how to approach policy analysis in general
  • Develop a personal checklist to assess aspects of public policy on science and technology questions

Developing a Policy Analysis Checklist for your use


For our purposes in week 5, “policy” represents a decision by government to do something, and policy analysis represents an assessment of that decision from a particular perspective. As a policy analyst, you may have to adopt a variety of different perspectives to understand the pressures on public policy. This week you will develop a one-page checklist for a sequence of actions and items to consider when analyzing specific policy questions relevant to science and technology.

The illustration above comes from an important book on evidence-based policy. Last week we discussed some of the differences between natural sciences and social sciences.  Here, one of our concerns is whether science policy is also social policy.  When governments make decisions about science and technology, about industrial innovation strategies, or about support to particular kinds of development, are they also, but implication, making decisions about social policies – who benefits, who pays, who cleans up?

The four different explanations of technological development described by Smardon in Ch. 1 (week 3) represent different combinations of public policy – either consciously, or by default:

  • explanation 1: national systems of innovation theory – requires integration of pubic and private innovation networks
  • explanation 2: institutionalist theory – requires supportive institutions in the state
  • explanation 3: the developmental state – requires a conscious policy
  • explanation 4: varieties of capitalism – requires decisions about the sort of regulation and economy to permit

Considering Canada’s actual pattern of dependent technological development over the 20th century (Smardon, Chapters 2 and 3), we might identify specific policies that contributed to the problems Smardon identifies.  There are several difficulties in doing so:

  • The level of national outcomes at which Smardon is assessing policy is a big picture, and it is the result of many policy decisions over a long period of time.
  • It also reflects international trends and factors over which governments have little control.
  • Some of the problems are amenable to incremental change over time, while others require radical policy shifts, but how do we know which is which, and what will be the impact of a policy change?
  • This relates to a major problem of any policy analysis: things take time. How long should we expect to wait before we know whether a policy is a success or failure?
  • Finally, all assessments of outcomes are filtered by interest: a policy may be good for one group and bad for another.  Some will pay, and others will benefit. How does the policy analyst balance

Bardach (2011) defines an eight-step process for assessing and making policy

  1. Define the problem
  2. Assemble the evidence
  3. Construct the alternatives
  4. Select the criteria for decision
  5. Project the outcomes
  6. Confront the trade-offs
  7. Decide!
  8. Tell your story

Note that this is a sequential process, and does not dictate the content of your checklist, i.e. the factors that you consider in each stage.

For the content of your checklist, we have to go back to week 4, and recall what different disciplines contribute to the assessment of policy. Are we concerned about political, economic, or social aspects of the policy’s impact?  Do we need to be informed by specific branches of natural science: engineering to assess the feasibility of transport options; environmental science to assess impact of a project on the environment; geological engineering to assess the stability of a building site?  Keep in mind that different analysts will have different perspectives, different stakeholders, and different pressures while conducting their analysis.  Consider, for example, a defence policy analyst in a majority Conservative government which has already committed to purchase the F-35, and the same analyst six months later working with a minister in a minority government from a party that made an election commitment to scrap the purchase.  How will those two positions affect an analysis of technology transfer policies related to defence spending?

Bardach (2011) provides an appendix which lists actions that governments can take, along with the reasons for which it might take particular actions:

  • Actions related to taxation (adding, dropping, changing, improving collection)
  • Regulation
  • Subsidies and grants
  • Provide services (add, expand, drop or reduce, privatize, monetize, etc)
  • Budget for agencies
  • Pass legislation (rights, responsibilities
  • Adjust the framework of economic activity (competition, concentration, tax incentives, public/private balance of service provision or production, crown corporations, etc)
  • Public or targeted education
  • Financing and contracting
  • Bureaucratic and political reforms

As you produce your personal checklist for assessing public policy, consider the impact of alternative government actions that might be expected to affect the desired outcomes.  Try to be explicit about what you need know in order assess a policy and project its outcome reasonably accurately. If you don’t know or understand the interaction of all these variables, what kinds of expertise do you need?

Required reading

Smardon, Chapter 4 “Another form of dependent technological development: post-Fordist accumulation in the neo-liberal era”.

1970s saw Canada move from Autopact protections towards the Canada-US FTA of 1987, acceleration of continental rationalisation. Canadian industry switched towards resource exports and limited manufacturing for export as protected markets diminished.

  • shifting to an export-driven strategy (Autopact – GATT – flat domestic demand)
  • innovation clusters and exports (Canada lacked innovation clusters)
  • declining relative productivity (due to relationships to US industry, i.e. structural constraint)
  • new patterns of production and trade in the post-Fordist era (high tech, networked not linear innovation)
  • relative trends in productivity (Canada’s productivity slides, but WHY?)
  • falling trade shares in research-intensive products
  • divergence in R&D
  • trade in technology services
  • declining R&D intensity and autonomy of Canadian subsidiaries
  • a contrary view
  • Conclusion

Optional reading

Bardach, Eugene (2011) A Practical Guide to Policy Analysis: The eightfold path to more effective problem solving. Los Angeles: Sage and CQ Press.  Introduction and Appendix B

Pawson, Ray (2006) Evidence-based policy: A realist perspective. London: Sage. Especially chapter 2 on building blocks of evidence, and figure 2.1, the basic components of realist causal explanation. Resources and additional readings online.


Complete required reading and prepare for seminar.

We are catching up this week on week 4 questions, and will also try to finish week 5 by Friday’s class. You will have next week to work on your assignments, and will present your toolkits in week 7.  I will adjust the remaining weeks in the schedule.

Self assessment questions

Starting with Smardon, chapter 4:

  • what is the main argument of the chapter?
  • why were growth opportunities in Canadian markets increasingly restricted?
  • what was the impact of opening Canadian markets through GATT?
  • as markets opened, why couldn’t Canadian firms take advantage of them?
  • what were the structural constraints that caused lower productivity in Canada?
  • what was the significance of machinery and fabricated metal products in the 1980s?
  • when did knowledge sector (telecommunications and computers) start to pick up as a major part of the innovation economy?
  • what are innovation clusters, and how did they work as the economy evolved in the 1970s?
  • interpret tables 4.2 and 4.3 – what do they tell us about productivity changes in Canada and the US?
  • Why did Canada’s trade share of research intensive products decline in the 1980s and 1990s?
  • What does Smardon conclude about Canada’s innovation networks, investment in manufacturing R&D, declining relative productivity, declining trade share of growth sectors, and gaps in R&D capacity?
  • What was the role of Nortel in R&D in Canada?
  • In what sector did R&D increase in Canada?

Covered in the lesson:

  • what is a systematic review?
  • why is it more suited to natural and medical sciences than to social sciences?
  • what are the basic components of realist causal explanation?
  • what actions can governments take, and why might they do so?
  • how does this relate to the narrative of Canadian innovation developed by Smardon?


Participation evaluation for week 5 Toolkit will consist of:

  • 60s: What is policy analysis?
  • 60s: Describe “post-Fordist accumulation in the neo-liberal era”
  • 5R: The key trends described by Smardon in Chapter 4 all relate to one phenomenon. What are the key trends and what is the central theme to which all the trends relate?  Discuss for 5 minutes. Present the results in 60s. There is an opportunity for a minority report as a 60c.
  • 60s: What is the timeline over which Canada’s situation changed, according to Smardon, and what does this suggest about the timeline over which policies would need to be implemented and monitored? Why is this a problem?
  • 60s: Based on your understanding of Canadian party politics ([Progressive] Conservative, Liberal, NDP) what challenges might you anticipate in crafting and implementing policies to address the problems Smardon has identified?
  • 60s: [we won’t get to this, but start thinking about it for your toolkit presentations] What could be done about this?

Questions and discussion