Evaluating Financial Sector Reforms: A Joint Task for Academia and Policymakers Statement prepared for the panel discussion "Improving Financial Resilience" at the T20 Summit "Global Solutions"

Check against delivery.

Since  the global financial crisis, major reforms have been launched which  aim to enhance the stability of the financial system, making future systemic crises less likely and reducing collateral damage in the real economy. Evaluating the effects of these reforms needs to be part of a structured process that answers three questions: Did the reform cause an outcome? Have reforms had similar effects across markets and jurisdictions? Has the reform achieved its overall objective? Economic research has advanced significantly over the past decades and provides a rich infrastructure that can be used to answer these questions.  Based on transparency, international coordination and independent assessments, academia and policymakers can join forces to conduct good policy evaluations.

I. The long shadows of financial crises

Ten years after the onset of the global financial crisis, its legacies still weigh heavily on the financial sector and the real economy. The crisis emerged from the core of the global financial system, but it has had significant economic and social costs globally. Output losses occurred, wealth decreased, unemployment went up - with all its social consequences - and public debt increased (Figure 1).

Costs of the Great Financial Crisis
Figure: Costs of the Great Financial Crisis

The Financial Stability Board (FSB) estimates that the cumulative output loss since the global financial crisis is in the order of 25 % of one year's world GDP (FSB 2016). Taking a longer-term perspective for the years 1970-2011 shows that financial crises were followed by output losses relative to GDP of 26 % for emerging economies and 33 % for advanced economies (Laeven and Valencia 2012). While the exact magnitude of the decline depends on country characteristics and the way the crisis is measured,[1] these numbers confirm that costs of financial crises are immense and long-lasting.

Since the global financial crisis, major reforms have thus been launched that aim to enhance the resilience of the financial system, making future systemic crises less likely and reducing damage done to the real economy. Financial market participants should only take risks if they are able to bear the potential losses without taking recourse to public funds.

Regulations have been amended to strengthen the private sector's capacity to buffer shocks, to reduce implicit public subsidies for the financial sector, and to promote strong, sustainable, and inclusive growth. G20 leaders have identified four main reform areas: making banks more resilient, ending too-big-to-fail, transforming shadow banking into market based finance, and making derivative markets safer.

Many financial sector reforms have entered uncharted territory and their impact still needs to be assessed. Because the implementation of many core reforms is well underway (FSB 2016), an initial analysis is now becoming possible. The FSB thus developed a Framework for the Post-Implementation Evaluation of the Effects of the G20 Financial Regulatory Reforms. At their meeting in Baden-Baden, Germany, in March 2017, the G20 reaffirmed their commitment to a structured evaluation of these reforms and the FSB published the draft framework for a public consultation (FSB 2017). The framework will be submitted to the G20 Leaders' Summit in Hamburg for endorsement and will subsequently be published.

II. Embedding evaluation in a structured policy process

Policy evaluation needs to be part of a structured policy process, which involves four steps:

Step 1: Specify the objectives of reforms

The objectives of reforms need to be defined as a benchmark against which progress can be measured. The paramount goal of financial sector reforms has been to realign private incentives, whereby private actors are to bear the costs of financial distress in the same way they might reap the economic benefits. At least two dimensions must be considered. According to a narrow view, goals of specific reform areas can be defined, such as the ability of individual banks to absorb shocks arising from financial and economic stress or to improve risk management and governance. From a broader perspective that looks at financial stability, the objectives might target the stability of the financial system as a whole, the orderly functioning of markets, the cost/availability of financing, and economic growth.

Step 2: Define intermediate targets

It is only possible to ascertain whether reforms are effective over a longer period of time. In the interim, one has to rely on a range of intermediate variables for monitoring. For more narrowly defined policy objectives, this includes measures of risk-based capital, liquidity and leverage as well as evidence of improved bank governance, risk management, or reporting practices. For broader objectives, one can use standard indicators of financial system resilience, the cost/level of financing, or market liquidity.

Step 3: Calibration of instruments and ex ante impact assessment

Once the policy objectives and intermediate targets have been established, instruments need to be calibrated accordingly. This requires an ex ante impact assessment.

Step 4: Ex post impact assessment

In a final step, once instruments have been applied and the effects have been observed over a sufficiently long period of time, the impact of reforms can be assessed ex post.

Currently, such a framework for the evaluation of post-crisis financial sector reforms is lacking at the global level. This is the gap that the FSB framework is about to close. It is about answering the question of whether the reforms have achieved their intended outcomes, whether they work together as intended, and whether they have had material unintended consequences. Such side effects may have to be addressed, but without compromising on the objectives of the reforms or by reducing resilience. The framework provides a common understanding of the elements required  for a "good" policy evaluation, and it will provide a basis for an informed and evidence-based discussion on regulatory policies.

Policy evaluation need not be designed from the ground up. In many policy areas, structured evaluations are routinely applied. Labor market policies and educational programs are often implemented following a testing phase, and evaluations are commonly applied in the area of development policies.[2] Like clinical drug trials in medicine, policy evaluation can help to analyze whether a policy, or "treatment", is effective, whether it works on the causes of the "disease" or whether it merely relieves its symptoms. It can also help to  identify any potential side effects. "How do regulations affect the stability of financial institutions?" is similar to asking "How does this drug affect the health of the patient?".

III. Addressing challenges of policy evaluation

Policy evaluation is challenging, as it requires dealing with attribution, heterogeneity, and general equilibrium effects. Embedding evaluation into a structured policy process makes these challenges explicit and makes it possible to address them. Transparency, international cooperation, and independent assessments are key elements of such a process.

a) Did the reform cause an outcome (attribution)?

Disentangling the costs and benefits of regulation is the key challenge of any evaluation. Financial reforms generate direct compliance costs for market participants, such as the costs of reporting requirements, and these costs tend to be recognized immediately. The benefits of a more stable and resilient financial system, by contrast, can be reaped only over the longer term and are more difficult to quantify.

The direct compliance costs need to be distinguished from costs of crises that are borne by market participants. The bailout of banks during the crisis shifted private costs to taxpayers and raised levels of public debt worldwide. Many reforms thus seek to realign incentives by withdrawing implicit public guarantees. As a result, risk-taking on the part of banks may decline, allocation of credit may change, and costs of debt may increase because creditors cannot rely on bailouts and thus demand higher risk premia. Profits in the financial sector may decline as a consequence of reforms.

Taken together, costs are shifted from the public sector to the (private) financial sector. An example from environmental policy may serve to illustrate the point. A Pigovian tax on chemical producers that pollute a river internalises the externalities of polluting the river by aligning private and social costs, thereby realigning the incentives to pollute. The direct result of this policy is lower production of chemicals, as pollution of the river is now part of the private sector's costs, and lower sectoral profits. However, these are not the costs of the reform but the intended consequence.  

Practical implementation of reform assessments needs to address the issue of a missing counterfactual. How should the benefits of the reforms that have been implemented be measured ex post? Is it possible to measure the benefits of reforms if no crises have occurred? Wouldn't the lack of a counterfactual render an assessment of the benefits of reforms impossible? These questions are not new, and they are in fact similar to cost-benefit analysis in other policy areas. It is in the very nature of counterfactuals that they cannot be observed. If monetary policy keeps the inflation rate at or below a target of, say, 2 %, how does one quantify the macroeconomic benefits of having avoided an acceleration of inflation? If educational policies increase school enrolment rates, how does one quantify the benefits of reduced youth unemployment and perhaps even reduced youth crime rates?

The need to quantify benefits of (economic) policies without knowing the counterfactual is not unique to financial sector reforms. There are two - complementary - approaches of dealing with them:

First, assessing the benefits of reforms can be done without directly measuring the broad objectives of reforms in terms of enhanced stability of (global) financial markets or a lower probability of financial crises. Indeed, the benefits of reforms can be assessed by looking at the channels through which excessive risk-taking in the financial sector materializes and through which risks to financial stability can build up. The focus can be on indicators that are closely related to factors contributing to financial sector stress. In the same vein, indicators might include the soundness of the financial sector and the contribution of the financial sector to real economic activity.

This then leads to the second question, namely the question of establishing a causal impact of reforms on these indicators and the construction of counterfactuals: How would the financial system have developed without the reforms having been implemented?

Constructing macroeconomic counterfactuals is difficult and not desirable. We do not wish to see a financial crisis in a country that has not undertaken financial sector reforms just so we can improve our understanding of the benefits of reforms in a country that has undertaken such reforms. So, while the "real" counterfactual is neither possible nor desirable, there are nevertheless many ways to construct synthetic counterfactuals that can serve a similar purpose and many ways to use econometric methods that make it possible to establish causality. There are many possibilities of dealing with the challenge of causal identification, even in the area of macroeconomics (Fuchs-Schündeln and Hassan 2015). But empirical methods alone are not enough; good conceptual and theoretical frameworks are also needed to decide what to look for in the data.        

b) Have reforms had similar effects across markets and jurisdictions (heterogeneity)?

Reform effects differ across countries, banks, and sectors of the financial services industry. This heterogeneity facilitates rather than prevents reform evaluation. Essentially, heterogeneity and institutional knowledge are important for good (causal) impact assessments. Comparing countries that enacted different types of reforms at different points in time, comparing developments before and after reforms have been initiated, comparing banks for which new regulations are more or less binding can help to establish causal effects. Understanding differential effects across markets or jurisdictions delivers valuable lessons for policy development that would be missed by looking only at averages.

Take, for example, the increase in capital requirements for banks after the crisis. Higher capital requirements increase the resilience of banks with regard to shocks and provide the basis for sustainable lending to the real economy. Trends in post-crisis bank lending have differed across countries and individual banks. A decline in lending can, for instance, be due to a decline in credit demand, more prudent lending behavior of banks in response to higher risks, or it can result from a decline in banks' credit supply. Such a decline in credit supply, in turn, can be caused by structural changes in the banking sector and the reallocation of market shares to non-bank lenders, it can be due to a re-orientation of banks' internal strategies, or it can be the consequence of tighter regulations. Experience shows that all of these factors play a role and that their relative importance differs.

What conclusions can be drawn from observing trends in bank lending over time? Is direct inference possible with regard to the intended and unintended consequences of reforms? Prima facie, aggregate time trends and averages provide only limited answers to these questions. In the extreme, basing policy decisions on such statistics alone can aggravate the situation. For example, regulations may be relaxed because of their alleged impact on credit supply, even if they did not play a major role in the contraction of credit in the first place. Banks may then be encouraged to lend excessively in risky markets - with potentially negative effects for financial stability.

c) Has the reform achieved its overall objective (general equilibrium)?

In order to assess the effects of reforms, it is ultimately necessary to conduct a general equilibrium analysis. Which markets are affected, what are the interactions between different reforms, how does the allocation of risk and credit change in an economy? These are questions which cannot be answered directly by microeconomic models that carefully identify causal effects.

At the same time, general equilibrium analysis is even more challenging than partial equilibrium modelling, in particular in situations in which markets dynamically adjust and many shocks hit the system simultaneously. Most of the time, financial markets are not "in equilibrium" but in a state of transition.

Rather than promising to deliver a comprehensive view of all market adjustments, general equilibrium analysis serves the purpose of identifying the relevant shocks, transmission channels, and market interactions. Rather than broadening the perspective even further, a general equilibrium perspective should help to focus on the relevant issues under consideration. 

IV. A joint task for academics and policymakers

Academics and policymakers can join forces to conduct good policy evaluations. Post crisis, the academic profession and economists in particular have been criticized for their inability to predict crises, for using the wrong models, and for applying the wrong concepts.[3] Large parts of this criticism ignore the significant progress that has been made with regard to economic analysis in performing meaningful causal evaluations, modeling heterogeneity, and exploiting microeconomic data. Enhancing the use of existing data, methodologies, and the skilled personnel that are employed both inside and outside policy institutions should thus be a priority.

Both academics and policymakers would benefit from engaging in better policy evaluations. Academia could engage in developing methodologies and in studying designs that contribute to societal welfare without compromising on academic rigor. Policymakers could draw on the rich expertise that is available and make better use of existing infrastructures. Moreover, some of the mechanisms that have been developed in academia to mitigate misaligned incentives and ensure transparency are readily applicable to policy evaluations.

Good policy evaluation needs to take incentives of the different stakeholders into account. For example, regulators have an incentive to overstate the "success" of reforms, industry has an incentive to focus on private costs, while academia has an incentive to focus on "novel" aspects. Policy institutions may lack resources to govern the process. Incentives can be managed according to the following three basic principles:

1. Transparency

Transparency is a precondition for a credible regulatory process. Goals and concepts need to be well-defined. Policy evaluation means being transparent about the goals of regulatory policies and what these policies have actually achieved.

Transparency can be accomplished by making information on evaluation studies, data (data confidentiality permitting), and codes available to relevant stakeholders. A centralized collection of evaluation studies can reduce the costs of obtaining relevant information for everybody. Such a repository can improve transparency by making it easier and less costly for (internal) evaluators and external stakeholder to keep track of evaluation work. Technical solutions can range from a website to more structured repositories. Other fields, such as development economics or medicine often rely on repositories in order to structure available evidence.[4]

Replicability is an important element of transparency. Replicability ensures that policy decisions are based on robust evidence and on results which survive changes in the set-up of the empirical model. This requires transparency about the models being used, access to data so that results can be replicated, and good documentation of the set-up of studies. Many of the required protocols have been developed in academia and can be applied to policy work. At the same time, policy evaluation would benefit from a greater appreciation of replication studies in academic work.

In order to ensure replicability, democratic data access is important. Robust evaluations have to rely on a wide range of methods and good data. Whenever possible, data used in the analysis should be made public. This facilitates the replication of results and can, thereby, help to ensure high quality standards.

2. International Coordination

Regulatory reforms have a global dimension and have been decided at an international level. Also, prudential policies have international spillover effects.[5] Speaking a common language is thus important when looking at the effects of reforms. Setting  standards, learning from good practices, and international coordination are vital. The FSB, in coordination with international standard setting bodies, can play an important role in this regard. Its proposal for a framework for policy evaluation comes at the right time and addresses the right issues.

3. Independence

Independent, objective assessments are needed to obtain an unbiased picture on the effects of reforms. There are several avenues towards independence. External experts might be involved that have a mandate for independent evaluations and do not have a stake in the formulation of policies. These can be existing institutions such as research institutes or policy advisory bodies. Internal experts might also provide independent assessments, in particular in cases where data cannot be shared with externals. Internal evaluations, however, are at greater risk of being subject to implicit or explicit biases. Therefore, appropriate institutional arrangements such as peer reviews, independence from policy groups, and direct reporting lines need to be in place.

V. References

  • Basel Committee on Banking Supervision (2010), An assessment of the long-term economic impact of stronger capital and liquidity requirements, Bank for International Settlements, August 2010.

  • Buch, Claudia M., and Oliver Holtemöller (2014), Do We Need New Modelling Approaches in Macroeconomics?, IWH Discussion Papers 8/2014.

  • Buch, Claudia M., Matthieau Bussiere, and Linda Goldberg (2017). Prudential policies crossing borders: Evidence from the International Banking Research Network. Vox EU. http://voxeu.org/article/prudential-policies-crossing-borders

  • Financial Stability Board (2016). 2nd Annual Report on the Implementation and Effects of the G20 Financial Regulatory Reforms. August. Basel.

  • Financial Stability Board (2017). Proposed Framework for Post-Implementation Evaluation of the Effects of the G20 Financial Regulatory Reforms. April. Basel.

  • Fuchs-Schündeln, Nicola, and Tarek Hassan (2016). Natural Experiments in Macroeconomics, in: Taylor, John B., and Harald Uhlig (eds.): Handbook of Macroeconomics, Elsevier, Vol. 2a: 923-1012.

  • Laeven, Luc, and Fabián Valencia (2012). Systemic Banking Crisis: An Update. IMF Working Paper 12/163, Washington DC.

  • Rey, Helene (2013). Dilemma not Trilemma: The global financial cycle and monetary policy independence. http://voxeu.org/article/dilemma-not-trilemma-global-financial-cycle-and-monetary-policy-independence

  • Verein für Socialpolitik (2015). Leitlinien und Empfehlungen des Vereins für Socialpolitik für Ex post Wirkungsanalysen. September. https://www.socialpolitik.de/docs/VfS-Leitlinien_Ex_post-Wirkungsanalysen.pdf

  • Wissenschaftlicher Beirat beim Bundesministerium für Wirtschaft und Energie (2013). Evaluierung wirtschaftspolitischer Fördermaßnahmen als Element einer evidenzbasierten Wirtschaftspolitik. Berlin.


Footnote

  1. See, e.g., Basel Committee on Banking Supervision (2010).
  2. For an evaluation framework applicable to a range of regulatory policies in the European Union, see http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htmhttp://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htmhttp://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm. The German Economic Association (Verein für Socialpolitik 2015) and the academic advisory council to the German Ministry of Economics (Wissenschaftlicher Beirat 2013) have likewise developed criteria for good, evidence-based policymaking.
  3. For a review of this discussion, see Buch and Holtemöller (2014).
  4. Examples include the Health Systems Evidence  from the McMaster Health Forum (https://www.healthsystemsevidence.org), the Cochrane Library (http://www.cochranelibrary.com), or, for development economics, J-PAL (https://www.povertyactionlab.org/) or the International Initiative for Impact Evaluation (www.3ieimpact.org).
  5. Buch, Busssiere, and Goldberg (2017) summarize the results of a cross-country research project on prudential spillovers through international bank lending. Rey (2013) discusses the importance of prudential policy for global financial cycles.