Peer review is a method used in many fields. It involves bringing outside experts into collaborative review of a body of work or project.
Peer review happens as a matter of course when institutions seek external guidance about their own expertise from outside experts or consultants. In academia, peer review is the predominant process by which research is judged. Material is sent to relevant outside experts and judged in relation to its contribution to the best current understandings in the field.
Peer review in the broad sense works best when it is complemented by peer exchange — that is, by bringing field-related experts together in person to discuss basic developments in their field. However, while peer exchange can be rich, it can also be empty. Just as conferences can be exciting and productive exchanges of knowledge, they can also be empty rehearsals for banal information delivery. They can quickly descend into career-sensitive jostling or acts of going through the motion. They can tend towards treating knowledge development as a competitive process. Through peer review, formal knowledge tends to be empowered at the expense of informal and implicit knowledge. In practice it often does not meet the test of normative reflexivity discussed elsewhere on the website.
However, for all of these limitations, when conducted well, peer review is the best method we have for assessing complex analytical or technical knowledge (see the Knowledge Circle). This potential explains why peer review remains important in contemporary intellectual life.
The Peer Review Process developed here is a response to these strengths and weaknesses. The Circles of Social Life Peer Review Process takes the robustness of peer reviewing and carefully qualifies its weaknesses. In this process developed by a Metropolis-Cities Programme team led by the City of Berlin the focus is on a major project in a city. Ideally this project should be in early development or at a critical juncture in its operation. This qualifies one of the common weaknesses of peer exchange events — empty rehearsals of information in gatherings where there is little immediately at stake beyond the cultural capital of the people in the room. This Peer Review Process is intended to have on-the-ground consequences.
For more information on the method see Chapter 9 of Urban Sustainability in Theory and Practice: Circles of Sustainability.
Phases in the Peer Review Process
The Peer Review Process follows the same seven-stage method as outlined in the Process Circle. As with all our methods it is not a pathway to be followed mechanically. In the description below, the high-level definitions of the stages remain as previously expressed, but each is elaborated with specific recommendations for practice based on extensive field experience.
The simplified process presented here begins with written preliminary documentation of the project prepared by the host city and distributed to the peers before they come to see the project. The process then includes a concentrated period — optimally, two or three days—where the peers are immersed in the on-the-ground reality of the project, and then have time to work through what they have seen with local experts and practitioners in an extended round of discussion. A report on outcomes and recommendations is produced in the aftermath.
Commit
Commit to responding creatively to a complex or seemingly intractable problem.
Because the Peer Review Process asks cities to open their work to a critical group of outsiders, taking on this process takes commitment — commitment of time and resources, commitment to openness and creatively, and commitment to rethinking current plans rather than just showcasing best practices. Once a project or programme of activity has been identified that is suitable for a peer review, the host city spends considerable time preparing a written overview of that project to present to the peers. This has the virtue of giving project managers time to reflect: it is not the usual cut-and-paste process, taking stuff off the website that has been used to document success. Key figures involved in the project need to commit to the documentary preparation and to two or three days of hosting the peers.
Engage
Engage local and global partners in the process of responding to the provisionally identified main issue.
A Peer Review at city level is a structured method aimed at extending mutual learning about a particular project or programme. Peers are invited on this basis. The process involves both review and exchange — possibly ongoing interchange. The success of the process turns substantially on appropriate peers being identified and then accepting the invitation in the collaborative spirit that it is extended. The invited team of around four to seven experts should include a range of people with skills oriented around two emphases: working with parallel projects and thinking broadly about the social whole. This means firstly engaging colleagues from other municipalities, technical experts, who know directly about the chosen general issue and its working intricacies. Secondly, and just as importantly, it means involving individuals with generalizing cross-domain knowledge that goes beyond the immediate project.
Assess
Assess the nature of the problem, and analyse the current situation in relation to the general issue in contention across all social domains.
The assessment stage is conducted from two main perspectives: internal and external. The internal assessment begins with an initial report written by local practitioners and experts of the host city. This report is crucial for giving the external peers insight into the machinations of the project. It also allows them to think about the comparable examples from their own cities and how lessons might be transferred. If a city has the capacity, it is a distinct advantage to also develop an urban profile of the whole city before the peer review meeting. This enables the project to be seen figuratively and dramatically in the context of the strengths and weaknesses of the city.
Define
Define the terms of the problem, and identify the most important critical issues that will provide the focus of action in relation to the general issue.
The definition stage is done mutually with the internal and external experts working together in discussion. It has two main phases. The first is to interrogate the critical issues that are affecting the project and refine what the city hopes to do about them (as critical objectives). This can be done in the first couple of hours of the second day (if two days is the chosen length of the workshop). Refining the critical issues and objectives that affect the project and then testing the relationship between those critical objectives is often more revealing that is usually expected. In fact, while assessing the compatibility of critical objectives and resolving possible tensions between them is fundamental to the success of any project, it is usually handled tacitly rather than as a focussed event. Such tensions always exist in some way, but they often they go unrecognized. It is worthwhile in the phase to test the range of critical issues and objectives against the four domains of the Profile Circle.
Implement
Implement measures to respond to the problem, and authorize the various aspects of the plan and its subprojects within project parameters.
This stage and the next are for the host city to decide upon. The Circles method provides tools for measurement and assessment.
Measure
Measure and monitor activities, and assess progress towards achieving the normative goal and objectives of the project.
Communicate
Communicate progress and strategies in relation to the project through public documentation, publication, and through engagement with stakeholders.
A key aspect of the communication stage is the writing and public dissemination of a final report on the recommendations. Here is an example. Ideally, a written report on the whole exercise should more broadly contextualize the recommendations and discuss the complexities of the project and its setting. However, what is really important for a peer review is that the host city informs the peers about which of their recommendations and suggestions are going to be implemented in the project. It would, of course, be positive for ideas to continue to be exchanged and further activities to be planned for after the review workshop, but in the busy working lives of such practitioners this is not always possible. Nevertheless, in a short period of the process, through intensive, open, trustful exchange of professional opinion, peer review can be the initial catalyst for fruitful cooperation between all those who were involved in the exercise—between the host and the peer cities and also between individuals.