Позитивные изменения. Том 3, № 4 (2023). Positive changes. Volume 3, Issue 4(2023) - Редакция журнала «Позитивные изменения»

8. Presence of an Evidence Base for the Social Technology Used

This criterion reflects whether the project is founded on social technology or practice substantiated by scientific theories, research findings in the relevant field, and the experiences and opinions of the beneficiaries.

9. Presence of a Monitoring System This criterion reflects whether the project involves regular data collection throughout its implementation. This includes maintaining statistics (such as inquiries, number of services rendered, visitor counts, calls), as well as gathering feedback through questionnaires, interviews, etc

This model is adaptable for other regions, and its various components — the evaluation criteria, indicators, and approaches for developing diagnostic tools — could be effectively utilized in formulating assessment models for federal projects and programs.

10. Presence of Indicators for Assessing Project Effectiveness

This criterion reflects whether the project has developed and is currently employing a system of performance indicators for each implementation phase. These indicators should be clearly described, with an elaborated assessment methodology, including tools and data collection procedures.

11. Presence of a Data Collection and Media Presence Monitoring System

This criterion reflects whether the non-profit organization, both overall and at the project level, has established and implemented a system for evaluating media effectiveness. This involves collecting and analyzing data regarding the project’s coverage in traditional and social media. The data should be regularly gathered, analyzed, and the results promptly utilized in future media interactions.

For the ‘assessment hygiene’ criteria (8–11), recommendations are provided only for those criteria receiving low scores (1 or 2). Here are sample recommendations for criterion 8 (Presence of an Evidence Base for the Social Technology Used):

Table 1. Example of a set of indicators for a priority area

If the project is being implemented for the first time without prior testing or implementation experience (e.g., in a different territory or for a different target audience), it is advised to:

• Conduct desk research based on scientific and other data sources. This research should aim to establish an evidence-based vision of the project’s outcomes using the embedded social technology;

• Conduct a stakeholder survey, including potential beneficiaries, is recommended to identify potential challenges and threats, and to preliminarily evaluate the nature and quality of the changes expected from the project, and the conditions required for the changes to take effect. Such surveys are mandatory if the beneficiaries include audiences with feedback limitations (such as children, individuals with mental or cognitive impairments).

If the project has been implemented before but lacks a sufficient evidence base, conducting surveys of at least three stakeholder groups involved in the project is recommended. These could include project team members, beneficiaries, funding organizations, partner entities, governmental bodies, etc. The survey aims to identify potential areas for project growth, outcomes, and conditions necessary for achieving these results. A desk study, as in the case of initial project implementation, is also recommended.

The outcome should solidify the evidence base for the social technology or practice used, backing it by scientific theories and concepts, research findings, and beneficiary feedback.

PROJECT ASSESSMENT ACCORDING TO SOCIAL IMPACT INDICATORS

The second component of the model involves project assessment based on social impact indicators.

These indicators, developed from analyzing previous bids and existing thematic support areas, reflect the qualitative impact of the project. In the Table 1 an example of a set of indicators for a priority area.

When the project team believes that existing indicators fail to adequately capture the project’s impact, or if there are changes in the thematic areas supported by the Fund, the list can be supplemented. Such modification is conducted according to a defined algorithm. For new indicators, a universal formula is suggested: “What changes have occurred in the area of…? How do you assess the project’s contribution to these changes? How do these changes apply to you?” The quantitative assessment scales for these changes are akin to those used for the primary (universal) list of indicators.

It is recommended to engage project stakeholders in formulating the indicators, which may include:

• State entities: municipalities, individual politicians (party representatives, lawmakers), and executive branch representatives;

• civil society: volunteer centers, NGOs, and universities;

• market representatives: business and state corporation representatives;

• direct beneficiaries.

It is crucial to ensure triangulation in the study, validating findings through interviews with representatives from at least three stakeholder groups (e.g., beneficiaries, their families, state administration representatives, businesses involved in the project, etc.).

An impact assessment questionnaire is recommended for gathering feedback from the beneficiaries. This can be used for online and offline interviews or surveys.

OVERALL ASSESSMENT MODEL

The Fund was also tasked with developing an Overall Assessment Model, encompassing the assessment of all projects that receive grant support. The final report collates all project coverage indicators and types of impact. The overall assessment of the cumulative project impact on regional development is designed to record three categories of data: UAP-based universal social impact criteria, overall change significance index based on indicator analysis, and an optional integrated qualitative impact assessment protocol. These are operated as follows:

• Projects are assessed against universal social impact criteria based on UAP;

• Data is aggregated into the change significance index;

• A separate study is conducted to learn the cumulative project impact on regional development, using a qualitative impact assessment protocol.

Econometric modeling analysis can be employed to further refine the overall assessment of social impact indicators based on the UAP. This approach can be viable once a comprehensive database of indicators is collected, enabling the establishment of causal relationships.

CONCLUSION

Summing up, we would like to mention a few characteristics of the model developed, which are noteworthy in our opinion.

The model, while not designed as a decision-making tool for supporting specific projects, has the potential to serve this purpose. Its primary function is to act as a “system of mirrors,” allowing for a multifaceted examination of the project-initiated changes from various perspectives. This approach helps to better