Indicators for assessing the risk of execution of state contracts with a long life cycle

Mishin Aleksandr ORCID: 0000-0002-0592-8524 PhD in Economics Associate Professor of the Department of Business Informatics, Faculty of Information Technologies and Big Data Analysis, Financial University under the Government of the Russian Federation 125167, Russia, Moscow, 49/2 Leningradsky Ave. mishin_au@mail.ru

Andriyanov Nikita Andreevich ORCID: 0000-0003-0735-7697 PhD in Technical Science Associate Professor, Department of Data Analysis and Machine Learning Faculty of Information Technologies and Big Data Analysis, Financial University under the Government of the Russian Federation 125167, Russia, Moscow, 49/2 Leningradsky Ave. naandriyanov@fa.ru

The article was prepared based on the results of research carried out at the expense of budgetary funds under the state assignment of the Financial University

Introduction

           The chosen direction of research, firstly, indirectly affects all the national goals established by Decree of the President of the Russian Federation No. 474 of 21.07.2020, since the efficiency of the Russian economy is largely determined by the efficiency of its public sector due to its high share in it; secondly, it directly supports the goal "d", which is to "achieve digital maturity of key sectors of the economy and social sphere, including healthcare and education, as well as public administration."

          This study is relevant, since the volume of public procurement is comparable to the revenue part of the federal budget. So, in 2022, the volume of public procurement exceeded the volume of 12.9 trillion. rubles according to 44-FZ (Galieva D. Small tender, yes expensive. The newspaper "Kommersant" No. 1 of 09.01.2023, p. 2) and 8.1 trillion. 223-FZ. (The volume of purchases in the B2B segment in 2022 decreased in the Russian Federation by 2% to 33 trillion rubles. The official website of the Interfax agency).

          Within the framework of the existing public procurement system in Russia, there are a number of problems of the effectiveness of procurement processes.

          Yudin Yu.A., Tarabukina T.V., Oblizov A.V. (2023) argue that "during the execution of the contract, all savings of public funds can be spent on finalizing or correcting violations under the contract" ^[1]. It is also impossible not to agree with Melnikov V.V. (2022) that "the legislation on the contract system does not provide for the expert community and customers a common understanding of what "efficiency of public procurement" is and what are the criteria for achieving it" ^{[2, p. 127]} and "there has been an ideological departure from understanding the importance of the effectiveness of ensuring the needs of achieving budget savings while creating equal access to suppliers and transparency of transactions" [ibid., p.125].

          It should be noted that this situation is observed even 16 years after the commissioning of the EIS and the accumulation of a huge amount of digital data. Thus, the share of failed and canceled contracts in the total number of notifications for 2022 amounted to 33.4% by the end of 2022, and more than half of this figure falls on transactions for which contracts were still concluded – 20.5% of the total number of notifications (calculated on the basis of data from the Consolidated Analytical Report on the results of monitoring purchases of goods, works, services for state and municipal needs, as well as purchases of goods, works, services by certain types of legal entities for 2022, developed by the Ministry of Finance of the Russian Federation, pp.45-46).

          Although the problems of the public procurement system in Russia are complex, the obvious point of applying efforts to optimize public procurement processes is the creation of a data-driven functionality built into the EIS for selecting suppliers and assessing the risks of executing government contracts. According to the research results of Styrin E. M., Rodionova Yu. D. (2020) "There is no predictive analytics system. On electronic platforms, only individual elements of proactive service provision can be identified" ^{[3, p.61]}.

          The existing functionality of the EIS allows the customer to set a system of criteria for selecting a supplier in manual mode, partially relying on the current legislative and regulatory requirements. At the same time, the supplier within the framework of the EIS still does not have access to raw data that allows profiling the supplier, and to recommendations based on big data analysis. Regulators – the Federal Treasury, the Federal Antimonopoly Service, the Ministry of Finance, the Accounting Chamber and other state authorities - are engaged in the analysis of these data. The result of such an analysis is regularly revised standards that are fixed in legislation and mandatory for all participants in public procurement (unless otherwise provided by law) (see Figure 1).

Figure 1. An enlarged scheme of the process of regulating the public procurement system in Russia

Source: developed by the authors

           The disadvantages of the process model presented above are:

1) The duration and cost of making changes. The change of standards occurs through the introduction of amendments to regulatory legal acts;

2)      The standards of the public procurement system are calculated only on the basis of indicators located inside the EIS circuit, and not on the basis of big data available to government agencies;

3)      Data analysis and machine learning (ML) technologies are not used enough.

          It is possible to significantly improve the efficiency of public procurement processes by automating the procedure for evaluating and selecting a supplier on the basis of recommendation ML technologies, or by adding the functionality of forecasting contract performance with certain parameters to the EIS. For the development of such models, the key task will be to identify significant features that allow predicting the effectiveness of the concluded state contract.

          The object of the study is the processes of monitoring the execution and risk assessment of the execution of government contracts, and the subject of the study is the indicators of risk assessment of the execution of government contracts in the Russian Federation.

          The purpose of the study is to identify indicators (signs) of government contracts that may be important for predicting their execution.

Review of existing research

          The analysis of literary sources in this area has shown that these problems have been attracting the attention of Russian and foreign scientists for quite a long time. In Western countries, there is no such regulated system of public procurement and the level of corruption may be less, the culture of contract fulfillment and fulfillment of obligations is significantly different, so there are no comprehensive studies in the perspective of the task of assessing the risks of already launched contracts.

          The most complete, though outdated study was conducted by Farzad Khosrowshahi (1999) ^[4]. In his work, many signs have been investigated, but it is entirely devoted to the analysis of the supplier exclusively, and not the contract or the signs of the procurement process. And this work came out at a time when there was still no big historical data. Its results are only partially applicable to Russian practice, as well as the work of Acheamfour Karikari V., Kissi E., Adjei-Kumi T., Adinyira E. (2020) ^[5]. Both works appeal to the features for which to automate the collection and calculation of data, for example, the reputation of the company, the experience of the staff, etc. El-Sawalhi N., Eaton D., Rustom R. (2007) also agree with these authors ^[6]. Eliseev D., Dmitry Romanov D. (2018) have created a fairly effective assessment forecasting model that includes only four key criteria and is well suited for Russian conditions ^[7].  Sozaeva D.A., Gonchar K.V. (2022) used BI-solutions for visual data analysis, and did not evaluate the statistical significance of features for the purposes of forecasting the execution of government contracts, indicating the importance of OKPD 2 as a key feature affecting the result of contract execution ^[8]. The authors also considered any early termination of the contract as a target feature, while in the framework of this study, the authors decided to consider the termination of the contract by agreement of the parties as a successful completion of the contract, since there were a lot of such cancellations after 2022, and they are associated with difficulties experienced not only by suppliers, but also by customers. The works of Sanjana P., Basavaraj K., Pranali D, Kashmira P. (2020) ^[9] and Zedan A. (1996) ^[10] do not contain fundamental additions to the aforementioned authors, highlighting the importance of construction projects as the most problematic in terms of performance efficiency.

          Table 1 presents the main studies in this area, containing a list of the most significant features of contracts for predicting their execution.

Table 1. Overview of existing research in the field of identifying key features for predicting the effectiveness of the execution of government contracts

Source: developed by the authors

          Tsyganova M. S., Chernushenko D. A., Buresh S. V. (2022) revealed the influence of the seasonal factor on procurement efficiency, although they did not directly address forecasting issues in their work ^[11]. In this study, when uploading a data set, this is taken into account by forced random sampling for various years and months due to restrictions imposed by the peculiarities of the operation of the EIS server at this time.

Research methodology

          As part of the study, the authors applied the approaches of Exploratory Data Analysis (EDA).

          Schematically, the methodology of the study is presented in Figure 2.

Figure 2. Stages of the study

Source: developed by the authors

          Data was uploaded using the standard EIS upload service. This service has a number of restrictions regarding the number of contracts to be uploaded per operation and the absence of a number of features in the uploaded data set (if they are present in the tools of the contract search and filtering interface). Data was uploaded by repeating similar requests to the EIS for different years and combining the received uploads into one data set with the addition of a feature not displayed in the upload manually.

          In addition to all twenty-nine signs available in the EIS for unloading contracts, the signs "Status" and "Reason for termination" were manually added to the data set, having the following values:

          Values of the "Status" attribute: "Execution completed", "Execution terminated".

          The values of the attribute "Reason for termination": "No", "Unilateral refusal of the customer from the performance of the contract in accordance with civil law", "Unilateral refusal of the supplier (contractor, contractor) from the performance of the contract in accordance with civil law", "Agreement of the parties", "Judicial act".

          Next, a target attribute was added to the data set, reflecting the presence or absence of problems with the execution of the state contract. The target attribute was assigned the value "1" if the attribute "Status" has the value "Performance terminated" and the attribute "Reason for termination" has one of the following meanings: "Unilateral refusal of the customer from the performance of the contract in accordance with civil law", "Unilateral refusal of the supplier (contractor, contractor) from the performance of the contract in accordance with in accordance with civil law", "Judicial act". Accordingly, if the contract is completed normally, then the "Status" sign has the value "Execution completed". Also, the contract within the framework of the study is considered successfully executed if the "Status" sign has the value "Performance terminated", but the reason for termination is not specified or has the value "Agreement of the parties".

          At this stage, the number of records in the dataset was more than 1 million. The unique identifier of the contract is the "Registry Entry number" field, and some registry entries with about thirty thousand rows were identified in the data set. Lines with the same numbers of registry entries differ in quantity, price, cost, identifiers of products supplied / services rendered (in fact, according to the lines of the contract specification). With the help of one VBA procedure, unique values of contract registry records were found and unloaded, and with the help of another, derived signs were added to the data set: "There were rows in the registry record", "Total units of products / services actually delivered", "Total cost of delivered products". The addition of these features made it possible to compress the data set to about 83 thousand records without losing important information about contracts. The choice of VBA as a data processing tool was due to the large amount of data. The codes of the developed procedures are shown in Figures 3 and 4.

Figure 3. VBA code-procedures for finding unique numbers of registry entries

Source: developed by the authors

Figure 4. VBA code-procedures for compressing a data set by unique numbers of registry entries

Source: developed by the authors

          In parallel with the data processing by standard services, a data set for supplier companies was unloaded from the Spark/Interfax service in order to enrich the original dataset. To do this, at first, based on the data of a compact source dataset, MS Excel tools found unique TIN suppliers. The generated csv file was used as input data for uploading a data set from the above service. The resulting data set enriched the original dataset with 135 features. Due to the large number of signs, their full list is not given in the study. The structure of the added features is presented in Table 2.

 Table 2. Overview of existing research in the field of identifying key features for predicting the effectiveness of the execution of government contracts

Source: developed by the authors

          Since the financial and economic indicators of suppliers have already been well studied by previous authors (their work has already been reviewed above), and the purpose of the study is to increase scientific knowledge, in order to find new valuable features, the authors focused on analyzing the significance of categorical features that, according to the results of the domain analysis, can be useful in assessing the risks of the contract, and in at the same time, they were not previously considered by other authors.

          In total, within the framework of the consolidated data set, the authors selected the following list of features for further analysis:

• Form of ownership;
• Organizational and legal form;
• Company Size;
• Budget level;
• Supplier's region;
• The first two characters of the OKVED 2 code;
• The method of placing the order.

          Further, conjugation tables (summary tables) were constructed for the selected categorical features. Its example for the attribute "Form of ownership" is presented in Table 3:

 Table 3. The conjugation table for the attribute "Form of ownership"

Source: developed by the authors

          Attention is drawn to the large spread of the values of the target attribute across the selected groups of the studied attribute. For the obtained data, the use of Pearson's chi-squared criterion is inappropriate, since the obtained values of the criterion will be unrealistic due to the presence of zeros in the sample and uneven frequencies. In this case, it is more correct to use the exact Fischer test, however, this method is poorly optimized for conjugation tables of the dimension required in this study. The need to calculate large factorials taking into account huge frequencies quickly causes a RAM overflow error. The problem will be solved in the framework of further research by the authors by developing a special numerical algorithm. Since the purpose of this work is to find a list of new significant features that could be used in forecasting the execution of state contracts, it was decided to use one-factor analysis of variance (ANOVA).

          Tables 4 and 5 show its results for the data given earlier in Table 3.

Table 4. Calculation of averages and variances for groups of the attribute "Form of ownership"

Source: developed by the authors

 Table 5. Results of one-factor analysis of variance of the attribute "Form of ownership"

Source: developed by the authors

           Based on the results of the calculation of ANOVA indicators, it can already be concluded at this stage that this feature does not exactly have significance and is not valuable for predicting the execution of contracts, since the P-value is greater than 0.05.

          Further, according to the described methodology, the ANOVA method will be applied to the remaining selected features.

Research results and their discussion

The author suggests using a number of indicators developed on the basis of data available in state information systems to assess the risk level of execution of a specific contract.

Table 6 shows the results of applying the ANOVA method to the feature "The first two characters of the OKVED 2 code".

Table 6. Results of one-factor analysis of variance of the feature "The first two characters of the OKVED 2 code"

Source: developed by the authors

           Thus, within the framework of the study, it was revealed that the industry affiliation of the supplier organization is important for predicting the execution of a government contract (the calculated P-value is less than 0.05). The tables of conjugation and calculation of averages and variances are not given, because the number of unique high-order values of OKVED 2 included in the sample was 84.

          The same decision was made with respect to the "Supplier region" attribute (see table 7):

Table 7. Results of one-factor analysis of variance of the "Supplier region" attribute

Source: developed by the authors

          This feature may also be statistically significant for predicting the target feature.

          Consider the "Order placement method" attribute (Table 8).

Table 8. Interface table for the "Order placement method" attribute

Source: developed by the authors

           Tables 9 and 10 show its results for the data given earlier in Table 3.

Table 9. Calculation of averages and variances for groups of the "Order placement method" attribute

Source: developed by the authors

Table 10. Results of one-factor variance analysis of the "Order placement method" attribute

Source: developed by the authors

         Based on the results of the calculation of ANOVA indicators, it can be concluded that this feature is not significant and is not valuable for predicting the execution of contracts, since it has a P-value of 0.1144.

          Next, we will analyze the "Company size" attribute (see Table 11).

Table 11. Interface table for the "Company size" attribute

Source: developed by the authors

           Tables 12 and 13 show its results for the data given in table 11.

Table 12. Calculation of averages and variances for groups of the "Company size" attribute

Source: developed by the authors

 Table 13. Results of one-factor analysis of variance of the "Company size" attribute

Source: developed by the authors

          Based on the results of the calculation of ANOVA indicators, it can be concluded that this feature "Company size" has absolutely no significance and is not valuable for predicting the execution of contracts, since it has a large P-value equal to 0.71.

          The data set under study contains about 51 variants of organizational and legal forms of suppliers. The highest frequencies of the attribute values are shown in Table 14.

Table 14. Frequency of values of the attribute "Organizational and legal form of the supplier"

Source: developed by the authors

          Table 15 presents the results of a one-factor analysis of variance of the values of the attribute "Organizational and legal form of the supplier".

Table 15. Results of one-factor analysis of variance of the attribute "Organizational and legal form of the supplier"

Source: developed by the authors

       Based on the results of the calculation of ANOVA indicators, it can be concluded that the potential significance of the attribute "Organizational and legal form of the supplier" for predicting the execution of contracts, since the P-value is extremely small.

          Consider the "Budget level" attribute (Table 16).

Table 16. Interface table for the "Budget level" attribute

Source: developed by the authors

           Tables 17 and 18 show its results for the data given above.

Table 17. Calculation of averages and variances for groups of the "Budget level" attribute.

Source: developed by the authors

 Table 18. Results of one-factor variance analysis of the "Budget level" attribute

Source: developed by the authors

          Based on the results of the calculation of ANOVA indicators, it can be concluded that this feature is not significant and is not valuable for predicting the execution of contracts, since it has a P-value significantly greater than 0.05.

Conclusions

          Thus, based on the results of the application of the ANOVA method, conclusions were drawn about the potential value for the purposes of forecasting the execution of government contracts of the following features:

- "Organizational and legal form of the supplier"

- "Supplier's region";

- "The first two characters of the OKVED 2 code";

          These signs are not statistically significant:

- "Supplier's form of ownership";

- "The size of the supplier company"

- "Budget level";

- "Order placement method".

          The obtained results can be applied by researchers in the course of cluster analysis, research data analysis, in the construction of an ensemble of models for forecasting the execution of government contracts.

          The practical significance of the study is provided by the complexity and complexity of the task of forecasting government contracts, a large volume of the generated sample.

          The conclusions do not contradict the research results of other domestic and foreign authors, however, they expand and deepen existing approaches to the search for new significant features based on information capabilities contained in state information systems and other sources of big data.

         In the future, the authors plan to expand the volume of the data set, continue and deepen the research analysis of data, and build reliable ML models.

You can simply select and copy link from below text field.