Skip to main content
SearchLoginLogin or Signup

The Evaluation of the Performance Quality for Publishers of Long-term Preservation of Digital Science and Technology Resources

Published onAug 29, 2024
The Evaluation of the Performance Quality for Publishers of Long-term Preservation of Digital Science and Technology Resources
·

Abstract – [Purpose/Significance] Since 2013, the National Digital Preservation Center (hereinafter referred to as the NDPC) led by the National Science and Technology Library has constructed 71 agreements for the long-term preservation of digital science and technology resources. However, there are problems, such as delayed submission and incomplete data from publishers, that negatively impact the long-term preservation of digital science and technology resources. Therefore, it is necessary to perform a performance quality evaluation to serve as an effective means of deterrent. [Method/Process] The key indicators in the long-term preservation agreement were extracted based on the implementation work. The evaluation indicators were enhanced through Delphi Method to establish a performance quality evaluation system for publishers. Subsequently, the NDPC institution conducted scoring and division of levels to identify common issues and propose incentive and deterrent mechanisms. [Results/Conclusion] The long-term preservation of digital science and technology resources lack robust business-oriented deterrents. It is recommended to establish incentive and deterrent mechanisms from combining long-term preservation to contract renewal, publicity of ranking, and data crawling authorization to urge publishers to enhance performance quality.

KeywordsLong-term Preservation of Digital Resources, Evaluation of Performance Quality, Incentive and Deterrent Mechanisms.

This paper was submitted for the iPRES2024 conference on March 17, 2024 and reviewed by Dr. Lee Pretlove, Willem Vanneste, Mikala Narlock and 1 anonymous reviewer. The paper was accepted with reviewer suggestions on May 6, 2024 by co-chairs Heather Moulaison-Sandy (University of Missouri), Jean-Yves Le Meur (CERN) and Julie M. Birkholz (Ghent University & KBR) on behalf of the iPRES2024 Program Committee.

Introduction

Digital science and technology resources are important for scientific and technological innovation and social development. The long-term preservation of digital science and technology resources, namely, through the continuous management and maintenance of digital information, ensures the long-term survival of digital information and ensures the authenticity, discoverability, accessibility and utilization of digital information [1].

In 2013, the National Digital Preservation Program (NDPP) was led by the National Science and Technology Library (hereafter referred to as NSTL) and cooperated with relevant domestic institutions to systematically and comprehensively implement digital science and technology resources, in particular, the localized long-term preservation of external digital science and technology resources, which can effectively solve the problem that digital resources cannot be reliably utilized in our country for a long time due to natural disasters, deliberate destruction, network barriers and other potential dangers [2]. In May 2023, the National Digital Preservation Center (hereinafter referred to as the NDPC) was officially launched. Based on the establishment of the National Science Library, the Chinese Academy of Sciences, the NDPC undertakes work related to the long-term preservation of NSTL digital science and technology resources and comprehensively promotes the long-term preservation of digital science and technology resources in China [3].

As of June 2023, the NDPC has reached 71 resources of 56 publishers in China, including electronic journal databases, electronic book databases, academic paper databases, and curated databases, and has built China's largest long-term preservation resource system covering both national and international high-quality digital science and technology resources. However, there are problems, such as delayed submission and incomplete data from publishers, that negatively impact the long-term preservation of digital science and technology resources.

In recent years, in terms of evaluating performance quality of long-term preservation of digital science and technology resources in China, a capability maturity assessment framework for long-term preservation service assessment was designed, and a digital preservation service maturity evaluation framework that meets practical needs was preliminarily developed; this framework can serve as a reference for institutions constructing and developing digital preservation services in China [4]. A number of indicators were selected according to the factors that affect the preservation of scientific data to construct a scientific data evaluation index system, which could provide suggestions for scientific data management [5]. Comparison and contrasting of LIFE and CDL-TCP can provide a reference for studies on the cost of digital preservation for related domestic archive institutions [6]. In general, although the evaluation framework for long-term preservation tailored to China's characteristics has been established from macroscopic perspectives, there is a lack of evaluation and incentive mechanisms, as well as deterrent on whether publishers submit long-term preservation data in long-term preservation implementation.

This paper will extract indicators from long-term preservation agreements based on implementation work. The evaluation indicators were enhanced through Delphi Method to establish a performance quality evaluation system for publishers. Subsequently, the NDPC institution conducted scoring and division of levels to identify common issues and propose incentive and deterrent mechanisms.

Construction of an Evaluation Index System for the Performance Quality of Publishers

Evaluation Principles

The evaluation criteria for assessing the performance quality of publishers are based on whether licensed materials have achieved complete and accurate preservation, as agreed upon between both parties. From a legal perspective, the most fair and just content is the long-term preservation agreement between the publishers and the NDPC. Therefore, the evaluation principle for assessing the performance quality of publishers is to evaluate their adherence to the agreed content in the long-term preservation agreement with the NDPC.

The NDPC has a long-term preservation agreement template. Both parties confirm the contents of the template through negotiation. The specific aspects covered by this agreement, such as the scope of preserved resources, institutional coverage, data format, and data transmission frequency, can be quantified as evaluation index that must be fulfilled by publishers. However, any additional data provided by them beyond what is stipulated in this agreement should not be considered when evaluating their performance quality.

Evaluation Objects

Evaluation objects: Six preservation institutions, which are responsible for implementing long-term preservation, were established by the NDPC. These institutions include (1) National Science Library, the Chinese Academy of Sciences (hereinafter referred to as NSLC), (2) Institute of Scientific and Technical Information of China (hereinafter referred to as CITIC), (3) Peking University Library, (4) Agricultural Information Institute of the Chinese Academy of Agricultural Sciences, (5) Institute of Medical Information & Library, the Chinese Academy of Medical Sciences and Peking Union Medical College, and (6) China National Publications Import and Export (Group) Co., Ltd. (hereinafter referred to as CNPIEC). As of June 2023, only the NSLC, CITIC, and Peking University Library have successfully implemented stable long-term preservation, which means that means that they have already signed  long-term preservation agreements with some publishers for a period of more than one year, and preserved data has been received since last year at the latest. Therefore, this evaluation will primarily focus on these three institutions.

Evaluated objects: Since a publisher may consist of various types of resources with different service teams assigned accordingly, to accurately evaluate performance quality, we initially consider "resources" as the minimum dimension for evaluation purposes. This includes 71 resources provided by 56 publishers, such as electronic journal databases, electronic book databases, dissertation databases, and factual databases. In future research and practice scenarios where further evaluation based on "publishers" is necessary, the average value calculation method will be utilized to organize and calculate the evaluation results.

Evaluation Cycle and Evaluation Subject

The six preservation institutions of the NDPC primarily serve the purpose of resource acquisition in addition to long-term preservation. One of the reasons for establishing long-term preservation cooperation with publishers is to achieve effective resource acquisition. Resource acquisition is typically based on a yearly cycle. Therefore, the evaluation cycle for assessing the performance of a publisher can be determined by aligning it with the resource acquisition cycle and the timing for self-audit verification. So, it is also based on a yearly cycle.

The evaluation subject consists of two aspects: resources and technology. The resources aspect refers to determining trigger events, identifying content that needs to be preserved, and agreeing upon specific arrangements for data delivery during negotiations between each institution’s resource team within the NDPC and the publisher. The technology aspect involves verifying whether the publisher has submitted all required data after receiving it. The resource team quantifies these aspects into measurable indicators, which are then provided to the technical team. The technical team evaluated the quality of the preserved data. In summary, both the resource team and the technology team complement each other in achieving comprehensive evaluations.

Extraction of Evaluation Indicators

The long-term preservation agreements of 71 domestic and foreign high-quality digital science and technology resources, realized by the NDPC, were organized, and the corresponding evaluation indicators were extracted from the agreement terms. Considering factors such as signing year, resource type, and adjustment of the agreement template leading to diversified evaluation indicators, a method of consolidating similar items was adopted to preliminarily extract a total of 19 evaluation indicators encompassing both performance indicators of the licensor (publishers) and licensee (NDPC).

As this paper aims to evaluate the performance quality of publishers, the performance indicators of NDPCs were excluded. Simultaneously, in practical long-term preservation implementation, data submission from publishers poses significant challenges. Therefore, after extensive discussions and analyses, technical aspects were prioritized as the most crucial evaluation content while appropriately reducing the emphasis on evaluating the resources.

Through repeated comparisons and integration along with referencing relevant research related to performance evaluation, three first-level indicators and twelve second-level indicators were confirmed, as shown in Table 1.

Table 1: Evaluation Indicators for the Performance Quality of Publishers

First-level Indicator

Second-level Indicator

Trigger Event

Trigger Event Activation

Trigger Event Execution

Legal Dispute

Type of Legal Dispute

Data Delivery

Data Delivery Frequency

Data Integrity

Data Quality

Data Delivery Format

Data Delivery Method

Data Delivery List

Data Delivery Notification

Service Response Efficiency

Resource List

The first-level indicator of "triggered event" assesses the publishers’ ability to adhere to the agreement and enable licensees/NDPC to provide long-term preservation services within the agreed-upon organization scope during trigger events. It comprises two second-level indicators: trigger event activation and trigger event execution. The trigger event activation means that if trigger event never happened before, or ever happened. The trigger event execution means that if the publisher agrees licensee to enable the service when trigger events happened. It is best that the trigger event never happens, which means that the publisher's service is relatively stable. Secondly, if the trigger event happens, the publisher can cooperate with us to enable the service. The worst scenario is when the trigger event occurs, but the publisher refuses to enable the service.

The first-level indicator of "legal dispute" evaluates whether the publisher has caused legal disputes due to noncompliance with the agreement. It encompasses one second-level indicator: the type of legal dispute. It includes no legal dispute, arbitration, and litigation.

The first-level indicator of "data delivery" signifies the performance of the publishers in data delivery. It includes nine second-level indicators: data delivery frequency, data integrity, data quality, data delivery format, data delivery method, data delivery list, data delivery notification, service response efficiency, and resource list. All of these second-level indicators are extracted from the terms of the agreement, which mean that if publisher is able to comply with the agreed content of all aspects. It is not explained one by one here, but can refer to the description of the evaluation content in the following evaluation index system.

Construction of the Evaluation Index System

After determining the evaluation indicators, experts in data analysis, experts in long-term preservation negotiations, and experts in long-term preservation data processing are selected, and several expert seminars are organized through Delphi Method  to ultimately determine the weight of the evaluation indicators and the corresponding score calculation standard for each evaluation content.

The sum of the weights for all the second-level indicators under each first-level index is 1. The weight distributions for the first-level indexes are as follows: trigger event (0.1), legal dispute (0.3), and data delivery (0.6). The weight distributions for the second-level indicators are as follows: trigger event activation (0.5), trigger event execution (0.5), legal dispute type (1), data delivery frequency (0.1), data integrity (0.15), data quality (0.15), data delivery format (0.1), data delivery method (0.08), data delivery list (0.08), data delivery notification (0.09), service response efficiency (0.10) and resource list (0.15).

The score calculation standard corresponding to each evaluation is divided into five grades: 0, 1, 5, 8, and 10. A score of 0 points indicates that the content cannot be evaluated due to a lack of data, while scores ranging from 1 to 10 represent poor, medium, moderately good, and excellent performance, respectively.

Based on the results, the performance quality evaluation index system for publishers can be formulated as follows:

Table 2: Evaluation Index System for the Performance Quality of Publishers

First-level Indicator

Second-level Indicator

Score Calculation Standard

Index Name

Weight

Index Name

Weight

Evaluation Content

Criteria for Scoring

Trigger Event

0.1

Trigger Event Activation

0.5

Never happened before

10

Ever happened

5

Trigger Event Execution

0.5

The publisher agrees licensee to enable the service

10

The publisher refuses licensee to enable the service

1

Not involved

10

Legal Dispute

0.3

Type of Legal Dispute

1

Not involved

10

Arbitration

5

Litigation

1

Data Delivery

0.6

Data Delivery Frequency

0.1

Able to comply with the agreed data delivery frequency.

10

Changing the frequency of data delivery, time-delay within 3 months

8

Changing the frequency of data delivery, time-delay within 3 to 6 months

5

Changing the frequency of data delivery, time-delay within 6 to 12 months

1

No data submitted

0

Data Integrity

0.15

Perfect

10

Not more than 5% of data missing

8

Not more than 10% of data missing

5

More than 10% of data missing

1

No data submitted

0

Data Quality

0.15

The necessary data elements are complete

10

Not more than 1% of necessary data elements missing

8

1%-5% of necessary data elements missing

5

More than 5% of necessary data elements missing

1

No data submitted

0

Data Delivery Format

0.1

Able to comply with the agreed data delivery format

10

Changing the data delivery format and notification in advance

8

Changing the data delivery format and no notification

5

No data submitted

0

Data Delivery Method

0.08

Able to comply with the agreed data delivery method

10

Changing the data delivery method, time-delay within 3 months

8

Changing the data delivery method, time-delay within 3 to 6 months

5

Changing the data delivery method, time-delay beyond 6 months

1

No data submitted

0

Data Delivery List

0.08

Able to attach data delivery list when submitting data

10

After reminding, data delivery list can be attached

8

Data delivery list can be attached but format fails to meet the specified requirements

5

No data delivery list

0

Data Delivery Notification

0.09

Able to notify in advance or according to the data delivery frequency agreed by both parties

10

Able to notify in advance after reminding

8

Able to notify in advance occasionally

5

No notification

0

Service Response Efficiency

0.1

The missing data could be completed within 15 working days upon receipt of supplementary notification

10

The missing data could be completed within 15 to 30 working days

8

The missing data could be completed within 30 to 60 working days

5

The missing data could be completed beyond 60 working days

1

Unsolved

0

Resource List

0.15

Able to submit resource lists on time and accurately

10

Able to submit resource lists on time and but inaccurately

8

Able to submit resource lists on time but some mistakes occurred

5

Delay submission of resource list

1

No submission

0

Performance Quality Evaluation Results of Publishers

Evaluation Empirics

According to the performance quality evaluation index system, a questionnaire method was adopted and independently scored by three NDPC institutions on 71 high-quality digital science and technology resources that have been preserved for a long time. The resource team primarily focused on trigger events and legal disputes as key indicators, examining whether there were any instances where publishers denied the NDPC's provision of long-term preservation services within agreed institutional scopes during trigger events, as well as whether any legal disputes arose due to noncompliance with agreements by these publishers. Meanwhile, the technical team concentrated on data delivery as a primary indicator, conducting comprehensive analysis of self-audit results to empirically evaluate factors such as timely receipt of preserved data, accuracy and integrity of data delivery, and responsiveness of publishes in addressing issues.

Evaluation Score Calculation Method

Based on the three institutions of the NDPC that have empirically scored the questionnaire, the total score of a resource is determined through a weighted average calculation method, which is adopted by taking into account the relative importance of different data items in the dataset. The weighted average calculation is a calculation formula in statistics. It will ensure that the contribution of each data item is better reflected, particularly when certain items hold more significance than others. By sorting the overall evaluation results, the comparability of the evaluation results is ensured.

The weighted average calculation method is as follows:

xˉ\bar x represents the weighted average, as well as the evaluating results of each resource. ff represents the weight, and xx represents the score of each item. Since the sum of ff is equal to 1, which means that nn is equal to 1. So, the calculation method in this paper is relatively simple, that is, x1f1+x2f2+...+xkfkx_1*f_1+x_2*f_2+...+x_k*f_k.

Evaluation Results

According to the performance quality evaluation index system, the weighted average calculation method can be utilized for evaluating results, which range from 4 to 10. To effectively assess the performance quality of one resource and emphasize the significance of evaluation, it is necessary to classify the evaluation results, facilitating subsequent release to publishers as a means of providing incentives or deterrents.

The grading evaluation method can be employed to determine score ranges based on evaluation results such as excellence, good, and unqualified status. Considering the research objective of this paper, with a maximum score of 10, three grades of A/B/C, are assigned according to the evaluation results. Grade A represents excellence characterized by timely data submission, high data quality, proactive problem-solving abilities, etc., with a score ranging between 9.496 and 10. Grade B indicates good performance where both parties agree upon submitting data within a specific timeframe while maintaining good data quality and prompt resolution of issues, whose score range lies between 7-9.495. Grade C signifies an unqualified status requiring improvement in performance quality with scores below 7.

Based on the NDPC assessment results, 60% of the resources belong to Grade A, 25% to Grade B, and only 15% to Grade C. Resources under grade C exhibit poor performance quality primarily due to significant delays in data delivery. On the other hand, resources categorized under grades A and B collectively represent approximately 85%, indicating that most publishers generally meet NDPC requirements regarding performance quality but need further enhancements in various aspects.

Common Problems of Publishers

Due to the absence of recent execution cases, a temporary performance evaluation cannot be conducted regarding the emergency start-up, backup, and transfer of preservation data in the preservation data system. Publishers are capable of fulfilling the agreement terms and actively dealing with any problems encountered in terms of agreement performance, and there have been no legal disputes, conflicts, or claims related to the agreement. With respect to the delivery of preserved data, there are several common issues, such as data integrity, timely data delivery and standardization in data format processing. In addition, there is a need for further improvement and optimization in performance quality regarding mechanisms and efficiency in problem handling. The common problems of publishers are as follows:

  1. Delayed delivery of data

Some publishers fail to deliver data according to agreed-upon timelines, typically with a 3-8 month delay. Certain publishers suggest providing long-term preservation data only after signing the license agreement.

  1. Worrying data quality

On the one hand, some publishers do not provide journals, lack volume, or repeatedly submit data. On the other hand, resource lists from publishers contain inaccuracies such as confusion between the starting year of long-term preservation and accessible years, ISSN errors, and inconsistent journal titles compared to publishers’ official websites.

  1. Lack of professional service teams

Most of the publishers lack a specified long-term preservation service team and are primarily served by sales and technical staff. This often leads to difficulties in resolving early-stage issues after personnel changes, as they have limited understanding of long-term preservation. Therefore, NDPCs need to repeatedly emphasize and train them on this topic. Additionally, some publishers exhibit a slow response in regard to providing missing data required for supplementation by NDPCs after the audit process, resulting in a supplementary cycle of approximately one month or longer.

Incentive and Deterrent Mechanisms for Publishers on Performance Quality

Incentive Mechanisms

Following the evaluation results of publishers, to grant honor to outstanding performance publishers and motivate other publishers to enhance their long-term preservation performance quality, the following four incentive mechanisms are proposed:

  • Publicity of performance quality rankings on the NDPC website, national conferences on long-term preservation and resource acquisition;

  • Commendation of excellent publishers during long-term preservation open day activities;

  • Provision of booths or presentation opportunities at national conferences on long-term preservation and resource acquisition for exceptional publishers;

  • The priority given to excellent publishers and their resources in terms of contract renewal.

Deterrent Mechanisms

Due to the absence of a significant penalty for violating the existing terms of long-term preservation agreements, three deterrent mechanisms are proposed for contract renewal, and long-term preservation data are obtained in a reasonable manner to address the underperformance of publishers:

  • Implementing NDPC negotiations for publishers with poor performance, where the performance quality of long-term preservation will be considered during contract renewal.

  • Inclusion of a clause in future renewal contracts stating that each institution of the NDPC could rely on its own technical strength to acquire long-term preservation data.

  • Long-term preservation rights are recognized as legitimate rights held by libraries for resource acquisition, and these rights are included as key evaluation indicators in the resource acquisition proposal.

Conclusion

The long-term preservation and sustainable utilization of digital resources is significant in China [7]. As the key driver for comprehensively advancing the establishment of China's long-term preservation system for digital science and technology resources, the NDPC should not only preserve all types of commercial digital resources but also ensure the complete and accurate implementation of established long-term preservation cooperation resources. The research conducted in this paper has made some progress in evaluating the performance quality of long-term preservation. However, there are still several areas that need further improvement. It is hoped that publishers can be encouraged to strictly adhere to the provisions outlined in the long-term preservation agreement, establish a service coordination and processing mechanism, and implement a performance quality evaluation as well as an incentive and deterrent mechanism to enhance performance quality. Together, these efforts will contribute to jointly promoting the construction of China's long-term preservation system.

Comments
0
comment
No comments here
Why not start the discussion?