 |
Website: Open source assessment Status: concluded Period: September 2019 - December 2019Funding: € 38.610,00 (VAT excluded) Funding organization: AgID (Agenzia per l'Italia Digitale) Person(s) in charge: |
Executive summary:
The project consisted in defining a procedure for assessing and comparing open source software, both technically and economically. Such procedure aims to assist Public Administrations in acquiring open source software.
The procedure was defined by the Nexa Center in cooperation with AgID and with the help of the Piedmont Region.
Background:
In May 2019, AgID (the Agency for Digital Italy) published the guidelines on acquisition and re-use of software for the Public Administration. The guidelines aim to help the Public Administrations in reusing and acquiring software, promoting the share and use of open source software. In particular, the guidelines specify that, when searching for software, the Public Administrations must first take into consideration and assess software already used and shared by other Public Administrations, then open source software, and eventually-only if any suitable software has been found in previous steps-off the shelf software can be considered.
Objectives:
In this context, the Nexa Center for Internet and Society, in cooperation with AgID, defined a procedure to assess and compare open source software. The procedure fully conforms to AgID guidelines. The procedure consists first of a technical and economic assessment of the software solutions, secondly, all the software solutions under evaluation are compared according to results of the assessment previously performed, and finally a rank from the most to the least satisfactory solution is produced.
Regarding the technical assessment, each software solution under evaluation is quantitatively assessed using the assessment criteria defined in AgID guidelines. Specifically, these assessment criteria are: coverage of functional and non-functional requirements, interoperability, protection of personal data, security, accessibility, the presence of a maintainer, the presence of support for installation, the dependencies from other software, the competence of Public Administration in the use of the software, the number of Public Administrations interested in the software, and the vitality of the software project.
Regarding the economic assessment, the Total Cost of Ownership (TCO) of the software is estimated. To do so, a new TCO model was produced, which resulted from a review of the scientific literature on TCO, an analysis of some TCO models available online and feedback received from the Piedmont Region. The TCO model so formulated takes into consideration both capex and opex costs of acquiring a software.
After each software under evaluation is economically and technically assessed, in the proposed procedure a multi-criteria decision making algorithm is adopted to compare all the software solutions. As input, the algorithm takes the results of the economic and technical assessments and the weights of the assessment criteria (which were also defined in this work); as output, it produces a rank from the most to the least satisfactory software solution.
Finally, a possible extension of the software assessment criteria was proposed in this work. In fact, it was proved that there exists a mapping between the assessment criteria of AgID guidelines and the software quality characteristics of the ISO/IEC 25010:2011 standard (system and software quality models). Therefore, it was proposed to also include the ISO/IEC 25010:2011 quality characteristics in the assessment criteria, at the discretion of the interested Public Administration.
Results:
The proposed procedure was tested by 7 Public Administrations in real cases. To analyze the effectiveness and ease of use of the proposed procedure, these Administrations were asked to reply to a survey. The main results of the survey are as follows. Among the 7 Public Administrations, one considered the procedure difficult to conduct, while the remaining ones considered it of medium difficulty or easy to compile, with respect to an average difficulty of their normal tasks. 6 over 7 Public Administrations completed the procedure in less than 4 hours and the remaining one between 4 and 8 hours.
The main outputs of this project are:
- A document describing the proposed procedure for assessing and comparing open source software.
- A spreadsheet, which allows Public Administrations to practically conduct the assessment and comparison procedure.
- The weights to be assigned to the assessment criteria, according to the category of software under evaluation.
The results of the project are also presented at the 129° Nexa Wednesday
