Felix provides several options for organisations to build evaluation templates. Templates can include provision for free text responses, N/A responses for evaluators, the ability for evaluators to skip questions, and a variety of numerical scoring scales.

When drilling into evaluation scores, the following principles are important to understand.

NOTE: As a reminder, templates are made up of one or more criteria, each criterion having a % weighting. Criteria can include one or more questions.

 

All questions are weighted equally within a criteria set

Felix does not currently allow each question within a criteria to be individually weighted. If a criterion has three (3) scorable questions, they are each worth 33%. If there are four (4) scorable questions, they are each worth 25%, etc.

 

For Yes/No questions, a Yes is always a 100% score

This is important when your users are setting up templates and the way questions are worded to get the right response from evaluators.

 

Free Text questions are not scorable

This might seem obvious, however, its important to highlight when a criteria set only includes free text questions.

 

Evaluation criteria will be 0% weighting if all questions are free text

As all free text questions are non-scorable, a criteria setup like this will be 0% weighting and effectively act purely as a way to capture qualitative feedback in the same location as quantitative feedback in an evaluation template.

 

Skipped questions or N/A responses are effectively removed from the scoring for a criteria

If a question was set as Optional or a N/A response option was offered on a scorable question, then Felix does not penalise or reward the vendor for this question. It is removed from the calculation for that criteria and the remaining scored questions are re-balanced.

Example: A criteria has four questions. Three are scored and one is left Unscored or with No Response.

The default behaviour is that all questions within a criteria are equally weighted.

 

Criteria 1
Question	Question Weighting	Score
1	25%	1/5
2	25%	1/3
3	25%	10/10
4	25%	Unscored or No Response

Re-balanced, so that we do not reward the vendor for the last question with 100% nor penalise them with 0%.

 

Criteria 1 (re-balanced)
Question	Question Weighting	Score
1	33%	1/5
2	33%	1/3
3	33%	10/10
4		Unscored or No Response

With a resultant score for this criterion of:

(0.33*0.2)+(0.33*0.33)+(0.33*1) = 50%

 

Criteria with all N/A responses are effectively removed from the scoring

There are possible cases where a criterion has all scorable questions answered with N/A (Unscored). In these cases, the criteria are effectively excluded and Felix does not penalise nor reward the vendor for this fact. It is removed from the calculation template and the remaining scored criteria are rebalanced.

 

Criteria with all skipped questions are given a score of 0

Unlike N/A responses from evaluators, where they feel they are not in a position to score this topic or where all questions in criteria are skipped (where they are not mandatory), then the criteria are given a score of 0 for that evaluation.

 

When combining scores across evaluations for the same template - each criteria is first averaged, then those averages totalled. 

In order to respect the intended weightings of criteria and any N/A responses that can distort this effect, Felix first averages criteria scores across completed evaluations, then sums these averages for a total.

Example: Two evaluations are completed by two users, using the same template. The template has three criteria. 

 

User 1
Criteria	Weighting	Score
1	50%	60%
2	20%	70%
3	30%	Unscored
TOTAL		=(0.6*0.5+0.7*0.2) / (0.5+0.2) = 61.7%

For User 1, they responded with all N/A responses in Criteria 3. Criteria 3 is therefore Unscored and does not penalise nor reward the vendor. The evaluation score is re-balanced across the scored criteria.

 

User 2
Criteria	Weighting	Score
1	50%	60%
2	20%	No Response
3	30%	40%
TOTAL		= (0.6*0.5+0*0.2+0.4*0.3) = 42%

For User 2, they skipped all responses in Criteria 2. Criteria 2 is therefore marked as No Response and receives a score of 0.

Felix then combines these two evaluations as follows:

Criteria 1: ( 60% + 60% ) / 2 evaluations scored = 60% of 50% weighting = 30%

Criteria 2: ( 70% + 0% ) / 2 evaluations scored = 35% of 10% weighting = 3.5%

Criteria 3: ( 40% ) / 1 evaluation scored = 40% of 30% weighting = 12%

For a total score for the vendor across the two (2) evaluations of 45.5% 

NOTE: Felix used to simply average the two evaluation scores, which in the example above would be 51.9% (which is (61.7 + 42)/ 2). This method was changed to better respect criteria in the case of N/A scores.