Metrics

class recsyslearn.accuracy.metrics.NDCG

Bases: AccuracyMetric

NDCG evaluator for recommender systems.

classmethod evaluate(top_n: DataFrame, pos_items: DataFrame, ats: tuple = (5, 10)) → Series

Compute the NDCG@k of a model by using its recommendation list. Returns the NDCG averaged over users.

Parameters:

  • top_n (pd.DataFrame) – Top N recommendations’ lists for every user. Columns: [‘user’, ‘item’, ‘rank’].

  • pos_items (pd.DataFrame) – Relevant items per user. Columns: [‘user’, ‘pos_items’].

  • ats (tuple, default (5, 10)) – The tuple of values at which to evaluate NDCG@k.

Raises:

  • ColumnsNotExistException – If top_n does not contain columns (‘user’, ‘item’, ‘rank’) or pos_items does not contain columns (‘user’, ‘pos_items’).

  • RecListTooShortException – If the top_n list does not contain enough items.

Returns:

The NDCG@n averaged over users, in the form (‘NDCG@k_0’, …, ‘NDCG@k_n’)

Return type:

pd.Series

class recsyslearn.beyond_accuracy.metrics.Coverage

Bases: BeyondAccuracyMetric

Coverage evaluator for recommender systems. Used formula can be found here https://doi.org/10.1007/s13735-018-0154-2

classmethod evaluate(top_n: DataFrame, items: list) → float

Compute the coverage of a model by using its recommendation list.

Parameters:

  • top_n (pd.DataFrame) – Top-N recommendations’ lists for every user with items or users already segmented.

  • items (list or array-like) – List of items in the dataset.

Raises:

ColumnsNotExistException – If top_n not in the form (‘user’, ‘item’, ‘rank’, ‘group’).

Returns:

The computed coverage.

Return type:

float

class recsyslearn.beyond_accuracy.metrics.Novelty

Bases: BeyondAccuracyMetric

Novelty evaluator for recommender systems. Used formula can be found here https://doi.org/10.1007/s13735-018-0154-2 where popularity is defined in terms of the segmentation of the item groups (e.g. short head -> 3, mid tail -> 2, long tail -> 1) or in terms of percentage of user-item interactions.

classmethod evaluate(top_n: DataFrame, popularity_definition='group') → float

Compute the novelty of a model by using its recommendation list and the segmented item groups.

Parameters:

  • top_n (pd.DataFrame) – Top-N recommendations’ lists for every user with items or users already segmented.

  • popularity_definition (str) – Either ‘group’ or ‘percentage’, to choose whether popularity is computed in terms of segmenting items/users according to the distribution of user-item interactions or if it is defined as the percentage of user-item interactions.

Raises:

ColumnsNotExistException – If top_n not in the form (‘user’, ‘item’, ‘rank’, popularity_definition).

Returns:

The computed novelty.

Return type:

float

class recsyslearn.fairness.metrics.Entropy

Bases: FairnessMetric

Entropy evaluator for recommender systems.

evaluate(top_n: DataFrame, rel_matrix: DataFrame | None = None) → float

Compute the entropy of a model by using its recommendation list.

Parameters:

  • top_n (pd.DataFrame) – Top N recommendations’ lists for every user with items or users already segmented.

  • rel_matrix (pd.DataFrame, default None) – Relevant items for users. It could be, for example, the items with a rating >= threshold.

Raises:

ColumnsNotExistException – If top_n not in the form (‘user’, ‘item’, ‘rank’, ‘group’).

Returns:

The computed entropy.

Return type:

float

class recsyslearn.fairness.metrics.KullbackLeibler

Bases: FairnessMetric

Kullback-Leibler divergence evaluator for recommender systems.

evaluate(top_n: DataFrame, target_representation: DataFrame, rel_matrix: DataFrame | None = None) → float

Compute the Kullback-Leibler divergence of a model, for a given target representation, by using its recommendation list.

Parameters:

  • top_n (pd.DataFrame) – Top N recommendations’ lists for every user with items or users already segmented.

  • target_representation (pd.DataFrame) – The target representation desired for each group.

  • rel_matrix (pd.DataFrame, default None) – Relevant items for users. It could be, for example, the items with a rating >= threshold.

Raises:

ColumnsNotExistException – If top_n not in the form (‘user’, ‘item’, ‘rank’, ‘group’) or if target_representation not in the form (‘group’, ‘target_representation’).

Returns:

The computed KL Divergence for the given target representation.

Return type:

float

class recsyslearn.fairness.metrics.MutualInformation

Bases: FairnessMetric

Mutual Information evaluator for recommender systems.

evaluate(top_n: DataFrame, flag: str, rel_matrix: DataFrame | None = None) → float

Compute the Mutual Information of a model by using its recommendation list.

Parameters:

  • top_n (pd.DataFrame) – Top N recommendations’ lists for every user with items or users already segmented.

  • flag (str) – Which actor of the recommendation scenario has been segmented (i.e. user).

  • rel_matrix (pd.DataFrame, default None) – Relevant items for users. It could be, for example, the items with a rating >= threshold.

Raises:

ColumnsNotExistException – If top_n not in the form (‘user’, ‘item’, ‘rank’, ‘group’).

Returns:

The computed Mutual Information.

Return type:

float