06 - Confusion and rates

More on rank

• giving a subject, giving some scores with templates, a rank is said that, such as we take the first k results and consider them.
• having a rank 20 means to find out that the correct match is somewhere in top 20 similarity scores

Key-definitions to avoid errors

• precision
  • fraction of relevant instances among the retrieved istances
  • Precision = TP / (TP + FP)
• recall (sensitivity)
  • fraction of relevant instances that were retrieved
  • TP/(TP+FN)
• f-score: a combination of both of them

Using the ML terms Positive (P) for A and Negative (N) for R

FRR = FN/(TP+FN) FAR = FP/(FP+TN)

They are clearly different, since they start from the two types of error!

FRR can be compared to Miss Rate or false negative rate (FNR)

FAR can be compared to Fall-Out or False Positive Rate

Offline evaluation

• it is important to have a recognition system offline
• all samples should be labeled, but this is not true in reality
• so we need to evaluate performance

In performance evaluation, each sample is presented as a probe, and they may play or not the role of being an impostor according to the gallery setup and on the possible identity claim.

1. Training vs Testing (TR vs TS):

In this scenario, the dataset is divided into two distinct sets: the training set (TR) and the testing set (TS). The training set is used to train the biometric recognition system, while the testing set is used to evaluate its performance.

Example: Let’s say you have a dataset of facial images from 100 individuals. You could randomly select 70 individuals for the training set and the remaining 30 individuals for the testing set. The algorithm learns from the facial features of the individuals in the training set and is then tested on the images in the testing set to assess its accuracy.

2. Probe Set vs. Gallery Set (P vs. G):

In this scenario, the dataset is divided into two sets: the probe set (P) and the gallery set (G). The probe set consists of biometric samples that need to be identified, while the gallery set contains the enrolled templates or samples against which the probe samples are compared for identification.

Example: Consider a fingerprint recognition system. The probe set could consist of fingerprint images captured from various sources (e.g., crime scenes), and the gallery set could contain the fingerprint templates of known individuals stored in a database. The system then matches the probe fingerprints against the gallery fingerprints to identify the individuals.

3. Templates in the Probe Set (P) Related to Gallery Set (PG) vs. Including Templates Not in the Gallery Set (P = PG U PN):

In this scenario, the probe set can be organized in two ways:

• P = PG: The templates in the probe set are always from subjects in the gallery set. In other words, the biometric samples to be identified are guaranteed to be enrolled individuals in the system.
• P = PG U PN: The probe set includes templates belonging to subjects not in the gallery set. This scenario accounts for the possibility of encountering unknown individuals during recognition.

Example: Imagine an iris recognition system where P represents the probe set, PG represents the gallery set, and PN represents templates of individuals not in the gallery set.

• P = PG: The probe set consists of iris images of individuals enrolled in the system. This scenario assumes that all individuals needing identification are already registered in the system.
• P = PG U PN: The probe set includes iris images not only from enrolled individuals but also from individuals not registered in the system. This scenario simulates real-world situations where unknown individuals might be encountered.