Validating Knowledge Structures

Gamma Index: It was proposed by Goodman & Kruskal (1972) as a way of validating the prerequisite relations and assumes that not every response pattern is represented by a prerequisite relation. It compares the number of response patterns that are represented by a prerequisite relation with the number of response patterns that are not.

γ = (N_c - N_d) / (N_c + N_d)

where N_c is the number of concordant pairs and N_d is the number of discordant pairs.

Generally, a positive Gamma Index supports the validity of the prerequisite relations and a negative Gamma Index violates the its validity.

   

Violational Coefficient: It validades prerequisite relations. For this purpose, the number of violations against the prerequisite relations are calculated. A low VC supports the validity of prerequisite relations.

VC = ∑ v_xy / n(|S| - m)      

Discrepancy Index: It was proposef by Kambouri et al. (1994) as a way of validating the knowledge space and allows for the evaluation of its quality. DI calculates the average minimal distance between each response pattern and the nearest state in the knowledge space.

DI = ∑_r min_K{ d (r, K) | r ε R } / n

     

Distance Agreement Coefficient: It was proposef by Schrepp (1999) as a way of validating the knowledge space and allows for the evaluation of its quality. It compares the average symmetric distance between the knowledge states and participant's response patterns (d_dat) with the average distance between the resulting knowledge states and the power set of items (d_pot).

d_dat calculates the average minimal distance between each response pattern and the nearest state in the knowledge space.

d_dat = ∑_r min_K{ d (r, K) | r ε R } / n

d_pot calculates the average minimal distance between the power set of all potencial response patterns and the nearest state in the knowledge space.

d_pot = ∑_P min_K{ d (P, K) | P ⊆ Q } / 2^|Q|

 

Note that d_dat is equal to the DI.

Generally, a low Distance Agreement Coefficient supports the validity of the knowledge space and a high Distance Agreement Coefficient violates its validity.

DA = d_dat / d_pot

 

Please note that for the relation-oriented measures, quasi-ordinal knowledge spaces should be used.

Example Spaces

As example data, knowledge spaces provided by the R package pks (Heller & Wickelmaier, 2013; Wickelmaier et al., 2016) are used. Concretely, the following spaces are used:

Density 97

Taagepera et al. (1997) applied knowledge space theory to specific science problems. The density test was administered to 2060 students, a sub structure of five items is included here.

Matter 97

Taagepera et al. (1997) applied knowledge space theory to specific science problems. The conservation of matter test was administered to 1620 students, a sub structure of five items is included here.

Doignon & Falmagne

Fictitious data set from Doignon and Falmagne (1999, chap. 7).

Please note that for "Matter 97" and "Density 97" the structures are not (quasi-ordinal) knowledge spaces. The VC and γ coefficients work on the underlying surmise relation, i.e. effectively on the closure of the knowledge structure under union and intersection.

References

Doignon, J.-P., & Falmagne, J.-C. (1999). Knowledge spaces. Berlin: Springer.

Heller, J. & Wickelmaier, F. (2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42, 49-56.

Schrepp, M., Held, T., & Albert, D. (1999). Component-based construction of surmise relations for chess problems. In D. Albert & J. Lukas (Eds.), Knowledge spaces: Theories, empirical research, and applications (pp. 41--66). Mahwah, NJ: Erlbaum.

Taagepera, M., Potter, F., Miller, G.E., & Lakshminarayan, K. (1997). Mapping students' thinking patterns by the use of knowledge space theory. International Journal of Science Education, 19, 283--302.

Wickelmaier, F., Heller, J., & Anselmi, P. (2016). pks: Probabilistic Knowledge Structures. R package version 0.4-0. https://CRAN.R-project.org/package=kst