Knowledge Space Theory
Knowledge space theory is a formal theory started in the eighties by Jean-Claude Falmagne and Jean-Paul Doignon, for the assessment of knowledge in humans beings. The initial objective was to construct "an efficient machine for the assessment of knowledge" (Doignon ,Falmagne 1985). Nowadays, with the diffusion of the computers, this "efficient machine" has the form of a computer program aimed at assessing knowledge, competence and learning of single individuals in a given field.
Psychometric theory (eg.classical test theory, or item response theory) was developed much earlier than KST, with the aim of assessing knowledge, ability, proficiency, etc producing a numerical score in a given area. In KST instead, the main goal of assessment is to provide a precise description of what the student knows and does not at a given moment.
KST has two main components:
The deterministic component draws upon discrete math theories (including set theory, lattice and order theory, combinatorics), to point out theoretically plausible combinatorial structures for the representation of knowledge and learning.
The probabilistic component brings about the probabilistic methods and models for the empirical validation and for the application of the deterministic models in real (noisy) contexts.
Some formal definitions:
- The Domain of knowledge is set \(Q\) of questions;
- The Knowledge state is the subset \(K \subseteq Q\) of all questions that an individual is able to solve;
- The Knowledge structure is a pair \((Q,\mathcal{K})\) , where \(\mathcal{K}\) is a collection of Knowledge states, containing at least the empty set and \(Q\) ;
- The Response pattern is the subset \(R\subseteq Q\) of the items correcly solved by an individual;
The objective of an assessment in KST consists of identify the Knowledge state of an individual in a given domain of knowledge \(Q\) by asking a few questions as possible. The assessment of knowledge can be accomplish in a probabilistc or deterministc way.In the deterministic approch is assume that a student's response to an item is correct if and only if the student masters it. So the response pattern and the knowledge state are the same.
According to the probabilistc approch in real situation for each item \(q \in Q\),two types of errors can occurs.
A Careless error occurs when item \(q\) belongs to \(K\) but the response is incorrect.
A Lucky guess occurs when item \(q\) doesn't belongs to \(K\) but the response is correct.
In the probabilistic approach of the Basic Local Indipendence Model (BLIM,Falmagne & Doignon, 1988a, 1988b) , the response pattern and the knowledge state need not to be equal.
Because this two types of errors there will be a proability \(\beta_{q}\) of a careless errors and a probability \(\eta_{q}\) of a lucky guess for each of the items \(q\in Q\). The careless error and the lucky guess probabilities are define as \(0\leq \beta_{q},\eta_{q}< 1\).
Let \(r_q\) be a random variable with realizations in \({0,1}\), representing the either correct (1) or incorrect (0) response to item \(q\). Then the conditional probability of observing a correct response to item \(q\) is:
The assessment procedure works iterativley by amministring an item at the time. In each step the procedure selects the new item to amminister, obtains a response from the students and updates the likelihood of the knowledge states. See for further details you can read the documentation.
This ShinyApp is a pedagogical demostration of how adaptive knowledge assessment works in KST. This demostration using a set of 23 items in the domain of fractions. For this set of items a knowledge structure contain 260 knowledge states was developed. The 23 items and the knowledge structure are used by the ShinyApp in conjunction with the assessment procedure described above.
The ShinyApp operates in two different modalities: either interactively or by simulating the responses of a student ( Start the simulation ). In both case is shown how the adaptive assessment work step-by-step.
Three main parameters can be manipulated:
- The lucky guess probabilities and the careless error probabilities . You can tune lucky guess and careless error of the items. Things have made simple by assuming tht all the items as same lucky guess and careless error probabilities. If you set this parameters to zero what you get is a deterministc assessment otherwise the assessment will be probabilistic
- The Stopping criterion . This parameter is the minimum level that the likelihood of a knowledge state must have to stop the assessment procedure.
Procede to the other Panel.
A Short Guide
This ShinyApp allows the user to perform his own assessment several times, using the interactive and the simulation modalities, changing the parameters of the procedure and observing step by step they affect the assessment results both in terms of accuracy and efficiency.
Step 1
After click on the assessment tab you can locate a slider for setting your lucky guess, careless error and stopping criterion parameters in the bottom left corner of the screen. (Please notice that once you have set this parameters you should avoid to change them during the assessment session)
Step 2
At this point you can start the assessment and answer to the first question which is locate in the middle of the screen. What you get is something similare this screenshot.
Step 3
When you finish with the interactive modality you can also experiment the simulation modality.
In first place you need to restarting the assessment by clicking Restart Button
After that you can switch to simulation modality by setting as the students true state of knowledge your own or other knowledge state changing the parameters (lucky guess, careless error and stopping criterion). In the simulation modality, the App simulate the response of a student with the given knowledge state. To be observe that a number of question in the simulation could sometimes be quite larger, to make the simulation faster in this case you can choice a larger number of simulation step for each click of the button Procede with the simulation using the Number of simulate answer input.
Step 4
At the end of the simulation you can always go back to the Step 1 and try other scenario. In the accompaing documentation ( Here the documentation ) you can find a number of questions for you. By finding an answer to eache of this question you should improve your understanding of the fondamental carateristic and meccanism of assessment procedure.