Welcome to ValidationQuant.com
This is my professional website. Validationquant LLC is my consulting firm.
Please download some of my presentations and publications from the Presentations tab.
Slides from many of the courses I taught, mostly on Model Validation, are here.
Please download some of my presentations and publications from the Presentations tab.
Slides from many of the courses I taught, mostly on Model Validation, are here.
What is Model Validation?
( A non-technical analogy with many buzzwords )
Models are stylized and simplified versions of the real world, which can be more easily manipulated than reality. The messiness of the real world is replaced with assumptions representing the informed opinion of the model designer. The reason models are useful is precisely these simplifying assumptions. There will almost always be unforeseen or unusual circumstances that “break” the model. A model can be “wrong” if it breaks too often, but models are never absolutely correct.
Model validation is an independent determination of whether the model is fit for its intended use. The key principle is “effective challenge” – the validation should be competent and thorough enough to find whatever flaws may exist, and the validator should be sufficiently knowledgeable and empowered, with proper incentives, to point out these flaws and ensure that they are addressed.
Models are not panaceas; they are just tools, like a can opener or an axe. To explain model validation, I will use the analogy of an axe. The intended use for an axe is to cut wood, in this example. The axe developer makes some assumptions, about which kind of wood is being cut, how strong the lumberjack is, how thick the logs will be, and so on. Model validation includes :
1. Verifying proper governance and control mechanisms – Does company policy ensure that the lumberjacks do not bring their own axes? Are broken axes reported to management? Do the lumberjacks get time to sharpen their axes? Do the lumberjacks get supplied with helmets, goggles, and gloves?
2. Verifying sufficiency of documentation – Does the documentation describe what kind of steel the axe blades are made of, how to sharpen and re-sharpen, how the handle is attached, how to train the lumberjacks with this new equipment, etc.?
3. Evaluation of conceptual soundness, including developmental evidence – Does the documentation explain why axes were chosen instead of chainsaws, and how these axes were tested on the intended kind of wood?
4. Verifying inputs – Look at the forest; are the trees the same kind of wood that the developer expected? Are the lumberjacks as strong as described by the developer?
5. Checking assumptions – Is the wood free of nails, as expected by the developer? Do the lumberjacks get sharpening stones? Is the wood in their forest the same kind as described in the documentation?
6. Checking the design – Are the handles the right size and shape to attach to the blades? Does the metal shop have the right kind of steel to make the blades as specified? If they buy the axes instead of making them, that axe is a vendor model and may require a separate validation.
7. Checking implementation - The validator tests that the handles are attached as specified; the edges are sharpened properly; the lumberjacks are all trained; etc. The validator cuts some wood to test the axe.
8. Benchmarking against alternative approaches – The validators cut wood with the axes being validated, some other axes, chainsaws, and hand saws, and document how each performed in terms of speed, sawdust and wood chips, quality of the cuts, etc.
9. Replication – The validator lumberjacks build axes according to the documentation, and cut the same kind of wood with the replica axes and also with the axes being validated. Do the replicas cut wood the same way, or does the documentation describe how to construct adzes instead of axes?
10. Checking the model's accuracy – Do the chips fly as described in the documentation? Do the axes stay sharp as long as described? Most important, does the axe actually cut the wood? Does the axe land squarely where it was aimed?
11. Demonstrating that the model is robust and stable – Do different lumberjacks get the same results? Does the axe cut well even when it gets a bit dull? Do the trees fall in the right direction even if they are slightly crooked?
12. Assessing potential limitations – Can the axe cut standing trees or only logs? Does it need special tools to sharpen? Can it cut through knotty or wet wood? Can it cut wood which has nails or spikes? Is the axe too heavy for some lumberjacks to use?
13. Sensitivity analysis – Is slightly softer wood slightly easier and slightly faster to cut? Does a slightly stronger lumberjack cut slightly faster?
14. Stress testing, including scenarios that are outside the range of ordinary expectations – The validator cuts harder and harder wood, until eventually the axe breaks or bounces off. The validator cuts softer and softer wood until the axe gets stuck.
15. Ongoing monitoring – Do the axes continue to perform as well as when the initial validation was performed, and is the quality of the cuts deteriorating? Are the axes needing replacement more often than expected? Are the lumberjacks using the axes to cut wood, or are they instead splitting coconuts or opening cans?
16. Monitoring model performance by periodic review, including back-testing – The validator checks back a year later to see if the lumberjacks have any complaints and to see if any unexpected problems with the wood, the axes, or the lumberjacks, have come up since the last review.
Models are stylized and simplified versions of the real world, which can be more easily manipulated than reality. The messiness of the real world is replaced with assumptions representing the informed opinion of the model designer. The reason models are useful is precisely these simplifying assumptions. There will almost always be unforeseen or unusual circumstances that “break” the model. A model can be “wrong” if it breaks too often, but models are never absolutely correct.
Model validation is an independent determination of whether the model is fit for its intended use. The key principle is “effective challenge” – the validation should be competent and thorough enough to find whatever flaws may exist, and the validator should be sufficiently knowledgeable and empowered, with proper incentives, to point out these flaws and ensure that they are addressed.
Models are not panaceas; they are just tools, like a can opener or an axe. To explain model validation, I will use the analogy of an axe. The intended use for an axe is to cut wood, in this example. The axe developer makes some assumptions, about which kind of wood is being cut, how strong the lumberjack is, how thick the logs will be, and so on. Model validation includes :
1. Verifying proper governance and control mechanisms – Does company policy ensure that the lumberjacks do not bring their own axes? Are broken axes reported to management? Do the lumberjacks get time to sharpen their axes? Do the lumberjacks get supplied with helmets, goggles, and gloves?
2. Verifying sufficiency of documentation – Does the documentation describe what kind of steel the axe blades are made of, how to sharpen and re-sharpen, how the handle is attached, how to train the lumberjacks with this new equipment, etc.?
3. Evaluation of conceptual soundness, including developmental evidence – Does the documentation explain why axes were chosen instead of chainsaws, and how these axes were tested on the intended kind of wood?
4. Verifying inputs – Look at the forest; are the trees the same kind of wood that the developer expected? Are the lumberjacks as strong as described by the developer?
5. Checking assumptions – Is the wood free of nails, as expected by the developer? Do the lumberjacks get sharpening stones? Is the wood in their forest the same kind as described in the documentation?
6. Checking the design – Are the handles the right size and shape to attach to the blades? Does the metal shop have the right kind of steel to make the blades as specified? If they buy the axes instead of making them, that axe is a vendor model and may require a separate validation.
7. Checking implementation - The validator tests that the handles are attached as specified; the edges are sharpened properly; the lumberjacks are all trained; etc. The validator cuts some wood to test the axe.
8. Benchmarking against alternative approaches – The validators cut wood with the axes being validated, some other axes, chainsaws, and hand saws, and document how each performed in terms of speed, sawdust and wood chips, quality of the cuts, etc.
9. Replication – The validator lumberjacks build axes according to the documentation, and cut the same kind of wood with the replica axes and also with the axes being validated. Do the replicas cut wood the same way, or does the documentation describe how to construct adzes instead of axes?
10. Checking the model's accuracy – Do the chips fly as described in the documentation? Do the axes stay sharp as long as described? Most important, does the axe actually cut the wood? Does the axe land squarely where it was aimed?
11. Demonstrating that the model is robust and stable – Do different lumberjacks get the same results? Does the axe cut well even when it gets a bit dull? Do the trees fall in the right direction even if they are slightly crooked?
12. Assessing potential limitations – Can the axe cut standing trees or only logs? Does it need special tools to sharpen? Can it cut through knotty or wet wood? Can it cut wood which has nails or spikes? Is the axe too heavy for some lumberjacks to use?
13. Sensitivity analysis – Is slightly softer wood slightly easier and slightly faster to cut? Does a slightly stronger lumberjack cut slightly faster?
14. Stress testing, including scenarios that are outside the range of ordinary expectations – The validator cuts harder and harder wood, until eventually the axe breaks or bounces off. The validator cuts softer and softer wood until the axe gets stuck.
15. Ongoing monitoring – Do the axes continue to perform as well as when the initial validation was performed, and is the quality of the cuts deteriorating? Are the axes needing replacement more often than expected? Are the lumberjacks using the axes to cut wood, or are they instead splitting coconuts or opening cans?
16. Monitoring model performance by periodic review, including back-testing – The validator checks back a year later to see if the lumberjacks have any complaints and to see if any unexpected problems with the wood, the axes, or the lumberjacks, have come up since the last review.