Can Model Checking be used to Validate AI Planner Domain Models?
Verification and validation are the two main components of the traditional method for determining whether or not a piece of software is error-free. Verification ensures that a software implementation meets a defined set of criteria (Schwabe et al., 2019). Validation is the process of confirming that the specified criteria accurately represent the end user’s demands. The importance of validation changes somewhat when dealing with AI and knowledge-based systems. A knowledge base is used to describe the world with which the program interacts, and a thinking algorithm distorts the base of knowledge while the program is executed. The effectiveness of the AI system is directly related to the precision of the base of knowledge about the actual world. Even if a system has a perfect reasoning mechanism, it will be useless if its representation of the world is inaccurate.
As a result, validating a component of the deployed system is essential to ensure it works as intended. The HSTS Planner is a model-based planning system used in the Remote Agent automated system control architecture. We are now exploring model checking to help verify the models used in this system. Inputs to the planner include the current state and the desired result, and outputs are the necessary steps to perform in order to accomplish the latter (Schwabe et al., 2019). The resulting plan is guaranteed to be compatible with a set of temporal constraints representing real-world restrictions on the regulated system and its surroundings, thanks to the planning algorithm. Tight coupling between constraints in domain models may soon lead to a situation where the interaction between constraints Nisonger is conceptually tractable. It is preferable, therefore, to have techniques to verify the model by eliminating model discrepancies and checking to see whether the model’s implicit attributes can be inferred from the collection of explicit constraints.
Constraint Model Specifications
The HSTS Domain Description Language (DDL) is an object-oriented constraints specification language developed using Allen’s periodic interval logic to define the models of the HSTS Planner domain. Classes and instances get defined by DDL models. A variety of system variables are included inside each object. If a state variable may be changed during scheduling, it is said to be controllable; otherwise, it is said to be uncontrolled. Instances of a class of robots, for instance, might be defined and created as follows:
(Define_Object_Class Robot
:state_variables
((Controllable Task)
(Controllable Location)))
(Define_Object Robot Robbie)
In this language, collections of labels may be used to define objects with no properties:
(Define_Label_Set Room (Kitchen Hallway LivingRoom))
The definition is provided for each ‘member value’ of a state variable. Predicates are used to describe members’ values and may include additional arguments. The value of a state is defined by the name of a condition and the values of its arguments (Dellermann et al., 2019). Objects, labels, and built-in types are all excellent choices for predicate arguments. For instance, the Robot class has defined the potential values for the Task and Location state variables. The Task state variable that belongs to the Robot class in this example may either have the value “Moving” (x,y), where x and y are locations, or it can have the value “Idle.” In the Robot class, the Location state may take the value ‘In Room’ (x), where ‘x’ is the robot’s location.
Determining the compatibility constraints that need to exist between various values of state variables is one of how a DDL model is limited. The restrictions are stated in terms of temporal periods, also known as tokens, during which a state variable retains a particular value or combination of values (Norgeot et al., 2020). During this time, the tokens are in effect. A collection of precedence temporal operators is used to specify the suitable connections between tokens. The operators make it possible to represent any temporal connections between the beginning and ending locations of two tokens. In addition, shorthand for standard temporal connections is supported by DDL. Token descriptors are used in the refining process for DDL compatibility requirements. A pattern representing the properties of a group of tokens is referred to as a token descriptor. These patterns may either refer to a single token or a series of tokens in succession (Dellermann et al., 2019). The following format is used for compatibility specifications: (master-token-descriptor compatibility-tree), where the ‘compatibility-tree’ is a binary search tree where the nodes are time relations and the leaves are tokens. Additionally, variables may be used to impose constraints on parameter values when transferring information between the parent token identifier and the token identifiers in the ‘compatibility tree.’
Model Checking for DDL
To prepare HSTS models as valid inputs for model checkers, we must first describe them in terms of stages and transformations between stages. Each object’s state variable values define the system’s state in HSTS (Schwabe et al., 2019). When the value of at least one stage variable shifts, it is said that a transition has occurred. To make it easier to describe what occurs during a transition, we use the convention where V (resp. V’) stands for the value of variable V before (resp. after)
(SV to P) (SV != P) AND (SV’ == P)
(SV from P) (SV == P) AND (SV’ != P)
The input languages of Spin [5], SMV [2], and Murphy [4] were translated using this general translation as a starting point. Due to variances in the source languages, the translated texts had to be somewhat adjusted for each target language. By way of illustration, a short program is included with both the Spin and Murphy translations that choose the next stage of the model according to the state transition relation (Norgeot et al., 2020). A state transition relation may be specified as a propositional formula in the input languages used for SMV, which significantly facilitates translation.
Results
This translation was tested on a scale model of a self-driving robot. The prototype was a battery-operated robot that could go from point A to point B and between. The robot’s objective was to x a hole in surface A. Out of a total of 569872 states, the model’s 66 temporal restrictions made 17320 of them attainable. The model’s constraints imposed the following restrictions:
((Robot. Task = Moving) contained by (Hole.Charge = Charge Full))
((Robot. Task = Fixing Hole) meets (Hole.Status = Hole Fixed))
The analysis that had been missed through testing discovered several possible flaws in the concept.
Since most of the robot’s modeling can be done in the subset of the language covered by the original translation, this is a reasonable assumption. Some quantitative timespan utilized in the model was lost in the recording process. In this case, we did not encounter any problems due to a lack of capacity to translate nonlocal restrictions. The research on the model’s robustness and livability revealed no significant issues. When nothing terrible happens, like in the absence of a stalemate, this is called a safety property (Schwabe et al., 2019). They are easily checked by running the model through a model checker to see whether any possible states in the model break the property. For a system to be considered “alive,” it must adhere to certain conditions, such as responding to a user’s questions or concerns at some point. Since they affect runs rather than states, they are more challenging to manage. Not all model checkers can validate liveness qualities; SPIN and SMV, for example, need a more complex state space exploration technique.
The SMV model checker is noticeably quicker than the competition for the first part of the language it supports (0.05 seconds vs. 30 seconds). This outcome occurred for two reasons. The fundamental reason for this is that the depth of the possible traces across the system was small compared to the depth of the state vector (Norgeot et al., 2020). SMV’s symbolic state space formulation and searching approach work better with this model. Sets of states are represented as logical predicates in this approach, eliminating the need to individually express each state (Clarke et al.,2018). The second is that, unlike the typical guarded programming language or computer language format for model checkers, SMV permits an explicit definition of a transition probability relation. Because of this, the translated DDL constraints may be transferred directly into the model-checking vocabulary without requiring further procedures to determine the next system based on the constraint. When the translation is extended to manage nonlocal restrictions and quantitative intervals, it remains to be shown whether SMV can keep this edge.
Conclusion
Some preliminary research indicates that model checking may be used to detect bugs in HSTS domain models successfully. Model checking’s comprehensive search capabilities will unearth modeling mistakes that a heuristic search during planning could miss. In the future, we want to make the translated language more expressive and automate the translation process after we have settled on the appropriate target language for model verification to be used by domain experts.
References
[1]. Clarke, E. M., Henzinger, T. A., & Veith, H. (2018). Introduction to model checking. In Handbook of Model Checking (pp. 1-26). Springer, Cham.
[2]. Dellermann, D., Lipusch, N., Ebel, P., & Leimeister, J. M. (2019). Design principles for a hybrid intelligence decision support system for business model validation. Electronic markets, 29(3), 423-441.
[3]. Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., … & Butte, A. J. (2020). Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nature medicine, 26(9), 1320-1324.
[4]. Penix, J., Pecheur, C., & Havelund, K. (1998, December). Using model checking to validate AI planner domain models. In Proceedings of the 23rd Annual Software Engineering Workshop, NASA Goddard.
[5]. Schwabe, K., Teizer, J., & König, M. (2019). Applying rule-based model-checking to construction site layout planning tasks. Automation in construction, 97, 205-219.
write