smyalygames/checklist-tester
1
0
Fork 0
mirror of https://github.com/smyalygames/checklist-tester.git synced 2025-05-18 14:34:12 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat(dissertation): write most of results in full

Browse Source
This commit is contained in:
Anthony Berg 2024-05-22 14:14:55 +01:00
parent ada9e447b1
commit 8743ae27f2
2 changed files with 224 additions and 136 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

360
pub/dissertation/chapters/results.tex
View File

@ -3,86 +3,165 @@
\begin{document} \begin{document}
\section{Final Prototype} \section{Final Prototype}
\subsection{Formal Model} \subsection{Formal Model}
\begin{itemize} % \begin{itemize}
\item The model is mostly designed to imitate a Boeing 737-800, % \item The model is mostly designed to imitate a Boeing 737-800,
as the types modelled, have user inputs which are different from % as the types modelled, have user inputs which are different from
other aircraft types % other aircraft types
\begin{itemize} % \begin{itemize}
\item For example, the Airbus A320 has push buttons whereas % \item For example, the Airbus A320 has push buttons whereas
they are not there on the 737-800 % they are not there on the 737-800
\item However, further user input types could be added to the model % \item However, further user input types could be added to the model
and as a result, further aircraft types could have their % and as a result, further aircraft types could have their
procedures run through the formal model % procedures run through the formal model
\end{itemize} % \end{itemize}
\item The \lstinline|Procedure| type makes sure that the items on % \item The \lstinline|Procedure| type makes sure that the items on
the procedure is completed in order, and if a step is missed, % the procedure is completed in order, and if a step is missed,
that would result in an invariant failure, resulting in the % that would result in an invariant failure, resulting in the
checklist test failing % checklist test failing
% TODO write more % % TODO write more
% Stuff like: % % Stuff like:
% - How items are handled % % - How items are handled
% - Functions such as auto complete or step by step % % - Functions such as auto complete or step by step
% - Why Aircraft? % % - Why Aircraft?
\end{itemize} % \end{itemize}
The formal model was designed using the Boeing 737-800 to create the
types for inputs types that exist on the aircraft, for example
switches, buttons, etc. This is significant as other aircraft
have different input interfaces, such as the Airbus A320, where the
majority of the inputs use buttons that click on as an alternative to switches.
However, further forms of aircraft input types can be added to the formal model
in the future which would allow for the formal model to be compatible with
a larger variety of aircraft.
There are multiple well-formed checks implemented through invariants,
pre- and post-conditions, with an example being the \lstinline|Procedure| type
having an invariant that makes sure that the items in the procedure/checklist is
completed in order, and if a step is skipped, it would result in an invariant violation,
meaning that the test for the checklist has failed.
\subsection{Checklist Tester} \subsection{Checklist Tester}
\begin{itemize} % \begin{itemize}
\item The main features of GUI have been completed, it has all the sections desired % \item The main features of GUI have been completed, it has all the sections desired
\begin{itemize} % \begin{itemize}
\item Projects can be created to split up different aircraft % \item Projects can be created to split up different aircraft
or revisions of checklists % or revisions of checklists
\item Procedures can be created and tested % \item Procedures can be created and tested
\item These procedures get tested in the flight simulator automatically % \item These procedures get tested in the flight simulator automatically
and gives the results of how the procedure has been doing in % and gives the results of how the procedure has been doing in
real time % real time
\end{itemize} % \end{itemize}
\end{itemize} % \end{itemize}
\subsection{Setting up Tests} All the desired sections of the GUI has been implemented, allowing for
\begin{itemize} the checklist tester to be used to meet the objectives of this project.
\item Each test is set up by defining each action in the procedure,
on the Procedure screen The GUI allows for projects to be created, allowing separation of aircraft and
\item To be able to define each action is supposed to do, it uses revisions of QRHs. In each project, checklists can be created, and the steps
the Dataref variables in X-Plane, which is what stores the state for the checklist can be defined and edited if needed. Once the steps in
of the aircraft. Each switch has their own unique Dataref the checklist are defined, the test for the checklist can be run
\item In the checklist tester then, each action asks for a and the Checklist Tester will automatically run each step of the checklist
Dataref and a desired goal value and show the progress through the checklist in real time and if each step
\item Some Datarefs are read only, but there are other Datarefs in the checklist is being completed correctly or failing.
for the item desired, but are only \enquote{command}s, which
can only be called and not have its value changed; this can be \subsubsection{Setting up Tests}
run by setting the desired goal value to be -988 (because XPC uses that value) % \begin{itemize}
\end{itemize} % \item Each test is set up by defining each action in the procedure,
% on the Procedure screen
% \item To be able to define each action is supposed to do, it uses
% the Dataref variables in X-Plane, which is what stores the state
% of the aircraft. Each switch has their own unique Dataref
% \item In the checklist tester then, each action asks for a
% Dataref and a desired goal value
% \item Some Datarefs are read only, but there are other Datarefs
% for the item desired, but are only \enquote{command}s, which
% can only be called and not have its value changed; this can be
% run by setting the desired goal value to be -988 (because XPC uses that value)
% \end{itemize}
Each test is set up by defining each step of the checklist from the
\textit{Procedure} screen in the Checklist Tester.
To be able to define what each step of the checklist is supposed to do,
it requires the dataref variable, which are the variables that
store the state of the aircraft in X-Plane, to be referenced for the specific
input in the aircraft for that step in the checklist. To identify dataref name required
for the specific input, there is an X-Plane plugin DataRefTool
which also allows to see the current state that the datatref is, and it is a read only
variable. Then, to set the desired goal of the step in the checklist, the input can be put
to the desired state in the flight simulator, and the value of the dataref can be taken
and be set in the Checklist Tester.
However, some aircraft in X-Plane have read-only dataref variables that can only
be modified by running a command, calling a specific dataref. So to be able to
test that step in the checklist, the desired state of the step can be set as
--988 (that value was chosen because XPC uses that value to not modify variables).
This will mean that the checklist tester will not attempt to change the variable
of the dataref.
\subsubsection{Running Tests} \subsubsection{Running Tests}
\begin{itemize} % \begin{itemize}
\item Tests are run by connecting to the flight simulator, X-Plane % \item Tests are run by connecting to the flight simulator, X-Plane
\item The tester goes through each action in the procedure one by one % \item The tester goes through each action in the procedure one by one
and waits for the current action to complete before proceeding on to % and waits for the current action to complete before proceeding on to
the next one % the next one
\item The checklist tester is not advanced enough to be able to control % \item The checklist tester is not advanced enough to be able to control
fly the aircraft; hence the tester would be able to engage autopilot % fly the aircraft; hence the tester would be able to engage autopilot
first, or control the aircraft themselves, where the checklist tester % first, or control the aircraft themselves, where the checklist tester
would be acting like a first officer % would be acting like a first officer
\end{itemize} % \end{itemize}
Running a test for a checklist requires an active instance of X-Plane
to be running with the plane loaded in, as the Checklist Tester
checks for an active simulator connection, otherwise it will not run.
Once the test has been started, the Checklist Tester goes through each action
in the checklist one by one and waits for the current step to complete before
proceeding to the next one.
The Checklist Tester is not advanced enough to control the flight controls
of the aircraft, meaning that the aircraft has to be flown manually,
have autopilot set manually, or add steps to control the autopilot in the Checklist Tester,
avoiding the need to set up the autopilot manually each time.
\subsubsection{Storing Test Results} \subsubsection{Storing Test Results}
\begin{itemize} % \begin{itemize}
\item There is a database storing the results of each of the tests % \item There is a database storing the results of each of the tests
\item Each tests store % \item Each tests store
\begin{description} % \begin{description}
\item[Time taken] for each of the actions in the procedure to complete % \item[Time taken] for each of the actions in the procedure to complete
\item[Start state] for the state that the action in the procedure was at % \item[Start state] for the state that the action in the procedure was at
\item[End state] for the state that the action in the procedure finished % \item[End state] for the state that the action in the procedure finished
the item at % the item at
\item[Overall test time] Stores the time taken from when the test started % \item[Overall test time] Stores the time taken from when the test started
to when the test ended % to when the test ended
\end{description} % \end{description}
\item This gives feedback/statistics for the checklist designers % \item This gives feedback/statistics for the checklist designers
to find areas of improvement on the procedure, such as one action % to find areas of improvement on the procedure, such as one action
in the procedure taking too long, may point out a potential flaw % in the procedure taking too long, may point out a potential flaw
to the designer and as a result aid finding potential alternative % to the designer and as a result aid finding potential alternative
options for that step in the procedure % options for that step in the procedure
\end{itemize} % \end{itemize}
Whilst checklists are being tested in the Checklist Tester,
there are multiple aspects being tracked and stored on the database to be
used as results for the tests that run. The results are stored on the
\lstinline|Test| and \lstinline|ActionResult| table, which can be seen on the
entity relationship diagram in~\autoref{fig:db-erd}, with the respective values
that are stored.
The aspects that the database store are the time taken for the entire
checklist, by taking the time when the test started, and when the last step in the
checklist was completed. These are stored as a start and end time on the \lstinline|Test|
table, in Coordinated Universal Time (UTC) format.
Each step that is tested in the checklist gets tracked separately in
the \lstinline|ActionResult| table, where the start and end state of the dataref
is tracked, with the start end end time in UTC format.
This gives feedback/statistics for the checklist designers to find
areas of improvement on the procedure, such as one action in the
procedure taking too long, may point out a potential flaw to the designer,
as a result aiding finding potential other options for that step in the procedure.
\subsection{Submitting a Pull Request for X-Plane Connect} \subsection{Submitting a Pull Request for X-Plane Connect}
% What I did: % What I did:
@ -121,82 +200,91 @@
\item Submitted the pull request stating the changes made\footnote{\url{https://github.com/nasa/XPlaneConnect/pull/313}} \item Submitted the pull request stating the changes made\footnote{\url{https://github.com/nasa/XPlaneConnect/pull/313}}
\end{itemize} \end{itemize}
\section{Problems Found}
\section{LOC?}
% TODO I don't know what LOC meant, was it Language of Choice?
\section{Reflection} \section{Reflection}
\subsection{Planning} \subsection{Planning}
\subsubsection{Gantt Chart} % Used Gantt chart to create a plan for what would be needed from this project
Used Gantt chart to create a plan for what would be needed from this project
\textbf{Pros:} % \textbf{Pros:}
\begin{itemize} % \begin{itemize}
\item Was useful for the first part because it set expectations % \item Was useful for the first part because it set expectations
of what was needed and how much time there was to complete them % of what was needed and how much time there was to complete them
\item Helped visualize the different components of the project % \item Helped visualize the different components of the project
\item Helped in the beginning being accompanied by a Kanban in Leantime\footnote{\url{https://leantime.io/}} % \item Helped in the beginning being accompanied by a Kanban in Leantime\footnote{\url{https://leantime.io/}}
\end{itemize} % \end{itemize}
\textbf{Cons:} % \textbf{Cons:}
\begin{itemize} % \begin{itemize}
\item Was not detailed enough, and a design document would have been useful % \item Was not detailed enough, and a design document would have been useful
to accompany the Gantt chart for each section % to accompany the Gantt chart for each section
\item The lack of detail was not helpful when falling behind as having % \item The lack of detail was not helpful when falling behind as having
attention deficit hyperactivity disorder (ADHD) % attention deficit hyperactivity disorder (ADHD)
added to the burden of feeling like each section was a massive project % added to the burden of feeling like each section was a massive project
\item Leantime's claim for being \enquote{built with ADHD [\ldots] in mind} % \item Leantime's claim for being \enquote{built with ADHD [\ldots] in mind}
felt misleading as navigating through it felt worse than using the front page % felt misleading as navigating through it felt worse than using the front page
of Stack Overflow\footnote{\url{https://stackoverflow.com/}} % of Stack Overflow\footnote{\url{https://stackoverflow.com/}}
\item Todoist\footnote{\url{https://todoist.com/}} was a good alternative though % \item Todoist\footnote{\url{https://todoist.com/}} was a good alternative though
\end{itemize} % \end{itemize}
\subsubsection{GUI Design} A Gantt chart was used to create a plan for what would be needed from this project
Figma was very useful in implementations as and when these parts of the project should be completed.
\textbf{Pros:} The Gantt chart was useful for the first part of the project because it set expectations
\begin{itemize} of what was required and how much time there was to complete them. It also helped
\item It helped with timing and knowing what to do visualize the different components of the project. Implementing the Gantt chart into
\item Made things feel manageable as it was split up to different sections Leantime\footnote{\url{https://leantime.io/}} was helpful at first as it was able to
\item Meant features will not be forgotten be accompanied by a Kanban.
\end{itemize}
\textbf{Cons:} However, there were multiple downfalls of the Gantt chart. One of the problem was that
\begin{itemize} the design of the Gantt chart lacked detail for each of the components. A way this could
\item Certain features being too simple and annoying to use have been fixed was by making the Gantt chart more detailed, or create a design document
\item A bit of a learning curve for using other components, compared to accompany the Gantt chart. The lack of detail was later made worse as when falling behind
to using plugins with attention deficit hyperactivity disorder (ADHD), it felt like a burden to progress
\end{itemize} as each section felt like a massive project, when in reality it could have been split up
into subtasks.
Leantime's claim for being \enquote{built with ADHD [\ldots] in mind} felt misleading
as navigating through it felt worse than using the front page of
Stack Overflow\footnote{\url{https://stackoverflow.com/}} as it was very cluttered to access
what was desired, such as the Kanban requiring multiple pages to navigate through.
For the future, it would be helpful to find an alternative to Leantime to aid in
progress tracking.
\subsection{Implementation} \subsection{Implementation}
\subsubsection{Checklist Tester} \subsubsection{Checklist Tester}
\begin{itemize} % \begin{itemize}
\item Implementing the GUI was useful to split up the sections required % \item Implementing the GUI was useful to split up the sections required
for the project and having a goal for what to be done % for the project and having a goal for what to be done
\item However, a bit too much time was spent on creating a GUI when it % \item However, a bit too much time was spent on creating a GUI when it
could have been used for development % could have been used for development
\item It was useful for motivational reasons to feel like something % \item It was useful for motivational reasons to feel like something
materialistic has been produced rather than something theoretical % materialistic has been produced rather than something theoretical
\item Was originally intended to be used to interact with custom % \item Was originally intended to be used to interact with custom
plugin for X-Plane as it would have been difficult otherwise % plugin for X-Plane as it would have been difficult otherwise
\end{itemize} % \end{itemize}
\subsubsection{Connecting to the Flight Simulator} Implementing the GUI was useful to split up the sections required for the project,
\begin{itemize} and having an informal requirement for each section of the project.
\item Would have been more useful to search a bit further if there was However, a bit too much time was spent on creating a GUI when it could have been
another plugin available, as found Dataref Editor on the X-Plane docs, used for development or creating a design document which would have aided in
so could have looked for a similar plugin for connecting to X-Plane productivity.
\item At first spent about a week developing a C++ X-Plane plugin from scratch,
requiring to figure out sockets However, implementing the GUI was useful to an extent as it provided motivation
\item At the same time finding out XPC exists and having wasted that time by having something tangible rather than something theoretical or a command line interface.
\item However, it did teach me more about understanding how sockets work and
more about C++ and setting up a project with CMake and adding packages with % \subsubsection{Connecting to the Flight Simulator}
vcpkg % \begin{itemize}
\end{itemize} % \item Would have been more useful to search a bit further if there was
% another plugin available, as found Dataref Editor on the X-Plane docs,
% so could have looked for a similar plugin for connecting to X-Plane
% \item At first spent about a week developing a C++ X-Plane plugin from scratch,
% requiring to figure out sockets
% \item At the same time finding out XPC exists and having wasted that time
% \item However, it did teach me more about understanding how sockets work and
% more about C++ and setting up a project with CMake and adding packages with
% vcpkg
% \end{itemize}
\section{Time Spent} \section{Time Spent}
\begin{itemize} \begin{itemize}

BIN
pub/dissertation/dissertation.pdf
View File

Binary file not shown.