checklist-tester/pub/dissertation/chapters/results.tex

\documentclass[../dissertation.tex]{subfiles}

\begin{document}
\section{Final Prototype}
\subsection{Formal Model}
% \begin{itemize}
%   \item The model is mostly designed to imitate a Boeing 737-800,
%     as the types modelled, have user inputs which are different from
%     other aircraft types
%     \begin{itemize}
%       \item For example, the Airbus A320 has push buttons whereas
%       they are not there on the 737-800
%       \item However, further user input types could be added to the model
%         and as a result, further aircraft types could have their
%         procedures run through the formal model
%     \end{itemize}
%   \item The \lstinline|Procedure| type makes sure that the items on
%     the procedure is completed in order, and if a step is missed,
%     that would result in an invariant failure, resulting in the
%     checklist test failing
%   % TODO write more
%   % Stuff like:
%   % - How items are handled
%   % - Functions such as auto complete or step by step
%   % - Why Aircraft?
% \end{itemize}

The formal model was designed using the Boeing 737-800 to create the
types for inputs types that exist on the aircraft, for example
switches, buttons, etc. This is significant as other aircraft
have different input interfaces, such as the Airbus A320, where the
majority of the inputs use buttons that click on as an alternative to switches.
However, further forms of aircraft input types can be added to the formal model
in the future which would allow for the formal model to be compatible with
a larger variety of aircraft.

There are multiple well-formed checks implemented through invariants,
pre- and post-conditions, with an example being the \lstinline|Procedure| type
having an invariant that makes sure that the items in the procedure/checklist is
completed in order, and if a step is skipped, it would result in an invariant violation,
meaning that the test for the checklist has failed.

\subsection{Checklist Tester}
% \begin{itemize}
%   \item The main features of GUI have been completed, it has all the sections desired
%   \begin{itemize}
%     \item Projects can be created to split up different aircraft
%       or revisions of checklists
%     \item Procedures can be created and tested
%     \item These procedures get tested in the flight simulator automatically
%       and gives the results of how the procedure has been doing in
%       real time
%   \end{itemize}
% \end{itemize}

All the desired sections of the GUI has been implemented, allowing for
the checklist tester to be used to meet the objectives of this project.

The GUI allows for projects to be created, allowing separation of aircraft and
revisions of QRHs. In each project, checklists can be created, and the steps
for the checklist can be defined and edited if needed. Once the steps in
the checklist are defined, the test for the checklist can be run
and the Checklist Tester will automatically run each step of the checklist
and show the progress through the checklist in real time and if each step
in the checklist is being completed correctly or failing.

\subsubsection{Setting up Tests}
% \begin{itemize}
%   \item Each test is set up by defining each action in the procedure,
%     on the Procedure screen
%   \item To be able to define each action is supposed to do, it uses
%     the Dataref variables in X-Plane, which is what stores the state
%     of the aircraft. Each switch has their own unique Dataref
%   \item In the checklist tester then, each action asks for a
%     Dataref and a desired goal value
%   \item Some Datarefs are read only, but there are other Datarefs
%     for the item desired, but are only \enquote{command}s, which
%     can only be called and not have its value changed; this can be
%     run by setting the desired goal value to be -988 (because XPC uses that value)
% \end{itemize}

Each test is set up by defining each step of the checklist from the
\textit{Procedure} screen in the Checklist Tester.
To be able to define what each step of the checklist is supposed to do,
it requires the dataref variable, which are the variables that
store the state of the aircraft in X-Plane, to be referenced for the specific
input in the aircraft for that step in the checklist. To identify dataref name required
for the specific input, there is an X-Plane plugin DataRefTool
which also allows to see the current state that the datatref is, and it is a read only
variable. Then, to set the desired goal of the step in the checklist, the input can be put
to the desired state in the flight simulator, and the value of the dataref can be taken
and be set in the Checklist Tester.

However, some aircraft in X-Plane have read-only dataref variables that can only
be modified by running a command, calling a specific dataref. So to be able to
test that step in the checklist, the desired state of the step can be set as
--988 (that value was chosen because XPC uses that value to not modify variables).
This will mean that the checklist tester will not attempt to change the variable
of the dataref.

\subsubsection{Running Tests}
% \begin{itemize}
%   \item Tests are run by connecting to the flight simulator, X-Plane
%   \item The tester goes through each action in the procedure one by one
%     and waits for the current action to complete before proceeding on to
%     the next one
%   \item The checklist tester is not advanced enough to be able to control
%     fly the aircraft; hence the tester would be able to engage autopilot
%     first, or control the aircraft themselves, where the checklist tester
%     would be acting like a first officer
% \end{itemize}

Running a test for a checklist requires an active instance of X-Plane
to be running with the plane loaded in, as the Checklist Tester
checks for an active simulator connection, otherwise it will not run.

Once the test has been started, the Checklist Tester goes through each action
in the checklist one by one and waits for the current step to complete before
proceeding to the next one.

The Checklist Tester is not advanced enough to control the flight controls
of the aircraft, meaning that the aircraft has to be flown manually,
have autopilot set manually, or add steps to control the autopilot in the Checklist Tester,
avoiding the need to set up the autopilot manually each time.  

\subsubsection{Storing Test Results}
% \begin{itemize}
%   \item There is a database storing the results of each of the tests
%   \item Each tests store
%   \begin{description}
%     \item[Time taken] for each of the actions in the procedure to complete
%     \item[Start state] for the state that the action in the procedure was at
%     \item[End state] for the state that the action in the procedure finished
%       the item at
%     \item[Overall test time] Stores the time taken from when the test started
%       to when the test ended
%   \end{description}
%   \item This gives feedback/statistics for the checklist designers
%     to find areas of improvement on the procedure, such as one action
%     in the procedure taking too long, may point out a potential flaw
%     to the designer and as a result aid finding potential alternative
%     options for that step in the procedure
% \end{itemize}

Whilst checklists are being tested in the Checklist Tester,
there are multiple aspects being tracked and stored on the database to be
used as results for the tests that run. The results are stored on the
\lstinline|Test| and \lstinline|ActionResult| table, which can be seen on the
entity relationship diagram in~\autoref{fig:db-erd}, with the respective values
that are stored.

The aspects that the database store are the time taken for the entire
checklist, by taking the time when the test started, and when the last step in the
checklist was completed. These are stored as a start and end time on the \lstinline|Test|
table, in Coordinated Universal Time (UTC) format.

Each step that is tested in the checklist gets tracked separately in
the \lstinline|ActionResult| table, where the start and end state of the dataref
is tracked, with the start end end time in UTC format.

This gives feedback/statistics for the checklist designers to find
areas of improvement on the procedure, such as one action in the
procedure taking too long, may point out a potential flaw to the designer,
as a result aiding finding potential other options for that step in the procedure.

\subsection{Submitting a Pull Request for X-Plane Connect}
% What I did:
% - gitignore
% - changing url for repo
% - testing everything still worked
\begin{itemize}
  \item Adding the Gradle build tools can be seen as being helpful
    for others, as it would allow for the XPC library to be added
    as a dependency, especially if the NASA Ames Research Center Diagnostics and Prognostics Group
    were to add it to the GitHub repository, it would mean that it would be easier for
    people to access Maven Packages for XPC
  \item Therefore, to help improve the experience for other people who would want
    to develop with the XPC Java library, it would be logical to submit a
    pull request
  \item But it did mean making sure that the contribution would be perfect and not contain problems % TODO Improve wording
\end{itemize}

\subsubsection{Testing}
\begin{itemize}
  \item The XPC Java library includes a JUnit 4 test, however, implementing this
    with Gradle proved useless, as it was not able to get the results from the
    tests, which would be bad for not being able to catch problems with new builds
  \item Therefore, the tests were updated to JUnit 5, where most of the changes were
    adding asserts for throws~\cite{junit:migrate}
    \footnote{The commit including the changes to the tests can be viewed here:
    \url{https://github.com/smyalygames/XPlaneConnect/commit/e7b8d1e811999b4f8d7230f60ba94368e14f1148}}
\end{itemize}

\subsubsection{GitHub}
\begin{itemize}
  \item Made sure to add generated build files to .gitignore
  \item Changed the URL of the repository in Gradle to NASA's repository so that
    the Maven package can be published correctly on the GitHub repository
  \item From the beginning anyways, made sure to have insightful commit messages
  \item Submitted the pull request stating the changes made\footnote{\url{https://github.com/nasa/XPlaneConnect/pull/313}}
\end{itemize}


\section{Reflection}
\subsection{Planning}
% Used Gantt chart to create a plan for what would be needed from this project

% \textbf{Pros:}
% \begin{itemize}
%   \item Was useful for the first part because it set expectations
%     of what was needed and how much time there was to complete them
%   \item Helped visualize the different components of the project
%   \item Helped in the beginning being accompanied by a Kanban in Leantime\footnote{\url{https://leantime.io/}}
% \end{itemize}

% \textbf{Cons:}
% \begin{itemize}
%   \item Was not detailed enough, and a design document would have been useful
%     to accompany the Gantt chart for each section
%   \item The lack of detail was not helpful when falling behind as having
%     attention deficit hyperactivity disorder (ADHD) 
%     added to the burden of feeling like each section was a massive project
%   \item Leantime's claim for being \enquote{built with ADHD [\ldots] in mind}
%     felt misleading as navigating through it felt worse than using the front page
%     of Stack Overflow\footnote{\url{https://stackoverflow.com/}}
%   \item Todoist\footnote{\url{https://todoist.com/}} was a good alternative though
% \end{itemize}

A Gantt chart was used to create a plan for what would be needed from this project
and when these parts of the project should be completed.

The Gantt chart was useful for the first part of the project because it set expectations
of what was required and how much time there was to complete them. It also helped
visualize the different components of the project. Implementing the Gantt chart into
Leantime\footnote{\url{https://leantime.io/}} was helpful at first as it was able to
be accompanied by a Kanban.

However, there were multiple downfalls of the Gantt chart. One of the problem was that
the design of the Gantt chart lacked detail for each of the components. A way this could
have been fixed was by making the Gantt chart more detailed, or create a design document
to accompany the Gantt chart. The lack of detail was later made worse as when falling behind
with attention deficit hyperactivity disorder (ADHD), it felt like a burden to progress
as each section felt like a massive project, when in reality it could have been split up
into subtasks.

Leantime's claim for being \enquote{built with ADHD [\ldots] in mind} felt misleading 
as navigating through it felt worse than using the front page of 
Stack Overflow\footnote{\url{https://stackoverflow.com/}} as it was very cluttered to access
what was desired, such as the Kanban requiring multiple pages to navigate through.
For the future, it would be helpful to find an alternative to Leantime to aid in
progress tracking. 

\subsection{Implementation}

\subsubsection{Checklist Tester}
% \begin{itemize}
%   \item Implementing the GUI was useful to split up the sections required
%     for the project and having a goal for what to be done
%   \item However, a bit too much time was spent on creating a GUI when it
%     could have been used for development
%   \item It was useful for motivational reasons to feel like something
%     materialistic has been produced rather than something theoretical
%   \item Was originally intended to be used to interact with custom
%     plugin for X-Plane as it would have been difficult otherwise
% \end{itemize}

Implementing the GUI was useful to split up the sections required for the project,
and having an informal requirement for each section of the project.
However, a bit too much time was spent on creating a GUI when it could have been
used for development or creating a design document which would have aided in
productivity.

However, implementing the GUI was useful to an extent as it provided motivation
by having something tangible rather than something theoretical or a command line interface.

% \subsubsection{Connecting to the Flight Simulator}
% \begin{itemize}
%   \item Would have been more useful to search a bit further if there was
%     another plugin available, as found Dataref Editor on the X-Plane docs,
%     so could have looked for a similar plugin for connecting to X-Plane
%   \item At first spent about a week developing a C++ X-Plane plugin from scratch,
%     requiring to figure out sockets
%   \item At the same time finding out XPC exists and having wasted that time
%   \item However, it did teach me more about understanding how sockets work and
%     more about C++ and setting up a project with CMake and adding packages with
%     vcpkg
% \end{itemize}

\section{Time Spent}
\begin{itemize}
  \item Time spent was recorded using Wakatime, other than time spent
    researching, which had to be recorded manually, using Leantime
  \item The time spent on GUI is also time spent on connecting other tools
    such as the VDMJ wrapper, XPC, and the database
\end{itemize}
\begin{figure}[!htp]
  \centering
  % TODO make it show how many hours were spent instead of percentages
  \begin{tikzpicture}
    \pie[sum=auto, text=legend]{%
      40/GUI,
      11.5/Database,
      2/Unit Tests,
      14.5/Configuring Connector,
      15.5/Formal Modelling,
      13/VDMJ Wrapper,
      6/Packaging XPC,
      32/Research%
    }
    % Used to make the 2 go to the outside of the pie chart 
    \pie[sum=134.5, hide number]{40/, 11.5/, 2/2}
    \pie[sum=134.5]{40/, 11.5/}
  \end{tikzpicture}
  \caption{Time spent on sections of project (in hours)}
\end{figure}

\end{document}