checklist-tester/pub/proposal/proposal.tex

\documentclass[a4paper]{article}

\usepackage{csquotes}

\usepackage{xcolor}

% Tables
\usepackage{xltabular}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{array}

% Gantt Chart
\usepackage{pgfgantt}
\usepackage{moresize}

% For referencing
\usepackage[citestyle=ieee]{biblatex}
\addbibresource{references.bib}

% This should be loaded last
\usepackage[colorlinks]{hyperref}
\usepackage{subfiles}

\author{Anthony Berg}
\title{Testing Quick Reference Handbooks in Simulators}

\hypersetup{
    pdfinfo = {
        Title = Testing Quick Reference Handbooks in Simulators Project Presentation,
        Author = Anthony Berg,
        Subject = Dissertation Project Presentation,
    }
}

\newcommand{\lfcomment}[1]{\textcolor{blue}{\textbf{LF}:~#1}}
%Uncomment below to remove comments from PDF yet keeping in the LaTeX :-)
\renewcommand{\lfcomment}[1]{\relax}

\begin{document}
% Title Page and contents
\begin{titlepage}
    \clearpage\maketitle 
    \thispagestyle{empty}
\end{titlepage}

% 4 PAGE LIMIT %
% Context
\section{Context}
\subsection{Introduction}
\subsubsection*{Context}
%\begin{itemize}
%    \item Designing Emergency Checklists is difficult
%    \item Procedures in checklists must be tested in simulators~\cite{nasa-design},
%      which usually means trained pilots test it, as the tests need
%      to work consistently~\cite{manifesto} (making sure it's not lengthy,
%      concise and gets critical procedures)
%      	\lfcomment{Testing for what? What's the baseline? I guess you will need some kind of evidence/argument that demonstrates that with and without the checklist results are markedly different. }
%    \item Checklists are usually carried out in high
%        workload environments, especially emergency ones
%\end{itemize}

Designing aviation checklists is difficult and requires time 
to test them in simulators and the real world.~\cite{nasa-design}
The simulators require trained pilots to test them to make 
sure that they work consistently~\cite{manifesto}, which tests 
that the procedures in the checklist are concise, achieves the goal
of the critical procedure, and will not take too long to complete.
These checklists are also carried out in high workload environments,
and this workload is elevated if an emergency were to occur.~\cite{caa-emergency}

\subsubsection*{Problem}
%\begin{itemize}
%  \item Testing procedures in checklists are often neglected~\cite{nasa-design}
%    \item There are some checklists that may not be fit
%        for certain scenarios - e.g. ditching (water landing)
%        checklist for US Airways Flight 1549 assumed at least one engine
%        was running~\cite{AWE1549}, but in this scenario, there were none
%         \lfcomment{What check lists and what scenarios?}
%    \item Some checklists may make pilots \enquote{stuck}
%      - not widely implemented, could be fixed with \enquote{opt out} points.
%      e.g. US Airways 1549, plane below 3000ft, could have skip to
%      later in the checklist to something like turn on APU, otherwise plane
%      will have limited control~\cite{AWE1549}.
%    	\lfcomment{Yes, or might make them ignore the checklist. What criteria does that?}
%    \item Checklists may take too long to carry out - Swissair 111
%    	\lfcomment{Yes. See Checklist manifesto test}
%\end{itemize}

Testing procedures in checklists is often neglected by designers.~\cite{nasa-design}
This is shown as there are certain checklists that are not fit for 
certain scenarios. An example of this is the checklist for ditching (water landing)
which would have been applicable to use on US Airways Flight 1549. This checklist 
assumed that at least one engine was running~\cite{AWE1549}, but this flight lost 
both of their engines, and if this checklist was used, it could have ended in 
an incident that could have resulted in people losing their lives.
If occurrences like happened more frequently, this could result in 
pilots losing their trust in checklists, which could result in pilots 
not using them, when they are designed to aid in situations where
they missing a critical step could be detrimental to the safety of 
everyone onboard the aircraft.~\cite{manifesto}

\subsubsection*{Rationale}
%\begin{itemize}
%    \item Test checklists in a simulated environment 
%      to find flaws in checklist for things like
%      \begin{itemize}
%        \item Can be done in an amount of time that will not endanger aircraft
%        \item Provides reproducible results
%        \item Procedures will not endanger aircraft or crew further (Crew referring to Checklist Manifesto with the cargo door blowout)
%      \end{itemize}
%      \lfcomment{Again, explain testing against what}
%    \item Results in being able to see where to improve checklists
%\end{itemize}

Therefore, to aid designers in testing checklists, this project 
will create a tester for checklists to find flaws in checklists 
by using simulators without the need of trained crew. This will 
test that the procedures in the checklist can be done in a reasonable
amount of time that will not endanger the aircraft and that the procedures 
will have reproducible results for the given goal of the checklist.
With this, the results can be used to show areas of improvement in 
the checklist.

\lfcomment{On CL book, Ch1 is about nature of where CL work best; Ch2 explains what a CL is and isn't; Ch3 you can ignore, it talks about checklist for unknown/unexpected scenarios (advanced CL) in building; Ch4 he discussed CL with chefs; Ch5 talks about CL failures and why; \textbf{Ch6 is about Boeing's CL ``factory''}; Ch7 and 8 are about applying the CL he came up with, here you get the examples of test/baseline criteria for CL; Ch9 he explains it in his own practice (this is daunting read)!.}
\subsection{Key Background Sources}
\begin{xltabular}{\linewidth}{p{8em} X}
  \toprule
  Resource & Info \\
  \midrule \endfirsthead
  
  \toprule
  \multicolumn{2}{l}{Key Background Sources (continued)} \\
  Resource & Info \\
  \midrule \endhead

  \midrule
  \multicolumn{2}{r}{Continued\dots} \\
  \bottomrule \endfoot


  \bottomrule \endlastfoot

  \multirow{2}{*}{\parbox{8em}{US Airways 1549 NTSB Investigation~\cite{AWE1549}}}
  & \textbf{Description:} An investigation carried out by the NTSB
      on an aircraft that suffered from a dual engine 
      failure from a bird strike forcing the pilots to land one
      the Hudson River. \\
  & \textbf{Reason:} The investigation found that the Quick Reference Handbook (QRH) was too
      lengthy and the pilots' used their experience to prioritize
      essential actions outside the QRH to keep the aircraft in control. \\

  \multirow{2}{*}{\parbox{8em}{Design Guidance for Emergency and Abnormal Checklists in Aviation~\cite{nasa-checklist}}}
  & \textbf{Description:} Provides the challenges researched
    requirements for designing aviation checklists.
    It also talks about the problems that occur in 
    checklist design process. This work was funded 
    by NASA as a part of Emergency and Abnormal Situations Study. \\
  & \textbf{Reason:} This will guide looking out for 
    certain aspects to test for in checklists, such as if
    certain actions require waiting, or if it could be
    completed in a different order. \\

  \multirow{2}{*}{\parbox{8em}{Designing Flightdeck Procedures~\cite{nasa-design}}}
  & \textbf{Description:} Guidance provided by NASA on the process of 
    developing checklists, which includes steps to focus on 
    and how to make a well designed checklist.\\
  & \textbf{Reason:} This report includes steps on testing checklists
    which is the focus of this project and will provide guidance 
    on how the tests should be carried out, such as carrying 
    out feasibility testing. \\

  \multirow{2}{*}{\parbox{8em}{The Checklist Manifesto~\cite{manifesto}}}
  & \textbf{Description:} Insight into the steps of implementing
  a checklist in medicine, by a public health researcher, 
  whilst learning about how checklists are designed and 
  used in industries such as aviation and construction. \\
  & \textbf{Reason:} Checklist designer from Boeing is interviewed
  where they go through the vital design choices to make them effective
  and how they gain pilots' trust to use checklists. \\

  \multirow{2}{*}{\parbox{8em}{Intraindividual Variability in Basic Reaction Time Predicts Middle-Aged and Older Pilots’ Flight Simulator Performance~\cite{pilot-reaction}}}
  & \textbf{Description:} Tested 236 pilots aged between 40-69 years old, 
  to measure their reaction times and how they are affected 
  by age. The tests were conducted by Stanford
  University and MIRECC, and results have been peer reviewed. \\
  & \textbf{Reason:} Gives insightful statistics into pilots 
  reaction times and how they will affect the performance 
  when carrying out procedures in checklists. \\
\end{xltabular}

% Aim (or hypothesis) and Objectives
\section{Aims and Objectives}
\subsection*{Aims}
%\begin{enumerate}
%    \item Test Checklists for flaws that can compromise the
%      aircraft or that the checklist will not take too long
%      and overload pilot further
%    \item Test multiple conditions for that checklist
%    \item Find problems in the checklist
%    \item Find how reproducible the checklist is
%\end{enumerate}
%
%\lfcomment{See above about "testing" what?}

To test checklists for flaws that could compromise the aircraft 
and to make sure that the checklist can be completed in a reasonable 
amount of time for multiple different conditions (such as the weather or pilot's reaction times)
that could affect the amount of time the pilots will have to complete the checklist.
As a result, this will also test the reproducibility of the checklist's goal.

\subsection*{Objectives}
\begin{enumerate}
  \item Research current checklists that may be problematic that 
    could be tested
  \item Research how pilots act within certain situations for 
    the checklists to be tested
  \item Implement a formal model that runs through checklists,
    with the research gathered, to produce an accurate test
    \begin{enumerate}
      \item Understand the relative states of the aircraft that 
        need to be captured
      \item Ensure the consistency of the checklist procedures 
        are tested
    \end{enumerate}
  \item Implement a manager for the checklist testing logic to 
    provide information about how the test results
  \item Connect the formal model to a flight simulator allowing
    for a more accurate representation of the aircraft systems
    and flight conditions
\end{enumerate}

% Planning
\section{Planning}
\subsection{Diagrammatic Work Plan}

% Make title format for gantt chart into week commencing
\ganttset{%
  calendar week text={%
    \startday/\startmonth%
  }%
}

\begin{ganttchart}[
  hgrid,
  vgrid={*{6}{draw=none}, dotted},
  x unit=1.3mm,
  y unit chart=5mm,
  time slot format=isodate,
  % bar formatting
  bar label node/.append style={align=right},
  % milestone formatting
  milestone label node/.append style={align=right},
  % group styling
  group label node/.append style={align=right},
  % vrule styling
  vrule/.append style={red},
  vrule offset=.2,
  vrule label node/.append style={align=center},
  % Font sizing
  title label font=\ssmall,
  bar label font=\ssmall,
  group label font=\bfseries\ssmall,
  milestone label font=\ssmall,
  today label font=\ssmall,
  vrule label font=\ssmall,
]{2024-02-26}{2024-05-12}
  \gantttitlecalendar{year, week} \\
 
  % VDM
  \ganttgroup{VDM Modelling}{2024-02-26}{2024-04-28} \ganttnewline
  \ganttbar{%
    Create Checklist\ganttalignnewline
    Functionality%
  }{2024-02-26}{2024-03-03} \ganttnewline
  \ganttlinkedbar{%
    Capture\ganttalignnewline
    Aircraft State%
  }{2024-03-04}{2024-03-10} \ganttnewline
  \ganttlinkedmilestone{Useable Model}{2024-03-10} \ganttnewline
  \ganttbar{%
    Simulate\ganttalignnewline
    Pilot Actions%
  }{2024-04-15}{2024-04-21} \ganttnewline
  \ganttlinkedbar{%
    Flight\ganttalignnewline
    Conditions%
  }{2024-04-22}{2024-04-28} \ganttnewline
  %\ganttlinkedmilestone{Finish Model}{2024-03-24} \ganttnewline

  % Frontend
  \ganttgroup{Manager}{2024-03-11}{2024-03-24} \ganttnewline
  \ganttbar{%
    Create\ganttalignnewline
    Frontend%
  }{2024-03-11}{2024-03-17}\ganttnewline
  \ganttlinkedbar{Link VDM}{2024-03-18}{2024-03-24} \ganttnewline
  \ganttlinkedmilestone{%
    Complete\ganttalignnewline
    Interactive VDM%
  }{2024-03-24} \ganttnewline
  \ganttlink{elem3}{elem7}
  
  % Simulator Connector
  \ganttgroup{%
    Simulator\ganttalignnewline
    Connector%
  }{2024-03-25}{2024-04-14} \ganttnewline 
  \ganttbar{%
    Learn\ganttalignnewline
    Simulator SDK%
  }{2024-03-25}{2024-03-31} \ganttnewline
  \ganttlinkedbar{Connect Logic}{2024-04-01}{2024-04-14} \ganttnewline
  \ganttlinkedmilestone{Functional Tester}{2024-04-14} \ganttnewline
  \ganttlink{elem9}{elem11}
  \ganttlink{elem13}{elem4}

  % Coursework
  \ganttgroup{Coursework}{2024-03-11}{2024-05-01} \ganttnewline

  \ganttbar{Create Video}{2024-03-11}{2024-03-20} \ganttnewline

  \ganttbar{Create Poster}{2024-04-15}{2024-04-21} \ganttnewline

  \ganttbar{Documentation}{2024-04-01}{2024-05-01}

  % Deadlines
  \ganttvrule[
    vrule label node/.append style={anchor=north east}
  ]{%
    Presentation\ganttalignnewline 
    Deadline %
  }{2024-03-22}
  \ganttvrule[
    vrule label node/.append style={anchor=north west}
  ]{%
    Poster\ganttalignnewline %
    Deadline%
  }{2024-04-24}
  \ganttvrule{%
    Dissertation\ganttalignnewline %
    Deadline%
  }{2024-05-08}
  % Holidays
  \ganttvrule[
    vrule/.append style={thin, blue},
    vrule label node/.append style={anchor=north west}
  ]{%
    Easter\ganttalignnewline
    Holidays%
  }{2024-03-25}
  \ganttvrule[
    vrule/.append style={thin, blue},
    vrule label node/.append style={anchor=north east}
  ]{%
    End of\ganttalignnewline
    Holidays%
  }{2024-04-22}
\end{ganttchart}

\subsection{Brief Explanation}
%\begin{itemize}
%  \item Pretty much put how long each part of objectives should take.
%  \item The deadlines for the Presentation, Poster, Dissertation
%    will be done at the same time as the programming
%  \item The work during the Easter Holidays is there with  
%    the expectation of taking a bit of a break, but to not
%    lose momentum once the holidays are over.
%  \item All the items on the gantt chart are more so the worst
%    case scenario for how long each item will take
%  \item The last part of VDM modelling is just in case I run out
%    of time, and they aren't the most important
%\end{itemize}

The Diagrammatic Work Plan provides guidance for how long each objective 
should take whilst making them more specific tasks. These tasks in the 
gantt chart are have a pessimistic estimation for how long they 
should take to complete, as it should give a buffer for falling 
behind due to unforeseen circumstances, such as getting ill.

The Easter Holidays is an important time to take a break to prevent
burnout, however, the decision to continue working on the dissertation 
during the holidays is to prevent the problem of getting up to speed again 
once the term starts, hence the tasks are simple which should allow for 
taking more breaks from work during the holidays.

The last tasks in the VDM modelling is not essential to provide a proof of 
concept of the checklist tester, as they are features that would improve the 
test quality. This would allow for redundancy if time were to become a problem 
as these tasks could be taken out, in favour of writing the dissertation,
and could be implemented at another time.

Coursework tasks will be done in parallel to programming to
be as efficient as possible. This will also allow for features that 
complement the coursework to be implemented. As a result, the 
prototype will enhance the poster and presentation.

\subsection{Risks}
%\begin{itemize}
%\item Time management - that's why the last part of modelling is
%  done after the simulator connector as it's not essential,
%  can be done manually
%\item Simulator not being good enough?
%\item Simulator could be too complex to be able to link with model.
%\end{itemize}

The main risk is time management, as falling too far behind could 
be detrimental. However, to prevent falling behind, certain days 
will be planned to include breaks, which is included in the plan,
to prevent a burnout. Even if a burnout were to occur, there should be 
enough spare time built up from the pessimistic deadlines to 
prevent falling behind schedule. 

There is also a small risk of the current home flight simulators not 
being suitable to test the formal model, as their SDKs may not have 
the tools necessary or the formal model may be too complex to connect to 
the simulator. If this were to be the case, there could be other 
another way to test, such as manually flying the plane in the simulator, 
but recording data in the model manually, to provide feedback on the 
checklist tests.

% END OF 4 PAGE LIMIT %
\clearpage

% Ethics
\section{Ethics}
\subsection{Ethics Checklist}
% Copied from "../../doc/proposal/Specification.pdf"
% Should be left unchanged, unless dealing with formatting
My project:
\begin{enumerate}
    \item Will \textit{not} involve working with \textbf{animals} or
        users/staff/premises of the \textbf{NHS}
    \item Will be carried out \textbf{within the UK or European
        Economic Area}
    \item Will \textit{not} have any impact on the \textbf{environment}
    \item Will \textit{not} work with populations who do \textit{not}
        have \textbf{capacity to consent}
    \item Will \textit{not} involve work with \textbf{human tissues}
    \item Will \textit{not} involve work with \textbf{vulnerable groups}
        (Children/Learning disabled/Mental health issues, etc.)
    \item Will \textit{not} involve any \textit{potentially}
        \textbf{sensitive topics} (Examples include but are not
        exclusive to body image; relationships; protected
        characteristics; sexual behaviours; substance use;
        political views; distressing images, etc.)
    \item Will \textit{not} involve the collection of any identifiable
        personal data
\end{enumerate}

\subsection{Ethical Considerations}
This project will involve referencing previous aviation
accidents which had resulted in deaths. However, it should not be a 
problem by being respectful towards everyone involved in those accidents.

This project will also not involve the use of any users,
so no data collection considerations will need to be taken
into account for.
\clearpage


% References
\nocite{*}
\printbibliography[heading=bibnumbered]

\end{document}