\documentclass[journal]{IEEEtran}
\usepackage{wrapfig}
\pdfoutput=1


\usepackage{longtable}
\usepackage[T1]{fontenc}
\usepackage{tikz}
\usepackage{hyperref}

\usetikzlibrary{arrows,calc,positioning}

\tikzstyle{intt}=[draw,text centered,minimum size=6em,text width=5.25cm,text height=0.34cm]
\tikzstyle{intl}=[draw,text centered,minimum size=2em,text width=2.75cm,text height=0.34cm]
\tikzstyle{int}=[draw,minimum size=2.5em,text centered,text width=3.5cm]
\tikzstyle{intg}=[draw,minimum size=3em,text centered,text width=6.cm]
\tikzstyle{sum}=[draw,shape=circle,inner sep=2pt,text centered,node distance=3.5cm]
\tikzstyle{summ}=[drawshape=circle,inner sep=4pt,text centered,node distance=3.cm]
\tikzstyle{v1}=[draw,minimum size=1em, minimum height=3em,text centered,text width=2.2cm]
\begin{document}

\title{The Transformational Effects of Applying Systems Engineering in Laboratory Scientific Research}

\author{Kieran~Niels~Bjergstrom$^{1,2}$, William~G.~B.~Huish$^{1,2}$, Michael~J.~de~C.~Henshaw$^{1}$ Member IEEE, Vincent~M.~Dwyer$^{1}$, and Mark~J.~Everitt $^{2}$*
\thanks{Quantum Systems Engineering Research Group, The Wolfson School of Mechanical, Electrical and Manufacturing Engineering$^1$ and The Department of Physics$^2$, Loughborough University, Loughborough, LE11 3TU, UK}
\thanks{* \href{mailto:m.j.everitt@physics.org}{m.j.everitt@physics.org}; Tel.: +44 (0)1509 223325}
}


\maketitle

\begin{abstract}
Through case study analysis of physicists working in quantum systems, we establish that knowledge of Systems Engineering (SE) will benefit scientists by ensuring that experimental apparatus is robust and fully meets the experiment requirements. We suggest a paradigm shift in the training of early career research scientists to ensure they have a strong appreciation of the systems approach and experience of applying SE techniques. Two case studies were used: the first was a detailed analysis of increasing levels of SE in the development of a 3D-printer for fabricating superconducting nano-circuitry. Applying SE techniques improved record keeping, reduced the risk of failure modes and took better account of future development through which scientific discovery may be exploited in practical devices. The second case study focused on a group of 30 quantum physics PhD students undertaking a short-course in basic SE, and used their opinions to check and add confidence to the findings of the first case study. A variety of tools were used in both studies and, in both cases, the Functional Failure Means Effects Analysis (FFMEA) technique was considered to be most useful. Recommendations are made for the future training of early career scientists to include some light-weight Systems Engineering. 
\end{abstract}

\begin{IEEEkeywords}
systems engineering, science, technology, Education, Functional Analysis
\end{IEEEkeywords}


\section{Introduction}

\IEEEPARstart{S}{ystems} Engineering is applied, as standard practice, to the development of many complex products. It serves (among other things) to analyse and improve design feasibility, track and record project progress, and develop objective criteria against which system and sub-system functionality can be verified and validated. Whilst this approach is typical in industry, it is unusual in the context of low TRL (Technology Readiness Level) \cite{mankins1995technology} laboratory research - an area in which novel, and complex, demonstrator devices are designed and developed. It is necessary that such devices reliably, and demonstrably, satisfy their operational requirements and, once proven, can be translated into a manufacturable product. Furthermore, in order to best capitalise on the novelty of an emerging technology, it is imperative that the development process occurs with minimal time and resource waste.

It is our observation that in low TRL science there are three predominant causes for resource wastage:
\begin{enumerate}
\item A lack of foresight in project design, resulting in an avoidable and excessive need to re-design and rework, either at a system or component level.
\item A need to significantly re-design or re-engineer the technology after demonstrator stage, either due a failure to take into account manufacturing constraints in the design, or due to an insufficient record of knowledge and decisions that inhibits the transfer of knowledge to those working in the higher stages of the R\&D chain.
\item Project drift, resulting in a deliverable that is not validated against its concept of operations (CONOPS).
\end{enumerate}
One may observe that these are fundamentally design and development problems that systems engineering seeks to alleviate. As a consequence, we considered it important that a study was made regarding the costs and benefits of applying a systems engineering approach to the development of novel technologies within the context of low TRL laboratory research, i.e. TRL 1 - Basic principles \cite{mankins1995technology}.

A case study research method has been used to understand the application of a Systems Engineering approach by scientists to applied research; we make the distinction between a systems (thinking) approach, that could be applicable to both applied and theoretical scientific research, and systems engineering, which incorporates both systems thinking and the techniques generally associated with the technical processes of ISO 15288 \cite{iso2015iec} (the Systems and Software Engineering Life Cycle Processes Standard).

The motivation for this investigation is three-fold: firstly, we postulate that more rapid exploitation of scientific discovery for societal or commercial benefit may be achieved by connecting the science and engineering communities more strongly.  Systems Engineering is the integrating discipline that brings together different engineering disciplines and business concerns into a complex product, it could, therefore, have a role in connecting scientists, so that the integration challenges of introducing new technologies within a wider system are better understood and planned.  By introducing \textit{design for...} methods at an earlier stage, we suggest that the researcher is more likely to achieve an effective, credible design, that avoids the need for extensive re-engineering as it moves out of the laboratory space.   Secondly, the systems engineering approach explicitly records the decision steps of design, providing traceability that may be required for a variety of reasons in the future, but not least system upgrade or recovery.  The employment of systems engineering in the laboratory may provide a deeper understanding in this respect, especially with emerging applications that exploit manipulation of quantum states \cite{Everitt2016}.  Thirdly, we wish to understand whether a systems engineering approach to experimentation will lead to more efficient design of apparatus and of the experimental campaign; with less rework, whilst at the same time encouraging innovation and discovery.  This last has a more general implication for the scientific approach of experimentalists.

\section{Methodology - Case Study Method} \label{sec-methodology}
The technical community is mostly familiar with the use of case studies to illustrate (or educate) the reader in practice, by providing examples of good or bad practice associated with technical endeavours (e.g. the NASA database of case studies, in which they are described as "... a tool for creating an opportunity for conversation." \cite{Nasa2009}).  However, the social sciences community regard case study as a method of enquiry, although there is a wide range of case study type according to the number of cases investigated, the level of detail, the size of case(s), the attention paid to context, and whether reporting is purely descriptive or includes evaluation \cite{gomm2000case}.  Case studies are a form of empirical inquiry which investigates phenomena within their real-life context and when the boundary between the phenomena and context are not clearly defined \cite{yin2013case}.  
%check how to include page numbers in book citations
The case study approach is regarded as an appropriate research strategy when one wishes to ask "how" and "why" questions but either cannot, or do not wish to, exercise control over the situation in question (unlike an experiment in which there are controllable independent variables) \cite{gray2009research}, and so a case study approach was determined to be a suitable form of inquiry for this research.  As noted above, case studies can be undertaken in a variety of ways and we here define the approach that has been taken in this research.

A case study can be used inductively or deductively \cite{gray2009research}. The case studies reported herein have been used deductively, in the sense that they are predicated on the assumption that \textit{applied scientific research will benefit from the application of systems engineering techniques.}  The investigation has, necessarily, been evaluative because it has sought to determine both positive and negative aspects of the application of the techniques. There are a variety of methods for collecting data in case studies; observation has been chosen as the method for this research and this has included both direct observation (by staff familiar with the situation but standing outside the activity) and participant observation in which the scientist (who was the participant in the activity) records their behaviour and impressions in an ethnographic way.  This form of data collection is very good for ensuring that the context is properly understood and it provides insight into the motives and behaviours of the subject.  However, the researcher must be cautious of biases that can affect interpretation of the data and also the Hawthorne effect \cite{monahan2010benefits}, whereby individuals modify their behaviour because they know they are being observed.  A fundamental aspect of case study design is to define the unit of analysis \cite{gray2009research}; this is similar, in systems engineering language, to defining the system of interest, but it is important to understand that it not only concerns the physical domain of study, but also the lens through which it is observed (i.e. what are the features that the researcher seeks to observe?).  This study includes four cases, characterised by the following units of analysis (Table \ref{table:cases}):


\begin {table} [h!]
\centering
\begin {tabular} {|l|p{4cm}|p{2.4cm}|}
\hline
\textbf{Case ID} & \textbf{Description} & \textbf{Physical Defn} \\
\hline
Case A1: & Development of 3D-printer project without Systems Engineering & 1 research student,  1 project\\
\hline
Case A2: & Systems methods applied to intractable problems of development of 3D-printer project& 1 research student, 1 project\\
\hline
Case A3: & Formal application of systems methods to whole project & 1 research student, 1 project\\
\hline
Case B: & Quantum Systems Engineering summer school projects & 30 research students, 6 projects\\
\hline


\end {tabular}

\vspace{2pt}
\caption{\label{table:cases} Case Studies:
The lens through which all cases are viewed comprises external observation + participant observation.
}
\end{table}

Case A(1,2,3) is a project to develop a 3D-printer to print superconducting nano-scale circuitry; the three case studies are successive in time and allow consideration of changes due to the introduction of systems engineering over a period of 15 months.  
The development of the research student and the development of his appreciation of systems engineering from a standing start also form part of the analysis.  These three cases, therefore, provide a form of comparison to highlight the impact of systems engineering.  Case B concerned observation of a cohort of 30 PhD students, all researching various aspects of quantum technology (and sponsored by the Defence Science and Technology Laboratory, Dstl) who spent five days learning about systems engineering and applied a handful of techniques to group design projects relevant to quantum systems.  Two students had previous experience of systems engineering and these acted as embedded researchers. The data collection is based on the feedback from the cohort at the end of the course, the observations of the embedded researchers, and the observations of teachers and mentors on the course.  This case study provides a level of triangulation with the more detailed investigations of Case A.  The feedback for Case B was obtained using a survey; this method of data collection can also be appropriate for addressing the "how" question, with the additional opportunity to ask "how much?".  The survey was a typical course feedback survey used for quality purposes.  Although numerical data was obtained concerning the value of the course, it is recognised that the sample size (30) is too small to be statistically valid and that it was not constructed according to the recognised principles of analytical surveys \cite{gray2009research}.  This case study has, therefore, discounted the numerical data collected in the survey, but drawn on the free text comments from participants to gain insight of the appreciation and understanding of systems engineering by the quantum science PhD students.  For the record, all students indicated that the training had been valuable and that they had acquired new knowledge.

In the following sections, each case is described in terms of: situation (context), observations, and evaluation.  By looking across all the cases, some conclusions are drawn at the end.

\section {Case Studies} \label{sec-case-study}
\subsection* {Case Study A: 3D-printer}

This case study considers the potential value added to a laboratory based, applied research project through the application of systems engineering methods and tools. It considers the progress made by a research student in a project to design and construct a 3D-printer capable of printing superconducting nano-circuitry. We evaluate how the project progressed under three distinct circumstances: 
\begin {itemize}
\item{without the application of any systems engineering methods (A1), as the researcher would commonly approach his tasks; }
\item{with the application of a limited set of systems methods, applied \textit{ad hoc} to intractable design problems (A2); and }
\item{with a more complete systems approach, from systems requirements to validation (A3). }
\end {itemize}
In cases A2 and A3, the researcher was guided by expert systems engineers, but ultimately made his own decisions about which methods to apply and how.  The student was supported by supervisors with expertise in quantum physics, 3D-printing, systems engineering, and reliability engineering.  Supervisor contact was frequent (at least once per week), so that the student was able to check his ideas and understanding with ease.  However, the experimental development was led by the student with academic support; this was deliberately not directive, favouring instead discovery of systems engineering and self-development on the part of the student.


Whilst the ability to pattern superconductors into useful circuitry at the nano-scale is not novel \cite{Chesca2015}, doing so via an additive method has not yet been achieved. Additive manufacturing has a distinct advantage over standard lithographic techniques, in that the patterning can be changed comparatively easily and quickly. Instead of having to make a new mask and install it into the deposition system, all that needs to be changed is the patterning program that the system receives. This makes additive techniques more appropriate for prototyping circuitry, where the designs may be subject to frequent changes and updates. The obvious disadvantage of the additive process is its lack of manufacturing speed when compared with lithographic techniques, making it unsuitable for mass production except as a large scale parallel print process \cite{Engstrom2014}.

\par It is expected that an additive manufacturing method will allow for easier fabrication of three-dimensional circuitry designs. This could facilitate and enable a number of quantum technologies, such as directional Superconducting Single Photon Detectors (SSPD), Superconducting QUantum Interference Device (SQUID) stacked arrays, and other devices that it is not possible to create in planar geometries. Such capabilities are likely to have significant demand in the design and prototyping of superconducting devices. The end goal of this project is the design and demonstration of a desktop 3D-printer unit, capable of printing superconducting nano-circuity, and priced to be significantly more affordable than alternative fabrication units. 

\subsubsection*{\label{caseA1descrption} Case A1: Development of 3D-printer project without Systems Engineering}
The initial design and development of the 3D-printer project was completed without systems engineering methods or any knowledge of them by the research student.  This phase of the work lasted nine months. Early project progress was far from smooth, with a significant proportion of components developed during this phase being mutually incompatible, over-engineered, or not feasible for use. On reflection, it seems likely that had we applied some systems engineering methods from the start of the project most of this would not have happened. One cannot, in hindsight, quantify the proportion of mistakes or adjustments that were avoidable through a more formalised design process. The research student in question subsequently developed knowledge and practical experience of Systems Engineering and so could, reflectively, identify the more obvious design issues that wasted significant resources, and would have been addressed through straight-forward application of systems engineering techniques. One such case was the design and fabrication of the substrate holder, which is reported in narrative form below.
\par  \textit{The substrate holder was required to hold a 10mm $\times$ 10mm substrate in place on a 3-axis stage, such that material could be deposited onto it (see Fig. \ref{Springmech}).}
%include schematic figure of design [Fig 1 is not the substrate holder, it is the printhead housing - W]
\textit{Critically, this required that there was sufficient stability such that patterns could be reproduced on a series of substrates with a high degree of similarity. In this case, the core requirements of the component had not been identified, and its functional constraints had not been defined.  As a result, it was over-engineered such that it could actively hold larger substrates, as well as the 10mm$\times$10mm test substrates that were required. The substrate holder was also a permanent fixture in the printer, which became a source of difficulties when taking the printer apart during development.}

\begin{figure}[!tb]
	%\vspace{-33pt}
 \centering   \includegraphics[width=0.7\linewidth]{Spring_mechanism.png}
    \caption{Schematic of the original substrate holder design with the spring mechanism clamp to hold the substrate in place, from a top down view. This design had to be discarded as both the spring loaded curved clamp and right-angled stop were thicker than the substrate, making the risk of crashing and breaking the pipette too high.}
    \label{Springmech}
\end{figure}

\textit{The substrate holder had a spring mechanism to secure the substrate. This was beyond the required scope, which meant that time was spent adding unneeded capabilities that, more importantly, introduced a number of critical failure modes. In fact, the original design never worked and the component had to be entirely re-engineered. This was because the substrate holder was originally designed to manage system parameters that were beyond its natural control; a requirements specification, and functional analysis, would have shown that the substrate holder only needed to 'hold the substrate in place for the duration of printing', as all other parameters were entirely managed by other components.}
\par\textit{Had we started  by identifying the simple and minimal requirements for the substrate holder, it does not seem unreasonable to think that the initial ideas for the design would have matched this scope. Thus, instead of trying to design complex spring mechanisms to hold the substrate in place from all sides (Fig. \ref{Springmech}),}
%ref figure
%where is the spring mechanism shown on the figure?
\textit{ which resulted in the holding mechanisms being taller than the substrate, and vastly increasing the risk of the print head crashing into them, the eventual solution - that of attaching it to the surface of the staging with a carbon tab - would probably have been achieved much sooner.  Proper requirements capture, coupled with techniques such as Functional Failure Modes and Effects Analysis (FFMEA) \cite{FFMEABHW}} %[Burge S. Functional Failure Modes and Effects Analysis (FFMEA) Alias System FMEA. In: The Systems Engineering Tool Box. Burge Hughes Walsh; 2009. p. 1–15.]}
% Available from: http://www.burgehugheswalsh.co.uk/Uploaded/1/Documents/FFMEA-Tool-v1-2.pdf
\textit{ and Functional Means Analysis (FMA) \cite{FMABHW}} %[Burge S. Function Means Analysis (FMA) Alias Morphological Analysis. In: The Systems Engineering Tool Box. 2006. p. 1–13.]}
% Available from: https://www.burgehugheswalsh.co.uk/Uploaded/1/Documents/FMA-Tool-v1-1.pdf
\textit{would almost certainly have avoided the design problems described above.  Whilst the original staging did not have to be entirely re-fabricated, it did need to be adapted and engineered to meet the final design requirements, resulting in time and material costs.}

Pursuit of \textit{interesting diversions} is an essential part of academic research and so following curiosity-driven research directions, which may eventually turn out to be false trails, should be encouraged.  Furthermore, these diversions help to develop the research student's broader domain knowledge in order to acquire expertise and context for their research subject and longer term science career.  However, the problem described above, and a number of similar problems during this phase of the research, concern design of the experiment in order to answer the scientific research questions.  The student's lack of awareness of project scope meant that whilst some interesting diversions were scientifically and developmentally valuable, others were clearly practically wasteful and resulted in unnecessary delays and material wastage.  

Methodologically, the above observations are obviously subjective and retrospection inevitably introduces bias.  Furthermore, it is risky to generalise from these observations because another research student may have spotted the problems without the benefit of systems engineering techniques.  But, the reflective comments of the research student indicate that he, at least, would have understood the project scope more fully, had he followed a systems engineering approach and we would argue that this illustrates that for some researchers, the adoption of systems engineering techniques will reduce the risk of unnecessary wastage during the design and development of experimental apparatus.



This experience has highlighted two issues in the research process associated with record-keeping and decision making, both of which affected verification.  "System Verification is a set of actions used to check the correctness of any element, such as a system element, a system, a document, a service, a task, a requirement, etc." \cite{SEBOKSysVer}.
Firstly, the only formal record during this phase was a regular lab book; there were no formalised and recorded systems requirements or concepts of operation. Consequently, the researcher had nothing against which to verify his design decisions; the most he could hope to do was verify component-level functionality. Secondly, as design decisions were not formally tracked there was no accessible and shared record of knowledge and decisions; this hindered the ability of the wider, supervisory group to assess the project's progress, and understand whether it was meeting its original goals. This was also a failure in verification. Had a systems process been adopted from the outset, the means would have existed to periodically and objectively verify development against the system requirements, and against the CONOPS.  It must be mentioned that reconciling project direction with project goals is not a one-directional process. As the development continues a much greater expertise in the area is developed, and it is sometimes only with this knowledge that informed decisions can be made. Some of these decisions may justifiably affect system requirements and concepts of operation, as is normal in the feed-back, feed-forward, nature of systems engineering. However, what is vital is that everybody involved in the project has the same idea of requirements and operation. Keeping a good record of knowledge does not only add value when translating up through to higher TRLs, but also when communicating across the TRL at which the team is working. In research science, where each individual is a specialist in their own area, the capability to horizontally translate knowledge is very valuable indeed.


\subsubsection*{Case A2: Systems methods applied to intractable problems of development of 3D-printer project}
This case study followed Case A1 immediately and lasted for four months. 
% fill in number of months
Several intractable component design issues had arisen during the first nine months upon which substantial resource (time and materials for prototyping) had already been expended.  A kind of \textit{trial and error} approach to component design had resulted in a series of component-level failures, some of which remained unresolved.  This case study is characterised by the introduction of a limited set of systems engineering techniques (rather than as a full suite of life cycle processes).  Specifically a functional requirements specification was created (i.e. the functional analysis and definition part of the system specification \cite{SEA5, FFMEABHW}). 

\textit{The print head housing is a critical component to the control of print quality parameters including consistency when printing, control over feature size, and reproducibility. It is also one of the few device components for this system that had to take the user behaviour into account. The user has to be able to load, remove, replace, and calibrate the print head. A large number of designs were trialled in the first nine months (Case A1), but all lacked fail safes, which led to print heads breaking. It was noted that none of the designs had taken into account a sufficiently broad set of failure modes; they had been designed to satisfy the requirements pertaining to print parameter control, but had not considered failure modes associated with integration into the wider system or user interaction. The designs had, therefore, lacked sufficient robustness.}

\textit{Figure \ref{Printhead} shows the finalised design of the print head housing. FFMEA was applied to this case (see Table \ref{FFMEA Printhead}) and the failure modes from previous designs were prevented by making the hole for the pipette (the cylindrical hole running throughout the height of the housing) smaller than the lower connector of the pipette. This prevented any possibility of the pipette falling through and breaking, as had been an issue previously. Secondly as the electrical contact (provided by a bolt through the threaded hole) was no longer responsible for holding the pipette in place, the risk of breaking the pipette by over tightening the screw was reduced dramatically. By having the cut out in the print head housing where the bolt comes into contact with the pipette, this risk was reduced even further. The height of the housing was also increased to reduce the amount of the pipette that was exposed outside the housing. The height increase was not extended over the whole area of the print head so the bolts for affixing the housing to the rest of the system didn't have to be changed.}
\begin{table*}[!t]
\resizebox{0.8\textwidth}{!}{\begin{minipage}{\textwidth}
\begin{@twocolumnfalse}
\begin{longtable}{|p{1.4cm}|p{1.8cm}|p{1.8cm}|p{1.2cm}|p{2.2cm}|p{2cm}|p{1.7cm}|p{1.6cm}|p{0.4cm}|p{4.5cm}|}
\caption{FFMEA of Print head housing} \label{FFMEA Printhead} \\

\hline \multicolumn{1}{|c|}{\textbf{Function}} & \multicolumn{1}{c|}{\textbf{Failure Mode}} & \multicolumn{1}{c|}{\textbf{Effects}} & \multicolumn{1}{c|}{\textbf{Severity}} & \multicolumn{1}{c|}{\textbf{Causes}} & \multicolumn{1}{c|}{\textbf{Probability}} & \multicolumn{2}{c|}{\textbf{Detection}} & \multicolumn{1}{c|}{\textbf{RPN}} & \multicolumn{1}{c|}{\textbf{Design Alterations}}\\
\cline{7-8}
& & & \textbf{of Failure} & & \textbf{of Occurrence} & \textbf{Method} & \textbf{Probability} & &\\
& & & & & & & \textbf{of Detection} & &\\ \hline
\endfirsthead
\hline
\endlastfoot
Hold pipette still & Bolt grip on pipette fails & Pipette crashes into substrate & 9 & Insufficient friction between bolt and pipette & 1 & Visible to user & 1 & 9 & Make hole smaller than the luer connector of the pipette, making falling through impossible.\\ \cline{2-10}
& Pipette is not held steady & Distorted printing pattern & 7 & Pipette wobbling during printing & 2 & Visible on camera & 2 & 28 & Pin the pipette in place using the electrical contact.\\\hline
Provide high voltage & High voltage circuit broken & No jetting will occur & 5 & Bolt-pipette contact broken & 2 & Visible to user & 1 & 10 & Cut out made in print housing so bolt-pipette contact is visible.\\\cline{5-10}
contact to pipette & & & & Wire-bolt contact broken & 3 & Visible to user & 1 & 15 & Nut installed on bolt to pinch wire in place.\\ \hline
\end{longtable}
\end{@twocolumnfalse}
\end{minipage}}
\end{table*}
% there should be a schematic, or perhaps reference to the previous figure, and a short description of what the print head does and how it all fits together.
% how sensitive is this component (commercially) - could we include the functional specification and FFMEA in an appendix?

\begin{table*}[!t]
\resizebox{0.8\textwidth}{!}{\begin{minipage}{\textwidth}
\begin{@twocolumnfalse}
\begin{longtable}{|p{1.3cm}|p{2cm}|p{2.4cm}|p{0.9cm}|p{2.3cm}|p{1.6cm}|p{3.2cm}|p{1.5cm}|p{0.4cm}|p{2.9cm}|}
\caption{FFMEA of Heater module} \label{FFMEA Heater} \\

\hline \multicolumn{1}{|c|}{\textbf{Function}} & \multicolumn{1}{c|}{\textbf{Failure Mode}} & \multicolumn{1}{c|}{\textbf{Effects}} & \multicolumn{1}{c|}{\textbf{Severity}} & \multicolumn{1}{c|}{\textbf{Causes}} & \multicolumn{1}{c|}{\textbf{Probability}} & \multicolumn{2}{c|}{\textbf{Detection}} & \multicolumn{1}{c|}{\textbf{RPN}} & \multicolumn{1}{c|}{\textbf{Design Alterations}}\\
\cline{7-8}
& & & \textbf{of Failure} & & \textbf{of Occurrence} & \textbf{Method} & \textbf{Probability of Detection} & &\\ \hline
\endfirsthead
\multicolumn{10}{c}
{{\bfseries \tablename\ \thetable{} -- continued from previous page}} \\
\hline \multicolumn{1}{|c|}{\textbf{Function}} & \multicolumn{1}{c|}{\textbf{Failure Mode}} & \multicolumn{1}{c|}{\textbf{Effects}} & \multicolumn{1}{c|}{\textbf{Severity}} & \multicolumn{1}{c|}{\textbf{Causes}} & \multicolumn{1}{c|}{\textbf{Probability}} & \multicolumn{2}{c|}{\textbf{Detection}} & \multicolumn{1}{c|}{\textbf{RPN}} & \multicolumn{1}{c|}{\textbf{Design Alterations}}\\
\cline{7-8}
& & & \textbf{of Failure} & & \textbf{of Occurrence} & \textbf{Method} & \textbf{Probability} & &\\
& & & & & & & \textbf{of Detection} & &\\ \hline 
\endhead

\hline \multicolumn{10}{|r|}{{Continued on next page}} \\ \hline
\endfoot

\hline
\endlastfoot

Mount substrate & Surface that substrate is on is no & Pipette crashes into substrate & 9 & Pipette is not a constant distance & 5 & Visible to user & 1 & 45 & *Commercially sensitive*\\ \cline{3-4} \cline{6-9}
& longer level & Pipette to substrate distance becomes too great for printing to take place & 4 & from the substrate during printing & 5 & Change of meniscus is observed on microscope camera & 2 & 40 & \\ \cline{3-4} \cline{6-6} \cline{8-9}
& & Deposition on substrate is uneven & 6 & & 5 & & 1 & 45 & \\ \hline
Controllably heat substrate & Insufficient heat supplied to the substrate & Solvent does not evaporate & 6 & Heater is not powerful enough & 2 & At maximum setting during testing heater does not get to required temperature & 3 & 36 & Modular design so heater is easily replaced. \\ \cline{5-10}
& & & & Heater is not installed close enough to the substrate & 1 & At maximum setting during testing substrate does not get to required temperature & 3 & 18 & Distance between heating element and module minimized.\\ \cline{5-10}
& & & & Heater has failed for unknown reason & 4 & No temperature change when heater is switched on & 1 & 24 & Modular design so heater is easily replaced.\\ \cline{2-10}
& Substrate gets overheated & Printing gets distorted & 6 & Thermal expansion of the substrate & 2 & Distortion in pattern discovered in  post print analysis & 6 & 72 & Model thermal expansion of substrates over temperature ranges used.\\ \cline{5-6} \cline{8-10}
& & & & Thermal drift of the carbon tab & 2 & & 6 & 84 & Operational procedure introduced to allow time for the system to settle after heating.\\ \cline{2-10}
& Electrical contact with the high voltage supply & Heater burns out & 10 & Voltage overload & 4 & System immediately starts losing temperature & 2 & 80 & Installation of electrical insulation layer in heater module\\ \hline
\end{longtable}
\end{@twocolumnfalse}
\end{minipage}}
\end{table*}


\begin{figure}[!tb]
\centering
    \includegraphics[width=0.7\linewidth]{Colourised_Printhead_x.png}
    \caption{Scale drawing of the print head housing design in use from as side on view. The blue areas show the main body of the print head housing, the pale yellow where the shafts for screws and the pipette are located from a cross-sectional view point. The white space shows a cut out so that where the screw that is used at the electrical contact meets with the pipette can be seen, so as to prevent the pipette breaking by over-tightening of the screw.}
    \label{Printhead}
\end{figure}
% 3D schematic may be clearer)

\textit{The research student felt that repeated re-engineering and adaptation of components during the first nine months had significantly delayed the project, and that the introduction of systems methods in this ad hoc manner had led to higher quality deliverables. The FFMEA process took between two and five hours to complete for a typical component; an uninformed design that had to be fabricated and tested at each iteration could cost more than a day per revision, on top of materials and fabrication costs.  Furthermore, during the initial phase, changing a key component would result in a chain of redesigns.  For the print head housing, both the substrate holder and pipette holder mounts had to be redesigned and re-fabricated to accommodate changes to the print head housing's dimensions, further increasing costs in time and materials. Had a systems approach been taken from the start, so that these components were designed in parallel, unnecessary costs would certainly have been avoided.}

\par \textit{The research student found it easy to learn, and successfully carry out a functional analysis and FFMEA for the components. An explanation and example proved to be sufficient guidance. The student noted that the most valuable outputs from these analyses came towards the end of completing the associated documents. By clearly listing and describing the `obvious' functions and associated failures, his attention was directed towards issues he had not intuitively foreseen. Additionally, failure effects common across functions and sub-systems became better known, and requirements for sub-system integration were better understood. Subsequent to acquiring a broader experience of systems engineering, the research student observed that carrying out an FFMEA was the activity that had made the most significant difference.}

\par \textit{The formal record, created by implementing these two systems engineering activities, proved to be beneficial in two ways.  Firstly, it provided clear traceability for failure events from observable effect to possible cause, which was a significant change from the blind disassembly and inspection that had previously been the working practice; this saved time. Secondly, it provided artefacts that were shareable with other project stakeholders to critique and check. Naturally, this enhanced the collaborative aspects of the project and enabled additional identification of issues that the research student had missed.  In one such case, a potentially dangerous failure was identified which could have led to the device building up a large static charge, the research student had neglected this issue as it is easily mitigated in a laboratory context.  However, this would be significant in a practical (commercial) device. It is precisely these sorts of observations that add value further up the TRL chain, by making a record (and, in this case, designing the fault out) at this stage, implementation risks are mitigated much earlier in the development programme.}

\par This case study could be colloquially described as \textit{toe in the water} adoption of systems engineering.  The techniques were chosen to specifically address the issues with which the research student was struggling, and applied as a \textit{post hoc} correction to design dilemmas.  The systems-lite approach, though, was suitable for gaining buy-in from an initially sceptical developer (the student) and ultimately resulted in both a change in working practice by the individual and an openness to explore systems engineering in more depth.  It is worth noting that the documentation created would have been suitable for carrying out verification tasks, but this was not undertaken during this phase.  The benefits to the progress of the project were obvious and apparent, but it is not possible to quantify the effectiveness.  The most obvious difficulty is in assessing the competence with which the student would have carried out FFMEA had he used it from the outset, instead of after he had already gained insight into causes of failures through the painful experience of witnessing several failures at first hand.  However, the issue of prior knowledge is not present in case A3, which follows.

% * <w.g.b.huish@lboro.ac.uk> 2017-06-01T17:08:56.483Z:
% 
%  Changes to behaviour prevented getting lost in the details
% 
% ^.
\subsubsection*{Case A3: Formal application of systems methods to the whole project}
% * <k.n.bjergstrom@lboro.ac.uk> 2017-06-01T22:53:14.577Z:
% 
% Validation and Verification - By developing a systems requirements there was a document against which sub-system performance could be validated. Verification is easy in lab science as the prototype is being perpetually tested, however there was no implicit means of validating sub-system performance against requirements.
% 
% ^.
Having observed the impact generated through applying a very limited set of systems engineering methods, there was interest in trying a richer system engineering approach: formalising concept of operations, specifying system and functional requirements, verifying component and system functionality, and validating the delivered system/component. This case lasted two months,
%add number of months
and concerned the development and integration of a complex sub-system (integrated heating element).  In this case, there had been no previous work on this design, although the researcher must have had the concept in mind, given prior work on the wider system.  The following Systems engineering process was conducted:

\begin{itemize}
\item Subsystem requirements capture and analysis
\item Functional Analysis
\item FFMEA
\item Design
\item Fabrication
\item Verification at various development stages
\item Validation
\end{itemize}

An integrated heating element is needed to controllably heat the substrate to prevent warping due to cooling after printing.  The particular challenges in this project were installing a heater in such a way as to not impinge on the movement of the staging, to not heat up parts of the printer other than the substrate and substrate mount, as some parts had unknown operating temperature ranges, and to prevent the high voltage system within the printer from electrically interfering with any element of the heater. This meant that the heater had to be electrically insulated from the printer without being thermally insulated, putting severe constraints on the materials that could be used as the electrical insulator. Additional complications arose from the need to avoid parts other than the substrate and substrate mount being heated, requiring the heating element to be in close proximity to both, and consequently to parts of the system that would be at high voltage.

% describe what is difficult and why

\textit{As with all systems development, the life cycle processes must be tailored to project size, organisational constraints, and other project needs.  With one research student and an advisory team of five academics (small team), it was important not to create} 
%check size and composition of advisory team
%increased from four to five academics, given the experimental advice provided by Kelly - K.
\textit{excessive systems engineering activity and documentation.  A curtailed set of technical processes, compared to the full list of ISO15288 \cite{iso2015iec}, was used (see above) and, within that, } 
%15288 ref
\textit{There were three aspects to the verification process, verifying the: physical compatibility (footprint, installation, etc.), electrical compatibility (power supply constraints, grounding, shielding), and operational functionality. The heater component was shown to be physically compatible with the system in terms of size requirements and weight restrictions, the latter being important as it had to be light enough to not impinge on the nimbleness of the staging. Electrical compatibility required the heater circuitry to be electrically insulated from the high voltage section, and the pre-existing wiring for the heater to be adapted for compatibility with the available power source. The heater was also shown to be able to heat a substrate on the staging controllably within the required temperature range. Each design iteration was verified against the sub-system's functional requirements, and mitigations for its determined failure modes. Once the component had been fabricated it was installed and tested, allowing its performance to be verified against its specifications and concepts of operations. Lastly, validation comprised of in-use testing, where it was established that the heating-element succeeded in relieving the issues it was intended to address. There was no formal customer sign-off more common with larger projects.}
% describe verification process exactly.
% Detail added, included some for validation- K.
\textit{The steps taken in the project were as follows:}

%check these steps to ensure they are all correct(ed)
%Steps 2 to 4 were conducted in parallel, rather than in order - W
%I don't believe we generated a functional flow diagram; we enumerated the required functions, but didn't place them in a formal structure - K.
\begin{figure}[!tb]
   \centering\resizebox{0.45\textwidth}{!}{
    \begin{tikzpicture}[
      >=latex',
      auto
    ]
      \node [intg] (kp)  {Pattern ink};
      \node [int]  (ki1) [node distance=1.5cm and -0.5cm,below left=of kp] {Move stage controllably};
      \node [int]  (ki2) [node distance=1.5cm and -0.5cm,below right=of kp] {Eject ink controllably};
      \node [int] (ki3) [node distance=1.5cm and 0.5cm,below left=of ki1] {Move between\\defined limits};
      \node [int] (ki4) [node distance=2.5cm,below of=ki1] {Move stage at defined speed};
      \node [int] (ki5) [node distance=1.5cm and 0.5cm, below right=of ki1] {Maintain constant distance between pipette and substrate during printing};
      \node [int] (ki6) [node distance=2.5cm,below of=ki2] {Control pipette voltage};

      \draw[->] (kp) -- ($(kp.south)+(0,-0.75)$) -| (ki1);
      \draw[->] (kp) -- ($(kp.south)+(0,-0.75)$) -| (ki2);
      \draw[->] (ki1) ($(ki1.south)+(0,-0.75)$) -| (ki3);
      \draw[->] (ki1) ($(ki1.south)$) -| (ki4);
      \draw[->] (ki1) ($(ki1.south)+(0,-0.75)$) -| (ki5);
      \draw[->] (ki2) ($(ki2.south)$) -| (ki6);
    \end{tikzpicture}}
    \caption{Functional flow diagram used to inform the functions defined in the FFMEA of the whole system}
    \label{functionalflowdiagram}
  \end{figure}
\begin{enumerate}
\item \textit{Requirements captured and documented based on the research student as customer; these were based on functional decomposition from the top level requirements of "The subsystem shall heat the substrate in a controllable way."}
\textit{\item A functional architecture was created as a functional flow diagram \cite{SEA5}, (see Figure \ref{functionalflowdiagram})}
\textit{\item A schematic design was created using MS Visio and iteratively developed (without the need for prototyping) by verification against the requirements specification, which included both the subsystem functional requirements and the integration requirements.}
\textit{\item Using FFMEA, critical failure modes were identified (see Table \ref{FFMEA Heater}) and those with a high Risk Priority Number (RPN) were designed out, through further design iteration.  RPN \cite{raheja2012design} is calculated simply through the product of severity, likelihood, and detection probability (i.e. how certain the failure is to be confidently attributed to the cause).  The criticality of the failure mode is the product of severity and likelihood.  It should be noted that these values are almost always subjective estimates, based on the estimator's experience or intuition.  In this project, the estimates were validated through discussion with supervisor expertise in 3D-printing.}
%Added ref for RPN: Design for Reliability, Pg. 82
\item \textit{Further design constraints were imposed. These were required for fabrication simplification; something the research student had not previously considered.}
\item \textit{The heating element was fabricated in line with the final design and integrated into the existing printer.}
\item \textit{The installed heating element was verified against the design requirements.  There were no specific tests to satisfy in this step, but verification was determined visually and operationally by the research student and his intuitive understanding of how the subsystem should work according to the design.}
\item \textit{The subsystem (heating element) was validated according to it meeting the original need (top level requirement).}
\end{enumerate}

\begin{figure}
\centering
\includegraphics[width=1\linewidth]{Heater_Element.png}
\caption{A side on schematic of the top part of the heater element, showing the separation of 3.75mm between the heater element and the top of the substrate. The rest of the heater element is not shown due to commercial sensitivity.}
\end{figure}

\textit{Based on the functional requirements specification, the FFMEA identified potential failures. The main failures highlighted are as follows. There is a risk of thermal drift of the carbon tab distorting the printing if the temperature changes significantly during printing. In order to prevent this a procedural instruction has been put in place to heat the substrate to temperature and allowing a short period of time for the system to settle before starting printing. There is also a risk that the substrate is not completely level on the staging, causing uneven deposition, failure to print, or in the worst case the tip to crash into the substrate. To prevent this a program to auto correct for height as the tip moves across the surface has been written. The potential for the heater burning out as a result of voltage overload from contact with the high voltage system is also a high risk failure mode, as this would require the heater part to be replaced. This lead to electrical, but not thermal, insulation being incorporated into the design.}
%add number of failures and refer to appendix if it is included.
\textit{By taking these into account a component design was created that has not had to be re-engineered since it fabrication, despite numerous subsequent changes to the whole system.  From a project perspective, it was the first major component that, once designed, did not have to be re-engineered, or adapted, due to design changes to other parts of the system. Installation of the element was achieved without difficulty. It worked to specification the first time, and was delivered on time (i.e. according to schedule). Although the initial design process took longer than in cases A1 and A2, the costs relating to the protracted iterative design process of these cases, and the knock-on effects of redesign being required elsewhere in the system, outweighed this initial investment of time. The researcher recorded spending thirteen hours on the initial design for the heating element, ten of which were spent on activities pertaining to systems engineering, and three on other design activities (such as drawing schematics). Fabrication, carried out by a third party, has consistently taken one working day for any component. Consequently, it is clear that a considerable time-saving was achieved by eradicating the need for repeated fabrication of numerous prototypes, as had been the case in the earlier project phases.}

Although the systems engineering carried out in this case study focused on a complex subsystem, rather than the whole system, and could be considered to be lightweight in some parts, it is clear that the influence of clearly stated, and verified requirements, and the application of FFMEA have considerably reduced the amount of rework that had dogged the previous phases.  Reduction, or removal, of rework is often quoted as a raison d'\^{e}tre for Systems Engineering \cite{shishko1995nasa,honour20046}.  
% NASA SP6105, June 1995 and Systems Engineering Centre of Excellence report 01-03, 2003 %Both added.
This case has a marked, and important, difference with the previous one (A2):  in this case the research student had no prior knowledge of the design article or \textit{trial and error} experience of its development.  A formal improvement that could have been instituted in this case would have been the prior and explicit definition of verification test criteria and test plan; retrospectively the research student remarked that this would have been important had the project been conducted by a larger group in which work was partitioned between its members.  It would provide certainty to group members regarding the performance of individual components that were outside their responsibility.  Overall, the research student gained an understanding of constraints, pre-sighting potential fabrication and integration issues, and an approach for good experimental design without the need for excessive prototyping.  This was a good example of how a systems approach develops \textit{hindsight in advance}.  As in case A2, the research student considered application of the FFMEA to have the most significant impact on the subsystem development.

No attempt was made to decide on the optimal amount of systems engineering for this case and additional work would be required to characterise the appropriate balance and intensity of systems engineering effort for applied science, experimental design. Furthermore, such optimisation should also consider the effect on the full life cycle; for instance, would additional systems engineering effort at the experimental stage lead to more, or less effective commercialisation?  The case study provides no answers to this, and related, questions.  We note the conclusions of Honour \cite{honour20099}, that investment in systems engineering is most effective if done early in the life cycle.  However, we can conclude that application of systems engineering has, in this case, improved the design and fabrication of experimental apparatus in terms of robustness and project timeliness.
 It would seem that a significant change in behaviour has taken place for the research student who, as a result of learning and applying systems engineering techniques, has developed a design philosophy that is more holistic and places a greater weight on accurate scoping and description of the design item than had previously been his practice.
 
\subsection*{Summary of case A}
To summarise the level of Systems Engineering used in cases A1, A2, A3, a simple V-diagram, that corresponds to the component developments, is presented in Figure \ref{fig:vdiag_1}.  The stages of this process are listed in Table \ref{V-table} and the activity carried out in each case are described.  It may be noted that deployment was only entirely successful for case A3, in which the most comprehensive application of Systems Engineering techniques was undertaken.


\begin{table*}[!ht]
\footnotesize
\begin{@twocolumnfalse}
\begin{center}
\begin{longtable}{|p{2cm}|p{4.8cm}|p{4.8cm}|p{4.8cm}|}
\caption{Table showing depth of systems engineering performed in each case study for each step in the V-diagram in figure \ref{fig:vdiag_1}} \label{V-table} \\
\hline \multicolumn{1}{|c|}{\textbf{V-diagram stage}} & \multicolumn{1}{c|}{\textbf{A1}} & \multicolumn{1}{c|}{\textbf{A2}} & \multicolumn{1}{c|}{\textbf{A3}} \\ \hline
\endfirsthead
\endhead

\textbf{User Requirements} & None & None & Bullet point of what the researcher required from the heater module were made.\\ \hline
\textbf {System Requirements} & None & FFMEA (see table \ref{FFMEA Printhead}) after initial designs had been attempted. & A functional analysis (see figure \ref{functionalflowdiagram}) was performed, leading to an FFMEA (see table \ref{FFMEA Heater}) \\ \hline

\textbf{Architecture Design} & The dimensions of the substrate mounting were constrained by the staging.& The maximum dimensions of the print head housing were constrained by the existing set up.& The maximum dimensions of the heater module were constrained by the existing set up.\\
\hline

\textbf{Component Design} & Only the size of the substrate was considered in the component design. & The design alterations considered in table 
\ref{FFMEA Printhead} were implemented. & The design alterations considered in table 
\ref{FFMEA Heater} were implemented.\\
\hline
\textbf{Component Manufacture} & Not considered. & Not considered. & The geometry of the design was simplified from circular to square to aid manufacture speed. \\ \hline
\textbf{Component Integration} & Yes & Yes & Yes \\ \hline
\textbf{Testing} & Testing only conducted on clamping the substrate. & The functions defined in table \ref{FFMEA Printhead} were tested. & The functions defined in table \ref{FFMEA Heater} were tested. \\ \hline
\textbf{Deployment} & Unsuccessful & Required several design and fabrication iterations & Successful \\ \hline
\end{longtable}
\end{center}
\end{@twocolumnfalse}
\end{table*}


\subsection*{Case B1 - Quantum Systems Engineering summer school}
Everitt et. al. \cite{Everitt2016} have argued that systems engineering is required to realise the opportunities presented by the most recent research in quantum mechanics.  Based upon the experiences related in case A, and on the delivery of two short workshops on systems engineers for quantum scientists, a curriculum for a five-day summer school in quantum systems engineering was developed.  This was delivered to thirty PhD students all sponsored by DSTL (UK Defence Science and Technology Laboratory) and mostly nearing the end of their second year (of three).  The curriculum, designed to provide an overview of systems engineering and training in a small number of useful techniques, comprised the following:

\begin{itemize}
\item Concepts of Operations (CONOPS), 
\item Systems requirements analysis using Structured Textual Analysis (STA) \cite{STABHW}, 

\item Functional requirements analysis, 
\item Function Means Analysis (FMA) \cite{FMABHW}
\item Brainstorming and negative brainstorming for concept generation (though participants generally used FMA with brainstorming only)
\item Functional Failure Modes and Effects Analysis (FFMEA), 
\item Quality Function Deployment (QFD) methods, 
\item Verification and Validation (V and V), 
\item Life-cycle analysis.
\end{itemize}
There were lectures in each topic and then students were expected to apply the techniques to a project over the course of the week. The project required them to design, and present to a panel of industry experts, a design concept for a commercially feasible complex technical product. They had to apply and demonstrate the use of systems engineering techniques in order to convince the judging panel that their technology was credible, and they were expected to highlight design parameters, features, opportunities, and novel solutions within their designs. Particular emphasis was placed on a problem-oriented design process, developing a solution to an existing problem rather than creating a technology with no application in mind.

\textit{Over the course of the summer school the students' perceptions of usefulness of systems engineering methods was tracked informally, and at the end of the week their opinions regarding the value of a systems engineering approach, and its applicability to scientific research and their own doctoral research, was solicited through feedback forms that were completed anonymously. Throughout the week observations regarding the approach students were taking to accomplishing their projects goals were recorded, in an attempt to observe how systems thinking affected their behaviour when tackling a complex problem under time-constrained circumstances.}


\textit{Feedback forms requested participants to identify positive and negative aspects of their experience of the summer school.    With respect to the merits of the systems engineering techniques that had been learned and the participants' systems appreciation there were no negative comments at all.  The benefits most frequently identified by participants were as follows:}
\begin{itemize}
\item \textit{the systems approach enabled a better appreciation of the problem requiring to be solved and ensured that the student began with the problem, rather than beginning with the solution}
\item \textit{the group discussions, using systems artefacts, clarified boundaries of the problem}
\item \textit{it was helpful to view problems from the perspective of other (stakeholders)}
\item \textit{provided a systematic way to work through a problem in a group context}
\end{itemize}

\textit{To conclude this case study, the following unstructured, but agreed comments from the four main course lecturers provide an insight into the benefits of educating scientists in some basic systems engineering.  Initial scepticism from some participants dissipated very quickly once work on the project began (on day two).  All groups were thorough in the }
\textit{application of the techniques at their disposal: STA, FMA,FFMEA, QFD.  As in the previous case studies, it seems that FFMEA 
was regarded as the most useful technique learned by the participants.  Virtually all the participants felt that the experience had been worthwhile; many felt that they were too far into their research to obtain significant benefit from applying systems engineering to their projects at the current stage, however, several remarked that they wished they had learned about systems engineering at the beginning of their PhD.}

To some extent, this case study had the purpose of testing the conclusions of case study A. It is different because, whereas Case A tested the implementation of System Engineering techniques in an ongoing PhD project, Case B provided training in the techniques to 30 PhD students from different institutions
 and sought their opinion regarding how they \textit{might} use the techniques within scientific research.  The consistency with which the Case B participants endorsed the benefits of Systems Engineering supports the conclusions drawn from Case A.
 
%%VDiagram figures here
    \begin{figure}[!ht]
    \centering\resizebox{0.45\textwidth}{!}{
    \begin{tikzpicture}[
      >=latex',
      auto
    ]

      \node (A1) at (0,0) {};
      \node [v1] (B1) [node distance=4cm and 0.2cm, below right=of A1] {Component Manufacture\vphantom{g}};
      \node [v1] (B2) [node distance=4cm and 0.2cm, below left=of A1] {Component Design};
      \node [v1] (ML1) [node distance=1cm and -1cm, above right=of B1] {Component Integration};
      \node [v1] (ML2) [node distance=1cm and -1cm, above left=of B2] {Architecture Design};
      \node [v1] (M1) [node distance=1cm and -1cm, above right=of ML1] {Testing};
      \node [v1] (M2) [node distance=1cm and -1cm, above left=of ML2] {System Requirements};
      \node [v1] (T1) [node distance=1cm and -1cm, above right=of M1] {Deployment};
      \node [v1] (T2) [node distance=1cm and -1cm, above left=of M2] {User Requirements};
      
      \draw[<-] (B1) -- (B2);
      \draw[->] (B1) -- (ML1);
      \draw[->] (ML1) -- (M1);      
      \draw[->] (M1) -- (T1);
      \draw[<-] (B2) -- (ML2);
      \draw[<-] (ML2) -- (M2);      
      \draw[<-] (M2) -- (T2);
      \draw[dashed] (ML2) -- (ML1) node [midway, above] {Verification};
      \draw[dashed] (M2) -- (M1) node [midway, above] {Verification};
      \draw[dashed] (T2) -- (T1) node [midway, above] {Validation};
    \end{tikzpicture}}
    \caption{Lifecycle diagram showing the order of systems engineering processes, as well as where verification and validations occurred. Table~\ref{V-table} shows which processes were used for each case.}
    \label{fig:vdiag_1}
  \end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}
\subsection{Critique of Methodology}
The methodology was described in section \ref{sec-methodology}, in which we explained the methodological influences on conclusions that must be considered.  In this subsection, we briefly discuss specific aspects of confidence in the conclusions to be drawn from the case studies.

For Case A, it is important to note that although the introduction of Systems Engineering methods is coincident with improvements in experimental prowess, it is also the case that the researcher (student) was developing his understanding of the project and his understanding of research techniques in general.  Thus, ongoing development of the researcher himself is a factor to be considered.  Improvements in project performance can be externally validated by supervision, but the reasons for improvement are based on the (subjective) view of the researcher.  His analysis may also be affected by his growing knowledge of Systems Engineering as the project proceeds, i.e. the linkage between Systems Engineering and performance may only be internally observable (by the research student) once he has acquired sufficient knowledge of Systems Engineering to rationalise cause and effect.

The summer school curriculum was built, at least in part, based on the knowledge gained during Case A, thus cases A and B are not independent. The fact that the outcomes of case B are coincident with case A should not be considered to be \textit{proof} of the conclusions or imply a generalisability of the results.  Case studies never validate the results of other case studies, but they may add (or reduce) confidence in the results; this is referred to as triangulation.  Thus, the results of case B provide added confidence in the conclusions given below in section \ref{sec-conclusions}, but they do not prove them.

\subsection{Changes in behaviour}
Since the 1990s, many commercial organisations have invested in Systems Engineering training because it has been viewed as organisationally transformational.  There are a variety of Systems Engineering competencies that can be developed to improve an individual's problem solving ability in complex environments \cite{frank2012engineering}; in this section we outline the development of the research students in cases A and B.

At the start of the PhD project the student displayed resistance to the position that  systems Engineering should be applied to laboratory research for the following reasons:
\begin{itemize}
\item he thought it would mainly be a paperwork exercise
\item that although it might support the wider R\&D chain, it would have no value to his individual work
\item that the imposition of a \textit{rigid} framework might inhibit creativity
\item that a cyclical process requiring verification and validation (V\&V) to advise an iterative design was not analogous to laboratory research
\item he found it difficult to imagine V\&V criteria in advance
\item that the time required to learn the techniques would ultimately outweigh any benefits in the context of a three-year doctoral programme
\item he thought that failure modes were intuitively obvious
\end{itemize}
After repeated failures to design an apparatus that was sufficiently robust and worked as required, a limited number of systems engineering activities were introduced.  Firstly, the student found that writing a good specification for components and keeping a record of requirements and failures enabled him to keep the whole design (rather than just individual components) in mind and, thus, enabled more effective integration with other components.  This also facilitated traceability, so that failures could be properly analysed and understood.  The FFMEA was initially based on observation of failures, but this quickly led the student to identify failure modes that had not occurred and had not previously been identified or understood.  This experience refuted one of the initial prejudices noted above.



By the end of case A2, the researcher had concluded that, without doubt, specific systems engineering methods could save time in a laboratory context. A very light-weight application of systems methods to component design had developed both an appreciation of designing for reliability, and of designing for integration. Furthermore, as the researcher began to take a systems approach from the beginning of any sub-system design processes, he noticed that he began to design for production.  This was an aspect that he had not previously considered at all. During the early stages of the project little consideration was taken as to how parts were manufactured. It was left to the expertise of the technician to mitigate any problems that were a consequence of poor design. This led to numerous parts being re-engineered at the point of fabrication, either by the technician if the change was minor, or in conversation with the researcher if major design changes were needed. This is a costly process, both in time spent by the researcher and increased fabrication costs; it was significantly reduced once a systems approach had been adopted.

The researcher found that when no systems engineering was applied (case A1), most re-designs were architectural changes, or attempts to achieve better performance. However, when a systems approach was taken (cases A2 and A3), with significantly more time being spent on the initial design process, further design iterations tended to focus on improved integrability, and ease of fabrication. As the design process had begun by specifying requirements, and understanding integration problems.


It was also noted that design choices in case A3 were made bearing in mind the aim of reaching higher TRLs, with the researcher independently making effort to ensure his designs would not limit the scope of a future product. For example, more recent components and sub-systems have been designed to be modular (enabling easy replacement or change).

The researcher also observed that once a record of knowledge had been established it would be much easier for additional people to join the project, and be able to quickly develop the understanding needed to contribute. It would also stop dead ends being explored multiple times and, conversely, prevent abandoned ideas being lost when they may have potential relevance in other situations.

The application of the systems methods was not simply a formulaic repetition of process, but constituted a change in the overall behaviour of the researcher in terms of both how he viewed and practised laboratory science.  The most significant change was the development of an holistic view of the research in terms of the developing system and the system development over time.  Practising the methods (as opposed to simply reading about them) has had a transformational effect on the behaviour of the researcher, instilling in him a systems minded approach.


With regard to case B, one week is not a sufficient period over which to observe unambiguously behavioural transformation as a result of new learning.  However, the fact that most participants could identify ways in which Systems Engineering could have been valuable in their PhDs suggests that there is the opportunity for behavioural change in the future.

\subsection{Role of Systems Engineering in Laboratory Science}
In well-defined experiments, the system boundary is clearly understood, the interactions of interest are established and reasonable assumptions can be made about insignificant interactions.  This relies on the researcher having a comprehensive understanding of the situation to be modelled in an experiment.  The scientist must indulge in some engineering in order to build the experimental apparatus in such a way that the physical set-up of the experiment will accurately represent the anticipated physical behaviours and exclude any undesired behaviours (or that unanticipated behaviours will be obvious to the experimenter, should they occur).  The case studies reported herein have indicated that a selection of tools used by systems engineers can aid the design of experiments to ensure robustness, clear definition, and affordable adaptability during the course of an experimental campaign.  They have indicated that time and resources may be saved (or at least not wasted) when a systems approach is taken to the design of experiments.  The role of Systems Engineers includes that of integrator (of systems components and relevant disciplines) and design and manufacture of experimental apparatus can often be considered to be a problem of integration.

We consciously endeavoured to keep the systems engineering knowledge and practice required by the scientist to a minimum (in both case studies), however, the question of how much (Systems Engineering) is enough? has not been attempted.  One could speculate that a set of principles may emerge, if physicists adopt Systems Engineering training, that match the systems engineering investment to the experiment complexity and level of resources.  An area that we have not yet explored, but is expected to be significant is that of systems reliability engineering.  It is clear from both studies that the FFMEA technique for identifying (and thence mitigating) failure modes was the one most valued by the scientists, of those that were tried.  One could argue that the case studies have considered a small sample of research students and that the need for Systems Engineering training for all post-graduate science students is not established, because these could be a minority in terms of experimental ability.  However, it is not our assertion that it is universally needed, but that it appears there is merit in training scientists in Systems Engineering in order to reduce the risk of experimental failure or resource wastage.  The students also noted that the techniques had enabled them to formulate and understand the problem better (case B1) and so it can be argued that Systems Engineering provides a useful and accessible tool set for problem formulation in complex experiments.

The case study outputs indicate that the participants felt that some Systems Engineering training for physicists (e.g. at the start of a PhD) would be beneficial from the point of view of managing experimental resources more effectively and of ensuring that an experimental campaign runs to schedule.

Many experimental programmes involve teams of scientists and the use of the Systems Engineering artefacts for sharing knowledge across a team of scientists has been highlighted in both case studies A and B.

The benefits of the systems approach to the future exploitation of research by developers working at higher TRLs has been considered, but has not been tested in this analysis.

There are, therefore, two main criteria that may be used to evaluate the usefulness of Systems Engineering in scientific experimentation: i) level of rework required in design and construction of experimental apparatus, and ii) efficiency with which scientific discovery can be turned into systems of social or commercial benefit.  The first criterion has been met in the case studies, but the second has not been attempted.

\section{Conclusions} \label{sec-conclusions}
Although this work was originally inspired by considering the extent to which Systems Engineering could be needed for experimentation and development of quantum technologies, we have arrived at a more general conclusion regarding the role of Systems Engineering in scientific experimentation.  The studies reported herein lead us to propose that laboratory science would benefit from adopting light-weight Systems Engineering processes and techniques to improve the robustness and efficiency of experiment design.  In this context, light-weight means a broad understanding of the systems approach, an appreciation of higher-TRL needs to exploit research in real devices and systems, and set of techniques for capturing and analysing requirements, concept generation, functional design, failure mode identification, and verification and validation.

Systems Engineering is not uninformed application of processes and tools, but also a way of thinking.  Therefore, we propose that there is a benefit to training early career scientists in basic Systems Engineering. Although adoption of the Systems Engineering techniques was relatively rapid in the case studies, it is recognised that there is a hearts and mind battle to be won to realise this proposal.  However, it seems that experiments are demanding increasing complexity, and tools are needed with which to deal with this.

Overall, the tools considered have been: CONOPS, STA, Functional requirements analysis, FMA, concept generation, FFMEA, QFD, V\&V approaches, and life-cycle analysis (see section \ref{sec-case-study}).  The technique that appeared to be most useful to the students was FFMEA; this had the particular merit of keeping students attention on the wider system.

\subsection*{Future Work}
Future work will examine a wider range of techniques and seek to establish the principles of what constitutes sufficient Systems Engineering for laboratory science.  This will include understanding the appropriate level of Systems Engineering knowledge for experimental scientists and thence develop a Systems Engineering curriculum for scientists.

The recommendations included in this paper constitute a change in paradigm for the training of scientists, future work will develop the detail of how such a paradigm shift may be achieved.




\section{Acknowledgements}
M.J.E, K.N.B, and V.M.D. would like to thank the Defence Science Technology Laboratory (DSTL) under Contract No. DSTLX1000096136R for their support through the grant \emph{Engineering for Quantum Reliability}. W. G. B. H. would like to thank Dr. Kelly Morrsion and Dr. Daniel Engstrom for advice in generating both the designs of components and which materials are potentially suitable for use in the printer, and Mr. Kevin Clarke and Mr. Iliya Dimitrov for their aid in manufacturing the components.

\bibliographystyle{IEEEtran}
\bibliography{lite}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{KNB}}]{Kieran Bjergstrom} is a doctoral research student in the Department of Physics at Loughborough University, working within the Quantum Systems Engineering Research Group. His research focusses on reliability engineering and systems engineering for quantum technologies. In particular, taking interest in understanding the costs and benefits of applying a systems engineering approach to the development of laboratory-based, low-TRL, technologies, and in the realistic modelling and simulation of quantum devices within dissipative environments. \end{IEEEbiography}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{WGBH}}]{William Huish} is a PhD research student in the Quantum Systems Engineering Research Group at Loughborough University. His research focusses on the potential of additive manufacturing to aid the development of quantum technologies. He graduated from Loughborough University with a Masters in Physics in 2015.
\end{IEEEbiography}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{MH}}]{Prof Michael Henshaw}
leads the Engineering Systems of Systems research group. He joined Loughborough University in 2006, after spending 17 years in the Aerospace and Defence business. He graduated with first class honours in Applied Physics and his PhD research concerned modelling laser fusion for energy generation and his post-doctorate research was in X-ray lasers.  Much of his current work concerns the integration of complex networked systems with contributions in Systems of Systems (SoS) and Cyber-Physical Systems (CPS) research.  He recently led the TAMS4CPS project, sponsored by the European Commission that has published a research agenda for modelling and simulation of CPS. He is a member of the NATO Technical panel in Systems Concepts and Integration, leading the integration and interoperability area and is chair of the recent NATO Systems of Systems Lecture Series.  He is also a co-chair of the IEEE SMC Society Technical Committee on SoS. He is an associate editor for the Aeronautical Journal and the Systems Engineering Body of Knowledge (sponsored by INCOSE and IEEE) in which he is responsible for the knowledge areas of Product Systems Engineering, and SoS Engineering.  Prof. Henshaw leads the Loughborough University global research challenge in Secure and Resilient Societies, which integrates technical, societal, and humanities disciplines to create solutions to natural and manmade threats to society.\end{IEEEbiography}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{VMD}}]{Dr Vincent Dwyer} is a Reader in Electronic Devices in the Wolfson School of Mechanical, Electrical and Manufacturing Engineering at Loughborough. He has a background in theoretical physics, including density functional theory for surface reconstruction calculations and the recursion method for generating the density of states, and he is internationally known in the areas of quantitative surface analysis and back-end-of-line reliability/physics of failure of integrated circuits. He is a regular member of the Programme Committee for the European Symposium on Reliability of Electronic Devices, Failure Physics and Analysis (ESREF), the major European conference in this area. He has made significant contributions to the study and analysis of electromigration in IC interconnects, dielectric breakdown of gate oxides and thermal failure of semiconductor junctions. The latter includes a model which has been used by all major IC manufacturers in protection circuit design; has been the subject of tutorials at major international conferences, and appears in all recent textbooks on the subject. Together with Mark Everitt, Vince jointly leads the Science-Wolfson School mini CDT in Quantum Technologies
\end{IEEEbiography}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{MJE}}]{Dr Mark Everitt}
is a Senior Lecturer in Quantum Control in the Department of Physics at Loughborough and leads the Quantum Systems Engineering Research Group (whose focus is: The application of Systems Engineering methods to accelerate Blue-Sky and low technology readiness level devices and technologies;
New Systems Engineering methods for quantum technologies industry especially in the areas of Quantum Design for Test, Reliability, Manufacture, etc.;
Next generation manufacturing processes for  quantum technologies;
Quantum reliability engineering; Computer aided quantum engineering solutions). He has made significant contributions to superconducting realizations of quantum circuits, the foundations of quantum mechanics (specifically the measurement problem and the quantum to classical crossover and phase space methods), non-linear dynamics and quantum chaos. Extending his interest in designing superconducting quantum circuits for quantum information processing, his current research interests include reliability assurance and systems engineering for quantum technologies, application and development of quantum phase space methods, engineered quantum environments, quantum-state meta-materials and the foundations of quantum mechanics. 
\end{IEEEbiography}

%\flushbottom
%\newpage



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}

