add test results

fault_injection_async25.tex

@@ -12,6 +12,8 @@
 \usepackage{orcidlink}
 \usepackage[shortcuts,acronym]{glossaries}
 \usepackage{subcaption}
+\usepackage{pgfplots}
+\usepackage{pgfplotstable}
 % Tikz because graphs are fun
 \usepackage{tikz}
 \usepackage{tikz-timing}
@@ -20,6 +22,7 @@
 \usetikzlibrary{positioning}
 
 \makeglossaries
+\renewcommand{\glstreepredesc}{\hspace{3em}}
 
 % Acronyms for the document
 \newacronym{dut}{DUT}{Design Under Test}
@@ -31,6 +34,7 @@
 \newacronym{sed}{SED}{Single Event Delay}
 \newacronym{prs}{PRS}{Production Rule Set}
 \newacronym{uvm}{UVM}{Universal Verification Method}
+\newacronym{eda}{EDA}{Electronic Design Automation}
 
 % Simple citation required command
 \newcommand{\citationneeded}{\textcolor{red}{[citation needed]}}
@@ -76,10 +80,7 @@ As a leading toolchain for asynchronous logic development, ACT offers a comprehe
 
 These innovations eliminate the need for additional injection-related logic within the design while also reducing development effort, as testing infrastructure for behavioral validation can simply be reused. Additionally, only the design under test needs simulation at the gate-level, while the auxiliary testing harness can stay at higher levels of abstraction. Finally, we also achieve a reduction in necessary injections by targeting high-fanout signals more heavily, triggering more errors per injection. 
 
-To validate our setup, we benchmarked it against existing fault-injection tools, demonstrating its performance in both simulation efficiency and the overall number of injections needed to achieve representative results.
-
-% Alternate abstract after test results
-% To validate our setup, we benchmarked it against existing fault-injection tools, demonstrating substantial improvements in both simulation efficiency and the overall number of injections needed to achieve representative results, thus enabling better scaling as target designs grow more complex.
+To validate our setup, we benchmarked it against existing fault-injection tools, demonstrating substantial improvements in both simulation efficiency and the overall number of injections needed to achieve representative results, thus enabling better scaling as target designs grow more complex.
 \end{abstract}
 
 \begin{IEEEkeywords}
@@ -88,18 +89,18 @@ asynchronous circuits, SET, fault-tolerance, cluster computing, computer aided d
 
 \section{Introduction}
 
-While commercial EDA tools offer convenient access to the current state of the industry, they are usually not customizable enough (as they tend to be closed source) or not even available for more specialized applications. 
+While commercial \acs{eda} tools offer convenient access to the current state of the industry, they are usually not customizable enough (as they tend to be closed source) or not even available for more specialized applications. 
 %This problem is well understood for asynchronous logic, as the commercial offerings' focus on mainstream synchronous designs limits functionality for everything outside their scope. 
-It is generally agreed that this EDA focus on mainstream synchronous design is one key reason for the hesitant adoption of asynchronous design styles.
-This lack of EDA support has somewhat been mitigated by the publication of the open source ACT toolchain by the Yale AVLSI group \cite{manoharOpenSourceDesign}. However, the local compute currently offered by ACT often does not suffice for tasks that are more laborious. 
+It is generally agreed that the focus on mainstream synchronous design is one key reason for the hesitant adoption of asynchronous design styles.
+This lack of \acs{eda} support has somewhat been mitigated by the publication of the open source ACT toolchain by the Yale AVLSI group \cite{manoharOpenSourceDesign}. However, the local compute currently offered by ACT often does not suffice for tasks that are more laborious. 
 
 Especially for those tasks that lend themselves nicely to a high degree of parallelization, cluster computing offers high potential for speed improvements. 
 For this reason, we have augmented ACT with the tool \texttt{action} which does just that - while offering a simple \acs{api} to vastly extend its functionality. Our goal was to create a framework to build on, and we here present a real world use-case to demonstrate this capability.
 
 Specifically, we consider the problem of fault-tolerance assessment through fault injection.
-With applications extending into harsh environments like space on the one hand, and feature sizes in the nanometer regime on the other hand, high-energy particles are getting more likely to cause erroneous behavior of transistors. Consequently fault tolerance is an increasingly desired property of digital circuits, be they synchronous or asynchronous. To this end, studying the behavior of selected target circuits under artificially injected faults is a vital means. Beyond a mere quantitative estimation of the error probability of a given design in a given environment, there are also more qualitative, conceptual questions that fault injection can answer. For instance, one may try to understand whether a mechanism like the temporal masking provided by flip flops in synchronous designs also exists in asynchronous designs, established by the communication protocol, and if so, on which parameters it depends.
+With applications extending into harsh environments like space on the one hand, and feature sizes in the nanometer regime on the other hand, high-energy particles are getting more likely to cause erroneous behavior of transistors. Consequently fault tolerance is an increasingly desired property of digital circuits, be they synchronous or asynchronous. To this end, studying the behavior of selected target circuits under artificially injected faults is a crucial tool. Beyond a mere quantitative estimation of the error probability of a given design in a given environment, there are also more qualitative, conceptual questions that fault injection can answer. For instance, one may try to understand whether a mechanism like the temporal masking provided by flip flops in synchronous designs also exists in asynchronous designs, established by the communication protocol, and if so, on which parameters it depends.
 
-In any case, to produce meaningful results such fault-injection experiments need to cover an ample parameter space, spanned by time and location of fault injection, fault parameters, target circuit properties and condition, just to name a few. In the literature, multiple attempts have been made to create tools for leveraging this ambitious task, and they have produced valuable results (see \cite{behalExplainingFaultSensitivity2021} for just one example). Yet, we feel that the chance to integrate such a fault-injection tool into a design environment, as well as the availability of cluster computing offers the opportunity for further improvement.
+In any case, to produce meaningful results, such fault-injection experiments need to cover an ample parameter space, spanned by time and location of fault injection, fault parameters, target circuit properties and condition, just to name a few. In the literature, multiple attempts have been made to create tools for leveraging this ambitious task, and they have produced valuable results (see \cite{behalExplainingFaultSensitivity2021} for just one example). Yet, we feel that the chance to integrate such a fault-injection tool into a design environment, as well as the availability of cluster computing, offers the opportunity for further improvement.
 
 %Exposing digital circuits to environments like space can break some of the most basic assumptions we make when designing digital circuits. Given the level of miniaturization we have access to, having high energy particles rain upon the millions of interconnects in an average design can introduce unexpected behavior. These undesired deviations from design specification, or \emph{failures}, need to be well understood to make predictions about a design's robustness.
 
@@ -256,13 +257,36 @@ Points to talk about
 \section{Results}
 \label{sec:results}
 
-Points to talk about
+\begin{figure*}[htbp]
+    \centering
+    \input{results/aggregated.tex}
+    \caption{Failure type distribution of tested circuit types}
+    \label{fig:results/aggregated}
+\end{figure*}
+
+
+\begin{figure*}[htbp]
+    \centering
+    \begin{subfigure}{0.4\textwidth}
+        \begin{center}
+            \input{results/deviation_num_injections_dims.tex}
+        \end{center}
+        \caption{varying the number of injections}
+        \label{fig:res/deviation_num_sims_dims}
+    \end{subfigure}
+    %\hfill
+    \begin{subfigure}{0.4\textwidth}
+        \begin{center}
+            \input{results/deviation_num_signals_dims.tex}
+        \end{center}
+        \caption{varying the number of selected signals}
+        \label{fig:res/deviation_sel_signals_dims}
+    \end{subfigure}
+    \caption{Variation of failure type rates in percentage points when}
+    \label{fig:res/deviation_dims}
+\end{figure*}
+
 
-\begin{itemize}
-    \item Compared to Behal: how many failures were we able to find with our new tool vs with the old tool
-    \item Compared to Behal: how efficient (failures found / injection) is this setup compared to previous attempts
-    \item Dflow: how do certain families of async and sync compare
-\end{itemize}
 
 \section{Conclusion}
 \label{sec:conclusion}

results/aggregated.csv

@@ -0,0 +1,5 @@
+Circuit,Timing deviation,Value failure,Coding failure,Glitch,Deadlock,Token count failure,Nothing
+DIMS,89,8,11,10,4,5,8
+DIMS rand,89,8,11,10,4,5,8
+DIMS large,89,8,11,10,4,5,8
+NCLX,96,8,nan,12,3,5,4

results/aggregated.tex

@@ -0,0 +1,37 @@
+
+\pgfplotstableread[col sep=comma]{results/aggregated.csv}\aggregateddata
+
+\begin{tikzpicture}
+    \begin{axis}[
+        width=\textwidth,
+        height=.25\textheight,
+        ybar=2pt,
+        enlargelimits=0.15,
+        ylabel={\% of total injections},
+        xtick=data,
+        symbolic x coords={DIMS, DIMS rand, DIMS large, NCLX}, % Group names on the x-axis
+        xticklabel style={rotate=45, anchor=north east},
+        ymin=0, ymax=100, % Y-axis range from 0 to 100
+        axis line style={-}, % Removes the top and right axis lines
+        legend style={
+            at={(0,-0.45)}, % Position the legend inside the graph
+            anchor=north west,
+            legend columns=7,
+            font=\small, % Adjust legend font size
+            draw=none, % No border around the legend
+        },
+        nodes near coords, % Option to show values near the bars
+    ]
+    
+        \addplot table[x=Circuit, y=Timing deviation] {\aggregateddata};
+        \addplot table[x=Circuit, y=Value failure] {\aggregateddata};
+        \addplot table[x=Circuit, y=Coding failure] {\aggregateddata};
+        \addplot table[x=Circuit, y=Glitch] {\aggregateddata};
+        \addplot table[x=Circuit, y=Deadlock] {\aggregateddata};
+        \addplot table[x=Circuit, y=Token count failure] {\aggregateddata};
+        \addplot table[x=Circuit, y=Nothing] {\aggregateddata};
+    
+        \legend{Timing deviation, Value failure, Coding failure, Glitch, Deadlock, Token count failure, Nothing}
+    
+    \end{axis}
+\end{tikzpicture}

results/deviation_num_injections_dims.csv

@@ -0,0 +1,9 @@
+Total simulations,Timing deviation,Value failure,Coding failure,Glitch,Deadlock,Token count failure,Nothing
+2338,0.37,-0.81,-0.02,-1.01,0.02,-0.33,-0.50
+3149,0.87,0.27,-0.40,0.14,-0.29,-0.04,-0.47
+4773,0.10,-0.71,-0.53,-0.58,-0.07,-0.46,0.12
+5015,-0.16,-0.01,0.54,-0.39,-0.08,-0.13,0.29
+7780,0.04,-0.06,0.89,-0.37,-0.05,-0.27,-0.08
+9583,0.34,0.00,0.93,-0.07,-0.09,-0.46,-0.29
+13317,0.25,0.21,1.12,-0.06,-0.03,0.01,-0.42
+25757,0.00,0.00,0.00,0.00,0.00,0.00,0.00

results/deviation_num_injections_dims.tex

@@ -0,0 +1,24 @@
+
+\pgfplotstableread[col sep=comma]{results/deviation_num_injections_dims.csv}\deviationsnuminjectionsdims
+
+\begin{tikzpicture}
+    \begin{axis}[
+        width=.95\textwidth,
+        xlabel={Deviation from max runs (\% points)},
+        ylabel={Number Simulations},
+        ylabel near ticks,
+        xmin=-2, xmax=2,
+        ymin=2000, ymax=26000,
+        scaled y ticks=false,
+        grid=both,  % Add a grid for better readability
+    ]
+    
+        \addplot table[x=Timing deviation, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Value failure, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Coding failure, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Glitch, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Deadlock, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Token count failure, y=Total simulations] {\deviationsnuminjectionsdims};
+        \addplot table[x=Nothing, y=Total simulations] {\deviationsnuminjectionsdims};
+    \end{axis}
+\end{tikzpicture}

results/deviation_num_signals_dims.csv

@@ -0,0 +1,7 @@
+Selected signals,Timing deviation,Value failure,Coding failure,Glitch,Deadlock,Token count failure,Nothing
+100,0.00,0.00,0.00,0.00,0.00,0.00,0.00
+90,-0.61,-1.00,-1.37,0.09,-0.38,-0.37,0.73
+70,0.37,-0.81,-0.02,-1.01,0.02,-0.33,-0.50
+50,0.00,0.13,-1.00,1.11,-0.19,0.16,0.46
+30,0.00,0.13,-1.00,1.11,-0.19,0.16,0.46
+10,0.00,0.13,-1.00,1.11,-0.19,0.16,0.46

results/deviation_num_signals_dims.tex

@@ -0,0 +1,24 @@
+
+\pgfplotstableread[col sep=comma]{results/deviation_num_signals_dims.csv}\deviationsnumsignalsdims
+
+\begin{tikzpicture}
+    \begin{axis}[
+        width=.95\textwidth,
+        xlabel={Deviation from max runs (\% points)},
+        ylabel={Number Simulations},
+        ylabel near ticks,
+        xmin=-2, xmax=2,
+        ymin=0, ymax=100,
+        scaled y ticks=false,
+        grid=both,  % Add a grid for better readability
+    ]
+        
+        \addplot table[x=Timing deviation, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Value failure, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Coding failure, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Glitch, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Deadlock, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Token count failure, y=Selected signals] {\deviationsnumsignalsdims};
+        \addplot table[x=Nothing, y=Selected signals] {\deviationsnumsignalsdims};
+    \end{axis}
+\end{tikzpicture}