Characteristic |
InConf |
Mid-InConf |
Mid-NonConf |
OutConf |
EndOfRoute |
p-value 2 |
|---|---|---|---|---|---|---|
| LateralDeviation | 0 (0, 0) | 1 (-1, 1) | 1 (-1, 1) | -2 (-5, 4) | 1 (-14, 17) | <0.001 |
| CorrectionAngle | 1 (1, 3) | 2 (1, 5) | 3 (1, 17) | 13 (2, 52) | 181 (181, 181) | <0.001 |
| WMA | 0 (0, 0) | 0 (-1, 1) | 0 (-1, 1) | -1 (-5, 4) | 1 (-14, 17) | <0.001 |
| Abs_WMA | 0 (0, 0) | 1 (1, 1) | 1 (1, 1) | 4 (2, 11) | 15 (7, 28) | <0.001 |
| ma4_lateraldev | 0 (0, 0) | 0 (-1, 1) | 0 (-1, 1) | -1 (-5, 4) | 1 (-14, 16) | <0.001 |
|
1
Median (Q1, Q3) |
||||||
|
2
Kruskal-Wallis rank sum test |
||||||
Aicraft’s Lateral Deviation - Part 2
svm
naïve bayes
gbm
knn
ctree
rpart
Assessment of different classification algorithms and their respective results
Introduction
As previously explained on Aircraft’s Lateral Deviation - Part I, understanding of an aircraft lateral intent is vital across the air traffic industry for safety, and flight efficiency among other factors. This task can easily become unwieldy when triaging millions of measurements per hour across the entire National Air Space (NAS). On this section, I present a variety of potential predictive models for the classification of an aircraft’s lateral intent, and which could improve automated detection systems across this field.
Disclaimer: The views and opinions expressed in this report are those of the author and do not necessarily reflect the views or positions of any of the entities herein referred.
Original Data
The first step was to ingest the tidy data generated in Part I. Note, as showing below the data class label with the least number of observations is “EndOfRoute” with ~85,000; indicating a potential option to downsample the data before fitting the models.
Scatter Plot of Matrices (SPLOM)
Pre-processing
Before proceeding to model training, I took three specific actions to ensure the proper conditioning of the data. First, I scaled the numerical attributes. The main objective of scaling the data was to address skewedness and data outliers. Second, I normalized the data to enhance the model performance. Normalization is critical to prevent any attributes with larger scales to dominates the model learning process. Lastly, I finished conditioning the data by downsampling it so all classes would end up with the same frequency as the minority class.
For duplication purposes, I utilized 10% of the conditioned data.
summary(sampled_df) LateralDeviation CorrectionAngle WMA Class
Min. :0.009301 Min. :0.0000002 Min. :0.00932 InConf :8517
1st Qu.:0.555604 1st Qu.:0.0056313 1st Qu.:0.55566 Mid-InConf :8517
Median :0.556482 Median :0.0267311 Median :0.55653 Mid-NonConf:8517
Mean :0.556627 Mean :0.2676967 Mean :0.55667 OutConf :8517
3rd Qu.:0.557358 3rd Qu.:0.3721161 3rd Qu.:0.55741 EndOfRoute :8517
Max. :0.935658 Max. :1.0000000 Max. :0.93549 Data Partitioning
Post-conditioning of the data, I applied a train-test split method in preparation to properly evaluate the performance of the models. The idea here is to divide the dataset into three distinctive subsets, training, testing, and cross-validating, and observe how the model generalize across these.
# to create a 50-30-20 subsets
set.seed(12345)
# training set with 50% of the data
trainIndex <- createDataPartition(sampled_df$Class, p = 0.50, list = FALSE)
train <- sampled_df[trainIndex, ]
# remaining 50%
remaining <- sampled_df[-trainIndex, ]
# test set (30%)
testIndex <- createDataPartition(remaining$Class, p = 0.60, list = FALSE)
test <- remaining[testIndex, ]
# cross-validation (20%)
crossval <- remaining[-testIndex, ]
# preview of the partitions
table(train$Class)
InConf Mid-InConf Mid-NonConf OutConf EndOfRoute
4259 4259 4259 4259 4259 table(test$Class)
InConf Mid-InConf Mid-NonConf OutConf EndOfRoute
2555 2555 2555 2555 2555 table(crossval$Class)
InConf Mid-InConf Mid-NonConf OutConf EndOfRoute
1703 1703 1703 1703 1703 Classification Models
Conditional Inference Tree
One of the implemented classification models was a conditional inference tree (CTREE). The CTREE method follows a “recursive partitioning framework” to split each evaluated class and “the outcome takes place based on the measured p-value of association between the observations” at hand. As an example of the steps taken for most of the models, here is a copy of the code and graphic.
set.seed(12345)
myFormula <- Class ~.
# start.time <- Sys.time()
model <- partykit::ctree(myFormula, control=ctree_control(maxdepth= ), data=train)
# end.time <- Sys.time()
# time.taken <- end.time - start.time
# train model
# table(predict(model), train$Class, dnn=c("PREDICTED", "ACTUAL"))
modelaccur <- sum(predict(model) == train$Class) / length(train$Class)
# prop.table(table(predict(model), train$Class, dnn=c("PREDICTED", "ACTUAL")))
# test model
model2 <- ctree(myFormula, data = test)
# table (testPred2, test$Class, dnn=c("PREDICTED", "ACTUAL"))
testPred2 <- predict(model2, data = test, method="class")
model2accur <- sum(testPred2 == test$Class)/length(test$Class)
# prop.table(table(predict(model2), test$Class, dnn=c("PREDICTED", "ACTUAL")))
# cross-val model
model3 <- ctree(myFormula, data = crossval)
# table (testPred3, crossval$Class, dnn=c("PREDICTED", "ACTUAL"))
testPred3 <- predict(model3, data = crossval, method="class")
model3accur <- sum(testPred3 == crossval$Class)/length(crossval$Class)
# prop.table(table(predict(model3), crossval$Class, dnn=c("PREDICTED", "ACTUAL")))For the CTREE depiction I used the partykit::ctree R package(Hothorn and Zeileis 2015). Here’s a glimpse of how the cross-validation model performs and shows the distribution of classes in the terminal nodes. The graphic greatly helps conceptualizing the model and provides a clearer image of the lateral deviations and status changes relationship.
plot(model3, type="extended", ep_args = list(justmin = .01),
drop_terminal=T, tnex= 1.3, gp=gpar(fontsize = 8, col="darkblue"),
inner_panel = node_inner(model, fill=c("white","green"), pval=T),
terminal_panel= node_barplot(model, fill=c(1:5), beside=T,
ymax=1.05, rot = 90, just = c(.95,.5),
ylines=T, widths = .90, gap=0.05, reverse=F, id=T),
margins = c(6,5, 5, 4), main ="Conditional Inference Tree\nLateral Adherence Status")
Naïve Bayes
The second classification model –Naïve Bayes (NB)—follows Thomas Bayes’ theorem of probabilistic classification judging the relationship between the probabilities of different events and their conditional probabilities. Per this theorem, the data is used to fit the model, assuming “the encompassing predictors are independent of the target variable classes”(Majka 2024). One note about NB was the capacity of the model to evaluate all the instances in a swiftly 1.60 seconds.
Multiclass Support Vector Machine (SVM)
The third model, a SVM was adapted using the one-versus-one approach, “C-classification” type, to assess the multiple classes within the observations. The kernel used for the calculation was a “radial”, the data was scaled, and the cost and gamma values were kept at default values with 10 and 0.1 respectively. It resulted in identifying 3,814 support vectors, reaching an overall accuracy, recall, precision, and F1 of 97%
Gradient Boosting Machine (GBM)
The fourth predicted model is an adoption of the typical ensemble model of generalized boosted regression modeling known as Stochastic Gradient Boosting Machine. This model is particularly “known as one of the most powerful techniques for building predictive models”. Reportedly, GBM scopes optimal performance when dealing with significant disparity across observations and it is highly recommended to use it for anomaly detection cases. In essence, the GBM framework takes the error or loss, of the prior and employs it to adjust the weights of the sub-sequential trees, thus minimizing the follow-on errors.
Models Review
At the end, all models were compared and evaluated for best performance. Although, all the models performed relatively well, there were unique traits to arrange these models in three groups. Case in point, top performers based on overall accuracy were the GBM, CTREE, and SVM-M. The second group included NB, and K-Nearest Neighbor (KNN). Finally, the third group with Recursive Partitioning and Regression Tree (RPART). The bellow image displays a comparison of models based on accuracy, and alternatively, the models’ classification error metrics.
Overall Performance Metrics
Overall, four key metrics were taken in consideration, specificity, sensitivity, kappa, and F1 score. As displayed, the classification models graph portray how each model performed, underlining different alternatives to further develop in support of the main objective of auto-classifying in-flight aircraft lateral deviations. The champion in this occasion was GBM with the highest probabilities across the metrics.
Summary and Recommendations
Essentially the project exposed a variety of features of importance regarding the examination of an aircraft lateral variability. In summary, deviation across some ARTCCs like ZLA and ZMA were more prevalent than others, particularly the ZAU or ZNY case. Also, the study demonstrated that the majority of the observations were within three standard deviations from the mean, indicating a consistency of data points across the distributions. Such insights were instrumental explaining inferences from the analysis across the large population. Moreover, the study validated the correlation and relative influence of the correction angle attribute [originally named Angle2NextFix] and the absolute weighted moving average of the lateral deviation [known as redLat] as a principal component or driver in the classification of the different lateral statuses.
As mentioned, the completion of this assessment highlights some successes and potential benefits to consider in order to improve the FAA’s decision support tools. However, the shallow level of the study is implicit and further exploration of the data, and validation, is highly advisable and/or testing of the models against operational data. Additional features to correlate could add value to a final proposition and utility of the assessment. Also, other data like GPS data, lat/log coordinates, weather data, approved flight plan details or supplemental sensor measurements could maximize new insights.
References
Hothorn, Torsten, and Achim Zeileis. 2015. “Partykit: A Modular Toolkit for Recursive Partytioning in r” 16: 3905–9. https://jmlr.org/papers/v16/hothorn15a.html.
Majka, Michal. 2024. “Naivebayes: High Performance Implementation of the Naive Bayes Algorithm in r.” https://CRAN.R-project.org/package=naivebayes.