OUR METHODS

MODEL DEVELOPMENT

1. Sample some data. The data could be the tweets related to energy in Canada that have been scraped, and/or it could be some datasets that we have gathered, such as the blog authorship corpus from Kaggle. The sampling method is determined by the classification task, such as universal personal values classification.
2. Split the sample into train and test sets. The test set will be loaded into Label Studio (not pictured) for annotation from domain experts. The train set will be sent to the data programming pipeline.
3. This command runs the data programming pipeline. It can be configured to select train parameters for the label model, whether or not there is a labeled test set created in step 2 to use in calculating class balance and further evaluating the labeling function and label model accuracy, the train set created in step 2, and the classification task that the train data is for.
4. The labeling functions use the LF Resources to heuristically label the train data. Each labeling function votes for its respective label, generating the columns of the label matrix, over all of the data points in the train data, generating the rows of the label matrix.
5. This label matrix is used as input to the label model, which models the correlation and conflicts of the labeling functions, de-noising the heuristic labels and approximating the true label.
6. The label model generates two new columns for the train set, label and probs. Label is the predicted true label for that data point, and probs is the probability distribution over all classes, where the label model selects the class with max probability in the multiclass case and classes above the knee in the probability distribution in the multilabel case. This train data is then evaluated for completeness, and all artifacts from the data programming pipeline including the label model metrics and labeling function summary are available to view using the Mlflow UI.
7. Once it is determined that the train data is of good quality, it is fed into the classification pipeline, which is run with this command. It can be configured to use different algorithms such as multinomial logistic regression and multilayer perceptron, depending on the classification task and whether it is multilabel or multiclass.
8. The feature matrix is some combination of feature vectors generated by algorithms such as the Universal Sentence Encoder and TF-IDF, where the desired features are chosen at step 7. Feature matrices are created for the train set and test set. The test set will ideally be the human-annotated data to measure external validity, but in the case that manual annotation has not been completed, the train data can be split into train and test sets (and test can further be divided into test and validation if needed for the classification algorithm, such as for early stopping).
9. The train set feature matrix created in step 8 and the train labels or probs (depending on the algorithm) created in step 6 are used as input to the classification algorithm chosen at step 7. The end model that is created through the training process is then validated using the test data. As with the data programming pipeline, all metrics and artifacts can be viewed using the Mlflow UI. The classification steps 7-9 are repeated until a good quality model is obtained.
10. Once a good model of the classification task is obtained, it is served with a dedicated prediction API endpoint for that particular classification task. This endpoint is used to obtain predictions for all relevant datapoints.

Using this model creation process, we can quickly include new data types such as images, test new LF Resources, add or modify classification tasks, and experiment with features to feed into the model, all without being constrained by available labelled data or existing models. This process is particularly useful for a multimodal approach to studying a variety of communication concepts where the data is constantly evolving.