198     #   APPEND the notenum into list TmovementChannelToTonic[key][0].
207     #   APPEND the notenum into list TmovementChannelToTonic[key][0].

SHOULD SAY:

198     #   APPEND the (notenum%12) into list TmovementChannelToTonic[key][0].
207     #   APPEND the (notenum%12) into list TmovementChannelToTonic[key][0].

That modulo 12 operator has to be in there to discard the octave (how far up the piano keyboard), which is irrelevant to our analysis.

Also if you cannot find CSC523Fall2022TimeMIDIOut.sorted.txt.ref, it is here:

~parson/DataMine/CSC523Fall2022TimeMIDI/CSC523Fall2022TimeMIDIOut.sorted.txt.ref

I fixed both problems & re-zipped the handout .zip file, but if you got an earlier version, you'll need this info.

    # ADDED 11/5 if you get an error ModuleNotFoundError: No module named 'xgboost', just comment out the import of xgboost.
    #    Thanks to Bob Elward for the catch. I updated the handout code late 11/4.

These are the attributes we will use in this analysis in fall2022concert_train.arff.gz and fall2022concert_test.arff.gz.
We discussed time-series data analysis and this dataset on October 31.

@attribute movement numeric        # A movement of a musical piece, conceptually a song, numbered 0 through 3.
@attribute channel numeric            # A MIDI channel, conceptually a musician playing an instrument, numbered 0 through 3.
@attribute command {noteon, noteoff}    # Whether the musician has just played or stopped playing a note.
                                                                 # handout code filters out noteoff because it adds no information to scale analysis,
                                                                 # then removes attributes command and velocity. It must wait to remove attribute
                                                                # command until it has removed noteoff instances so it can find them.
@attribute notenum numeric            # The note 0 through 127 being played. Think of a piano keyboard.
@attribute velocity numeric              # How hard the note is played, irrelevant for scale analysis.
@attribute tick numeric                    # The time with the (movement, channel) sequence, needed for note lagging
@attribute ttonic numeric                 # The so-called "do note" or tonic or root, which is the key pitch of the scale.
                                                         # Initially ttonic is tagged data by the score generator.
                                                         # Handout code derives an empircal ttonic by taking the statistical mode of the notes
                                                         # played in a given (movement, channel).
@attribute tmode {Ionian,Mixolydian,Lydian,Aeolian,Dorian,Phrygian,Locrian,Chromatic} # scale being played



Figure 1: Distribution of tmode in the training dataset

We will classify tmode from other attributes. The test dataset has an identical but independent distribution to the training data. My genmidi.py Jython script generated training and testing data using different pseudo-random number seeds with distributions seen in that link. You do not need to understand the music theory in genmidi.py to do the assignment.



Figure 2: Distribution of normalized notes in the original data colored by correlation to tmode

In Figure 2 a normalized note is the notes distance from the tagged ttonic, which is the generator's intended "do" note in the tmode. My preparation script extracts an actual "do" note as the statistical mode (most frequently occurring value) of the normalized notes. Here is some diagnostic output from that script.

Computed and tagged tonic by (movement,channel):
('test', 0, 0) -> [7, 7]
('test', 0, 1) -> [7, 7]
('test', 0, 2) -> [7, 7]
('test', 0, 3) -> [7, 7]
('test', 1, 0) -> [9, 9]
('test', 1, 1) -> [9, 9]
('test', 1, 2) -> [9, 9]
('test', 1, 3) -> [9, 9]
('test', 2, 0) -> [7, 7]
('test', 2, 1) -> [7, 7]
('test', 2, 2) -> [7, 7]
('test', 2, 3) -> [10, 7]
('test', 3, 0) -> [7, 7]
('test', 3, 1) -> [7, 7]
('test', 3, 2) -> [7, 7]
('test', 3, 3) -> [7, 7]
('train', 0, 0) -> [7, 7]
('train', 0, 1) -> [7, 7]
('train', 0, 2) -> [7, 7]
('train', 0, 3) -> [7, 7]
('train', 1, 0) -> [9, 9]
('train', 1, 1) -> [9, 9]
('train', 1, 2) -> [9, 9]
('train', 1, 3) -> [9, 9]
('train', 2, 0) -> [7, 7]
('train', 2, 1) -> [7, 7]
('train', 2, 2) -> [7, 7]
('train', 2, 3) -> [1, 7]
('train', 3, 0) -> [7, 7]
('train', 3, 1) -> [7, 7]
('train', 3, 2) -> [7, 7]
('train', 3, 3) -> [7, 7]

All extracted tonics are the same as the tagged tonics except for these two: