This week we will look at getting some general intuition into the complex topic of machine learning. A number of readings are assigned to this week, all quite technical (and many pivotal in the development of AI). My strong recommendation will be to watch the Grant Sanderson video – a master class in exposition from an expert in conveying visual intuitions around the mathematics of AI – and then download and scan or skim-read as many of these papers as possible. I would spend a little time on (a) Markov, (b) Mikolov et al., and (c) Vaswani et al, as each of these papers marks a critical conceptual as well as technical transition in the history of generative AI.
For people without technical backgrounds, do not worry if none of this makes any sense! While a proper understanding of deep learning would require many courses, we will be breaking things down in a way that hopefully provides some intuition in class.
Viewings / Readings:
*Highly recommended* Sanderson, G. (2024). Visualizing transformers and attention | Talk for TNG Big Tech Day ’24, https://www.youtube.com/watch?v=KJtZARuO3JY
Church, K. W., & Mercer, R. L. (1993). Introduction to the Special Issue on Computational Linguistics Using Large Corpora. Computational Linguistics, 19(1), 1-24. https://aclanthology.org/J93-1001/
Cope, B., & Kalantzis, M. (2023). Generative AI Comes to School (GPT and All That Fuss): What Now? Educational Philosophy and Theory, 13-17. https://doi.org/10.1080/00131857.2023.2213437
Markov, A. A. (2006). An example of statistical investigation of the text Eugene Onegin concerning the connection of samples in chains. Science in Context, 19(4), 591-600. https://alpha60.de/research/markov/DavidLink_AnExampleOfStatistical_MarkovTrans_2007.pdf
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. https://arxiv.org/abs/1310.4546
Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11), 39-41. https://dl.acm.org/doi/10.1145/219717.219748
Radford, A. (2018). Improving language understanding by generative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379-423. https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017 [2023]). Attention Is All You Need. arXiv, 1706.03762. https://doi.org/10.48550/arXiv.1706.03762
Assignment(s):
In your respective Google Sheet tab, respond to one or more of Week 3’s questions.