This isn’t “I want to believe”, this is “it would be irresponsible to not consider”.

  • 0 Posts
  • 4 Comments
Joined 1 year ago
cake
Cake day: September 3rd, 2023

help-circle

  • Fun fact, when the jet stream gets perturbed like that and develops the sinusoidal deviations that we are experiencing, it’s called a Rossby wave.

    These waves are actually super normal as the jetstream shifts with the seasons and moves north/south, especially when in a La Niña phase of the ENSO, which we are in right now.

    The Hadley circulation cells whose boundaries define the jet stream are driven by convection. The US lies right along a jetstream boundary between two cells, and just downwind from the pacific ocean, so our weather is particularly sensitive to the temperature differences across the pacific ocean.

    El Niño patterns have a hot equatorial pacific ocean which drives significant convection on the southern cell of the jet stream crossing the US, stabilizing it. La Niña patterns have a smaller gradient between the temperatures in the cells to the north and south of the relevant jet stream, especially as climate change relatively warms the arctic faster, leading to higher amplitude destabilizations during La Niña patterns like we are experiencing now.

    More fun facts about these Rossby waves: they have been proposed as the mechanism to drive the eddies that end up forming planets in protoplanetary disks around baby stars (see the wikipedia page for Rossby waves above), and as the mechanism behind the hexagonal shape of Saturn’s polar cell. Worth noting that the exact mechanism for that hexagon is still highly debated, but Peter Gierasch used to have a fun model using a modified record turn table to create a rossby wave that formed a hexagon as a proof-of-concept that has stuck with me.


  • Try asking one to write a sentence that ends with the letter “r”, or a poem that rhymes.

    They know words as black boxen with weights attached for how likely they are to appear in certain contexts. Prediction happens by comparing the chain of these boxes leading up to the current cursor and using weights and statistics to fill in the next box.

    They don’t understand that those words are made of letters unless they have been programmed to break each word down into its component letters/syllables. None of them have been programmed to do this because that increases the already astronomical compute and training costs.

    About a decade ago I played with an LLM whose markov chain did predictions based on what letter came next instead of what word came next (pretty easy modification of the base code). It was surprisingly comparably good at putting sentences and grammar together when working at the letter-scale. It also was horribly less efficient to train (which is saying something in comparison to word-level prediction LLMs) because it needs to consider many more units (letters vs words) leading up to the current one to maintain the same coherence. If the markov chain was looking at the past 10 words, a word-level prediction has 10 boxes to factor into its calculations and trainings. If those words have an average of 5 letters, then letter-level prediction needs to consider at least 50 boxes to maintain the same awareness of context within a sentence/paragraph. This is a five-fold increase in memory footprint, and an even greater increase in compute time (since most operations are at least of linear order and sometimes more).

    That efficiency hit would allow for LLMs to understand sub-word concepts like alphabetization, rhyming, root words, etc. The expense and energy requirements aren’t worth this modest expansion of understanding.

    Adding a General Purpose Transformer just adds some plasticity to those weights and statistics beyond the markov chain example I use above.


  • NASA has led hard science employers in diversity metrics since its founding. They hire the best that they can, regardless of their identity. They’ve hired fascists like Von Braun. They’ve hired Hidden Figures like Katherine Johnson. There is a good write-up with facts and figures in the intro to the book version of Hidden Figures. NASA in the 1960s was a moonshot to combat racist hiring practices in STEM in the South. NASA’s strategies for hiring the best from any background worked, and they kept working in spite of inconsistent support from the white house in later decades.

    That’s what I used to say. Last few years were rough, and now I’m unemployed largely due to a mismanagement of DEI-like programs that is historically uncharacteristic of NASA. I know that it’s only anecdotal evidence, but as a minority misserved by these programs, I have some sour feelings towards them. I hope that removing them will allow NASA to focus on its previous diversity strategies that worked so well in decades past, including when I was first coming up through the system during the Obama administration.

    Diversity, Equity, Inclusion, and Accessibility are fabulous values that make science and engineering products better. NASA’s wonder and its execution on that wonder changes the world. DEIA values make sure that those changes do the most good and help the most people. DEIA at its best is a strategy of listening.

    All that said, I find it hard not to be a little happy at seeing that the latest iteration of these programs are getting rethought, even if it may take four years before anyone is allowed to try rebuilding them. They had become punitive and more interested in exclusion of “wrong thinking” than inclusion of diverse and different viewpoints. No more DEI is a bad thing. No more DEI Police is a good thing.