What does “temperature” on the Y-axis refer to?
I’m not a hundred percent sure, but afaik it has to do with how random the output of the GPT model will be. At 0 it will always pick the most probable next continuation of a piece of text according to its own prediction. The higher the temperature, the more chance there is for less probable outputs to get picked. So it’s most likely to pick 42, but as the temperature increases you see the chance of (according to the model) less likely numbers increase.
This is how temperature works in the softmax function, which is often used in deep learning.
Super helpful, thanks!
https://youtu.be/wjZofJX0v4M your answer from the 22:00 mark on.
37 is well represented. Proof that we’ve taught AI some of our own weird biases.
What’s special about 37? Just that it’s prime or is there a superstition or pop culture reference I don’t know?
Watch this:
I didn’t know either, but it seems to be an often picked ‘random’ number by people. Here is an article about it, I didn’t read it though.
https://youtu.be/d6iQrh2TK98?feature=shared
Just a number dumb monkeys believe to be “more random”.
Probably just because it’s prime. It’s just that humans are terrible at understanding the concept of randomness. A study by Theodore P. Hill showed that when tasked to pick a random number between 1 and 10, almost a third of the subjects (n was over 8500) picked 7. 10 was the least picked number (if you ditch the few idiots that picked 0).
Maybe randomness is a label we slapped on shit we don’t understand.
I remember watching a lecture about probability, and the professor said that only quantum processes are really random, the rest of things that we call random is just the human inability to measure the variables that affects the random variable. I’m an actuarie, and it’s made me change the perspective on how I see and study random processes and how it made think on ways to influence the outcome of random processes.
…which is kind of a hilarious tautology, because “quantum processes” are by definition “processes that we are unable to decompose into more basic parts”.
The moment we learn about some more fundamental processes being the reason for a given process, it stops being “quantum” and the new ones become “it”.
Even quantum just appears random I think. it’s beyond our scope of perspective, it works in multiple dimensions. we only see part of the process. That’s my guess though it could be totally wrong
it’s a matter of interpretation, but generally the consensus is that quantum measurements are truly probabilistic (random), Bell proved that there can’t be any hidden variables that influence the outcome
Interpretation for sure. Bells theory and then it being proven winning a Nobel prize to me only proves more we really don’t understand the world around us and only perceive what we need to survive. And that maybe we should be less standoffish to ideas that change our current paradigm, because we obviously have a lot to learn.
Didn’t Bell just put that up as a theory and it got proven somewhat recently by other researchers? The 2022 physics Nobel Prize was about disproving hidden variables and they titled their finding with the catchy phrase “the universe is not locally real”.
If you discount the pop-culture numbers (for us 7, 42, and 69) its the number most often chosen by people if you ask them for a random number between 1 and 100. It just seems the most random one to choose for a lot of people. Veritasium just did a video about it.
37 is my favorite, because 3x7x37=777 (three sevens), and I think that’s neat.
Wrong. Two hints:
7x7=9 at the end, not 7.
30x30=900, already more than 777.
One hint: 3x7=21, 21x37=777.
When in doubt, use a calculator.
Oh I am sorry. I did not see the x sign between 3 and 7. Lol.
? My calculator definitely thinks that 3x7x37=777. Did you read it as 37x37 instead?
Yes. Thanks. Sorry.
You don’t even need a calculator for a quick calculation, take the closest value of 10: 3x7=21x37 or easier 20x40 = 800 which is close to the actual number, 777.
Another fun fact: if you ask people to pick 2/3rds of a number everyone else picks when asked the same question, the correct number is drumroll 24.
I don’t like the inclusion of 37%, it’s 1/e that isn’t even 37%, is only that because of a pretty arbitrary rounding. Veritasium videos are usually OK, but this one is pretty meh.
Sorry but pop culture from were? I don’t recognize any of those numbers.
Lucky number 7.
42 is the meaning of life in The Hitchhikers Guide to the Galaxy.
And 69…nice!
I’m guessing this is for US and UK culture? Probably a lot of other former and current English colonies
It’s not the meaning of life. It’s the Ultimate Answer to Life, the Universe, and Everything. Nobody knows what the Question is.
Thanks. I Borked that one up
Is there some human sciences theory as to why?
Thanks!
What about 57
Heinz Ketchup?
I think you mean heinz 57 the steak sauce…
not this again.
it’s ketchup mfer, 57 varieties of tomatoes!
Unsolicited fact: Heinz picked the number 57 at random, it just sounded like good marketing at a time when things were general marketed as “tonic #4” and the like.
(well, maybe not fact, more like probable truth)
I’m curious about that too. Something is twisting weights for 57 fairly strongly in the model but I’m not show what. Maybe its been trained on a bunch of old Heinz 57 varieties marketing.
Wesley Snipes
My art professor wrote a book about famous artists and thinkers dying at 37: Raffaello, Parmigianino, Valentin de Boulogne, Cantarini, Watteau, Van Gogh, Toulouse-Lautrec, Tancredi, Gnoli, Manai, Majakovskij, Rimbaud, Byron, Mozart, Robespierre
https://www.ibs.it/trentasette-mistero-del-genio-adolescente-libro-flavio-caroli/e/9788804734017
Not a great book tbh.
Only dudes, though, right?
Why would that need to be proven? We’re the sample data. It’s implied.
The correctness of the sampling process still needs a proof. Like this.
What you’ve described would be like looking at a chart of various fluid boiling points at atmospheric pressure and being like “Wow, water boils at 100 C!” It would only be interesting if that somehow weren’t the case.
Where is the “Wow!” in this post? It states a fact, like “Water boils at 100C under 1 atm”, and shows that the student (ChatGPT) has correctly reproduced the experiment.
Why do you think schools keep teaching that “Water boils at 100C under 1 atm”? If it’s so obvious, should they stop putting it on the test and failing those who say it boils at “69C, giggity”?
Derek feeling the need to comment that the bias in the training data correlates with the bias of the corrected output of a commercial product just seemed really bizarre to me. Maybe it’s got the same appeal as a zoo or something, I never really got into watching animals be animals in a zoo.
Hm? Watching animals be animals at a zoo, is a way better sampling of how animals are animals, than for example watching that wildlife “documentary” where they’d throw lemmings of a cliff “for dramatic effect” (a “commercially corrected bias”?).
In this case, the “corrected output” is just 42, not 37, but as the temperature increases on the Y axis, we get a glimpse of internal biases, which actually let through other patterns of the training data, like the 37.
“we don’t need to prove the 2020 election was stolen, it’s implied because trump had bigger crowds at his rallies!” -90% of trump supporters
Another good example is the Monty Hall “paradox” where 99% of people are going to incorrectly tell you the chance is 50% because they took math and that’s how it works.
Just because something seems obvious to you doesn’t mean it is correct. Always a good idea to test your hypothesis.
Trump Rallies would be a really stupid sample data set for American voters. A crowd of 10,000 people means fuck all compared to 158,429,631. If OpenAI has been training their models on such a small pool then I’d call them absolute morons.
A crowd of 10,000 people means fuck all compared to 158,429,631.
I agree that it would be a bad data set, but not because it is too small. That size would actually give you a pretty good result if it was sufficiently random. Which is, of course, the problem.
But you’re missing the point: just because something is obvious to you does not mean it’s actually true. The model could be trained in a way to not be biased by our number choice, but to actually be pseudo-random. Is it surprising that it would turn out this way? No. But to think your assumption doesn’t need to be proven, in such a case, is almost equivalent to thinking a Trump rally is a good data sample for determining the opinion of the general public.
What’s the y axis?
The temperature scale, I think. You divide the logit output by the temperature before feeding it to the softmax function. Larger (resp. smaller) temperature results in a higher (resp. lower) entropy distribution.
I don’t understand any of these words, I need to take a math class or something
Higher temperature -> more chaotic output
I still don’t understand.
More yellow more common, more blue less common
Temperature is basically how creative you want the AI to be. The lower the temperature, the more predictable (and repeatable) the response.
HA, funny that this comes up. DND Beyond doesn’t have a d100, so I opened my ChatGPT sub and had it roll a d100 for me a few times so I could use my magic beans properly.
I use the percentile die for that.
Also an excellent method.
Opened up DND Beyond to check since i remember rolling it before and its there, its between D8 and D10, the picture even shows 2 dice
That’s helpful. Thank you.
But why use Chatgpt for that? Why not a duck duck go action? I just don’t understand why we’re asking a LLM whose goal is consistency, not randomness, to do random
Roll two d10, once for each digit, and profit?
I guess you’d need 10 to represent 0, and if you got 2x 10 that would be 100?
Yup! Also one has to mind the order in which one rolls the dice. Since 10 and 5 could be either 05 or 50. As a bonus, if you roll them in order of “tens” to “ones”, getting 10 on the first dice has added suspense since the latter dice determines if it is going to count as a low roll of 0X (by rolling 1-9 on the next dice X) or if it is going to be a max roll of 100 (by rolling another 10).
holy crap, the answer to life the universe and everything XD
More than likely it’s because of that book and how often it’s qouted
Yes, but it’s significant because the prompt was to choose a number. I realize computers can’t really be random, but if we needed to just select a popular number…we can already do that!
Computers can be random with special hardware.
Care to elaborate?
There are devices that measure radioactive decay for operations where truly random numbers are very important. Or something like that, I am not an expert, sorry.
Interesting. As I understand it, pure computing (not sensors recording external data) are incapable of generating truly random numbers. But I’m obviously not an expert either!
I’ve been using “Perfect Passwords” for years, which apparently generate nearly random passwords from server noise, but he admits it’s still not truly 100% random…
Cloudflare uses a Livestream of lava lamps for their randomness in their encryption. That being said I’m not sure how uniquely random this is.
https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/
37
In a row?!
Try not to suck any dick on the way to the parking lot!
Which model?
When I tried on ChatGPT 4, it wrote a short python script and executed it to get a random integer.
import random # Pick a random number between 1 and 100 random_number = random.randint(1, 100) random_numberdoes the neural network actually run scripts or is it pretending
It generates code and then you can use a call to some runtime execution API to run that code, completely separate from the neural network.
That’s not answering the question though.
“Pick a number between 1 and 100” doesn’t mean “grab two d10” or write a script.
So what? It figured out The Answer, big whoop.
Get back to me when it figures out The Question.
NEEDS MOAR 69 FELLOW HUMAN
Well… I was not aware Chatgpt could make simple graphs.
A human made the graph
LMs aren’t thinking, aren’t inventing, they are predicting what is supposed to be answered next, so it’s expected that they will produce the same results every time
This graph actually shows a little more about what’s happening with the randomness or “temperature” of the LLM.
It’s actually predicting the probability of every word (token) it knows of coming next, all at once.
The temperature then says how random it should be when picking from that list of probable next words. A temperature of 0 means it always picks the most likely next word, which in this case ends up being 42.
As the temperature increases, it gets more random (but you can see it still isn’t a perfect random distribution with a higher temperature value)Except it clearly doesn’t produce the same result every time. You’re not making a good case for whatever you’re trying to say.
They add some fuzziness to it so it doesn’t give the exact same result. Say one gets a score of 90, another 85, and other 80. The 90 will be picked more often, but they sometimes let it pick the 85, or even the 80. It’s perfectly expected, and you can see that result here with 42 being very common, but then a few others being fairly common, and most being extremely uncommon.
I petition to rename ChatGPT to DeepThought based on these results.
I spent an afternoon once playing Infinite Craft, which uses some sort of LLM behind the scenes to do it’s combinations.
At one point I got 007, and found 007+007 = 0014.
The maths gets wild though, and because it’s been trained on text, it has no idea when it comes to combinations of numbers it hasn’t seen before. I spent ages trying to get it to 69420 and just couldn’t, although I could get 42069.
Only 1000 times? It’s interesting that there’s such a bias there but it’s a computer. Ask it 100,000 times and make sure it’s not a fluke.
Ask humans the same and most common numer is 37
For very different reasons though. 37 is what people think is the most random, because humans are dumb. The LLM here tried to choose the most likely.
HOW DID THE TRUCK GET INTO SPACE??
Love that episode though.
I saw that YouTube video as well.
Hello Veritasium enjoyer
In his video, he shows that the more common answers are actually 42 and 69.
I discards them because they’re picked for a reason rather than a human genuinely trying trying to pick a random number, but they’re still way more common than 37.
That’s because they asked the internet for those polls. The internet thinks they’re funny by picking the meme numbers. So I can understand why they chose to omit those numbers from their results.
What are you referring to?
Most probably this: https://www.youtube.com/watch?v=d6iQrh2TK98
Thanks, I’ll have a look
YouTube STEM educator. 15 million subscribers. Probably in the top 5 STEM educators on the platform.
He released a video on the number 37 two weeks ago, with 6 million views.
I know veritasium but I hadn’t seen the video. Thanks, I’ll check it out.
I thought I’d give you context just in case, as your question was vague. You might not have consumed YouTube and was blissfully unaware. :)
