See all Y Combinator transcripts on Youtube

Speaker 1

00:00

You guys are brothers, right? Yeah, we are.

Speaker 2

00:02

Yeah, okay.

Speaker 3

00:03

He's the older 1, I'm 2 years younger.

Speaker 1

00:06

Okay, and what made you want to start Format's library?

Speaker 3

00:11

So, just for the people that don't know what it is, Format is a platform for annotating papers. And so if you want to think about it, you imagine a PDF view in your browser, and then you have annotations on the side that support LaTeX and Markdown. And so you can add annotations in parts of papers that you think are particularly tough to understand or you think you could add more content there.

Speaker 3

00:36

But so it's something that we've done. The 4 of us that started Format, we all have a technical background. And so after college, we kept on reading papers. And every once in a while, we had this internal journal club where we would read a paper and present it to the others.

Speaker 3

00:56

So I remember, for instance, presenting a few years back, presenting the Bitcoin paper to Luis and Mika, which don't have a CS background. And so you kind of have to go into, for instance, for the Bitcoin, you might have to go into, OK, what's a hash function? What's public key encryption? And so we were already doing this.

Speaker 3

01:14

And we knew that you also have this behavior offline in places like universities. And so we wanted to take that experience and bring it online. And we thought there was a lot of content that you end up producing while you're trying to read a paper, which can be the most dense piece of content that a human can read sometimes, right? The language can be incredibly spartan, and sometimes there's a step in some paper that they say, oh, this should be obvious, but then you look at it, and it's like, okay, I don't get it.

Speaker 3

01:47

And so we knew that there was a lot of content there that you end up producing while trying to understand a paper and we wanted to bring that online.

Speaker 1

01:56

Because, Luiz, you were in physics

Speaker 2

01:58

before. I studied physics together with Mika and João and Taimo went to MIT. Taimo studied economics and you studied CS. So a lot of the papers are around physics, math, economics, biology, CS, right?

Speaker 1

02:16

Yeah. Yeah, because that was, you kind of like solved the cold start by just annotating yourself.

Speaker 2

02:20

Exactly. Right.

Speaker 1

02:21

And now it's more about getting the author in there.

Speaker 2

02:24

Exactly. That was the kind of the growth act.

Speaker 3

02:26

We start our first paper was the Bitcoin paper. Yep. And still

Speaker 1

02:30

the most commented. Right.

Speaker 3

02:31

Yeah. That 1 is a good number of comments. It has been there for the longest and it was quoted or just there are a bunch of news sites that have pointed back to it.

Speaker 1

02:40

Oh, okay.

Speaker 3

02:41

It's like, okay, if you wanna read it, go to the annotated version. But We had a few cool people comment there.

Speaker 2

02:50

Laurence Lessig commented on the Bitcoin paper.

Speaker 3

02:53

A bunch of people from the Bitcoin community Exactly. Commented there. Yeah.

Speaker 3

02:59

But the larger goal with Fermat is to try to move things in the right direction, meaning move science towards what people call open science. And so that encompasses a number of things from open data, which means just sharing the data that you've used for publishing whatever research you might be publishing. And you want to share that and make that easily accessible to people so that if they want to replicate the results that you got or use it in their own research, they have an easy time doing that. So that's open data.

Speaker 3

03:31

You also have just publishing the code that you used or the algorithms that you've used and making those more easily available to people. There's also open publishing, which means just publishing in papers that are not behind, or in journals that are not behind paywalls. So there's a lot of things that are within open science, all of those. And then there's also, so we want to push things in that direction and also try to build a platform that makes it easier for people to collaborate.

Speaker 3

04:03

And we think there are a lot of things that could be happening nowadays where people could be collaborating, scientists could be collaborating remotely a lot more than they are. Or that's at least the way we think. But it's starting to change where we've had for the paper, the Erdos.

Speaker 2

04:25

Yeah. I think this is actually a trend. We're seeing more and more people collaborating online around papers. So for instance, there is this famous example around a problem called the Erdos discrepancy.

Speaker 2

04:38

And this problem is a famous problem that was posed by Paul Erdos, which is like this famous mathematician 80 years ago. And Terence Tao, the field's medalist, was trying to solve the problem. And he put it on his blog that he was trying a certain approach to solve the problem. And then there was this guy from Germany that just wrote a comment there, like the size of a tweet.

Speaker 2

05:02

And he said that the Erdos problem had a Sudoku-like flavor, and that some of the machinery that they were using to solve the Sudoku problem could be used there. And that was actually the key to correct the problem. And they ended up publishing a solution to the Yardosh discrepancy problem, which was probably 1 of the biggest milestones in number theory in 2016. And that was all thanks to a comment on his blog and to the fact that they were collaborating online around solving that problem, which was also a polymath problem.

Speaker 2

05:36

The polymath project was a project started by these other Fields medalists called Tim Gowers. And they were trying to, it was actually a social experiment to see if it was possible to solve math problems online and collaborating around math problems online. And yeah, and they were able to solve it, thanks to that comment.

Speaker 3

05:57

Because you kind of see, right, You look at GitHub, and then you think of the impact that GitHub has had for open source. Open source, of course, existed much before GitHub. But it has really allowed a lot more people to come in and be able to get into open source and start contributing.

Speaker 3

06:16

And there are a number of other really interesting platforms. Right. You have Wikipedia just for more general knowledge or you have Stack Overflow for just programmers helping each other. And we think that there could be something similar to that.

Speaker 3

06:31

But for science in general. Right.

Speaker 1

06:33

Well because did you listen to the Rogan with Peter Atiyah?

Speaker 3

06:37

No. Parts of

Speaker 2

06:38

it. Mika listened to that.

Speaker 1

06:40

Yeah, that was a really good 1. And he talks about, I don't know if They're talking about the archive in particular around publishing papers but he talks about having full time staff.

Speaker 3

06:50

Oh yeah.

Speaker 1

06:51

Just scrubbing the data looking for interesting information coming out. And again, like in the context of Stack Overflow. That's the place where like programmers find specific answers to problems.

Speaker 1

07:04

Whereas with the archive, like good luck. Yeah. Good luck finding that stuff. Yeah.

Speaker 1

07:09

And so have you guys thought about addressing like just discoverability in the context of particular fields?

Speaker 3

07:17

The, it's a really tough problem. Like For instance, paper recommendations, it's really hard to.

Speaker 1

07:24

Because you're just doing 1 a week right now. Yeah. In addition to the browser extension.

Speaker 3

07:29

And We also have our tool that is used internally at universities and research groups for people that they're reading papers together and they add annotations. But for now, we have the weekly journal. So we release a paper every week that we select and we annotate it, or somebody in the community annotates it.

Speaker 3

07:49

And then we have the archive extension that adds a bunch of features on top of archive, like BibTeX extraction, reference extraction, and comments. And eventually, definitely, like recommendation engine and making it easier to discover papers that are relevant to you, that's something we definitely want to add onto our archive extension. But it's a tough problem.

Speaker 2

08:17

It is. Yeah, initially we started Fermats, as John said, as a journal club. And then we saw that people liked the interface, the commenting interface, and liked reading the annotation.

Speaker 2

08:30

So now we are starting to expand and turn formats into more of a platform. And that's why we decided to do the archive Chrome extension. Because archive, for people that don't know what it is, it's basically a place where papers leave before they go to journals in the form of preprints. So they are like drafts before they go to journals.

Speaker 2

08:55

And what we did is we built a Chrome extension that basically allows people to see all the commenting interface on archive papers. And so you don't have to go to another website. You're just reading archive papers, and you see the comments on the site if you have the Chrome extension installed.

Speaker 1

09:12

Well, and a lot of these papers don't even have comments on the page.

Speaker 3

09:14

They don't.

Speaker 1

09:15

Like, best case, you're emailing the author?

Speaker 3

09:17

Exactly. Yeah.

Speaker 1

09:18

Yeah. They don't have.

Speaker 2

09:20

So what Archive does, it's basically they just host papers. That's the core functionality of Archive. And so 1 of the things that we noticed is that, especially for areas like machine learning and deep learning, archive is super important.

Speaker 2

09:36

Because the new papers are coming out at such a high rate that people don't wait before the papers go to journals, before they start working on top of it and using the stuff that other people discover. So all the papers are published on archive. And so you need a way to distinguish good quality work from bad work if you are reading a paper on archive that hasn't been peer reviewed or something about machine learning. And I think that's why the Librarian Extension is so important in fields such as machine learning.

Speaker 1

10:13

So does the Librarian Extension have a rating mechanism as well? Like, how do you distinguish good from bad work?

Speaker 2

10:19

Right now, it's only through the comments. But we are actually thinking about implementing some sort of rating system for papers.

Speaker 3

10:28

And we're probably going to also, we've been thinking about that for a while now. We're probably going to run a few surveys to our audience to, because you could do it in a number of ways, like rating a paper, you could do it, obviously there's likes or dislikes or upvotes and downvotes. So you could either just have an holistic rating for the whole paper.

Speaker 3

10:49

You could also imagine rating it on a number of different aspects of the paper. It could be about, OK, how big is their data set if they're using some data set? Or what do you think about their methods? So you could have a more complex rating system.

Speaker 3

11:05

And so we've been thinking about that a lot. And we're just trying to figure out what makes the most sense there. But that's also definitely in, like, we would love to add that to Archive, or to the Chrome extension for archive.

Speaker 1

11:20

Yeah, so how do you think the collaboration plays out then? Because I understand how, you know, say for instance, you know, you're a physicist, you start commenting on someone else's paper, you start a discussion that creates a new project, right? Do you think you'll go further than there?

Speaker 1

11:35

Like, are we talking about like forking and that kind of stuff?

Speaker 3

11:39

Yeah, that's, I think you could, there's a lot of things that you could do if you, once you have a platform that has more people in it and that they're doing more stuff in it. And so that's why the way we've been growing Fermat is with a goal far in the future where we are a much broader platform. And so right now, but right now we were focused mostly on solving problems that people have nowadays.

Speaker 3

12:09

And actually, we were largely inspired for our archive extension by the survey that the Archive guys did where they, right, they had, I don't know how many people, but they surveyed the people that use Archive and then published a paper where they describe the problems that those people reported while using Archive and the things that they most wanted to see, the features that they most wanted to see. And then the archive folks just said, hey, we're just going to be the platform to build upon. And we're not going to do all of these things that people would like us to do. But here it is.

Speaker 3

12:45

This is what people want to see. There's anybody else that wants to work on this, here are the results of the survey. And since then, they've actually done a pretty great job of building an API and wanting to become more of a platform. And so There's a lot of ways that we envision that you could have collaboration around science.

Speaker 3

13:06

And so, yeah, like forking a paper or forking some type of research. Or data. Exactly, or data. There's a lot of things that you could do there.

Speaker 3

13:17

It's not something that we're focused on right now. Right now, we're just trying to solve these problems that people have pointed out and create a place where people can just post comments and discuss around a paper.

Speaker 2

13:29

An example of the problems that people mentioned was like, for instance, reference extraction. So if you go to a PDF, you have at the bottom of the paper, you have the references that they used. And most of the times, when people want to search the references, they have to copy the text in the PDF, put it on Google, and try to find the link to the paper.

Speaker 2

13:49

And 1 of the things that we did with our Chrome extension is we allowed that. They just click on a button in the Chrome extension, and then they see a list of references with links to the papers. So that was 1 of the features that was most requested by the archive users. And our idea was, initially, we wanted really to convince people to install the Chrome extension.

Speaker 2

14:08

And so let's solve the hair on fire problems that they are describing here. And then once we have people using the Chrome extension, and then we can expand into open collaboration around papers since they're already there. Yeah, so that was the growth.

Speaker 1

14:23

Do you guys know of anyone working on publishing negative results? This is something that I've been fascinated with. And basically the problem is that as an academic, you're not incentivized to publish negative results because you want to publish things that have high impact so you can get a job or a tenure position or just get people to even care about your work, right?

Speaker 1

14:45

And so they don't publish. Do you know anyone like working on that?

Speaker 3

14:50

I know of researchers that are studying that field a lot, but unfortunately for some of these things you just, that's a very large problem and people are becoming more aware of that. And with that, you have negative results. You also have people doing a lot of research into p-value hacking.

Speaker 3

15:09

Yeah. Would you

Speaker 1

15:11

explain that?

Speaker 3

15:12

Yeah. Yeah, so p-value, it's essentially a standard that people use in order to know if the results that you have obtained out of some experiment that you've run are worthy of being published. And so, and that has worked for the most part, that has worked fine until now, but, or I mean, that's arguable, But people are looking into it and thinking, OK, should we do things differently? And should we be much more stricter with what's considered the golden standard to publishing?

Speaker 3

15:45

And we've thought of doing things there with Fermat, so that if you're looking at a paper, to have an idea, OK, how relevant is this paper? This is more specific for certain areas, like if you're talking about medicine or biology, where that is really important, like the statistical significance of the results that you're presenting. That's all, right? That's the most important thing.

Speaker 3

16:12

And so we've thought of doing something with Fermat there, either via some API where you could send us the DOI of a paper and we would send you some information regarding the p-value or something, or with a Chrome extension where you'd see that information displayed very prominently saying, hey, there might be some p-value hacking here or this is very solid research. Because there is a very big problem and people are realizing how prevalent it is, especially in things like economics and biology. Nutrition.

Speaker 1

16:51

I mean it came about, I was just talking to a friend who's doing a PhD at Cambridge in bio.

Speaker 2

16:57

That's a big thing.

Speaker 1

16:58

Yeah, and only by attending a conference in the States did he realize that there was someone in Australia working on the exact same problem as him concurrently and they're failing at the same types of experiments but because they don't publish them, Like no 1 knows the results, no 1 knows the methods, and essentially like these, you know, traveling salesman type problems that people are so excited about quantum for, like trying all these permutations, are happening at a smaller scale, but no one's publishing anything.

Speaker 3

17:30

So

Speaker 1

17:30

like the progress isn't happening.

Speaker 3

17:32

Yeah, and part of it is just the way research is done and you come into it and you're trying to find some correlation usually. You will be trying to find some trend in the data and you are going to usually have that bias. You're trying to find some correlation in publishing that.

Speaker 3

17:51

And so, yeah, you might need to change things dramatically in order to get people to start publishing negative results, which are like could be incredibly useful for other researchers. Yeah. That's yeah. But there are a bunch of people working on that.

Speaker 3

18:11

There's this researcher at Stanford. I'm forgetting his name. It's John. And then I forget his last name.

Speaker 3

18:17

But he actually just went on this podcast, Econ Talk. And he talked.

Speaker 1

18:20

Oh, really? I love Econ Talk.

Speaker 3

18:21

Yeah. So you should listen to that podcast. And actually, Taimur has been talking to that professor. I think he's a professor at Stanford.

Speaker 3

18:30

And yeah, and he has analyzed more this subject, but more relating to economics, I believe. But yeah, he's found a lot of the things that we're talking about here, they're prevalent also in economics. Cool.

Speaker 1

18:46

Let's go into the Twitter questions. So we have a ton of quite. You guys are very popular on Twitter.

Speaker 1

18:50

So congrats on your great following. Let's see. Let's start with something broad. Tanner Goblinstein asks, what are the most interesting papers you've read in the past couple of years that are not widely known?

Speaker 3

19:08

That, that's interesting. We end up, like I end up reading all sorts of papers from different areas. Like- How

Speaker 1

19:16

do you get the papers actually?

Speaker 3

19:18

It's just like a

Speaker 2

19:19

random walk. Really? Yeah.

Speaker 2

19:22

The random walk. It's

Speaker 3

19:23

funny. Yeah. Or sometimes you'll think, for instance, a few months ago I got like a Fitbit to track like my sleep. And so I wanted to read papers about sleep.

Speaker 3

19:38

And so that just got me into a random walk around, like research around sleep. And then I found a bunch of interesting things. I ended up annotating a paper about a big study in Finland that was done regards to the association between sleep and mortality. There are a bunch of really interesting things that I learned from there.

Speaker 3

19:58

For instance, that like If you sleep less than 7 hours, that's associated with higher mortality. But if you sleep more than 8 hours, that is also associated with higher mortality. Really?

Speaker 1

20:09

Yeah. So have you changed your life based on that?

Speaker 3

20:12

Yeah. No, I try. Well, not that I was usually more on the end of not sleeping enough. But there was also another thing from that research that apparently sleep quality doesn't matter as much, at least for mortality, which is kind of counterintuitive.

Speaker 3

20:27

But it seems that just sleep quality is very closely related to the amount of sleep that you're

Speaker 1

20:33

getting.

Speaker 3

20:34

So like 7 hours of like okay sleep versus 7 hours of great sleep that's kind of hard to distinguish. Kyle Yerganka Seriously?

Speaker 1

20:42

So you could like sleep on an airplane your whole life and live as long?

Speaker 3

20:45

Dr. Kim Kite Yeah, yeah, apparently. Maybe your life will be a little bit more miserable. So it's hard sometimes to pick the favorites, but there is 1, for instance, there is 1 that is also kind of random, but it's a paper published in the 90s about the Simpsons paradox and the hot hand phenomenon in basketball.

Speaker 3

21:08

So the hot hand phenomenon in basketball is you think that, OK, because they just made a field goal, the next 1 they have a higher chance of making it. And so there's this researcher that in the 90s looked at a data set from the Celtics to to see if for free throws, if that if that was true. And so before, they had asked students at Stanford and Cornell, like 100 students, if they thought that, OK, if they just made the first free throw, for the second 1, are they higher? Did they have a higher chance of making it or not?

Speaker 3

21:46

And there was something like 68 of the 100 students that were asked that agreed. And they thought that that was true. And these are people from Stanford and Cornell. And so then they looked at this.

Speaker 3

22:00

And so what they found back in the 90s, what they found was that actually that seemed not to be the case, right? That from your second free throw is not, you're not more likely to make it if you made the first 1. But what they found is that you're just more likely to make it on your second 1.

Speaker 1

22:21

Objectively. Significantly. Yeah. OK.

Speaker 3

22:24

And so this was done in the 90s with, like, I don't know how many free throws, but maybe like 5,000. They looked at some data from the Celtics. Just across

Speaker 1

22:31

the Celtics. Yeah. And

Speaker 3

22:32

then I went and got a data set from Kaggle with like 600,000 free throws. Free throw shots? And I reran the same, right, reran the same algorithms that they ran for the study in the 90s and then looked at what the results were.

Speaker 3

22:49

And yeah, and so the pattern is pretty clear that just on their second free throw, they're just much better at it significantly, regardless of their first 1. And yeah, it doesn't matter if they made their first 1 or if they missed. Yeah, and then that paper kind of then tried to explain why people think that there is a hot hand phenomena. And that is related to the Simpson's paradox, which for people that don't know what the Simpson's paradox is, it's also really kind of changed my world view a little bit once I learned more about the Simpson's paradox.

Speaker 3

23:32

But it's basically, what it says is that you can get 2 valid conclusions out of the same data, depending on how you split it. So An example is, for instance, that between 2000 and 2013, the average or the median wage for high school dropouts in the US is dropped. For high school graduates, it also dropped. For people with an undergrad degree, it dropped.

Speaker 3

24:04

And for people with a graduate degree or higher, it also dropped. So across the board, for all of those segments, the median wage dropped. But in aggregate, it went up. And so you look at it, and it's like, OK, what's going on here?

Speaker 3

24:22

And it turns out is that what happened is that a lot more people got a degree. So they just shifted towards higher education. So that's why you get, on average, it going up. And then for each 1 of these segments, it goes down.

Speaker 3

24:38

And so the Simpson's paradox is that depending on how you cut the data, you might get different results. But that could be valid. In this case, it's pretty easy to understand that you should be like, what's the right way to look at this data. But in some other cases, it's not clear whether or not you should include this variable and cut the data in some different way.

Speaker 3

24:58

And so relating it back, Like for this basketball issue, what it was is that if you looked, the results were different whether you looked on a player by player or if you looked at the aggregate. Once you collapse it all into the same table,

Speaker 2

25:13

you

Speaker 3

25:14

get different results rather than when you looked at it player by player. And so if you collapse it, I think, I forget exactly the way it went. But if you collapse it, it might have been that you indeed saw.

Speaker 3

25:30

You didn't see the hot hand phenomenon, but if you looked at it player by player, you saw it. And so they're arguing that that's why people add the idea. That's why you get like 68 students out of 100 saying that they believe in the hot hand phenomenon. Yeah, yeah, yeah.

Speaker 3

25:43

And so, yeah, so some of the papers, like that's really random.

Speaker 1

25:46

It's just like it's funny you're getting these just like little tidbits of trivia. Yeah. Is it has it been relevant to you in terms of physics?

Speaker 1

25:54

I mean you're basically you're working on software now right. Yeah. But I

Speaker 2

25:58

yeah I also end up discovering really cool physics papers. So for instance, my 2 favorite papers are actually written by Freeman Dyson. 1 of them is when he proposed the concept of a Dyson sphere.

Speaker 2

26:14

It's just 1 page. And he basically explained how an advanced civilization would need more energy than the energy that we can generate on Earth. So we would have to go to a star and build a cap around the star to extract the energy of a star. But It's funny because it's like with really simple math and physics equations, he was able to derive, OK, is this sphere stable?

Speaker 2

26:39

Is it going to eat indefinitely? And so it's a really interesting paper. And the other 1 that I really like is 1 about Feynman's derivation of Schrodinger equation and also written by Freeman Dyson. And it just shows, you know, Feynman's intuition about quantum mechanics.

Speaker 2

27:00

And it's also really simple and easy to read, even if you don't have a physics background. But 1 of the things that I noticed from trying to find papers and annotating all these papers was that in the 60s and all through the 20th century, all these discoveries and all these papers were mostly like 1, 2 pages. And yeah, it's so funny. And also fairly simple to read.

Speaker 2

27:32

But the discovery of the neutron, it's like maybe 1 column just. The discovery of the positron, the Dyson sphere paper, they're really, really short papers and fairly accessible.

Speaker 1

27:48

Why do you think they've gotten so long? Is it sort of like, you know, David Foster Wallace citing a million things because he doesn't have confidence?

Speaker 2

27:55

I think it's also a consequence of a field developing. You just have, you know, more complex questions and so it's harder to write. I think

Speaker 3

28:07

they're also a little bit more detailed as to the methodology and the format of papers has gotten a little bit more formal in that sense where people follow a very specific format. And I think that has added on to it. But yeah, nowadays they tend, like the gravitation wave that we annotate, that's relatively, that's what, like 15 pages?

Speaker 2

28:29

Maybe. It would be interesting to analyze the constraints in terms of size that the journals were imposing like 50 or 60 years ago compared to what they are doing now. If they are forcing people to write, they were forcing people to write shorter pages, shorter papers back then. Not sure.

Speaker 2

28:49

But I mean, if the discovery of the positron paper was published today, I bet it wouldn't be just a single column. Yeah.

Speaker 1

28:59

Well, Are they intended to be more reproducible now?

Speaker 2

29:03

Good question. Maybe. Maybe.

Speaker 2

29:07

Yeah, I think. Or maybe it's just more complex problems that they are tackling now. It might be the case.

Speaker 3

29:16

Yeah. It's definitely not going back, it seems. You don't really see a trend anywhere of shorter papers. But yeah, it's interesting.

Speaker 3

29:26

You go back to the 60s and 50s and it was pretty nuts.

Speaker 1

29:29

Man, glory days. Yeah. All right, Cool.

Speaker 1

29:32

So let's go to another question. Polaris 7 asks, what are the necessary ingredients in a good and impactful good and impactful science writing?

Speaker 2

29:42

This is also a good question. I don't I don't think that I'm qualified to, or like I haven't published that many papers to know that. But 1 of the things that we noticed, or at least I noticed from reading papers, is that sometimes it's not like the discovery paper that is the most impactful paper.

Speaker 2

30:05

So for instance, I just remember when quantum electrodynamics was discovered, there were 3 guys working on that problem. So Feynman, Schwinger and Tomonaga. And they were sort of working independently on that problem and publishing papers on quantum electrodynamics. And the most impactful paper was actually published by Freeman Dyson, who at the time took the time to analyze all the work and kind of unified the work of Feynman, Tomonaga, and Schwinger, wrote a paper that helped other researchers understand what quantum electrodynamics was back then and helped really spread their work.

Speaker 2

30:56

So it was actually the most impactful paper.

Speaker 1

30:59

So in other words, clear writing. Exactly.

Speaker 3

31:02

Yeah. Clear writing. Yeah. It's also, I mean, the question here is impactful scientific writing.

Speaker 3

31:08

And so you have, of course, writing papers. And then you also have just scientific writing in the sense of making some concept more, explaining that to a more general audience. And so I think there's also, it's also the same where you want to make it clear and you want to make it accessible. But for instance, even like something like the Bitcoin paper, where it is like, I mean, I studied photography in college and even like it took me a few reads through it to actually get it.

Speaker 3

31:41

And it's a beautiful paper, but it's definitely not, It's a very Spartan language and you want to read every sentence. And so it can be very challenging to approach it. And I think definitely you always benefit if you can make it as clear and accessible as possible. Because you never know the audience that is going to be, end up reading your paper.

Speaker 3

32:06

Of course you can expect other people in your field are going to read it, but sometimes things can be useful, especially like interactions between math and physics. Things can be useful in different fields. And so I think it's always beneficial for science if you try to make it as accessible

Speaker 2

32:23

as possible. What does impact mean? Yeah, well, that's

Speaker 1

32:27

a question as well. Yeah. Did you see that 1 from Adam?

Speaker 1

32:31

Adam Baybut asks basically the metrics for value add.

Speaker 3

32:35

Yeah, exactly.

Speaker 2

32:36

What does impact mean? If it's the number of citations that you get or just the number of people that learn about a certain subject because of a paper. So in that way, a review paper can have a really big impact compared to a discovery paper.

Speaker 2

32:54

And so it's 1 of the problems that we also think about a lot. These metrics and what are the incentives in science and what makes people, you know, want to publish a paper or, you know, why should people worry about clarifying a paper and make it understandable to as many people as possible. Do they have the incentives to do that? How can you create incentives to do that?

Speaker 2

33:23

And then sometimes, if the metric is just number of citations, sometimes it's not aligned to making the paper understandable and comprehensible to a large audience.

Speaker 1

33:34

I mean is that a is that a question that you guys have to tackle because you know on 1 hand you want to illuminate these papers that people could potentially learn from. Then on the other hand you're running a site with content right And you want things that are going to capture attention. So I saw you have the Charlie Munger post on there.

Speaker 1

33:53

Right.

Speaker 2

33:54

Mika annotated the Charlie Munger paper. Other co-founder.

Speaker 1

33:59

So it's like squarely non-technical paper. But Charlie Munger has millions of fans across

Speaker 2

34:05

the world.

Speaker 3

34:05

So you

Speaker 1

34:05

kind of have to balance those 2 things.

Speaker 3

34:08

Yeah, and yeah, it's not easy. And citations are definitely a proxy, right? If the paper is getting cited a lot, it has some sort of importance, but it's definitely not perfect.

Speaker 3

34:21

And if you look at the most cited papers in these different fields, you might be surprised that there might not be the ones that you expected to be. I certainly remember looking at like the most cited papers in computer science, And they're definitely very impactful, but you might, some of them, I remember reading through those 10, and some of them I'd never heard about before. And so, yeah, and sometimes very important, well, this is more specific for certain fields, very important concepts or discoveries never really get published in 1 paper that then gets a ton of citations. That knowledge gets spread in some other way.

Speaker 3

35:01

And so there are, yeah, citations that are not perfect. But I wouldn't say that we have a great answer for that. What's a better proxy and how you should go about it? And I don't think anybody really right now has a better answer to, or not that we've heard about, but yeah, it's an interesting problem.

Speaker 3

35:24

We'll see what people start using in the future. Because yeah, You could measure impact or how many people are talking about it on social media.

Speaker 2

35:35

There's many blog posts are written about this paper. Or if you have code, if you have a public repo, how many forks do you have on your repo?

Speaker 3

35:47

Yeah, or like for certain. And then it depends on field by field, right? So if you take bio, then bio papers can have a very direct, can be used very directly, say, in industry, right?

Speaker 3

36:02

You can publish a paper about a drug, and then that can be used worldwide and save lives. So there are like, for that field, maybe you can, there are a bunch of other metrics that you could use there to calculate the impact of a paper. But for the more traditional science, like physics and math, sorry, yeah, that's it's hard. Okay.

Speaker 1

36:26

Question up top, Arsalan Yarvesi asks, It's basically about working in public and in the speed of publishing. They say, since scientific papers usually go through scrutiny and evaluation before getting published, how do you cope with not being always updated and up to speed in a world with daily news and contributions. This kind of relates to what we were talking about before in relation to people publishing to the archive before they really test it out.

Speaker 1

36:55

Where do you guys fall in that dynamic of like publishing as soon as possible? Like with something like machine learning where things are just getting put out all the time versus going through a peer review for getting something out.

Speaker 3

37:09

And this this kind of loops back loops into peer review, which is the whole world unto itself that people are talking a lot about. For us, generally, or say for a weekly journal, we generally are not publishing the most recent research.

Speaker 1

37:27

And

Speaker 3

37:28

there is definitely, like sometimes, there's a lot of us having to catch up to even, I remember annotating a paper about like, this machine learning algorithm to play one-on-one poker.

Speaker 1

37:43

And

Speaker 3

37:43

this was like, out of my league, I had to go like spend a good amount of time there researching it and also figuring out, okay, how relevant is this? I also don't because, you know, I'm not in the field, so it's hard for me to gauge, okay, what's the impact of this paper? So Yeah, there's sometimes it takes us a lot of reading up before we can actually say, okay, this is worth publicizing and having our audience or it's worth our stamp of approval and say, hey, you should read this.

Speaker 3

38:16

I think you'll like it. And it can take a while sometimes. But in the future, like looping back to peer review, that's also something that I think the system nowadays does not seem to be perfect for the way things work nowadays. And we would love to see either VIA Fermat or some other platform to try to tackle that and try to do something to make the peer review a better system or to change it significantly.

Speaker 3

38:50

I think there's a lot of work left to be done there, which can have a very significant impact in science. That's part of the most, 1 of the most important aspects of science is just, OK, having a very skeptical mindset, looking at it with a very critical eye, and seeing, OK, is this something that we can build upon? Is this something that we're going to add to our foundations to build more science upon this? And so that's a very important aspect of science and I think it's not perfect and could be better.

Speaker 1

39:26

So Anvil Rotterdam asks, Have you ever thought about building a tool for annotating books? Something like what Patrick Collison was talking about in this thread where he basically says I'd pay a lot more for books if I could see the highlights, annotations, and marginalia of friends or people I follow.

Speaker 2

39:42

Yeah. No, it's, it's, I think it's, it's actually a really, really good question. And, And we have a friend, Jess Riedel, from the Preliminary Institute. He's a researcher there that wrote about this on his blog.

Speaker 2

39:58

And I think that besides annotating academic papers, it also makes total sense to annotate books. And especially kind of introductory books about science. And he gives this example of a book that is used by thousands of students to learn classical mechanics called Goldstein. And there is a section on that book where they talk about this transformation called the Legendre transform.

Speaker 2

40:31

And he does a bad job at explaining what it is. But apart from that section, the rest of the book is awesome. It's really nice if you want to learn classical mechanics. But if I want to write a book that does a better job at explaining the Legendre transformation, it has to be net better than the Goldstein book so that anyone will adopt that book.

Speaker 2

40:55

Otherwise, people just keep using the Goldstein book. So It would make sense for books to be annotated and also be open source so that in that sense, you would just commit a new chapter, a new explanation for that, and keep all the other chapters, and then just change that bit instead of having to write a new book and then convince people to adopt your book just because of that. So I think it makes total sense to do that.