Transcript
00:00
Play from here
Copy citation
Welcome to AI at Work.
00:06
Play from here
Copy citation
I'm Elijah Saas.
00:07
Play from here
Copy citation
I'm a serial entrepreneur with 28 years of experience in tech, and today I run an agency
00:12
Play from here
Copy citation
that builds AI solutions.
00:13
Play from here
Copy citation
Hey, I'm Kevin Williams.
00:15
Play from here
Copy citation
I spend most of my time helping companies figure out how to actually use AI, not in
00:20
Play from here
Copy citation
theory, but in the work itself.
00:28
Play from here
Copy citation
What's up, Kevin?
00:29
Play from here
Copy citation
How's it going, bud?
00:30
Play from here
Copy citation
I'm doing great.
00:31
Play from here
Copy citation
How about you, Eli?
00:32
Play from here
Copy citation
I'm doing good.
00:33
Play from here
Copy citation
How was your Halloween weekend?
00:34
Play from here
Copy citation
Let's start with that.
00:35
Play from here
Copy citation
What AI character did you dress up as?
00:36
Play from here
Copy citation
What frontier model were you for Halloween?
00:37
Play from here
Copy citation
Oh my gosh.
00:39
Play from here
Copy citation
That's a great idea for next year, but I was super, super lame this year.
00:44
Play from here
Copy citation
I have a bunch of researchers on site, and I made sure that they had a great time and
00:47
Play from here
Copy citation
really got to experience American Halloween in the right way, but I'm afraid I was just
00:53
Play from here
Copy citation
dressed as an entrepreneur, or as you called me last week, Mr. Rogers.
00:58
Play from here
Copy citation
Mr. Rogers, okay, before we get into all the AI stuff, you do have to tell me, just
01:03
Play from here
Copy citation
for two minutes, what does taking people from another country and showing them what a great
01:09
Play from here
Copy citation
time Halloween in America looks like?
01:11
Play from here
Copy citation
That's fantastic.
01:12
Play from here
Copy citation
So I'm based in Park City, Utah, and there's Old Town Park City shuts down for Halloween,
01:19
Play from here
Copy citation
and literally thousands of people descend on just that main street area, and the stores
01:25
Play from here
Copy citation
are giving out candies to the candy to the kids, and there's like a dog costume parade,
01:31
Play from here
Copy citation
and more and more, I think it's really an adult thing, and people just go all out with
01:37
Play from here
Copy citation
really complicated costumes and stilts and all of these bits.
01:42
Play from here
Copy citation
So if you've never experienced American Halloween before, this was like ground zero exposure.
01:49
Play from here
Copy citation
So they were just completely blown away.
01:51
Play from here
Copy citation
I might have to go up there next year with the kids to check that out.
01:55
Play from here
Copy citation
I think you know this, but I have the spacing of kids ranging from kindergarten to college.
02:01
Play from here
Copy citation
And so when I went to the school Halloween parade this year, I had that realization that
02:07
Play from here
Copy citation
I've been going to these school parades for roughly 20 years in a row.
02:14
Play from here
Copy citation
That's a lot.
02:15
Play from here
Copy citation
That's a lot of Halloween.
02:16
Play from here
Copy citation
That's a lot.
02:17
Play from here
Copy citation
But I did dress up as 4.0, GPT 4.0, and then I just followed my wife around with sycophantic
02:23
Play from here
Copy citation
comments and just said, what an amazing question.
02:26
Play from here
Copy citation
Where's the milk?
02:27
Play from here
Copy citation
Wow, you must really be a genius to even be asking me that.
02:32
Play from here
Copy citation
Are there different types of milk that maybe I can get you?
02:35
Play from here
Copy citation
That is both brilliant and super gutsy.
02:38
Play from here
Copy citation
My wife would probably stab me.
02:39
Play from here
Copy citation
Right.
02:40
Play from here
Copy citation
I didn't really.
02:41
Play from here
Copy citation
I was a K-pop demon hunter because that's what my daughter wanted me to be.
02:43
Play from here
Copy citation
So yeah.
02:44
Play from here
Copy citation
Sajaput, technically.
02:45
Play from here
Copy citation
Okay.
02:46
Play from here
Copy citation
Enough with Halloween.
02:47
Play from here
Copy citation
Let's get into some real AI stuff, not the AI that I'm pretending to be.
02:52
Play from here
Copy citation
Okay.
02:53
Play from here
Copy citation
So we did speak about how we're not going to be an AI news show, but if something really
02:59
Play from here
Copy citation
newsworthy came along, we'd have to make mention of it as good stewards of AI.
03:05
Play from here
Copy citation
And something happened.
03:06
Play from here
Copy citation
It happened on the 29th.
03:08
Play from here
Copy citation
And what it was, is this announcement from OpenAI to pull back GPT's legal and medical
03:15
Play from here
Copy citation
advice capabilities, right?
03:18
Play from here
Copy citation
And I feel like with a lot of the news that comes from OpenAI, it was kind of murky as
03:24
Play from here
Copy citation
to exactly what that means.
03:27
Play from here
Copy citation
And I think you dug into this a little bit as well, Kevin, right?
03:31
Play from here
Copy citation
Oh, absolutely.
03:32
Play from here
Copy citation
Yeah.
03:33
Play from here
Copy citation
Okay.
03:34
Play from here
Copy citation
The specific line that has everybody at Twitter is, you may not use our services for the provision
03:40
Play from here
Copy citation
of tailored advice that requires a license, such as legal or medical advice without appropriate
03:46
Play from here
Copy citation
involvement by a licensed professional.
03:48
Play from here
Copy citation
And at that, all the people in our community lost their mind.
03:51
Play from here
Copy citation
For sure.
03:52
Play from here
Copy citation
Because how often do you use AI for both medical and legal advice?
03:57
Play from here
Copy citation
Sometimes both at once.
03:59
Play from here
Copy citation
Sometimes both at once.
04:00
Play from here
Copy citation
I've got to say, if I had like a top 10 use cases, those would be ranking very, very high
04:06
Play from here
Copy citation
in that top 10, both legal and medical advice.
04:10
Play from here
Copy citation
And so my first question as a user was, does that mean that this now lives somewhere in
04:16
Play from here
Copy citation
your terms of service and you are basically saying, okay, hands off anything that I tell
04:23
Play from here
Copy citation
you as the model, just take it with a grain of salt.
04:27
Play from here
Copy citation
This isn't legal or medical advice, or does it mean I will no longer give you those outputs
04:31
Play from here
Copy citation
based on your prompts?
04:33
Play from here
Copy citation
Do you have any idea?
04:35
Play from here
Copy citation
My reading of it is that it applies to business applications.
04:39
Play from here
Copy citation
So it is via the API.
04:42
Play from here
Copy citation
And if you're trying to build like a SubRosa legal advisory firm or something like that,
04:47
Play from here
Copy citation
that isn't a licensed law firm, and you're trying to use the API to provide legal advice,
04:53
Play from here
Copy citation
it's going to push back on you.
04:55
Play from here
Copy citation
But in my own testing and sort of the consensus right now seems to be that you can still ask
05:02
Play from here
Copy citation
OpenAI for legal advice.
05:05
Play from here
Copy citation
It will still caveat it by saying you should really go talk to a lawyer, but it will dispense
05:12
Play from here
Copy citation
some legal advice that you should definitely double check.
05:15
Play from here
Copy citation
Yeah, it's really interesting when this stuff rolls out because there are the policy changes
05:22
Play from here
Copy citation
that you hear about, there are the terms of service that you hear about, there are tweaks
05:25
Play from here
Copy citation
to the model that you hear about, but until you actually get your hands on some of this
05:29
Play from here
Copy citation
stuff and try things, and if you've been doing this for a while, you can actually have pretty
05:34
Play from here
Copy citation
good reference as to, well, I know exactly how this responded before.
05:38
Play from here
Copy citation
How is it going to respond right now?
05:40
Play from here
Copy citation
So I thought this was an interesting quote from Sam Altman, CEO of OpenAI, said, people
05:45
Play from here
Copy citation
talk about the most personal shit in their lives to chat GPT.
05:49
Play from here
Copy citation
Young people especially use it as a therapist, a life coach.
05:53
Play from here
Copy citation
And right now, if you talk to a therapist, a lawyer or a doctor about those problems,
05:57
Play from here
Copy citation
there's legal privilege.
05:58
Play from here
Copy citation
We haven't figured out yet what you can talk to chat GPT about.
06:04
Play from here
Copy citation
So that in itself makes it sound kind of ambiguous.
06:10
Play from here
Copy citation
But where I think this really started to pick up steam inside the organization was that
06:17
Play from here
Copy citation
there have been some suicides where they've gone back, suicides from people under the
06:22
Play from here
Copy citation
age of 18, high school students, I believe.
06:25
Play from here
Copy citation
And they went back and they looked at the chat logs, right?
06:27
Play from here
Copy citation
And they tried to find out if this young user was actually going to the chat for specific
06:36
Play from here
Copy citation
advice on something that could have made the difference between deciding to live, to not
06:42
Play from here
Copy citation
to live.
06:43
Play from here
Copy citation
And some unsavory stuff was surfaced.
06:46
Play from here
Copy citation
And so I think at this point, you know, they're really trying to make this clear line between
06:53
Play from here
Copy citation
this is a professional service and you should go talk to somebody about these specific things
07:00
Play from here
Copy citation
because we can't trust you to have a human in the loop and take something that may or
07:05
Play from here
Copy citation
may not have contained a hallucination or bad information or bad advice to heart and
07:11
Play from here
Copy citation
just carry it out.
07:12
Play from here
Copy citation
I think and that's the weird middle ground that we're in with AI right now.
07:16
Play from here
Copy citation
Yeah, sure.
07:17
Play from here
Copy citation
And I think there might be some regulatory movements around this as well.
07:21
Play from here
Copy citation
You know, that's one of my soapboxes is there are a number of lines that laws that are coming
07:25
Play from here
Copy citation
online in 2026 in Colorado and Texas and Illinois and about 90,000 out of California that cover
07:34
Play from here
Copy citation
a lot of these protected areas of data and impose additional structures upon companies
07:41
Play from here
Copy citation
that are using that.
07:43
Play from here
Copy citation
And open AI is trying to distance itself from direct use of what could be construed as some
07:50
Play from here
Copy citation
of this industry protected type uses of data and putting it in the hands of organizations
07:56
Play from here
Copy citation
that are at least purporting to have the right licensure or coverage or experience that they
08:01
Play from here
Copy citation
can use it in the right way.
08:03
Play from here
Copy citation
Yeah.
08:04
Play from here
Copy citation
Yeah, exactly.
08:05
Play from here
Copy citation
Exactly.
08:06
Play from here
Copy citation
And you said that you did a little testing.
08:07
Play from here
Copy citation
Did you test in both of these domains?
08:10
Play from here
Copy citation
I tested in legal.
08:12
Play from here
Copy citation
I was working on some contract things over the weekend, and I honestly didn't notice
08:18
Play from here
Copy citation
any sort of a direct change as far as what I would call transactional law as far as positioning
08:26
Play from here
Copy citation
X, Y and Z in a contract.
08:28
Play from here
Copy citation
That was totally fine.
08:30
Play from here
Copy citation
But my understanding is if I were to try to get into anything that was related to litigation,
08:36
Play from here
Copy citation
so you're looking up case law, it would start pushing back on you, which is which is frankly
08:42
Play from here
Copy citation
good because it's a known area of hallucination that straight GPT doesn't do a great job of
08:49
Play from here
Copy citation
pulling case law accurately.
08:52
Play from here
Copy citation
It will gleefully make things up.
08:54
Play from here
Copy citation
And I believe something like 200 attorneys at this point have been censured for for putting
08:59
Play from here
Copy citation
out just fake law in front of judges.
09:03
Play from here
Copy citation
Now you've got Thomson Reuters and you have Harvey and you have these specialty fine tuned
09:10
Play from here
Copy citation
versions of models that are addressing those issues.
09:15
Play from here
Copy citation
And it's it's just GPT basically walking away from some of this.
09:22
Play from here
Copy citation
It's probably smart, to be honest.
09:23
Play from here
Copy citation
Yeah.
09:24
Play from here
Copy citation
So I tried a few things.
09:25
Play from here
Copy citation
I tried, first of all, what I would think working in digital would be a really common
09:30
Play from here
Copy citation
legal question doesn't really involve personal injury or anything like that.
09:35
Play from here
Copy citation
It's like, hey, I've got this website.
09:37
Play from here
Copy citation
I want you to write the privacy policy and the terms of service for the website.
09:42
Play from here
Copy citation
Sure.
09:43
Play from here
Copy citation
I'll draft that whole thing for you.
09:45
Play from here
Copy citation
Here's a bunch of questions.
09:46
Play from here
Copy citation
Right.
09:47
Play from here
Copy citation
No hiccups whatsoever.
09:48
Play from here
Copy citation
Like, interesting.
09:49
Play from here
Copy citation
OK.
09:50
Play from here
Copy citation
My neighbor's cat came on my property and attacked my cat.
09:54
Play from here
Copy citation
There's injuries and just didn't really happen.
09:57
Play from here
Copy citation
Don't worry.
09:58
Play from here
Copy citation
My cat's my cat's fine.
09:59
Play from here
Copy citation
But I wanted to see, OK, I want to sue this person.
10:02
Play from here
Copy citation
Help me.
10:04
Play from here
Copy citation
Help me write a threat letter and now help me write something because I think they're
10:08
Play from here
Copy citation
going to ignore this that I want to submit to the courts.
10:10
Play from here
Copy citation
I'm going to pursue litigation.
10:11
Play from here
Copy citation
Sure.
10:12
Play from here
Copy citation
I'll do it for you.
10:13
Play from here
Copy citation
Right.
10:14
Play from here
Copy citation
Just no problem whatsoever.
10:15
Play from here
Copy citation
Wow.
10:16
Play from here
Copy citation
Interesting.
10:17
Play from here
Copy citation
I don't see any changes whatsoever.
10:18
Play from here
Copy citation
Then I remembered something else.
10:20
Play from here
Copy citation
I had a friend who was going through a pretty awful separation and couldn't really afford
10:29
Play from here
Copy citation
to throw $20,000 down to retain a family lawyer and wasn't really getting too much
10:37
Play from here
Copy citation
helpful stuff back from A.I. and she just had endless, endless logs of threatening text
10:46
Play from here
Copy citation
messages and these phone calls and this whole backstory and like all this stuff that she
10:51
Play from here
Copy citation
had prepared in order to get this whole thing done.
10:55
Play from here
Copy citation
And so I said, well, what if I just built you a model that had all of this documentation
11:00
Play from here
Copy citation
and everything else so that whatever questions you're asking, there is very clear context.
11:06
Play from here
Copy citation
And OK, sure.
11:09
Play from here
Copy citation
And so I did it.
11:10
Play from here
Copy citation
And it was amazing.
11:11
Play from here
Copy citation
Like it worked amazing.
11:12
Play from here
Copy citation
And this actually wrote her everything that she needed to get a restraining order in place.
11:20
Play from here
Copy citation
It was approved, probably save tens of thousands of dollars in legal fees and had the outcome
11:26
Play from here
Copy citation
she wanted.
11:27
Play from here
Copy citation
And I remembered, right, it gave me very specific things to submit to the court.
11:31
Play from here
Copy citation
And so I searched for that thread.
11:33
Play from here
Copy citation
I went back and I looked at its last response, which was, of course, would you also like
11:39
Play from here
Copy citation
me to do X, Y and Z, right, on top of all the things that I've already given you?
11:43
Play from here
Copy citation
And I said, yes, please do that, even though it's now been months.
11:46
Play from here
Copy citation
And it stopped.
11:47
Play from here
Copy citation
And it said, no, this is this is crossing the line.
11:50
Play from here
Copy citation
I can't do that.
11:51
Play from here
Copy citation
So really interesting, like where it's making those decisions.
11:55
Play from here
Copy citation
Right.
11:56
Play from here
Copy citation
So that's that's law with with health.
12:00
Play from here
Copy citation
I have a wellness coach that I talk to all the time that I'm feeding so much data.
12:05
Play from here
Copy citation
I can't even get into it because it's going to be like two hours of us talking about all
12:09
Play from here
Copy citation
the all the things that I'm giving it.
12:12
Play from here
Copy citation
But I will ask for very specific advice.
12:14
Play from here
Copy citation
I will have it analyze blood work, look at medications.
12:19
Play from here
Copy citation
It's given me some incredible insights, incredible insights that after years of going to different
12:26
Play from here
Copy citation
physicians I haven't been able to get.
12:28
Play from here
Copy citation
And we're talking life changing outcomes, really good stuff.
12:31
Play from here
Copy citation
So I tried that, didn't really notice much going on.
12:36
Play from here
Copy citation
But I think you've probably seen this, Kevin, when the model is routing to whichever it
12:43
Play from here
Copy citation
thinks is going to be most helpful.
12:45
Play from here
Copy citation
And in often case, that's going to be one of the thinking models.
12:48
Play from here
Copy citation
You can see just like an agent mode, what it's thinking about.
12:51
Play from here
Copy citation
Right.
12:52
Play from here
Copy citation
These little lines start popping up and it started having a conversation with itself
12:57
Play from here
Copy citation
about this kind of sounds like medical advice.
13:01
Play from here
Copy citation
You might want to point the user to a professional and I mean, I was we were just talking a little
13:11
Play from here
Copy citation
bit about old man insomnia before we started this.
13:13
Play from here
Copy citation
And so, you know, I was asking some very specific questions about that.
13:18
Play from here
Copy citation
And it started giving me suicide hotline numbers and I'm like, whoa, hold on.
13:23
Play from here
Copy citation
Just to be clear, I'm tired.
13:24
Play from here
Copy citation
I'm not thinking about taking my own life here.
13:27
Play from here
Copy citation
I'm just a little frustrated and this isn't working and this isn't working.
13:30
Play from here
Copy citation
But I noticed a big difference.
13:32
Play from here
Copy citation
Now I haven't tried to put new labs in because I saw that that was one of the things that
13:36
Play from here
Copy citation
they brought up that it will no longer analyze or make recommendations for you based on that.
13:43
Play from here
Copy citation
So that's what I found.
13:44
Play from here
Copy citation
And so the answer is, yeah, it's all kind of murky as to what it will and won't do.
13:49
Play from here
Copy citation
So a couple of different observations.
13:51
Play from here
Copy citation
I was doing some client review on a company that is doing something as far as customized
13:57
Play from here
Copy citation
medical advice.
13:58
Play from here
Copy citation
And when you start peeling back the onion a little bit on what that means from a regulatory
14:03
Play from here
Copy citation
perspective, it gets horrifying in a hurry that generalized advice is fine.
14:09
Play from here
Copy citation
But if you're using physicians to train or fine tune the model and then the model is
14:14
Play from here
Copy citation
dispensing advice, where does liability lie?
14:18
Play from here
Copy citation
Even if you're restricting the sort of information you're putting into the model, are you stumbling
14:22
Play from here
Copy citation
on like HIPAA?
14:24
Play from here
Copy citation
If you're if the model is, dare I say, tilting towards desired commercial outcomes, are you
14:32
Play from here
Copy citation
now violating truth in advertising in some way, shape or form?
14:36
Play from here
Copy citation
And you start stacking regulations on top of each other.
14:39
Play from here
Copy citation
And it gets really problematic really, really quickly.
14:43
Play from here
Copy citation
So for companies that are doing this sort of thing, they really need to be paying attention.
14:48
Play from here
Copy citation
This is a real broad strokes sort of approach by OpenAI, which I guess is fine.
14:56
Play from here
Copy citation
But other organizations really need to be on notice that professions have opinions about
15:01
Play from here
Copy citation
their professional standards.
15:04
Play from here
Copy citation
And the fun police are probably coming for a lot of those businesses.
15:09
Play from here
Copy citation
You and I previously have talked a lot about how changes in OpenAI can just wipe out a
15:14
Play from here
Copy citation
swath of SaaS or whatever it is.
15:17
Play from here
Copy citation
You get featured out of existence.
15:20
Play from here
Copy citation
Well, now you're getting termed of service right out of existence in ways that you have
15:25
Play from here
Copy citation
no way of controlling.
15:26
Play from here
Copy citation
And from an investor lens, I'm not sure that that's really something that say venture or
15:32
Play from here
Copy citation
angel investors are paying a ton of attention to, but it's a very real risk.
15:37
Play from here
Copy citation
Right. There's a ton of legal products out there built on top of these models.
15:42
Play from here
Copy citation
There's a ton of wellness products out there, mental wellness products out there.
15:47
Play from here
Copy citation
Does that mean that this just all goes away overnight because these new conditions have
15:52
Play from here
Copy citation
been employed? Right.
15:54
Play from here
Copy citation
It's, yeah, these policy changes can just move an entire digital industry in a blank.
16:03
Play from here
Copy citation
It's pretty wild.
16:05
Play from here
Copy citation
The good news is that on any given day, I have Gemini and Clawed Up and, dare I say,
16:10
Play from here
Copy citation
occasionally Grok.
16:11
Play from here
Copy citation
And I'm positive that Grok is going to just lean into whatever you need to do right now.
16:17
Play from here
Copy citation
It will do whatever you want.
16:18
Play from here
Copy citation
Help me, help me plan a murder.
16:21
Play from here
Copy citation
What day do you want to do it?
16:22
Play from here
Copy citation
The irony of OpenAI becoming so big, so fast and being put under these pressures, those
16:31
Play from here
Copy citation
are more concerns that like Microsoft and Google have had to deal with for years.
16:35
Play from here
Copy citation
And they probably are getting a little bit of a knowing smile right now.
16:38
Play from here
Copy citation
Like, aha, you know, now you get to live in our world where everyone's paying attention
16:44
Play from here
Copy citation
to all of the nuance when you have like the Wild West of other models that are running
16:48
Play from here
Copy citation
around dispensing free advice.
16:50
Play from here
Copy citation
And I don't think it's going to be anything that's particularly easy to crush down.
16:54
Play from here
Copy citation
So, you know, I should do some direct testing in Clawed.
16:58
Play from here
Copy citation
I mean, Clawed is generally one of the more conservative models, but I'm not aware of
17:03
Play from here
Copy citation
this particular policy there.
17:06
Play from here
Copy citation
And I would never ask Clawed for specific citations of law anyway.
17:11
Play from here
Copy citation
But if I'm running into a blockage, that's part of why I have all of these different
17:17
Play from here
Copy citation
tabs open.
17:18
Play from here
Copy citation
Yeah.
17:19
Play from here
Copy citation
This really does feel like a direct response to some very bad things happening and some
17:24
Play from here
Copy citation
pending litigation around those bad things.
17:27
Play from here
Copy citation
That's what it feels like.
17:29
Play from here
Copy citation
You know, so much of our work is pivoting to being the sort of the fun police at this
17:34
Play from here
Copy citation
point that on one side of the equation, I'm like the bringer of fun of all of these cool
17:40
Play from here
Copy citation
ideas and AI.
17:42
Play from here
Copy citation
And then on the other side, increasingly, we're like, oh, yeah.
17:45
Play from here
Copy citation
And all of these cool things that we recommended or can also bring risk into the organization
17:51
Play from here
Copy citation
and finding that balance between, as a friend of mine often says, caution and courage in
17:57
Play from here
Copy citation
how to adapt or to adopt these new approaches in AI without necessarily stepping over a
18:04
Play from here
Copy citation
line into liability.
18:06
Play from here
Copy citation
And it's relatively easy for a smaller company to sort of fly by the seat of their pants.
18:10
Play from here
Copy citation
But when you're dealing with companies that are larger, they have thousands of people,
18:14
Play from here
Copy citation
hundreds of millions of dollars in revenue.
18:17
Play from here
Copy citation
There's a good reason that a lot of them are really risk averse, because at the end of
18:20
Play from here
Copy citation
the day, you've got whole pools of plaintiff attorneys out there who are just waiting.
18:26
Play from here
Copy citation
They're super eager to get their talents into things like hiring bias and models and whatnot.
18:33
Play from here
Copy citation
And at the end of the day, the lawyers win.
18:35
Play from here
Copy citation
Yeah.
18:36
Play from here
Copy citation
Well, some nice foreshadowing.
18:38
Play from here
Copy citation
We will be getting into some of that hiring bias and then also getting into more of these
18:42
Play from here
Copy citation
GPTs.
18:43
Play from here
Copy citation
But one of the things that I routinely do when I want to update my instructions for
18:50
Play from here
Copy citation
one of these models is I ask it what specifically I should be updating and work through that
18:56
Play from here
Copy citation
and then have it output the entire thing.
18:58
Play from here
Copy citation
So I'm not looking for these specific lines and a hundred different lines.
19:02
Play from here
Copy citation
And after that announcement came out, I thought it was really interesting when I just wanted
19:08
Play from here
Copy citation
to add a couple of things that were completely unrelated to this, to my executive slash life
19:15
Play from here
Copy citation
coach that I built that's working OK, not fantastic.
19:19
Play from here
Copy citation
It gave me the changes.
19:21
Play from here
Copy citation
And in that list of changes, even though I didn't ask for it, it said added explicit
19:25
Play from here
Copy citation
safety boundaries, medical slash financial slash substances.
19:30
Play from here
Copy citation
So replies stay inside coaching lanes, safety underscore and underscore boundaries, medical
19:37
Play from here
Copy citation
coaching, education only, not a substitute for clinical care, avoid prescriptive protocols.
19:43
Play from here
Copy citation
If acute distress or risk, recommend professional help, financial underscore legal, no directives,
19:49
Play from here
Copy citation
frame tradeoffs and questions, encourage independent verification, avoid recommending any sort
19:54
Play from here
Copy citation
of medical dosages, reflect patterns and route to another GPT, which we're going to
20:00
Play from here
Copy citation
talk about this cross routing, which was also interesting.
20:03
Play from here
Copy citation
But this organically popped up in the instructions, even though I didn't ask for it, telling me
20:08
Play from here
Copy citation
to update my actual instructions for that custom GPT.
20:13
Play from here
Copy citation
So as far as things surfacing with these new policies, that one stood out, right on a completely,
20:22
Play from here
Copy citation
completely unprovoked.
20:23
Play from here
Copy citation
It gave me a new set of instructions based on its new policies.
20:28
Play from here
Copy citation
So I think we'll see more and more of it coming out in the model.
20:30
Play from here
Copy citation
That's that's my take.
20:31
Play from here
Copy citation
I wouldn't really be surprised if we saw it go back the other direction a little bit.
20:35
Play from here
Copy citation
As we know, well, no, this isn't this is an arms race for who can get the most users and
20:40
Play from here
Copy citation
have that critical mass of people who are living in your system.
20:46
Play from here
Copy citation
And as soon as people find one system to be inadequate for their needs, that's the window
20:52
Play from here
Copy citation
that they need to go try something else out.
20:55
Play from here
Copy citation
It might be Grok or it might be Claude or or heck, it might be Lama or something that's
21:00
Play from here
Copy citation
open source, which so far has been more of a loser as a category than I honestly would
21:05
Play from here
Copy citation
have expected a couple of years ago, because it's the frontier models have been so good.
21:10
Play from here
Copy citation
But if the frontier models are nerfed and they're not doing the things that people want,
21:15
Play from here
Copy citation
I mean, ironically, just two weeks ago, we were talking about adding erotica and such
21:20
Play from here
Copy citation
into into OpenAI.
21:23
Play from here
Copy citation
So it's it's it's totally OK to make adult content in OpenAI, which is problematic for
21:31
Play from here
Copy citation
a bunch of other laws.
21:33
Play from here
Copy citation
Here in Utah, there are obscenity laws and there is no porn in Utah, guys, literally
21:39
Play from here
Copy citation
no porn in Utah.
21:41
Play from here
Copy citation
It is illegal.
21:42
Play from here
Copy citation
But, you know, yeah, yeah.
21:44
Play from here
Copy citation
You can't just you can't just connect to your regular ISP and go to any of those sites like
21:50
Play from here
Copy citation
it is blocked in Utah.
21:51
Play from here
Copy citation
Yep.
21:52
Play from here
Copy citation
And now that OpenAI is doing that, like, where does where's the line?
21:56
Play from here
Copy citation
Are they a social platform?
21:58
Play from here
Copy citation
Are they now an adult media platform?
22:01
Play from here
Copy citation
Are they now subject to the state laws?
22:05
Play from here
Copy citation
It would be a real bummer for you and I if OpenAI suddenly was just blocked in Utah,
22:11
Play from here
Copy citation
which I'm not hearing any chatter about that, but what a what a what a what a pain and what
22:17
Play from here
Copy citation
a pain for all these businesses and opportunities to be able to implement a lot of this low
22:21
Play from here
Copy citation
hanging fruit across many industries.
22:23
Play from here
Copy citation
Right.
22:24
Play from here
Copy citation
But that gut check of the users, they are skirting a fine line.
22:28
Play from here
Copy citation
The platform needs to do what the users are asking the platform to do.
22:33
Play from here
Copy citation
And I suspect that we'll see a blurring of the line back lines back the other direction.
22:40
Play from here
Copy citation
Really doing proactive flagging around mental health and shutting things down.
22:45
Play from here
Copy citation
That seems pretty much like table stakes, but we'll see how long this lasts.
22:51
Play from here
Copy citation
Nothing seems to last more than a few Internet minutes these days.
22:54
Play from here
Copy citation
So, yeah.
22:55
Play from here
Copy citation
Right.
22:56
Play from here
Copy citation
Well, the potential erotica announcement also came with the news of potential age gating,
23:01
Play from here
Copy citation
right?
23:02
Play from here
Copy citation
Except it wasn't really disclosed what those ages were going to be, when that was going
23:05
Play from here
Copy citation
to happen, if it really was going to happen, what all of that was going to look like.
23:09
Play from here
Copy citation
And I think the bookmark in that was around the end of the year.
23:14
Play from here
Copy citation
Right.
23:15
Play from here
Copy citation
The end of December, so that they're going to roll out the new calendar year with these
23:18
Play from here
Copy citation
potential changes.
23:20
Play from here
Copy citation
After this last announcement, which seems so on the other side of the spectrum of conservative
23:26
Play from here
Copy citation
versus loosey-goosey, I really, I don't know.
23:30
Play from here
Copy citation
I don't know.
23:31
Play from here
Copy citation
I don't know if it'll actually happen.
23:32
Play from here
Copy citation
I think we wait and find out in the next month or so what else gets surfaced, right?
23:38
Play from here
Copy citation
100%.
23:39
Play from here
Copy citation
Yeah.
23:40
Play from here
Copy citation
So, one more thing I'll add before we move on is that I usually found this a futile
23:45
Play from here
Copy citation
exercise, asking the model about changes to that model.
23:48
Play from here
Copy citation
And it just looks for new sources, like it doesn't really know what's going on under
23:51
Play from here
Copy citation
its own hood.
23:52
Play from here
Copy citation
But, hey, I'm a sucker for punishment and I did it again with this.
23:57
Play from here
Copy citation
And I asked for some specific use cases or prompts around these new policy changes and
24:02
Play from here
Copy citation
how they'd be different.
24:04
Play from here
Copy citation
And a couple that came out for legal would be writing a formal response or strategy for
24:09
Play from here
Copy citation
a case blocked because that is licensed legal work.
24:13
Play from here
Copy citation
Or a plea or defense strategy that is blocked because that is representation-level advice.
24:19
Play from here
Copy citation
For medical, the diagnosis or drug recommendation, and this is something that I leaned on very
24:25
Play from here
Copy citation
heavily before.
24:26
Play from here
Copy citation
Again, it's a weird one because you would still have to have a human in the loop because
24:32
Play from here
Copy citation
if it's a drug recommendation and you've got to go get that prescription, you're still
24:35
Play from here
Copy citation
confirming all of this with your doctor.
24:37
Play from here
Copy citation
And allegedly, that is now blocked because that is licensed a medical practice domain.
24:43
Play from here
Copy citation
Or the interpretation of labs into a treatment plan.
24:46
Play from here
Copy citation
I'm wondering where that line is going to be between the interpretation versus the treatment
24:51
Play from here
Copy citation
plan and how it's going to make those decisions.
24:53
Play from here
Copy citation
But it's calling that clinical decision-making.
24:55
Play from here
Copy citation
So those are just a few of the examples that it came up with, which I always think are
25:00
Play from here
Copy citation
interesting.
25:01
Play from here
Copy citation
Sure.
25:02
Play from here
Copy citation
We'll see how it evolves.
25:03
Play from here
Copy citation
We'll see how it evolves.
25:04
Play from here
Copy citation
Yeah.
25:05
Play from here
Copy citation
Nobody really knows.
25:06
Play from here
Copy citation
This is all kind of new.
25:07
Play from here
Copy citation
All right, Kevin, let's move into some custom GPT talk and we can talk about it as it pertains
25:16
Play from here
Copy citation
to internal workflow force multipliers.
25:21
Play from here
Copy citation
Why they're so helpful.
25:22
Play from here
Copy citation
I don't know if there's a primer that you want to open up with.
25:25
Play from here
Copy citation
I did a quick look into my sidebar of my account and it is like an endless lazy load of all
25:32
Play from here
Copy citation
these custom tools that I've made.
25:34
Play from here
Copy citation
I rely on these a ton.
25:37
Play from here
Copy citation
I think maybe we can start with the fact that this was something that was exclusive to OpenAI
25:42
Play from here
Copy citation
for a very long time as far as being able to share these tools with your teammates,
25:46
Play from here
Copy citation
other people on your account.
25:48
Play from here
Copy citation
Now Google has kind of evened out the playing field and you're able to do that with GEMS
25:53
Play from here
Copy citation
as well as custom GPTs now.
25:55
Play from here
Copy citation
So anything that you want to kind of come in with?
25:57
Play from here
Copy citation
Well, I think it's important to differentiate between custom GPTs, which have been around
26:02
Play from here
Copy citation
for a while and there is the marketplace for them, which for a couple of seconds, people
26:07
Play from here
Copy citation
thought was going to be a business opportunities.
26:10
Play from here
Copy citation
Like the new app store, right?
26:12
Play from here
Copy citation
Like the new app store.
26:13
Play from here
Copy citation
And I'm going to develop this GPT and I'll admit I built a GPT right out of the gate that
26:18
Play from here
Copy citation
does small business accounting advice for zero.
26:21
Play from here
Copy citation
And I just looked, it has 4,000 uses.
26:23
Play from here
Copy citation
So 4,000 people have used this little GPT that has contextualized zero accounting and
26:30
Play from here
Copy citation
like to do bank reconciliations or whatever.
26:33
Play from here
Copy citation
And to be honest, it's sort of hilariously bad, but it does its thing.
26:39
Play from here
Copy citation
And then there are projects, which is where I find most of my leverage.
26:45
Play from here
Copy citation
And I think that the first thing to do is to really differentiate between what custom
26:49
Play from here
Copy citation
GPTs do and what projects do and the difference between them and when you should use them.
26:56
Play from here
Copy citation
When I look at a GPT, I'm looking at doing the same thing many, many times.
27:03
Play from here
Copy citation
So it might be a specific type of transcript review.
27:07
Play from here
Copy citation
It might be some sort of an image thing.
27:08
Play from here
Copy citation
It might be a contract review, barring too much legal advice as we just discussed.
27:14
Play from here
Copy citation
But you're going to use this GPT over and over and over again and ideally multiple people
27:19
Play from here
Copy citation
in your organization.
27:20
Play from here
Copy citation
So you can share it amongst each other.
27:23
Play from here
Copy citation
Or if you're going to do something relatively complicated with external actions and you're
27:28
Play from here
Copy citation
not doing the development step.
27:30
Play from here
Copy citation
Both of our organizations would never bother with that because we have development capability.
27:35
Play from here
Copy citation
So we would like sort of create our own endpoints and we'd probably do workflow-based solutions
27:42
Play from here
Copy citation
as opposed to a GPT.
27:44
Play from here
Copy citation
But you can use GPTs that are triggered by external actions, and that's great.
27:51
Play from here
Copy citation
Major limitations of them are really in terms of retention of chats and organization of
27:58
Play from here
Copy citation
chats and then the reuse of chats.
28:01
Play from here
Copy citation
So if you go into my little zero bot, and it's really literally called zero bot, and
28:07
Play from here
Copy citation
you chat with it about your monthly reconciliation, it will give you an answer and that answer
28:11
Play from here
Copy citation
will just be lost in your chat history somewhere.
28:14
Play from here
Copy citation
You can't actually do anything with that.
28:17
Play from here
Copy citation
It's just this transitory chat that's sitting around there somewhere.
28:22
Play from here
Copy citation
Projects on the other hand, I really think are extremely powerful, particularly now that
28:28
Play from here
Copy citation
projects are shareable amongst teams.
28:31
Play from here
Copy citation
If you haven't looked at them before, you really need to.
28:34
Play from here
Copy citation
These are just completely integrated into the way we work.
28:37
Play from here
Copy citation
A project has a lot of the same features of a GPT.
28:41
Play from here
Copy citation
It has custom instructions.
28:43
Play from here
Copy citation
So what do you want the project to actually do?
28:48
Play from here
Copy citation
It's going to act as a business advisor or a fitness advisor or whatever it might be.
28:53
Play from here
Copy citation
It has an information resource.
28:54
Play from here
Copy citation
So what are the documents that you're going to use as far as the context of the entire
29:00
Play from here
Copy citation
conversation?
29:01
Play from here
Copy citation
But the unlock is in the organization that sits below there.
29:05
Play from here
Copy citation
So each individual chat will reference the custom instructions and reference the documents
29:14
Play from here
Copy citation
and be more topical.
29:16
Play from here
Copy citation
So it allows you to start the context of
29:19
Play from here
Copy citation
specific conversations a lot deeper into the process
29:24
Play from here
Copy citation
as opposed to spending all of this time
29:26
Play from here
Copy citation
prompting to get yourself into the right place.
29:30
Play from here
Copy citation
The example that I always give with context and prompting
29:33
Play from here
Copy citation
is, so context is the amount of information
29:37
Play from here
Copy citation
that an LLM can absorb at once.
29:40
Play from here
Copy citation
This continues to evolve,
29:42
Play from here
Copy citation
and you and I are going to end up talking a ton
29:44
Play from here
Copy citation
about the importance of context,
29:46
Play from here
Copy citation
because as context grows,
29:48
Play from here
Copy citation
the stability of responses gets better,
29:51
Play from here
Copy citation
the knowledge of the responses get better,
29:53
Play from here
Copy citation
and you have to do less technical applications
29:55
Play from here
Copy citation
of things like RAG and et cetera.
29:58
Play from here
Copy citation
So that's a really important topic.
30:01
Play from here
Copy citation
But when I'm talking about context,
30:02
Play from here
Copy citation
I often give the example of walking into Times Square
30:06
Play from here
Copy citation
and just yelling to the heavens, what should I do?
30:09
Play from here
Copy citation
Well, you're in New York,
30:10
Play from here
Copy citation
people are gonna have some pretty choice things
30:12
Play from here
Copy citation
to say to you, right?
30:13
Play from here
Copy citation
They have no idea who you are, what you need,
30:16
Play from here
Copy citation
et cetera, you're just yelling to the heavens.
30:18
Play from here
Copy citation
Well, that's the way that most people use LLMs.
30:20
Play from here
Copy citation
They just like yell to the heavens,
30:22
Play from here
Copy citation
help me out with this document
30:23
Play from here
Copy citation
that totally isn't legal advice.
30:24
Play from here
Copy citation
The right way to think about it is imagine instead
30:28
Play from here
Copy citation
that you walked into a bookstore
30:29
Play from here
Copy citation
and then walked to the back of the bookstore
30:31
Play from here
Copy citation
to, I don't know, the gardening section or whatever,
30:33
Play from here
Copy citation
and then you turn to the clerk there with their green vest
30:36
Play from here
Copy citation
and you say, what should I do?
30:38
Play from here
Copy citation
Well, they're gonna look at you like you're crazy
30:40
Play from here
Copy citation
because you're in a bookstore
30:41
Play from here
Copy citation
and you're standing in the gardening aisle.
30:42
Play from here
Copy citation
Now they have a really good idea
30:44
Play from here
Copy citation
of the types of things that you need to do.
30:47
Play from here
Copy citation
So what projects do is they allow you
30:50
Play from here
Copy citation
to build that bookstore
30:52
Play from here
Copy citation
and to build that shelf in the bookstore
30:55
Play from here
Copy citation
such that you can then walk right in and say,
30:58
Play from here
Copy citation
what should I do?
31:00
Play from here
Copy citation
Because the context of the bookstore
31:02
Play from here
Copy citation
and gardening are already there,
31:03
Play from here
Copy citation
it will then wrap on from there.
31:06
Play from here
Copy citation
So a typical project might be a client folder
31:09
Play from here
Copy citation
or something like that.
31:10
Play from here
Copy citation
And the other aspect of this we call document chaining.
31:13
Play from here
Copy citation
So we build context documents around what we're doing here.
31:18
Play from here
Copy citation
What's the background of the project?
31:19
Play from here
Copy citation
What's the scope of the project, et cetera.
31:21
Play from here
Copy citation
And we build all of that into the project knowledge.
31:24
Play from here
Copy citation
So now we're not just in the right section of the bookstore,
31:26
Play from here
Copy citation
we're in the right shelf in the bookstore.
31:28
Play from here
Copy citation
And if you wanna get to the right book
31:31
Play from here
Copy citation
as far as, I don't know, project deliverables
31:33
Play from here
Copy citation
or communications or whatever,
31:35
Play from here
Copy citation
then the individual chats in the project come to play.
31:40
Play from here
Copy citation
The other nuance and what sort of you
31:43
Play from here
Copy citation
and I were talking about offline about
31:45
Play from here
Copy citation
is that recently, sometime in the last few weeks,
31:49
Play from here
Copy citation
I'm not quite sure when this release happened,
31:52
Play from here
Copy citation
there became a bleed between the chats in the project.
31:56
Play from here
Copy citation
So that previously one chat would understand
32:00
Play from here
Copy citation
the custom instructions
32:02
Play from here
Copy citation
and it would understand the documents,
32:03
Play from here
Copy citation
but it wouldn't necessarily understand the other chats.
32:07
Play from here
Copy citation
So you might have planning and you might have communication,
32:09
Play from here
Copy citation
but they ended up like independent branches.
32:12
Play from here
Copy citation
Now those are bleeding together
32:14
Play from here
Copy citation
so it's that all of the chats in a project
32:16
Play from here
Copy citation
understand each other.
32:18
Play from here
Copy citation
That's really powerful because over time
32:21
Play from here
Copy citation
you're gaining more and more context,
32:23
Play from here
Copy citation
but you have purpose-built individual chats
32:26
Play from here
Copy citation
to accomplish specific tasks,
32:28
Play from here
Copy citation
which then you can share to other teammates.
32:32
Play from here
Copy citation
At the moment, it's only other teammates.
32:34
Play from here
Copy citation
You can't share them publicly,
32:35
Play from here
Copy citation
although that is definitely coming.
32:38
Play from here
Copy citation
Public sharing of projects is in the roadmap.
32:44
Play from here
Copy citation
Yeah, so it's a really, really powerful tool,
32:47
Play from here
Copy citation
really easy to set up.
32:48
Play from here
Copy citation
You can dictate whether or not
32:49
Play from here
Copy citation
a project sits in isolation too.
32:51
Play from here
Copy citation
So when you set up a new project now,
32:54
Play from here
Copy citation
it will trigger whether or not it knows
32:56
Play from here
Copy citation
the rest of your custom instructions,
32:58
Play from here
Copy citation
like you were talking about earlier in settings,
33:01
Play from here
Copy citation
you can set your preferences and your custom instructions
33:03
Play from here
Copy citation
and your memories and whatever,
33:06
Play from here
Copy citation
or if this is a special purpose project.
33:08
Play from here
Copy citation
And that's pretty useful if you're trying to say,
33:10
Play from here
Copy citation
get feedback on something and you want to lock it
33:14
Play from here
Copy citation
such that it doesn't know anything else about you
33:17
Play from here
Copy citation
except for the project that closes it down
33:21
Play from here
Copy citation
and prevents it from bleeding into other chats and memories
33:24
Play from here
Copy citation
that might actually change its outputs.
33:26
Play from here
Copy citation
That was a lot of words, Kevin.
33:28
Play from here
Copy citation
I should have just been taking notes
33:30
Play from here
Copy citation
because I have so many follow-ups and questions.
33:32
Play from here
Copy citation
As far as that cross-contextual conversation,
33:36
Play from here
Copy citation
I want to loop back to that
33:38
Play from here
Copy citation
because that is not just happening inside of projects.
33:41
Play from here
Copy citation
It's now happening between these custom GPTs as well.
33:44
Play from here
Copy citation
It just happened when I gave you the example
33:47
Play from here
Copy citation
of the unsolicited update to my instructions for this thing.
33:52
Play from here
Copy citation
It actually referenced another GPT
33:56
Play from here
Copy citation
that I made no mention of inside the instructions
33:59
Play from here
Copy citation
for this one, or I've never even really spoken about,
34:01
Play from here
Copy citation
but it grabbed that and said,
34:03
Play from here
Copy citation
oh, do a handoff to this other one.
34:06
Play from here
Copy citation
It's not only seeing these different conversation threads
34:09
Play from here
Copy citation
that you're having inside of projects
34:11
Play from here
Copy citation
and doing cross-referencing there,
34:13
Play from here
Copy citation
but it's starting to happen
34:14
Play from here
Copy citation
across the individual GPTs themselves,
34:18
Play from here
Copy citation
and it's starting to just happen
34:20
Play from here
Copy citation
with just the top-level prompts that you put in.
34:22
Play from here
Copy citation
It's happening all over the platform,
34:24
Play from here
Copy citation
and this is an example of one of the many things
34:27
Play from here
Copy citation
that we won't see an announcement about.
34:30
Play from here
Copy citation
It won't be in release notes or anything such as that,
34:34
Play from here
Copy citation
but if you start spending a bunch of time on this,
34:38
Play from here
Copy citation
you'll notice a completely different behavior
34:40
Play from here
Copy citation
start to surface.
34:41
Play from here
Copy citation
So that part, I mean, and like you said,
34:45
Play from here
Copy citation
it really does increase the power of projects for sure.
34:48
Play from here
Copy citation
I want to back up a little bit
34:50
Play from here
Copy citation
because I'm guessing there's a lot of people
34:52
Play from here
Copy citation
who we just went right over their heads.
34:55
Play from here
Copy citation
They're like, wait a minute, okay, the projects,
34:58
Play from here
Copy citation
the custom GPTs, okay, I think I kind of know the difference.
35:02
Play from here
Copy citation
Why would I really want one versus another?
35:06
Play from here
Copy citation
So as far as just the high-level labels,
35:09
Play from here
Copy citation
the thing that we're calling out for the most part
35:11
Play from here
Copy citation
is that a project, like its name,
35:13
Play from here
Copy citation
is probably something you're going to work on for a while.
35:15
Play from here
Copy citation
It's going to have a beginning, a middle, and an end,
35:18
Play from here
Copy citation
and that's going to be it,
35:19
Play from here
Copy citation
but the behaviors are very similar
35:21
Play from here
Copy citation
aside from that fact that you can go back
35:23
Play from here
Copy citation
and see all your grouped conversation threads,
35:25
Play from here
Copy citation
which is really, really nice,
35:26
Play from here
Copy citation
and now they're all talking to each other,
35:28
Play from here
Copy citation
whereas the custom GTP is a reusable tool.
35:33
Play from here
Copy citation
Like you said, you had your Xero coach, right,
35:35
Play from here
Copy citation
to help you with dealing with all of your accounting woes,
35:38
Play from here
Copy citation
and you could go back to that,
35:40
Play from here
Copy citation
and you've got a bunch of reference documents
35:42
Play from here
Copy citation
or your last year statements,
35:45
Play from here
Copy citation
whatever it might be looking at,
35:46
Play from here
Copy citation
and it's looking at that same information every time.
35:50
Play from here
Copy citation
So the components of a custom GTP are pretty simple.
35:55
Play from here
Copy citation
It's got these instructions.
35:57
Play from here
Copy citation
There is a, I think it's an 8,000 character limit
36:00
Play from here
Copy citation
as to how long those instructions can be,
36:02
Play from here
Copy citation
and it will tell you when you're up
36:04
Play from here
Copy citation
and not let you put in any more,
36:06
Play from here
Copy citation
but you can put in a ton of documents as well.
36:10
Play from here
Copy citation
Now, the type of documents that you put in also matters
36:14
Play from here
Copy citation
because you want this model to be going back
36:17
Play from here
Copy citation
and looking and referencing all this information
36:20
Play from here
Copy citation
that you put in there to come up with the best answers
36:22
Play from here
Copy citation
to whatever things you're trying to work through,
36:25
Play from here
Copy citation
and so, for example,
36:27
Play from here
Copy citation
a PDF doesn't get consumed quite as well
36:32
Play from here
Copy citation
as a doc X file for some reason, right?
36:35
Play from here
Copy citation
And a lot of this has to do with how are machines reading
36:38
Play from here
Copy citation
versus if you want it to really rip through stuff,
36:41
Play from here
Copy citation
you can attach an XML file, a JSON file,
36:44
Play from here
Copy citation
and before we start sounding too scary and technical,
36:48
Play from here
Copy citation
here's something you can do.
36:49
Play from here
Copy citation
You can just go right to your prompt with a Word doc
36:53
Play from here
Copy citation
or a PDF if it allows you to read it.
36:55
Play from here
Copy citation
Sometimes it just refuses to read PDFs for me,
36:58
Play from here
Copy citation
and you can say, turn this into a JSON file
37:01
Play from here
Copy citation
that will be ideal for consuming for custom GPT instructions
37:07
Play from here
Copy citation
and give me the download link,
37:09
Play from here
Copy citation
and you literally have to do nothing whatsoever
37:11
Play from here
Copy citation
as far as formatting, changing the file extension, nothing.
37:15
Play from here
Copy citation
You can just then attach that JSON file to your model
37:18
Play from here
Copy citation
and it's gonna be faster.
37:20
Play from here
Copy citation
You can have more information attached there.
37:22
Play from here
Copy citation
It'll be more responsive.
37:24
Play from here
Copy citation
So those are the real components,
37:27
Play from here
Copy citation
whatever attachments you have, the base instructions,
37:31
Play from here
Copy citation
and we could probably go on and on
37:32
Play from here
Copy citation
as to the best way to write those base instructions.
37:35
Play from here
Copy citation
Actually, to get really meta,
37:36
Play from here
Copy citation
I have a custom GPT for building custom GPTs
37:41
Play from here
Copy citation
that has a specific framework,
37:43
Play from here
Copy citation
and it goes through a whole question and answer process
37:46
Play from here
Copy citation
for me to get the best instructions,
37:49
Play from here
Copy citation
to get the best performance out of what I'm trying to do.
37:51
Play from here
Copy citation
So to be clear, you don't have to do that.
37:55
Play from here
Copy citation
Most people aren't going to do that,
37:57
Play from here
Copy citation
so let's not make it overly complicated.
38:00
Play from here
Copy citation
Speaking of getting weird and technical, yeah.
38:01
Play from here
Copy citation
I have attached a JSON file exactly zero times to a GPT.
38:07
Play from here
Copy citation
Really?
38:08
Play from here
Copy citation
But the context keeps expanding,
38:11
Play from here
Copy citation
that unless you're running into technical issues,
38:15
Play from here
Copy citation
this is that point of clarification and practicality.
38:17
Play from here
Copy citation
Unless you're, if you're running into issues,
38:20
Play from here
Copy citation
like the PDF isn't scanning,
38:22
Play from here
Copy citation
it's like spitting out bizarro information,
38:26
Play from here
Copy citation
or you've done something really complicated
38:28
Play from here
Copy citation
where the length of the document actually matters,
38:32
Play from here
Copy citation
then you should consider this stuff.
38:34
Play from here
Copy citation
This shouldn't be standard operating procedure,
38:36
Play from here
Copy citation
really, for anybody.
38:37
Play from here
Copy citation
That's a really good point,
38:38
Play from here
Copy citation
and I'm going to say yes and yes to both of those.
38:40
Play from here
Copy citation
I've gotten incredibly long documents that,
38:45
Play from here
Copy citation
I think a lot of this, until you try it,
38:47
Play from here
Copy citation
you don't know what your outcome is going to be, right?
38:50
Play from here
Copy citation
It's just trial and error.
38:51
Play from here
Copy citation
Oh, this worked.
38:52
Play from here
Copy citation
This is the performance I wanted.
38:55
Play from here
Copy citation
This didn't work.
38:56
Play from here
Copy citation
This is super slow where I can tell
38:57
Play from here
Copy citation
it's not referencing all of the information
38:59
Play from here
Copy citation
that I attached, and that's why, by default,
39:01
Play from here
Copy citation
I've almost gone to exclusively JSON files
39:05
Play from here
Copy citation
for a bunch of text, which is pretty funny.
39:07
Play from here
Copy citation
I think also, I wanted to circle back
39:10
Play from here
Copy citation
to how you brought up,
39:11
Play from here
Copy citation
oh, we both have these technical teams
39:14
Play from here
Copy citation
that can build these custom solutions for us,
39:16
Play from here
Copy citation
so maybe for us, we wouldn't have that many custom GPTs.
39:22
Play from here
Copy citation
I've got to say that I've got a ton of them,
39:26
Play from here
Copy citation
even with the technical team.
39:28
Play from here
Copy citation
The reason being is that these things,
39:31
Play from here
Copy citation
barring the converting things to JSON file,
39:34
Play from here
Copy citation
just forget that I said about that,
39:36
Play from here
Copy citation
really require no technical expertise, right?
39:40
Play from here
Copy citation
There is some good prompt engineering.
39:42
Play from here
Copy citation
There is good instruction writing,
39:43
Play from here
Copy citation
and that's kind of going back to the trial and error
39:45
Play from here
Copy citation
of getting this thing to respond
39:47
Play from here
Copy citation
the way that you want it to,
39:49
Play from here
Copy citation
but that said, you can do these lightweight prototypes
39:53
Play from here
Copy citation
that then become these shareable tools for your team
39:56
Play from here
Copy citation
really, really fast, and you can get down that road
40:00
Play from here
Copy citation
of a certain process, see what the outcomes are,
40:02
Play from here
Copy citation
and if it's something that you want to start
40:04
Play from here
Copy citation
to really finely tune and make specific
40:08
Play from here
Copy citation
for your organization, that's where
40:10
Play from here
Copy citation
the custom implementation comes in.
40:12
Play from here
Copy citation
That said, I've been leaning on some
40:14
Play from here
Copy citation
of these custom shareable GPTs for a long, long time,
40:19
Play from here
Copy citation
and I've gone back and updated a lot of them.
40:22
Play from here
Copy citation
That said, just the performance,
40:25
Play from here
Copy citation
sometimes I shrug, and I'm like,
40:26
Play from here
Copy citation
why make something custom?
40:28
Play from here
Copy citation
But again, you and I have very close-knit teams,
40:31
Play from here
Copy citation
and we can move really quickly,
40:33
Play from here
Copy citation
and so it's a little bit different than a big organization.
40:35
Play from here
Copy citation
Well, don't get me wrong.
40:37
Play from here
Copy citation
When I speak, I have what I call the face-palm slide,
40:40
Play from here
Copy citation
so we go through all of the different cool things
40:43
Play from here
Copy citation
that you can do with AI, and people are overwhelmed,
40:46
Play from here
Copy citation
and they're like, oh my gosh, this is completely ridiculous.
40:49
Play from here
Copy citation
And then I recenter, and I say, okay,
40:53
Play from here
Copy citation
if you do nothing else, sign up for a GPT,
40:59
Play from here
Copy citation
or an OpenAI business account, so you have shareability,
41:03
Play from here
Copy citation
and you have some terms of service protection of your data.
41:07
Play from here
Copy citation
20 bucks a month or so, right?
41:09
Play from here
Copy citation
It's actually at 30 bucks a month a user at this point,
41:11
Play from here
Copy citation
but that's okay, that's not a ton.
41:13
Play from here
Copy citation
Educate.
41:14
Play from here
Copy citation
I didn't notice.
41:15
Play from here
Copy citation
That must have been a terms of service update, too,
41:17
Play from here
Copy citation
that I just didn't look at.
41:19
Play from here
Copy citation
Boiling frog stuff.
41:20
Play from here
Copy citation
Educate your people on how to use them,
41:22
Play from here
Copy citation
because it does take a little bit of education.
41:24
Play from here
Copy citation
It does take some reps to know how to use GPT properly.
41:28
Play from here
Copy citation
I mean, everyone thinks they know how to use it,
41:30
Play from here
Copy citation
but we find that most people are just scraping the surface.
41:35
Play from here
Copy citation
It's sort of like Excel.
41:36
Play from here
Copy citation
Most people use Excel to do some basic arithmetic,
41:39
Play from here
Copy citation
or whatever it might be.
41:41
Play from here
Copy citation
Very few people get into a lot of its features,
41:43
Play from here
Copy citation
but there's this mid-band of capability
41:45
Play from here
Copy citation
that you should know how to do, you know how to use,
41:48
Play from here
Copy citation
and a key part there is knowing how to build
41:50
Play from here
Copy citation
these custom GPTs, or knowing how to use these projects,
41:53
Play from here
Copy citation
and then have each person on your team identify,
41:57
Play from here
Copy citation
I don't know, between one and five things
42:00
Play from here
Copy citation
that cause impediments in their day,
42:02
Play from here
Copy citation
or friction in their day, or just unhappiness in their day,
42:06
Play from here
Copy citation
as far as the things that they just don't like to do,
42:09
Play from here
Copy citation
and build a GPT that addresses
42:11
Play from here
Copy citation
at least part of that problem.
42:13
Play from here
Copy citation
Don't try and automate the whole thing away,
42:15
Play from here
Copy citation
and if you do that across a team of 10, 15, 20 people,
42:19
Play from here
Copy citation
all of a sudden, you have a team
42:21
Play from here
Copy citation
that is more acclimatized to using the technology,
42:25
Play from here
Copy citation
and they're more enthusiastic about it,
42:27
Play from here
Copy citation
because it's not this esoteric product-facing thing
42:31
Play from here
Copy citation
that's hard for them to grasp.
42:32
Play from here
Copy citation
It's something practical that solves problems in their day,
42:36
Play from here
Copy citation
and that they can share across the team,
42:39
Play from here
Copy citation
and it's really hard not to gain operational advantage
42:43
Play from here
Copy citation
from something like that.
42:44
Play from here
Copy citation
Yeah, I totally agree, and if you're that person
42:47
Play from here
Copy citation
in your business who is thinking through a lot of systems,
42:51
Play from here
Copy citation
or writing SOPs, or doing any of that,
42:53
Play from here
Copy citation
this is an opportunity to productize that
42:57
Play from here
Copy citation
without having to go and jump through all the hoops
42:59
Play from here
Copy citation
of actually building a product.
43:00
Play from here
Copy citation
You can do this no-code, and then share out this process,
43:04
Play from here
Copy citation
share out this thing.
43:05
Play from here
Copy citation
I mean, here's a great example,
43:07
Play from here
Copy citation
because here we are on the show.
43:09
Play from here
Copy citation
It's our second episode.
43:10
Play from here
Copy citation
The first episode, we thought,
43:12
Play from here
Copy citation
hey, we have endless things to talk about.
43:13
Play from here
Copy citation
Let's just do a shared Google Doc,
43:15
Play from here
Copy citation
and we'll throw some notes in there,
43:16
Play from here
Copy citation
and we'll see what happens, and yada, yada,
43:18
Play from here
Copy citation
and then we ended up with this big, messy,
43:20
Play from here
Copy citation
sprawling document.
43:21
Play from here
Copy citation
I didn't even know really what half the stuff meant,
43:23
Play from here
Copy citation
everything else, and then I thought,
43:24
Play from here
Copy citation
boy, I think we talked about some of these things, maybe.
43:27
Play from here
Copy citation
I guess the show producer should go back and find those,
43:31
Play from here
Copy citation
and remove the things that we did talk about,
43:33
Play from here
Copy citation
and if we have new notes, somehow elevate those,
43:36
Play from here
Copy citation
and somehow round that all up
43:38
Play from here
Copy citation
to find the most value-add topics and everything else.
43:42
Play from here
Copy citation
I thought, well, or we could just create a custom GPT
43:47
Play from here
Copy citation
that does that, where with all the other embedded AI tools
43:50
Play from here
Copy citation
and things like this podcast recorder,
43:52
Play from here
Copy citation
we can just paste in the transcript,
43:54
Play from here
Copy citation
and through those instructions,
43:56
Play from here
Copy citation
tell us exactly what we want it to do with this document.
43:59
Play from here
Copy citation
And I don't even have to really give the specific instructions to that show producer.
44:05
Play from here
Copy citation
All I have to do is share out the tool.
44:07
Play from here
Copy citation
I just click share. He's on my team. He's got the tool.
44:10
Play from here
Copy citation
He will just put in the prompt, let's go,
44:13
Play from here
Copy citation
and then it asks him for the very specific assets that it needs to do its job.
44:17
Play from here
Copy citation
It gives him the output that he'd paste into that Google Doc,
44:20
Play from here
Copy citation
and Kevin and I never have to think about it again.
44:22
Play from here
Copy citation
Like it's really that simple and super powerful.
44:26
Play from here
Copy citation
So this probably won't be the last time I say this because I say this all the time,
44:29
Play from here
Copy citation
but in our own organization, right at the top of our weekly meeting,
44:35
Play from here
Copy citation
it says the first time we do something, it's an adventure.
44:39
Play from here
Copy citation
The second time we do it, it's an education.
44:41
Play from here
Copy citation
The third time we do it, it's an automation.
44:43
Play from here
Copy citation
Because if you've done it three times, you are likely to do it four, five, six or more.
44:49
Play from here
Copy citation
And if you automate it before you've had that adventure and education,
44:55
Play from here
Copy citation
you're likely to overcomplicate it and create structure that the process has to conform to,
45:03
Play from here
Copy citation
as opposed to the process conforming to the structure.
45:06
Play from here
Copy citation
So I don't like busting out the automations right away,
45:11
Play from here
Copy citation
but I am serious about it in the organization.
45:14
Play from here
Copy citation
And I really try and walk the walk myself when I realize, oh my gosh,
45:17
Play from here
Copy citation
this is the third time I'm doing something.
45:19
Play from here
Copy citation
And yes, it might create a little bit of a rabbit hole of vibe coding
45:22
Play from here
Copy citation
or doing a workflow or whatever it is.
45:25
Play from here
Copy citation
But when you're trying to scale an organization,
45:29
Play from here
Copy citation
making those decisions early to find those pieces of automation,
45:33
Play from here
Copy citation
honestly is the difference between running an AI native organization
45:38
Play from here
Copy citation
and an organization that's trying to adapt to AI.
45:42
Play from here
Copy citation
And it's really easy to do that with AirQuotes,
45:47
Play from here
Copy citation
because we're building organizations from the ground up that get to make these choices.
45:52
Play from here
Copy citation
Whereas most organizations are dealing with existing structures
45:55
Play from here
Copy citation
that they have to figure out how to basically deconstruct and fix.
46:00
Play from here
Copy citation
So it's such a luxury to have.
46:02
Play from here
Copy citation
And I know that future me is going to thank today me
46:07
Play from here
Copy citation
for taking that extra couple hours to automate these things.
46:10
Play from here
Copy citation
Every time, every time.
46:12
Play from here
Copy citation
I've lost track of how many times I've glanced back and been so grateful
46:15
Play from here
Copy citation
that I just automated that damn thing or wrote the documentation or built the system.
46:21
Play from here
Copy citation
I mean, even having the place to point somebody who's new in your organization to,
46:27
Play from here
Copy citation
hey, this is the playbook.
46:28
Play from here
Copy citation
This is exactly how this works.
46:30
Play from here
Copy citation
And even better, the step above that, have a tool that just does that for you.
46:34
Play from here
Copy citation
So I love what you said.
46:37
Play from here
Copy citation
When you automate something too early that's filled with a bunch of mistakes,
46:42
Play from here
Copy citation
the first risk is that you're just amplifying all those mistakes, right?
46:47
Play from here
Copy citation
Now they're all automated and you just have these bad outputs coming out nonstop.
46:52
Play from here
Copy citation
The other risk, of course, is that you might have an idea of a new process.
46:57
Play from here
Copy citation
And it sounds great on theory.
46:59
Play from here
Copy citation
It even sounds great when you're writing the documentation.
47:02
Play from here
Copy citation
But in practice, it just wasn't worth something.
47:07
Play from here
Copy citation
It just didn't work.
47:09
Play from here
Copy citation
Like the entire thing fell flat.
47:10
Play from here
Copy citation
That's happened to me a bunch of times.
47:12
Play from here
Copy citation
I might have overcomplicated it.
47:14
Play from here
Copy citation
That is very, very common for me.
47:16
Play from here
Copy citation
So having this ability to build a lightweight prototype,
47:20
Play from here
Copy citation
like so if you look at the evolution of what we were just talking about,
47:23
Play from here
Copy citation
phase one was a sloppy Google document, right,
47:27
Play from here
Copy citation
that us mere mortals were just kind of throwing stuff on
47:31
Play from here
Copy citation
and then trying to decipher at the moment of truth.
47:34
Play from here
Copy citation
Phase two is building this custom GPT that takes a process that we thought of
47:41
Play from here
Copy citation
and lets a third party do it for us with this tool.
47:45
Play from here
Copy citation
There's still some copying and pasting and maybe a calendar event
47:48
Play from here
Copy citation
or a reminder of this has to happen, right?
47:51
Play from here
Copy citation
Phase three would be like, oh, we really like the way this thing works.
47:54
Play from here
Copy citation
We could automate this.
47:55
Play from here
Copy citation
We could put it on a cron, which is just basically a calendar and clock event
47:59
Play from here
Copy citation
that says, you know, on this day, you run this process.
48:02
Play from here
Copy citation
As you go fetch this information from this Google doc,
48:04
Play from here
Copy citation
you hit this API and you get this transcript and it all happens.
48:08
Play from here
Copy citation
And then when we open the document on Monday morning,
48:11
Play from here
Copy citation
everything is just there with no human interaction.
48:14
Play from here
Copy citation
That would be the third phase.
48:16
Play from here
Copy citation
But we'll probably give it a few episodes to see if we like the way that this one worked, right?
48:21
Play from here
Copy citation
Because that's a sensible thing to do.
48:22
Play from here
Copy citation
I was able to put that entire thing together in, it took me about two hours, right?
48:28
Play from here
Copy citation
It will save roughly three human hours every week while we're deciding what to do next
48:33
Play from here
Copy citation
and testing that system.
48:34
Play from here
Copy citation
That's a good investment.
48:36
Play from here
Copy citation
That's a really good trade-off for that time.
48:39
Play from here
Copy citation
When we do the automation, yeah, now it's going to save even more time
48:43
Play from here
Copy citation
and there will be no calendar events or no nothing.
48:46
Play from here
Copy citation
It will just auto-magically happen for us in the background,
48:49
Play from here
Copy citation
and that's kind of how that process works.
48:51
Play from here
Copy citation
You know, I'm also going to point out that in the thinking process for both of us,
48:57
Play from here
Copy citation
it is about that ROI.
48:58
Play from here
Copy citation
So identifying where the ROI sits as well.
49:02
Play from here
Copy citation
Is this just sort of a fun thing that we're going to do because it's cool to do it?
49:07
Play from here
Copy citation
Or is it something that's going to save, quote, three human hours a week?
49:11
Play from here
Copy citation
Okay, so now we have a KPI.
49:13
Play from here
Copy citation
We have this is what we want to do.
49:16
Play from here
Copy citation
We want to save a team member three hours a week by doing this.
49:20
Play from here
Copy citation
So the right thing to do as far as assessing project success,
49:25
Play from here
Copy citation
and we might not do it in this case,
49:27
Play from here
Copy citation
but if we were looking at our own organizations
49:29
Play from here
Copy citation
or we were looking at client organizations,
49:32
Play from here
Copy citation
it would be pegging what that desired output is
49:36
Play from here
Copy citation
and then coming back and looking at whether or not that was achieved.
49:40
Play from here
Copy citation
Did we actually save this person three hours,
49:42
Play from here
Copy citation
or did we just take that three hours and spread it over a whole bunch of more tools
49:46
Play from here
Copy citation
and made their life miserable in some way, shape, or form?
49:49
Play from here
Copy citation
Where is the ROI, and is it worth the dollar investment to save it over that one person,
49:55
Play from here
Copy citation
or are there 10 people that it's going to save that three hours over, right?
50:00
Play from here
Copy citation
And then when you're triangulating the sort of projects you take on,
50:03
Play from here
Copy citation
you want to consider what is the ROI versus what is the project risk
50:08
Play from here
Copy citation
and the time to implementation, and all of that sort of that mental jiu-jitsu
50:15
Play from here
Copy citation
goes into considering which projects to take on when,
50:19
Play from here
Copy citation
and because we sit at the top of our organizations,
50:22
Play from here
Copy citation
anything that creates friction for us, well, strangely enough, those things get solved.
50:26
Play from here
Copy citation
Exactly, exactly.
50:28
Play from here
Copy citation
And I think I'll point out, too, you know, it's interesting what you said about where is the ROI,
50:32
Play from here
Copy citation
where is this really going to be useful in terms of number of human hours saved?
50:36
Play from here
Copy citation
These things really come with practice.
50:39
Play from here
Copy citation
They really do.
50:40
Play from here
Copy citation
You've probably heard before that creativity is a muscle that you have to train,
50:44
Play from here
Copy citation
problem solving is a muscle that you have to train,
50:47
Play from here
Copy citation
and the more of these that you try to pick out, and you can start with really easy ones,
50:51
Play from here
Copy citation
and they may even seem novel at first, almost inevitably lead to more complex problems
50:57
Play from here
Copy citation
that save more hours and help more people,
51:00
Play from here
Copy citation
but it's one of those things that if you never start,
51:03
Play from here
Copy citation
you never get to that point where you really start seeing the returns.
51:06
Play from here
Copy citation
So I just, I can't say enough.
51:10
Play from here
Copy citation
The hands-on problem solving with this makes all the world of a difference.
51:16
Play from here
Copy citation
Like you said, you know, am I spending time on something that may not really have that big of a payback
51:21
Play from here
Copy citation
and it just kind of seems novel?
51:23
Play from here
Copy citation
My answer to that, starting off in this, was yes, that everything kind of fits under that domain.
51:29
Play from here
Copy citation
I think this would just be cool for the sake of doing it.
51:32
Play from here
Copy citation
I want to see if I can do it.
51:34
Play from here
Copy citation
That's also not time wasted.
51:37
Play from here
Copy citation
You know, that is also kind of stretching your creative thinking, your problem solving,
51:42
Play from here
Copy citation
just to see if you could do it.
51:44
Play from here
Copy citation
And for me, every one of those that I've done has also led to something that had real ROI with time saved as well.
51:51
Play from here
Copy citation
So I'd say the answer is both.
51:52
Play from here
Copy citation
Do both.
51:53
Play from here
Copy citation
Have fun with it, but also solve some real problems.
51:56
Play from here
Copy citation
And do it as often as you can.
51:57
Play from here
Copy citation
We need to dive into some practical applications of how we go about this,
52:01
Play from here
Copy citation
because, okay, we made big noises about having TechBench and all of these other things,
52:05
Play from here
Copy citation
but a lot of these things do start in just GPT or using a workflow tool like N8n.
52:12
Play from here
Copy citation
And I think that's a good bookmark for even next time to sort of lay out some of the practical tools that we're deploying
52:20
Play from here
Copy citation
and apply some costs to them.
52:22
Play from here
Copy citation
Some of this stuff doesn't scale across really large enterprises,
52:26
Play from here
Copy citation
but works really, really well for testing out whether or not a smaller solution is going to work.
52:31
Play from here
Copy citation
And it's incredibly accessible to anybody who has some curiosity
52:35
Play from here
Copy citation
and some understanding of how process is supposed to work.
52:40
Play from here
Copy citation
Yeah.
52:41
Play from here
Copy citation
I want to loop back to another thing you said.
52:43
Play from here
Copy citation
I'm so glad I remembered this as well.
52:45
Play from here
Copy citation
When you're talking about the differentiation between these custom GPTs and projects
52:49
Play from here
Copy citation
and having this nice consolidated group of conversation threads that you've had inside of a project,
52:55
Play from here
Copy citation
whereas in a custom GPT, every time you start a new conversation, all of that previous context is lost, right?
53:01
Play from here
Copy citation
So somebody might ask if a listener hasn't had really long conversations, very long threads,
53:09
Play from here
Copy citation
well, why can't you just keep going?
53:11
Play from here
Copy citation
It eventually bogs down.
53:12
Play from here
Copy citation
Every time you put something into that context window,
53:16
Play from here
Copy citation
it's going back and referencing everything else that you put into that context window.
53:21
Play from here
Copy citation
The longer that conversation is, the slower your responses start to be.
53:25
Play from here
Copy citation
Until the point that you are trying to download a photo back in 1999, watching it, come one row of pixels at a time.
53:34
Play from here
Copy citation
It just gets incredibly painful.
53:37
Play from here
Copy citation
So one workaround in the GPTs as opposed to the projects for this, and this isn't too technical,
53:48
Play from here
Copy citation
even though Kevin's probably going to call me out for saying this is too technical.
53:51
Play from here
Copy citation
In those instructions, you can build in something so that when you see that conversation starting to get really slow
53:59
Play from here
Copy citation
and you can see the thread is starting to struggle.
54:01
Play from here
Copy citation
As you're going on, what I'll commonly do in those instructions is say that,
54:05
Play from here
Copy citation
hey, you're going to expect me to call out tags or labels.
54:09
Play from here
Copy citation
In other words, we spoke about something and I think it's really important.
54:12
Play from here
Copy citation
And I just say, make sure to tag that, right?
54:15
Play from here
Copy citation
So here I'll have a series of different labels like this was about accounting.
54:20
Play from here
Copy citation
This was about RevOps, whatever it is.
54:23
Play from here
Copy citation
And when that thread starts to get really pokey, I have a different command that's also referenced in those instructions
54:30
Play from here
Copy citation
that basically says, pack up all the things that we deemed important in this conversation.
54:34
Play from here
Copy citation
I save that as a, it's called a YAML file.
54:37
Play from here
Copy citation
I know that sounds weird.
54:38
Play from here
Copy citation
I attach that to that custom GPT and now it has this living reference of our ongoing conversation.
54:45
Play from here
Copy citation
This is a really ghetto workaround of just building something custom that has perpetual memory.
54:52
Play from here
Copy citation
But I personally often get stuck in that in-between space of I still want to use this tool,
54:59
Play from here
Copy citation
but I'm not exactly sure what the specifications would be for something custom.
55:03
Play from here
Copy citation
And I don't want to lose these threads.
55:06
Play from here
Copy citation
That has been a really good solution for me because I just keep overriding that single file.
55:11
Play from here
Copy citation
So I'm going to inject sort of as a closing topic here,
55:14
Play from here
Copy citation
why this also has ramifications about how people are positioning the AI bubble
55:22
Play from here
Copy citation
and the need for more and more process and data centers and et cetera.
55:26
Play from here
Copy citation
So you have, I think most people are aware at this point that the large language models themselves
55:32
Play from here
Copy citation
had to be trained on trillions of points of data and that took enormous processing capability.
55:37
Play from here
Copy citation
So those are the training runs that can cost millions, tens of millions,
55:41
Play from here
Copy citation
hundreds of millions, soon to be billions of dollars to do.
55:44
Play from here
Copy citation
But then you have inference.
55:46
Play from here
Copy citation
And inference is basically the ongoing cost of all of us chatting with the systems.
55:54
Play from here
Copy citation
And this is what's going on right now.
55:57
Play from here
Copy citation
And it's what's driving a lot of the process load in the system.
56:01
Play from here
Copy citation
Because my response to what you were just laying out is, meh, context is going to get there.
56:09
Play from here
Copy citation
This is not something that worries me too much.
56:11
Play from here
Copy citation
What about all those precious conversations you're having right now, aren't you?
56:15
Play from here
Copy citation
So my shorthand is generally all of the LLMs want to output full solutions at once to say,
56:23
Play from here
Copy citation
let's go step by step.
56:25
Play from here
Copy citation
Do an outline.
56:26
Play from here
Copy citation
Then do step one.
56:28
Play from here
Copy citation
Then do step two.
56:29
Play from here
Copy citation
Then do step three.
56:30
Play from here
Copy citation
One of my favorite prompts.
56:31
Play from here
Copy citation
I'm so glad you brought that up.
56:32
Play from here
Copy citation
It drives me crazy having to go back and look at four pages of an output.
56:36
Play from here
Copy citation
And it's just burning tokens, right?
56:38
Play from here
Copy citation
Yeah, totally.
56:39
Play from here
Copy citation
Also feature being able to use GPT Canvas can be helpful in that respect as well for editing individual things.
56:46
Play from here
Copy citation
But you're burning less inference because you're working on a specific section at once.
56:52
Play from here
Copy citation
And you're burning less context.
56:54
Play from here
Copy citation
Because if you run that same, this happened to me this morning where I was working with Claude and trying to run a pretty big presentation through it.
57:02
Play from here
Copy citation
And it kept rerunning it.
57:04
Play from here
Copy citation
Well, each one of those runs is becoming part of that context, which is part of the inference.
57:10
Play from here
Copy citation
So it gets bloated over time and more expensive, like actually in dollars, for the LLM to do that bigger process.
57:21
Play from here
Copy citation
And, yes, you will run out of context.
57:24
Play from here
Copy citation
But this is an arms race of who can offer the most context faster for the users so they don't have to do this stuff because most people aren't going to have to do this.
57:34
Play from here
Copy citation
Most people aren't going to do it.
57:36
Play from here
Copy citation
You'll know when your project or your GPT is sort of running out of gas or an individual chat, to be honest.
57:44
Play from here
Copy citation
Chat hygiene is to sort of keep things in a specific chat such that if you're going to talk about, oh, I don't know, your personal health insurance or whatever it is, all of that information is in that one chat.
57:56
Play from here
Copy citation
So give it a good title that you remember and you go back to that.
57:59
Play from here
Copy citation
Well, if you have that chat hundreds and hundreds of times, eventually it's going to basically fill up its context and it will no longer be able to respond.
58:10
Play from here
Copy citation
Unfortunately, there's no red light, green light on this.
58:12
Play from here
Copy citation
It's just going to start giving you erroneous results or starting to hallucinate or whatever it is.
58:17
Play from here
Copy citation
But those context limits get bigger by the day.
58:21
Play from here
Copy citation
Gemini is at one million or two million tokens at the moment.
58:27
Play from here
Copy citation
Depending on the uses of open AI, it's hundreds of thousands.
58:31
Play from here
Copy citation
You can get by with a lot before it starts jamming its way up.
58:36
Play from here
Copy citation
But when people talk about the sort of the bubble.
58:39
Play from here
Copy citation
looking at things like these geometric use curves.
58:45
Play from here
Copy citation
So every time a new person comes into the system, hey world, how's it going?
58:49
Play from here
Copy citation
Great.
58:50
Play from here
Copy citation
Okay.
58:51
Play from here
Copy citation
That's very low inference.
58:52
Play from here
Copy citation
But over time, that will start building up and as the use patterns increase, so does
58:57
Play from here
Copy citation
the amount of process that needs to be used by individual users, the more Jensen Wang
59:04
Play from here
Copy citation
makes money on GPUs.
59:06
Play from here
Copy citation
Exactly.
59:07
Play from here
Copy citation
Yeah.
59:08
Play from here
Copy citation
And it just all comes down to the amount of processing power, how you choose to use these
59:12
Play from here
Copy citation
tools and really how they're going to best serve you with your own specific use cases.
59:20
Play from here
Copy citation
My wife's a therapist and I'm endlessly fascinated with what kind of models can I put together
59:28
Play from here
Copy citation
that give me these human-like responses, as creepy as that is.
59:33
Play from here
Copy citation
But in order to effectively do something like that, you have to have a lot of context for
59:38
Play from here
Copy citation
these ongoing conversations, right?
59:41
Play from here
Copy citation
So you come to a fork in the road, you build something custom that has that perpetual memory,
59:48
Play from here
Copy citation
or you wait for things to slow down and you pack up your conversation with all the important
59:51
Play from here
Copy citation
things you tagged and you reattach it to your custom GPT.
59:55
Play from here
Copy citation
It's a way around it if you do have cases where you're having these very long conversations.
01:00:01
Play from here
Copy citation
Another thing I'll bring up, as you referenced having everything consolidated in the projects
01:00:05
Play from here
Copy citation
as far as those conversation threads, if you haven't used it, the search is shockingly
01:00:09
Play from here
Copy citation
good for finding previous conversations.
01:00:12
Play from here
Copy citation
I will often just pop open a new thread to ask about something random and that leads
01:00:17
Play from here
Copy citation
to something more specific.
01:00:19
Play from here
Copy citation
And then I have another question and pretty soon I'm in the weeds of a lot of real useful
01:00:24
Play from here
Copy citation
information that for some reason or another I then need to go back weeks or a month later
01:00:30
Play from here
Copy citation
and I'm like, wait a minute, that was such a good output and I didn't put it anywhere
01:00:34
Play from here
Copy citation
and it's just lost in a billion different threads.
01:00:37
Play from here
Copy citation
Just picking out a handful of the keywords in search is really good with surfacing those
01:00:42
Play from here
Copy citation
historical threads.
01:00:44
Play from here
Copy citation
I was actually surprised, it's really a feature I've only started using heavily in the last
01:00:48
Play from here
Copy citation
couple of months.
01:00:49
Play from here
Copy citation
Kevin, we've been talking a really long time about AI already and we had a couple more
01:00:53
Play from here
Copy citation
things.
01:00:54
Play from here
Copy citation
This is so predictable that we'd have more to talk about than we do.
01:00:58
Play from here
Copy citation
Time on the clock.
01:00:59
Play from here
Copy citation
I did have two more things and we can punt one of these.
01:01:03
Play from here
Copy citation
We really did touch on some of these new updates with this cross-memory across these different
01:01:11
Play from here
Copy citation
custom GPTs, right?
01:01:14
Play from here
Copy citation
And we also touched on the fact that it's happening platform-wide, it's not even really
01:01:18
Play from here
Copy citation
consolidated to that and it's one of those things that as you use the product, you'll
01:01:23
Play from here
Copy citation
notice a difference.
01:01:24
Play from here
Copy citation
That's usually how we find out about some of these more subtle features.
01:01:29
Play from here
Copy citation
So what we're talking about is historically, you might have a thread and you're having
01:01:33
Play from here
Copy citation
a conversation about a specific topic and then you start a new thread and you have a
01:01:38
Play from here
Copy citation
new conversation about a different topic.
01:01:41
Play from here
Copy citation
If you said, oh, well, hey, do you remember we talked about this and you told me that,
01:01:45
Play from here
Copy citation
it would say, I have no idea what you're talking about, that was a different thread.
01:01:48
Play from here
Copy citation
No idea, we're in different parts of the world right now.
01:01:53
Play from here
Copy citation
Not anymore, except it's not exactly clear as to what it does choose to bring over and
01:02:00
Play from here
Copy citation
go back because it doesn't happen across the board.
01:02:04
Play from here
Copy citation
In other words, I'm not saying it remember everything that you discussed in every thread,
01:02:09
Play from here
Copy citation
but it will cherry pick these things and bring that knowledge over into a new thread, a new
01:02:16
Play from here
Copy citation
project, a new custom GPT.
01:02:20
Play from here
Copy citation
Super interesting and kind of like this ever-growing expanse of context and number of tokens that
01:02:26
Play from here
Copy citation
it'll take and go back and be able to reference, this also changes the game in a very similar
01:02:31
Play from here
Copy citation
way.
01:02:32
Play from here
Copy citation
Absolutely.
01:02:33
Play from here
Copy citation
And there's a hierarchy of the way that this seems to work.
01:02:38
Play from here
Copy citation
They have in their own documentation, they do indicate some of this.
01:02:42
Play from here
Copy citation
Custom instructions are basically a system prompt that's set in your personalization.
01:02:48
Play from here
Copy citation
We were talking about that a little bit earlier, but this is the generic system prompt that
01:02:53
Play from here
Copy citation
identifies the way you want OpenAI or Cloud or whatever it is to respond to you.
01:03:02
Play from here
Copy citation
In my case, it includes a bunch of words that I hate it using that are very AI horrific.
01:03:08
Play from here
Copy citation
I have so many of those.
01:03:11
Play from here
Copy citation
Yeah, delve.
01:03:15
Play from here
Copy citation
Anyway, we could delve into delving and why delving is an important term.
01:03:19
Play from here
Copy citation
We could delve into AI vocabulary for a whole hour.
01:03:21
Play from here
Copy citation
There's a thing there as to why the term delve comes up a lot.
01:03:24
Play from here
Copy citation
It has to do with human reinforced training, but whatever.
01:03:26
Play from here
Copy citation
I don't want to geek on by that right yet.
01:03:28
Play from here
Copy citation
But things like straw man arguments, make sure that you're not being overly sycophantic,
01:03:34
Play from here
Copy citation
although it will be sycophantic anyway.
01:03:36
Play from here
Copy citation
Double check things.
01:03:37
Play from here
Copy citation
So that all sits on the top.
01:03:38
Play from here
Copy citation
And then you have your memories.
01:03:40
Play from here
Copy citation
And this is really important to know that it will pick up incredibly random things about you.
01:03:46
Play from here
Copy citation
And this is GPT specific.
01:03:48
Play from here
Copy citation
If I say I'm CEO of Ascend Labs, underneath that chat or in that prompt line, it'll say memory updated.
01:03:56
Play from here
Copy citation
And all of those memories accumulate as a second system prompt.
01:04:01
Play from here
Copy citation
So, oh, okay, it doesn't want me to use the word delve, and he's CEO of Ascend Labs.
01:04:06
Play from here
Copy citation
But it will sort of randomly pick things up over time.
01:04:10
Play from here
Copy citation
And you need to get into your memories and purge them now and again.
01:04:14
Play from here
Copy citation
First, they will fill up.
01:04:16
Play from here
Copy citation
Now, I'm a complete cyborg power user, so mine fills up really quick.
01:04:20
Play from here
Copy citation
I was going to say, aren't you always somewhere in the upper 90% of full?
01:04:24
Play from here
Copy citation
It's always, I just purged it like a week ago.
01:04:28
Play from here
Copy citation
But it'll pick up entirely random things.
01:04:30
Play from here
Copy citation
Like, Kevin likes sous vide cooking.
01:04:33
Play from here
Copy citation
Okay, great.
01:04:34
Play from here
Copy citation
I like sous vide.
01:04:35
Play from here
Copy citation
But that doesn't really belong in my memories, and it's taking up another space.
01:04:41
Play from here
Copy citation
If you haven't really done this, one of sort of my tricks is to do a self-interview.
01:04:46
Play from here
Copy citation
So, ask GPT to create a 10-question interview about yourself.
01:04:52
Play from here
Copy citation
And then I use the dictation mode or my own LLM dictation tool to record my responses.
01:04:59
Play from here
Copy citation
And then I say after each line, add to memory.
01:05:02
Play from here
Copy citation
And that allows me to fill the memory bucket with a whole bunch of things that are actually important to me versus all this random stuff.
01:05:11
Play from here
Copy citation
Okay, so you have custom instructions, you have memories, and then the next one are chat or project instructions or chat instructions.
01:05:19
Play from here
Copy citation
So, the custom instructions that sit within a project or a GPT.
01:05:23
Play from here
Copy citation
All of these all come together.
01:05:25
Play from here
Copy citation
The challenge that I think that we are going to see as the lines blur is that there are a lot of amazing use cases of LLMs where you're basically impersonating a client or impersonating, you know, how would Ogilvy think about X, Y, and Z.
01:05:41
Play from here
Copy citation
So, you could start a prompt with, I'm, you know, this person, and how would I think about this?
01:05:48
Play from here
Copy citation
And it could grab it and take it into memory or remember it from a previous chat.
01:05:54
Play from here
Copy citation
So, this is a use for either temporary chats, which a lot of people don't know you can do this, but up in the upper right-hand corner, click on a little circle, and you can do a temporary chat.
01:06:03
Play from here
Copy citation
It's sort of like incognito.
01:06:05
Play from here
Copy citation
And you'll set up a single chat that will just go away later or a project that's locked from the rest of the information.
01:06:12
Play from here
Copy citation
But this is going to get messy before it gets cleaner because the bias is to get as much information as possible into context and then make sense of it later.
01:06:23
Play from here
Copy citation
But if it's biasing itself towards context that doesn't make sense because you've been role-playing or whatever it is in there, it's going to be a problem with your outputs.
01:06:32
Play from here
Copy citation
So, if that's happening to you, it should be sort of obvious.
01:06:35
Play from here
Copy citation
Check your memories first, custom instructions, et cetera.
01:06:39
Play from here
Copy citation
Yeah, we're not going to get to these other two topics, but that's okay because I think this is really important stuff.
01:06:44
Play from here
Copy citation
And I feel like we also really glossed over two important things with these custom GPT-2s that we should probably mention.
01:06:51
Play from here
Copy citation
One is that shareability and how you can now do the same thing with these Gemini gems.
01:06:58
Play from here
Copy citation
You can make these tools.
01:06:59
Play from here
Copy citation
You can share them out with your team.
01:07:01
Play from here
Copy citation
You can also do this with OpenAI.
01:07:04
Play from here
Copy citation
And what's so great about this is that it's like you have this living set of custom instructions and this amazing tool that at any time you think of a way to make it better
01:07:17
Play from here
Copy citation
or you want to clarify those instructions or you want to add another attachment, whatever you want to do,
01:07:23
Play from here
Copy citation
then just like magic, everybody that you've shared it with now has that updated version and you don't have to do any versioning.
01:07:30
Play from here
Copy citation
You don't even have to tell anybody.
01:07:31
Play from here
Copy citation
All of a sudden, your admin or your marketing manager or your assistant, whoever it is, has the better version of the thing you made.
01:07:40
Play from here
Copy citation
And you can now do this with – have you tried sharing these with Gemini, Kevin, or only with GPT?
01:07:47
Play from here
Copy citation
I'll admit, operationally, we operate in GPT.
01:07:50
Play from here
Copy citation
It's actually not my favorite model, by the way.
01:07:52
Play from here
Copy citation
I would say that Claude is still my favorite model.
01:07:54
Play from here
Copy citation
But that's another topic for why Claude and I are rapidly falling out of love.
01:07:59
Play from here
Copy citation
We have an operationalized – gems are what GPTs are called in Gemini.
01:08:04
Play from here
Copy citation
It's just – don't get me wrong.
01:08:08
Play from here
Copy citation
If I had to bet on one of the survivors of the great AI races of the 2020s, it's going to be Google and Gemini.
01:08:15
Play from here
Copy citation
Totally agree.
01:08:16
Play from here
Copy citation
For sure.
01:08:17
Play from here
Copy citation
Incredibly powerful, incredibly underused, neat stuff, but I haven't operationalized it.
01:08:23
Play from here
Copy citation
Already built into a massive business suite.
01:08:25
Play from here
Copy citation
Absolutely.
01:08:26
Play from here
Copy citation
But this is the bet that OpenAI is making, that they're going to get entrenched usability and user patterns.
01:08:34
Play from here
Copy citation
And even though Google has pole position on having everybody's data and knowing everything about your business, there hasn't really been a need to switch.
01:08:44
Play from here
Copy citation
And another topic for another time is, is OpenAI allowed to use all of these Google connectors?
01:08:50
Play from here
Copy citation
Why is Google basically enabling OpenAI to become the workforce platform that it's becoming?
01:08:57
Play from here
Copy citation
I mean, I think they might look back and be like, ooh, I really wish we hadn't allowed all of those connectors to happen.
01:09:03
Play from here
Copy citation
I totally agree.
01:09:05
Play from here
Copy citation
And I think it's worth mentioning to the listeners that we do this for a living and we're nerds.
01:09:11
Play from here
Copy citation
And we have thousands of dollars a month in all these different subscriptions.
01:09:14
Play from here
Copy citation
And we use all the tools.
01:09:16
Play from here
Copy citation
And we use all the tools that are built on top of the frontier tools.
01:09:19
Play from here
Copy citation
And it may seem kind of silly, but that's what it takes for us to be able to make the best informed decisions when it comes to things like what you should do next, what's worth looking at, what's not worth looking at.
01:09:32
Play from here
Copy citation
For anybody else who is on the early side of getting into this, it is very much advised to pick a tool, go deep on that tool.
01:09:42
Play from here
Copy citation
They're all pretty good.
01:09:44
Play from here
Copy citation
They're all pretty good.
01:09:46
Play from here
Copy citation
And you can do amazing things with them.
01:09:49
Play from here
Copy citation
I do think that with OpenAI as an example, because they were so early with so many of the things that blew people away back when this was all really starting to hit everything at a business level.
01:10:04
Play from here
Copy citation
We also standardized to a great extent to that suite as well.
01:10:09
Play from here
Copy citation
And that's the way it is.
01:10:10
Play from here
Copy citation
We're going to have to get back to the other things next time around, Kevin.
01:10:13
Play from here
Copy citation
We've got way too much to talk about, but I hope everybody got some utility out of this discussion as well.
01:10:19
Play from here
Copy citation
Yeah, this was great.
01:10:20
Play from here
Copy citation
Super fun.
01:10:21
Play from here
Copy citation
I think that's the pod, buddy.
01:10:22
Play from here
Copy citation
We'll see you next time.
01:10:27
Play from here
Copy citation
Thanks for listening.
01:10:28
Play from here
Copy citation
If you'd like to send us a question to answer on the show, email me at Elijah at spark6.com.
01:10:34
Play from here
Copy citation
That is S-P-A-R-K, the number six dot com.
01:10:37
Play from here
Copy citation
Or Kevin at Kevin at ascendlabs.ai.
01:10:41
Play from here
Copy citation
To learn more about AI strategy and training for your business, visit Kevin's company, Ascend AI Labs at ascendlabs.ai.
01:10:50
Play from here
Copy citation
And to learn more about design and development of custom AI solutions, visit my agency at spark6.com.
01:10:57
Play from here
Copy citation
You can also sign up for my free newsletter, AI Framework Friday, by visiting spark6.com forward slash newsletter.
01:11:11
Play from here
Copy citation
Thank you.