Ash (00:06.879)
Alright, welcome to Night School. My name is Ash Tullawat. I am the CTO of Gauntlet AI. I'm excited to have all of you joining us today. Please, please, please, let's start by introducing yourself here in the chat. Tell us where you're signing in from. Let us know if you're an engineer. Let us know why you joined night school. And we'll get started in a few minutes. Now, what usually happens at night school is we'd like to give everybody About three to five minutes just to make sure that You know, they can come in, get settled, understand what's going on, we give a little briefing on what night school even is, and then we move forward.

Ash (00:52.317)
I am Ash. I am the CTO of Gauntlet AI. I've been CTO for over over two and a half years now. Gauntlet started back in 2024. We've trained over 1,200 engineers at this point just through Gauntlet programs, gauntlet corporate training, or gauntlet-related partnerships. So really excited to be joining you guys here for today's session. So we have Solution Architect from Bloom Tech, shout out. We have Connor from Austin, Texas. If Sugarland, Idaho, New York, Texas, Seattle. Love it. James from San Diego.

Ash (01:34.015)
Anybody joining us internationally? Usually we usually have hey Pam, GT mom in the house. usually we have somebody from Australia. Panama. There you go. That's amazing. All right, again, we're gonna give everybody just another minute. There's about 200 people or so signed up for this event. Usually about hundred or half of them will attend. And so we'll get started. Is there s some reason why you specifically joined this session? You saw the title. We're going to tell you about the difference between a vibe coder and AI engineer. We're going to talk about planning, specs, and we're going to talk about decision logs. What is the big reason that you decided to click I'm going to attend? Would love to know that in the chat. I do hate AI slop. Sam Bodine, my code is trash. That's why I'm here. Harness architecture. Fair. Totally fair.

Ash (02:30.811)
I I really resonate with Mark what with what Mark is saying. One one of the biggest things here at Gauntlet is we don't try to tie ourselves to a single tool. What we want to do is make sure that we're thinking of the design and the architecture and the thought process behind something, not just saying, Hey, this is the best tool and you have to use it. I think somebody who has the right thought process, the right planning, the right method can actually use any tool and be successful. Minimum viable factory. Yes, that is an open source project that I have. I think it is broken right now, but it does lay out how I would be creating a factory from scratch. All right, I'm gonna get started. Today we're gonna be covering vibe coding versus AI engineering. We're gonna be talking about planning, we're gonna be talking about specs, decision logs, and a thought process for anybody wanting to go to that next level. You might have just started vibe coding, and I think The gateway to become a real software engineer, to even talk talk about infrastructure, to talk about security, is actually much more open than it ever was before. So today I'm gonna be focusing on what the differences are, how you can sort of take yourself to the next level to through three key steps, which are gonna be planning, spec creation, and decision logs. Now, when you think about vibe coding, you think about almost it's almost become like a slur, right? Everyone says, like, hey, you're just a vibe coder. You just vibe coded that design. Hey, you just vibe coded that little dashboard or application. I actually think there's nothing wrong with vibe coding inherently, right? It is actually generally a very good thing to learn no matter where you are inside of your company. If you're in marketing, if you're in sales, if you're in product management. If you're in engineering, if you're in something else, everyone should be coding with AI because learning to communicate, learning to help understand exactly what you need to do to get the AI to perform in the manner that you want it to is the crux of vibe coding. And there's nothing wrong with having a level of clarity, method, some sort of method, some sort of context generation.

Ash (04:49.791)
Or preparation tool that you're going to be using to making sure that the AI performs well on the other side. So I want to start off by setting the stage and saying, vibe coding is totally fine for prototyping, for getting started, for even doing a day-to-day small aspects of things. But what we want to do and what a lot of people in this chat probably wanna do is become an AI engineer. And I think there's a couple of steps you can take to go from vibe coder to AI engineer, and they all involve architecture, design. Trade-offs, specs, planning, and decision logs. Now, if we think about the spectrum, and this is the sort of spectrum of vibe coding to AI engineering, there's one little aspect I always like to think about, which is everything is not super binary. If you are vibe coding, that doesn't mean that you have to accept everything that the AI generates. If you're an AI engineer, that doesn't mean that you need full understanding of every single line. And what every single parameter and what every single type is doing? I think what the true balance is that the future is someone who could manage a factory of agents to go out and build software. What does that actually mean? Meaning if we had an assembly line and this assembly line sort of walked us down the creation of a ticket or the completion of a ticket or the creation of other different types of software from scratch, what would the human being be doing, offering judgment? deciding on architecture, reviewing what code was generated. And as a result, moving forward down that pipeline as managers, as observers, as individuals who were making sure that the quality was super high. You want to make sure that as we talk about the subject, you don't want to fall on each end of this spectrum. You want to find your perfect balance. Hey, you might be a platform engineer, so you might need a little bit more understanding of what each file does. Hey, you might just be a product manager. So if you understand the code a little bit less, but if you're able to prototype and save your developers your a ton of time, that's really important. Decide where you fall on the stack, depending on the role you have at your company or the role that you want to have in an engineering organization. Then decide how much you want to delegate off to AI and how much judgment you want to maintain while being in the driver's seat. The tools are the same for AI engineering and vibe coding.

Ash (07:15.681)
But there's four key differences I like to highlight. A true AI engineer retains ownership. At the end of the day, if your AI-genated slot causes a production issue, your CTO or your VP of engineering is not going to go to AI and be like, what happened? That's never going to happen. It's going to be somebody that comes to you and says, hey, it looks like you generated this code. Why didn't you review it? And so for anybody looking to sort of break into AI engineering or get to that next level of AI native development, then you need to be in the driver's seat. You need to have ownership. You need to be able to manage multiple agents at once. You need to have teams of agents working for you based on frameworks and harnesses that you design and you implement. I want everybody to understand that the software development lifecycle does not just include generating code. One of the biggest things my mentor taught me when I first started was there's a difference between software engineering and there's a difference between writing code. AI can write code really, really well, but that does not mean it can write test, review those code, and do your CI CD pipeline at that same level. It is up to you to think about the full cycle. Think about the generation of the spec, what the user actually wants, what the ticket looks like, and how that same ticket is then outputted on the other side. Be able to debug. Right, if something goes wrong and you leave everything up to AI, then how are you gonna go through the code base and figure out what went wrong? And finally, you want to scale that prototype to production. What's happening now is there's a lot of these AI demos in the market. We worked with this AI website builder company that had like this demo website builder for like a year and a half. It had like this big system prompt, and they were not improving it. They had no evals, they had no testing, had no security. The security was just what the basic things that the A had already suggested. And the goal was to this was just a nice thing to show their leadership. That's not what we're trying to do. We wanna build AI tools that are actually gonna cause a difference in the way someone solves a problem that is actually going to be consequential in helping somebody do work day to day. So you wanna go to production. If you're in vibe coding right now, you wanna think about security, you wanna think about scalability, you wanna think about testing. Ash (09:38.38) You want to think about evals. You want to build a harness. And that is what is going to separate you from a regular vibe coder to being an true AI engineer. There was a study done by Metter. I this was pretty popular on X, so you guys might have already seen this. But what happens is in the enterprise right now, everybody's being given tools. Here's clawed code. Here's cursor. Here's codex. Go figure it out. Go be faster. And that does not happen. You need some level of training to get to the other side. You need to go and try these tools out continuously to build something called an AI-first methodology. This is essentially the process by which you code with these agents. This is the process by which you divide up work, clearly define specs, and actually generate something on the other side. What I see in the market right now with our hiring partners and our clients is there's two big bottlenecks that exist. It is in the planning stage or the product management stage. Where there's just so much things that can be built now with AI, but there's not enough tickets or time to make sure they're clear clearly delineated and ready for AI to actually work on. The other is code review, where you have all of these PRs just sitting waiting for an engineer to come in and review them. And it could be like hundreds of PRs before somebody even gets to them. And because AI generates this code so fast, because there has to be ownership, there's always gonna be a bottleneck on review. And so what we want to make sure is we don't let this AI go unmanaged. Meaning we clearly communicate to this AI exactly what we want to build, how it should build on top of this brownfield code base, and then review that output based on acceptance criteria. And this is the concept known as loops. Loops are essentially a way for us to make sure that we're putting in the right contract, the right output is then being generated, it's matching acceptance criteria. And if it doesn't, then we're gonna retry and restart all over again. The problem is not seven 70% of the code that we generate. It's that last mile. The edge cases, the error handling, the security, the architecture. So if you take away nothing from this night school class, take away this. The last mile is not busy work. It's what separates you from between prototype and production.

Ash (11:59.146)
Is what separates you from being a vibe coder and an AI engineer. This last mile is where all of our focus should go. I know a lot of companies that might one shot the first 70%, but spend months just making sure that the last mile is perfect. And that is what we're going to be focusing on today. How do we make sure that that last 30% is as perfect as it should be, and that AI agents actually augment the generation of this last part? Okay, when vibe coding meets production, there's a couple of things that need to sort of happen. You need to make sure that there's a level of clarity between all the tasks that are being done. So you finish this application, let's say 70 to 75% of the way there, then you need clarity as to exactly why this is so important and why individuals need to make sure that, hey, this one feature is done over other features. So I'll give you an example of this. Let's say you go out into your user base and you say, hey, what is the best part of our application? Your product managers come back, and once your product managers come back, they see, hey, this is the X, Y, and Z, everything that we need to build out. You let's say you give that to AI. AI does not have enough context for it to decide this ticket should be done over the other ticket. The goal should be that we are dictating what is worked on first, what is the most impactful. And then clearly dictating that through specs. The goal of this program is to make sure that the planning stage is clear and the review stage is clear. So after the generation process, after it's gone through your factory, after it's gone through all those agents, then you can say this is the acceptance criteria, these are the unit tests that might run, these are the security checks that might run, and this is what's actually going to happen at the end of that entire process. I'll stop there for one question that we're getting here in the chat, which is How much of an emphasis is there in software architecture in the Gauntlet program? On every Tuesday, after you get a challenge here in Gauntlet, so if you don't know how Gauntlet works, so we get a challenge on Monday morning at 9 a.m. And it's essentially due at the end of the week. Each challenge you must pass to stay within the program. On Tuesday, you have to submit something called an architecture defense. So for that challenge, you actually map out the architecture. You think about the system design, the trade-offs you're doing, and what you're trying to maximize.

Ash (14:22.155)
then you have to defend that with your classmates to ensure that your understanding of architecture and design is high and that you're able to communicate this in an interview and communicate this at your larger engineering team. So it is very important to the Gauntlet program because as AI gets better and better at generating code, the only thing that will matter is system design, is security, is systems thinking, is architecture, how you're going to approach the solving of that problem.

Ash (14:53.237)
Alright. So if we think about the SDLC, there used to be something called document-driven development. Document-driven development is actually pretty uncommon. I doubt that many of you actually know what that is. This is something that existed, I would say 15 to 18 years ago. One of our principal engineers, Zach, was the one that initially told me about it. What document-driven development means is that we have a single document that everybody sort of agrees upon. And then they do some sort of waterfall approach to actually pursuing and actually completing something. And so what at the time this was like very slow, right? Because everything was done manually. So you'd have to like make sure that everybody was on the same page. And then eventually is that there was something like Agile that came about. You might know Jira, you might know product management and tickets and the creation of that. The reason I bring this up is as we get closer and closer to coding agents taking over the coding process. Something called spec-driven development is becoming more and more common. What spec-driven development means is that you have a contract or a spec between yourself, the developer, and the agent or the AI. And that contract could be thought of as an API contract, could be thought of as like a technological decision that this is the exact thing you're gonna work on. This is the scope of work that is involved, these are the files that you're gonna touch. And this is going to be the exact acceptance criteria for what you're coding. That spec is then handed off to any coding agent of your choice. That could be clawed code, cursor, codex, whatever. And the goal of this spec is on top of brownfield projects, you're able to pursue and produce something that is consistent, that matches your engineering standards. And any engineer can actually initiate it. Meaning, if I generated a spec, handed it off to one of my fellow engineers. They can hand it off to a coding agent, manage the coding agent, and still produce the same thing I produced. That is what becomes your source of truth for work being done. Why do I bring this up? Because the biggest problem in vibe coding versus AI engineering is context management. There's so much context in the enterprise. There's context that product managers have. There's context that designers have. There's context that only you have. There's context that users aren't telling you.

Ash (17:19.305)
And AI is unable to actually grab all that context dynamically. We need to distill in this spec all the possible information to make sure, hey, when it's doing this task, whatever a normal developer would require to complete this task, it is already understanding of that information. And so that's where this new changing STLC comes in. Planning is the new bottleneck. I know teams that are sort of pushing through tickets at record pace. And oftentimes it's not how fast you build it, it's what to build, when to build it, and what is the most impactful. Oftentimes ambiguity is the problem here. So what tends to happen is we're not able to put in all the information in the ticket correctly. So the spec that is being generated does not have all the information. Oftentimes we think that we want one thing, but then as we work through that spec, you then have to change it and come back. And so as we think about how the software development lifecycle changes, the first problem to think about is context management. A vague input is always going to lead to an inconsistent output. Our goal as AI engineers is to focus on upfront, making sure that these specs are clear, they have acceptance criteria, they follow some sort of standard. And this is the way AI engineering goes from vibing to actually producing output that is consistent. I know companies that are building giant factories across their entire engineering organization to augment them. These factories include QA tester agents, for full stack developer agents, product manager agents. And the problem is not actually the factory, right? It does a pretty good job 80 to 90% of the time. And oftentimes you have to make small tweaks. But the problem is deciding how what work is actually going to go into that factory? How are we going to set up this input prompt for the factory? What context or information do we need to grab across the factory to make the factory actually work as it should? We wanna go from prompting to context engineering, but if context lives across multiple different people, then we need to actually go out and grab all this context before we can actually do something with it. So once Ash (19:40.95) we have some sort of methodology on top of what we're doing, then we can say, hey, let's move forward and let's do something. That's to say if we think about spec driven development right now, what does it actually mean? It means that let's say we generate a PRD for a new epic or feature or series of stories that we're going to add on top of our Brownfield project. This epic series of stories, let's say, has to do with a new checkout flow. We then have to go to the designer and grab all their new Figma designs. We then have to go into the product manager and say, hey, these are the new features. This is why I was built this way. This is what to think about. Then we have to go into our sort of platform engineer, make sure all our integrations work properly to decide what is actually going to be integrated. Then we have to go to our full stock engineer to go and implement it. Then we have to go to our back end engineer to think about our backend of the databases. Well, all of this information needs to be in the spec. So then we can hand it off to our factory to do the whole job end-to-end. And so that's what I'm getting at. If you want to go from vibe coding to AI engineering, the first step is better planning. Better planning so you can generate specs that are clear for AI. These specs should have clear outcomes. They should have scoped boundaries, meaning, hey, this is the exact task you're working on. Let's not worry about X, Y, and Z. These are the files that you should be focusing on. Let's not think about X, Y, and Z. These are the constraints, the engineering standards, the constitution, the guidelines, and the architecture. These are the decisions that we've made previously inside of this code base that must stay intact. This is the breakdown of tasks that you must be doing. This is how to verify what you did is correct. The better we get at generating specs, the better output we will get out of our coding agents. Most of the time, what are we doing? Most of the time, all we're doing is giving very vague prompts to Claud Code on exactly what we want to do. A lot of people just use the screenshot tool in a Mac and say, hey, there's an error. Or they often say, hey, there's this like error happening in the logs, fix it. As engineers, if you want to be a true AI engineer, put yourself in the driver's seat. Why is it going wrong? What is going wrong? And how should you fix it? What is your recommendation? And then maybe have AI do the research. The first step to bridge this gap is planning. And the reason we bridge that gap.

Ash (22:05.907)
is with good planning comes good architecture, comes good spec creation, and good spec creation actually leads to reliable agents. Okay, I'm gonna take a second there to answer this question. When practicing this at home, is this principally a difference between a spec and creating a plan with the agent before letting it build? Great question from Philip here. Philip, what I want you to do is to emulate an engineering organization. I want you to try to build a factory where there's multiple agents working together. And the way I want you to practice this is to decide how you're gonna get all the data from all the different roles inside of an engineering organization. How that data is gonna be formatted and what's gonna look like. And then I want you to use this process of creating a spec to see if your factory functions as you think it would. Then grade your spec, grade your output, give feedback to your factory, and then move that forward. The goal being, I want you to master the process of gathering context, organizing context, and then giving that context to AI. So actually, it's not just generate specs at home and practice it. It's going to be go build a factory that emulates a real company, then give it a spec and run the entire experiment. Question after that is as someone early mid career, one of the biggest challenges I've experienced is that everyone expects you to be fast before you're ready. This makes it hard to stand out when applying for jobs. Since the only way I'm able to keep up and bypass ATS is by leveraging AI to adapt and build projects with newer technologies. What would you recommend I try differently? Great question. what I would do, and this is also my experience as a CTO coming in.

Ash (23:52.684)
And my experience getting a lot of people hired out of the Gauntlet program. Everything is about what you build. you can try to go through the front door of a company. Instead, I would recommend going through the side. The biggest way to get hired right now is to contribute to open source AI repositories. I know so many people who are making meaningful contributions to the Open Claw project, for example, that are then getting job offers or interviews on the other side. Instead of just generating AI slop resumes and then applying to a lot of jobs. I would recommend working on real projects in AI, contributing to open source projects, and then letting those contributions lead to networking opportunities and eventually jobs for you. This will also teach you the process because you'll go through a lot of projects, but it will also build your network of individuals that are actually doing stuff in AI. So that'd be my big recommendation. Don't go through the front door, go through the side. The other thing I'll talk about is everything is about projects, and that's why Gauntlet is so project-based. Right, we have lectures, we have classes, we have something called rounds, we have a bunch of mentors walking around, but everything is project-based, meaning you have to get your hands on a keyboard and you have to actually do something to build and ship and move to production. So the more you can build and ship and move to production, that is going to help your chances, 10x to actually do something and get noticed. You have to get noticed, right? There's a huge pile of AI slop resumes that come my way every single week. But the people who stand out are the ones that build real things that catch my attention.

Ash (25:32.843)
Okay, let's talk about a spec. What does a spec look like? So this is a real sort of diagram of what you might put in inside of a spec. When I say a spec, a spec is just an MD file. A spec is essentially a source of truth that can be transferred from the human to the coding agent. And at the end of the day, that's just text for an LLM. It could be images or reference material, but most of it is gonna be text. What you're gonna do is start with the exact outcome you're looking for on top. Then you're gonna talk about what's out of scope, what the constraints are, what the acceptance criteria is, and maybe a task list of exactly the task you would do. This is very similar to if you've been a software engineer for a number of years. What we would do is once we received a ticket from a product manager, we would actually transform that ticket into a checklist of things that we were working on. We would transform that ticket into all the different criteria or constraints that we have to keep in mind. And then finally what we would do is start working on that task one by one to actually code it manually. So what we want to do with the coding agent is to use this document as a way for them to decide everything already in advance. That means that if I'm working in claw code or cursor, I turn on plan mode, drop my spec in, make sure I load in my memories, and then on the other side I actually try to generate something. Usually what happens is there's six key things that you want to really make sure you outline, which is a target, meaning what am I exactly building? The boundary, so that that includes the scope, the files, what it should focus on, what it shouldn't focus on, a way to prove that boundary or that target. That could be test, that could be unit test, end to end test, could be CI, could be even just a agent actually going into the browser and trying something. And then make sure that has a loop, meaning, hey, if this doesn't work, try this. Hey, if this doesn't work, you can always ask me questions. The goal here is you want to create this loop by giving it some criteria and giving it a goal to make sure it continuously goes until it's achieved. These specs become real IP and data for the companies that we work with. So I'll give you an example of what happens at one of our clients. They do a

Ash (27:58.644)
Spec kit, which is by GitHub. It's open source and you can actually look at it now. There's also a bunch of spec driven frameworks on GitHub. SpecKitty, Spec Kit are two of them. And essentially what you can do is you can fork Spec Kit and turn it into your own layer on top of your coding agent. And what Spec Kit does is it adds a bunch of commands on top of Cloud Code or Cursor Codex so you can actually orchestrate spec driven development at scale. This company, what it's doing is taking all the specs that they've been creating, the ones that led to merge tickets or rejected tickets or failed tickets, and actually storing them inside of S3. So they can see exactly what they've been building, how the product has improved over time, the direction that they're moving in, what they're focusing on, what ideas they've tried before, and they might not want to try again. And so your specs become your IP, your data. They become the source of information. That you can reuse over and over again. Decision logs are the next level to this. As you're doing development, I'm talking on code bases that are millions of lines of code. Decision logs are going to be the non-negotiables. They're going to be things that you've done in the past that should be considered when coding. And they're going to shift the premise to like, here's exactly what's been decided that you should build, to here's exactly what's been decided, but this must stay true as well. Why is this important? Because when you work in a development team, when you work with other engineers, you're doing this inherently every single day. You're saying, hey, the front end should always follow these guidelines. Hey, we're always going to use Postgres as our database. Hey, every time we locally set up our database, we're going to use a Docker container. These are decision logs or decisions you're making as an engineering team. And the thought process here is can you take that spec? Add in decision logs and then give it to a coding agent so that it's not going to go haywire or hallucinate on top of what you've done. My goal here is I want you to get the engineering standards you have. I want you to get maybe some of the things that go wrong when people code in your code base. I want you to get things that you would teach an intern if they were joining your software development team. And these become a shared constitution. This shared constitution

Ash (30:25.235)
is added to the context window of any coding agent that you decide to use. And as a result, it will maintain your code base based on the decisions of your manager, the decisions of your team, or decisions that were made in the past. The way this works is let's say we put a spec into our coding agent, we would have already loaded into memory the decision logs. They would be either a skill, they'd be inside of a memory object, and so our coding I agent could access it whenever it needs to. The power of decision logs is it's essentially acceptance of what the coding agent should be doing across the team. Meaning your product manager has read this so they understand. Meaning your design has read this so that they understand. And when you're working in an enterprise code base on core engineering team, when it's not just you, right? You're a true AI engineer working with two or three other people, you can use it as a way to make sure that these things are never touched. Or never broken. Now, the question that I often get is why do you need decision logs when you have like a clawed MD, when you can have rules on top of your coding agents? Well, the reason is oftentimes these decision logs are dynamic, right? Oftentimes you want to be able to version them. Oftentimes you want to be able to revert them. Oftentimes non-technical individuals are making these decisions because they could involve product or design. What you want to be able to do is port all of this in through some sort of spec-driven framework. Meaning just because this framework exists and these commands exist, then as a result, you're able to make sure, hey, all of the company's data and IP or decisions we've made, the coding agent already knows. The factory already knows. This also is the first step into the inside of the creation of a harness. When I define harness, what is actually a harness? A harness you can think of as the jacket on top of the LLM, the Batman suit on top of your brain, right? It's like the tools, the evals, the memory, and the context that keep you inside of a certain boundary or area when it when you're building or doing a certain other task. I'd like to think of it as like jackets. I can put on different agents and they can become and do different task-specific jobs. So for example, I have a single LLM configuration.

Ash (32:48.533)
I put on the marketing harness and all of a sudden they become a content creator or a content marketer. I have the same LLM, the same configuration. I put on, let's say, the full stack development harness or jacket, and they become a developer that I can use. This is how you should think of agents. And the reason your decision logs can become harnesses is because essentially a harness is just a set of evals, memory, context, et cetera, that makes sure that the agent acts a certain So these logs are essentially the first step. These are the criteria you must always meet. These are the rules you must never break. These are the technologies you must always work in. This forces your agent to actually work the way a true engineer would inside of an enterprise code base. All right, we have more questions, so I'll stop there for a second.

Ash (33:41.386)
Is there ever an instance where I don't know, this might be a a typo, is favorable over natural language? Gherkin is favorable over natural when writing specs. This is a good question. I think that I would actually keep it natural language because a spec is for humans as well. So this always goes back to something I am pretty adamant about keeping, which is at the end of the day. Compliance, leadership, technology, and engineering, there's gonna have to be someone responsible for actually owning the code, the errors, and managing these agents. And at the end of the day, the spec is as much for the AI as it is for the human. So I would actually maximize so that humans can give all the information very easily into a spec. If you do it that way. Then they actually want to do it and it actually produces something useful on the other side. So maybe there's a slight better difference in like performance on the other side, but I think maximizing for the human condition is actually more.

Ash (34:55.199)
Next question here is Do you find it easier to try to write a spec like this from scratch versus using the AI to help generate the spec and then feed it back to maybe a different model to plan and implement? Great question here from Jared. Jared, when I'm building a PRD, which is I think is different from a spec, I like to use AI a lot. Oftentimes I'll be green fielding or maybe generating a whole new feature on my code base. So AI does a good job of helping me transition and adding the constraints. Thinking of ideas on how to approach it. The reason why I like AI to review my specs, maybe have certain standards for them, but at the end of the day, I do make that a human process, is because I'm not trying to just build specs for myself. I'm trying to build them for, let's say, a 300% engineering team. And my goal is always going to be use this spec as a way to consolidate all the knowledge, all the tribal knowledge that your engineering team has. And so there's no way AI is gonna have that. I know people are building knowledge graphs, I know people are building company MCPs, I know people wanna grab information from different areas, so on and so forth. But I still think at the end of the day, it's always going to be something that you have to sort of go back to these humans to make sure that they're adding to. Then you can have AI check it. You can have AI say, is something missing? You can have AI say, Hey, is something going to actually change over time? But in the beginning, I think I would not involve AI in the process.

Ash (36:32.735)
Next question is how do you recommend structuring the build process for an entire new product like like an MVP versus individual features? Great question here. If I'm building an entire new product, I'm actually v one shotting the first 60% of it. So not the whole thing. What I'm gonna do is I'm gonna generate a PRD. I'm gonna turn plan mode on with so the maybe the best coding agent that I'm the most experienced in, and I'm actually gonna let it generate about half of the application. Then for the next half of the application, I'm going to go feature by feature through phases. And after each phase, I'm going to have acceptance criteria. And these acceptance criteria are things I'm going to check for myself. So that's how I like to green field a project. The more common thing, which is if I'm adding individual features to brownfield, then my goal being that I'm going to actually take the large code base, millions of lines of code, decide exactly where I'm going to add something, and then over time add those step by step.

Ash (37:33.687)
Philip's saying, I've been trying to review Byron's workflow lecture. Something I'm slightly confused on is when a spec is needed. Do you need a spec for the whole project? One for each shippable piece, or both? Great question here from Philip. The way I like to delineate this is a PRD is at the project level. That's your entire project, your source of truth, that has generally everything about the app you're building. The PRD is usually broken up into tickets. You know, it's this this is scoped or work. These could be epics or stories. And then that is then generated into specs. Specs usually are represent one PR of work. They usually are lower on the story points. And they are more sort of granular than maybe a ticket, epic, or a story might be. The goal being that one ticket or one story might be multiple specs. The reason why you do it this way on a really large code base or engineering team is because what you're essentially doing is giving the engineer the opportunity to give that spec to multiple coding agents. So one of the things I love doing is I'll use a cheaper model to see if if the spec could be solved by it. And then over time, if it the spec can't be solved by that cheaper model, I'll then use a more expensive model. I'll actually have four different models try the same spec. And then I'll accept the spec that I think was implemented the best. So to answer your question directly, Philip, if I'm on an engineering team, the spec is after stories. One story can be broken up into multiple specs. If I'm working individually, what I like to do is I will use the PRD to get, as I said, fifty to sixty percent of the way there. And I'm just using specs to add features. So one spec per feature is totally enough. Kyle is saying, What does this look like in your harness? Is it a slash command you run? Or is this a part of the MD file and the model that will run? Great question from from Kyle here. I'll answer the question in two ways. So there's something called a global harness. I mean, I don't know if these terms are have been coined yet or not, but there is the agent harness and sort of the factory harness. The factory harness is like the SOP or the operating model for the entire factory of agents that are building software for you.

Ash (39:48.298)
In that scenario, that factory harness is usually ported as a slash command, as some sort of wrapper on your coding agent, or a skill. If it's like an agent harness, meaning this specific agent should only act in this specific way, the way that is sort of ported in is usually the system prompt, usually some sort of command directly, or some sort of a skill that is only attached to that individual agent. So to answer your question, it depends on. What the harness is governing. Is it governing a ton of agents together, or is it governing a single agent? If it's governing a single agent, then it's a spec, a system prompt, or some sort of command. If it's governing multiple agents inside of a system, then it might be something deeper. it might be a constitution, a rule set, a file, MCP skill, so on and so forth.

Ash (40:42.679)
Good question. How can you involve junior devs in this workflow if they don't have enough context experience in the foundations to write specs and ADRs? Great question here from Nalamar. What I like to do for junior developers is actually have the specs created for them, and then all they're doing is managing the workflow and the agent themselves. This gives them an opportunity to actually learn the process of review, testing, securing, and deploying applications at scale. I also like to involve them in the process of creating these specs. Meaning I will usually team them up with a mid level engineer or a manager that I trust to show them how we're generating these specs. 'Cause I want them to generate these specs. I wanted them to generate these logs. And then over time they're they go from just managing these agents to actually coordinating and doing contracts with these agents. What is your opinion on knowledge graphs as it related to how an agents interact with them? I have a very controversial opinion on knowledge graphs, so but I'll say it either way. I think knowledge graphs usually are sound like a great idea and they work for the first six months, but then over time they degrade and nobody actually maintains the ontology of what's going on. If you don't know what a knowledge graph is, it's essentially nodes and edges that can represent knowledge over time. Meaning, let's say we're talking about all of the investments that Berkshire Hathaway made, then the nodes might be companies that Warren Buffett invested in, connected via some other connections or time period or different categories. And then you can actually traverse that knowledge graph and figure out different sorts of information. My take on knowledge graphs is if they were perfect and then information in them did not decay, then they'd be a very good idea. But of the companies that have tried to build knowledge graphs,

Ash (42:32.427)
They usually work for the first three to six months. They start decaying. People hate maintaining them. And then as a result, they become less and less useful over time. The thing I really prefer is a something called agentic rag, meaning as a developer, we don't have a knowledge graph when we start doing something. We actually go out and grab all the information ourselves and then start coding. Similarly, I think if we're building a software factory or if we're building some sort of harness or a developer agent, then we should implement agentic rag. which is like an agent router with all the fragmented data sources you have. And then you go out into those data sources and then you're actually trying to grab the information as you need it. Rack is saying, when you say 60% one shot, how do you decide what that is? Great question here, Rack. I usually what I like to do is take the PRD and then just estimate it based on the number of tickets I produced out of it. So if I have 20 tickets, try to get to 11 or 12. Marcus, riffing off your knowledge graph question, were your thoughts on leveraging ERD? We have mermaid markdown and specs. I love that. I love having architecture diagrams and mermaid inside of my specs. I think that Most of the time you should be spending when you're greenfielding an application is on your architecture diagram. Like having all the components down, having all the data transferred down, having all the API contracts down, having all the security rules down. That is your goal as an engineer. In fact, when I'm hiring, that's probably one of the questions I ask. Hey, I'm gonna ask you to build something from scratch. If you're if I told you to build something from scratch, what would be your process of working with these AI agents? And the answer I would be looking for is somebody who would spend maybe three to four hours at least just walking through all the possibilities and actually asking many different stakeholders and people what the right thing to build was and what the most impactful thing to build was. In Gauntlet, what happens is we like to do something called pre-search, meaning once you are given a challenge, it's your job to then go research that challenge, understand what we're asking you to do, what the crux of that challenge actually is.

Ash (44:41.759)
And then do research on other ways people might have implement implemented this. Then you're gonna do an architecture defense where you're gonna build out that diagram, understand that security, and then you're gonna move forward. So I think those concepts of planning are super important that set you apart from a vibe coder to an AI engineer. I'll take one more question, then I'm gonna finish off the deck and do a little bit more questions. Question from Chris here is how do you keep different data sources secure in terms of a gentic rag? Great question from Chris here. First and foremost, let's talk about the different data sources. You can have a vector store that you're doing similarity search on top of. You can have a structured data SQL store where you create the query and then you actually grab the data. You can have MCP connectors, and your MCP connectors can be API connections or CLI connections as well. Or you can have transactional information where you can just call a tool and hit an API and get information back. In terms of security for just the transactional information, That is the old-fashioned way. You know, you have a JWT token, you're authorized user only. You grab the information only if you're authorized. Via MCP becomes a little bit harder, but the way I like to do it is if you're using an MCP to gather information, you can also use the MCP to actually go out, grab information via different rules or criteria, and limit the amount of JSON objects inside of your MCP so the right person only gets the data that they deserve. Finally, for your SQL store or that, you can have row level security, you have multi-tenancy going on. The goal here is you still have a token associated with all those interactions. Meaning you can't run that SQL query unless that individual is authorized. And then finally, if it's your vector store, that can already, you know, you have the same token authorization through MongoDB, through Pinecone, through Quadrant that you can use. So the goal here being the person logged in gets the tools that they need, gets the information they need, only the information. That is available to them based on the rules that we have established. Okay, to finish this out, an ADR is going to be a decision record, meaning we are making a call as an engineering organization that this is a rule that is never going to change. Teams are actually committing this to their GitHub repositories. This becomes a constitution. Rules that can't be infringed upon by the AI agents. Why is this so important?

Ash (47:08.789)
Because these rules govern scalability, security, processes, product management, design, brand guidelines. There's so many things an ADR can cover. And the combination of a spec and an ADR often can actually get you really far when you're working on top of Brownfield code bases. And so going from VID coding to AI engineering is also figuring out change management. It's figuring out how you can take how what exactly an engineering team is doing, write that down somewhere. So that coding agent almost becomes a part of what you want to do. If you think of your coding agent as a coworker, then you can actually figure out: hey, what information should go into the spec, what ADRs do we need, and how can I make sure that every single time I'm working on a ticket, this is automatically in there. So for example, I know a company that will automatically generate the specs, get the specs reviewed, attach them to the ticket or the user story, try to find relevant ADRs, port all that information into a cloud code instance. And then have a developer just manage that end-to-end until it's deployed. I think that level of context management is what the skill is going to become, right? This is what software engineering is becoming. This is what factories might look like. And this is what you might be doing with all your different AI coworkers. Okay, if I were to put it together, the big takeaway here is this loop. You have some sort of intent inside of your individual startup. Your engineering organization that you want to build something. That intent is then turned into a roadmap, a PRD, an architecture diagram, and eventually it gets translated to specs. The spec inside of plan mode, your coding agent turns into a plan, some sort of task, some sort of phase diagram, some sort of step by step solution. Then it actually generates the code based on acceptance criteria. We run evals on top to make sure this is actually working. The tests are actually passing. This actually looks like the what the code that we are expecting. And then we can continuously iterate if there is a failure, if something doesn't pass, if it doesn't meet our quality check, if we want to change something on top. The big unlock from vibe coder to AI engineer is the creation of loops. Each of you should be building loops. Engineers no longer single-handedly prompt these agents.

Ash (49:31.521)
Like they're not just sending one-liners continuously to these agents. They're generating specs, they're generating ADR logs, they're adding those to plan mode inside of their coding agents, they're establishing a loop via acceptance criteria and quality checks, and then just letting letting that root loop run continuously. One of our hiring partners, in fact, the CTO was telling me that he is continuously coding, even though he's not sitting at his computer. And that's not just through dispatch on top of Cloud Code. He has multiple managed agent instances that are just running loops, trying different things, working on tickets at the same time, working using different models to work on the same ticket, trying different greenfield projects to see what might hit, testing different features. If you could clone yourself to work on all the different ideas you had, that is now possible with AI. through cloud manage agents, cloud management agents, cursor cloud agents, and all they're doing is establishing these loops. But how do you establish these loops? There's three things you need for the loop. You need some level of intent, some level of planning, some level of architecture and security diagramming. You need your specs and your ADRs, your documents that sort of keep everything in check. And then you need evals so that you can evaluate the output of that code and make sure it matches what you're looking for. Agents are just in the middle of that loop, and you you have to generate the different the ends of that loop end to end. Alright. That is what I wanted to cover today. If you take away something from this session, and I'm gonna be doing QA for another five, ten minutes after this, the first thing I want you to take away is that start writing specs. Almost every single engineering team I know is writing specs. That means they're taking their PRDs. They're transforming them into stories and tickets and epics. They're generating specs or contracts for their coding agents. And then using those specs as sources of truth for the exact task that the coding agent is being evaluated on. If the coding agent is deemed as your coworker, then you need a consistent and systematic way to actually understand if they're doing work correctly. With specs, what you get to do is you get to run evals on all your specs, meaning, hey, this spec was medium difficulty.

Ash (51:55.254)
It involved these files and it was a successful run by the coding agent. And you can actually map this end to end and actually see that your workflow is improving. The second thing you need to think about is ADRs. Decision logs become the first step in building a harness for your company. They're essentially the rules that shouldn't be broken. They're the constitution that should always be followed. And they're the things that are gonna keep your coding agent in check as you're building the things that you are building. And I've kept it super simple. Right, these are MD files. They could be turned into skills, they could be turned into rules, they could be turned into just context window tokens. Our goal should be that start with the basics. Can we build these files out? Can we make sure that these files exist when we're communicating with the agent? The goal is after that is to stop slop. Right? So, what we want to do is start using these files inside of plan mode, inside of our checking mode, inside of the test suites that we have to make sure that actually matches what we're looking for. And finally, your role is not disappearing. Our goal is not just to be faster without a plan. Our goal is should be to be sharper, to be more precise. Because if we're sharper and more precise, we'll be faster no matter what. All right. Let me look at some of these questions coming in.

Ash (53:21.609)
At times there will be a need to add referencing to some other external spaces. For that example is Confluence Revo. When we do this, how do we ensure that the spec doesn't hallucinate? This is not this is not avoidable because we want to do continue dev for an existing application with AI now. Okay, that is for sure. The spec is not going to be perfect, but right, but oftentimes if it's ninety-five-97% of the way there. It is still something that you can move forward with. So, for example, if you are grabbing information from Jira, grabbing information from Notion, oftentimes, yes, there will be some contradictory information. But the power of coding agents, and the reason why we still have a software engineer on top managing everything is because we want to make sure that even if that discrepancy exists, you're going to still go through the process of actually end-to-end implementing this. That's the first step. The second step is. Have product managers review some of the specs that are being generated. I know people who generate a skill to just make sure that the spec has certain criteria met end to end. I know people that make sure that each of the criteria can be like deterministic. You can run something to check that it's correct. They're based off API calls, so maybe you can go actually go into the APIs and check them as well. You can have a specific model or SLM to just check the the quality of the specs as well. Spec quality is important, totally agree, but I think The the process of creating these specs, the planning, the architecting, that alone takes you from just prompting endlessly without a sort of an end in sight, to actually getting there more than 70% of the time. Kyle is saying working with agents that give human coding timelines. How do you go about providing an AI timeline? Great question from Kyle here. Kyle, I just assume that everything will take much less time than I plan it to. So for example, if I'm trying to map something to a timeline, my goal is always to think about how much a human would do manually and maybe just cut that timeline by a factor of three.

Ash (55:36.675)
that is just like a rule of thumb. There's actually no data to support that. I think every engineering organization or manager usually has that rule of thumb where they know that hey, most of their engineers are moving at X, Y, and Z pace. I know there's tools coming out now to measure like coding merge rate or like AI coding efficiency and stuff like that. So maybe that will give us a number. but what I like to do is not set a timeline or not have AI set a timeline. I like to take the human timeline. And to sort of reduce it by a factor of three.

Ash (56:11.233)
Great question from Kyle here. How to use loop versus goal. I like to use goal when it's the more overarching thing that I'm trying to do. So for example, if I am building an epic and that epic is the creation of a Harry Potter game end-to-end, my goal would be: hey, create the the broom flying mechanics for my Harry Potter game. My loop is going to be, let's say, generate the actual broom. Based on these acceptance criteria. So it's more of a task in the larger goal, whereas the goal is a more overarching material.

Ash (56:53.195)
Yes, that is the spec kit. I also like Spec Kitty. Spec Kitty has a Kanban board that it generates automatically for your spec generation. Thank you, Larry. What's my favorite model? I am big on like trying to save all the I'm big on efficiency right now. So like I care less about using Opus or GPT-5.5. What I like to really do is we have a lot of self-hosted open source models here at Gauntlet. And what we're trying to figure out Is like a proxy layer, like the prompt or the request goes in, goes through my proxy layer, and then it decides based on spoke scope and difficulty and our record to actually orchestrate which model it should go to. so I'm big on like most of the time you don't even need a frontier model. that is my take. I think most of the time you need like a sonnet or a quen, and that should get you all the way there. and you might need an Opus model only if you need to e excal escalate based on like failure states. So long story short, I'm big on open source models right now. We're self-hosting a lot of open source models. I really like the composer model from cursor, but my goal no longer is to use the most expensive model. My goal is now to use the most efficient model.

Ash (58:16.983)
thank you, Chris. Jonathan's asking any Rex on how to orchestrate multiple models together and maximize tokens. Jonathan, I'm creating a solution for that now. I don't think something exists right now. I think you might have to create one for yourself. like a simple proxy that just helps you decide which model to use when. I haven't seen a solution. I know there's a lot of like control plane companies, but they're all like governance companies. They're not like token maximization companies, if that makes any sense. this recording will be shared with you after the session. The last thing I'll leave you with is if you would like to join classes like this all the time, think about applying to Gauntlet AI. Gauntlet AI cohort six will be starting after July 4th. And the goal of this cohort is to just take it to the next level. We always love taking it to the next level with our gauntlet program. If you could imagine an ecosystem where laundry, hotels, office space, flights, everything is covered. And food. And you can just build all day long with a thousand like for a thousand hours with a hundred people who are all in on AI. That is what Gauntlet is. So if you ever do consider going all in on AI, please, please, please consider Gauntlet. Let me answer the last few questions as we come to the end here.

Ash (59:36.979)
Any advice in operating out of highly regulated I just missed that question. Any advice on operating in highly regulated environments where we can only access Gemini models? possibly have the privilege of an 80k. Yes. again, the model does not matter. The harness does. So even if you're just stuck with a specific Gemini set of models, or if you're stuck with an open set open source set of models, if you take a build a really good skill, have a really good harness with the right context and memory, you can actually become just as good as a developer without having the frontier models. So That would be my recommendation. Think about all the other things you can control, not the model. Think about the skills. Think about the context management. Think about the memory. Think about the MCPs. That is what is going to set you apart. Question after that was

Ash (01:00:53.345)
Do you do you all have a library of tools you find most useful? I think Gauntlet people are all over on Twitter, so that'd be my sort of go to place. Twitter seems to be the the place where gauntlet people should go. we also follow all of our challengers. So if you go to the gauntlet account, the people we follow are g all gauntlet people. Can are students on OPT eligible to apply to Gauntlet? Unfortunately, no they're not. We also work with government agencies, so it is required for you to be a United States citizen. or a permanent resident. What type of laptop would you recommend? Doesn't matter. any regular PC or Mac that you'd use for coding should work fine. Cloud code, cursor, and codec should work on your laptop. in terms of deployment, we do everything in the cloud. curious what makes a model efficient versus just expensive. Is it through the power of the open source model that is self-hosted? I think Patrick, the metric I always like to use is cost per token, meaning how much How much how many how much money did I spend on the number of tokens to get the same pull request merged? So if I had used the same models, the open source one and the most expensive one, then at the end of the day, when both of those pull requests were merged, how many what was my token cost? I would want the best bang for my buck. That means I used less tokens for the same merge event. Ash (01:02:17.95) Christopher is saying, What are your age limits for Gaulin? I believe you have to be eighteen or Sonny will be sharing out the slides. Gauntlet is eighteen or older, please. No, you should build projects to get into Gauntlet. You will not be accepted if you haven't built something before. If you don't have any formal engineering experience and want to get into Gauntlet, I would start off by doing some contract gigs. There's contract gigs across the board. you can go on upwork and grab something, do some gigs on the side. I want to see you build, right? So the big reason why people get rejected from Gauntlet is going to be that lack of core engineering experience. But again, everybody starts somewhere. When I first got into software development, I worked my ass off at a dev shop for like 70 hours a week doing ticket after ticket after ticket after ticket, and I was completely underpaid. The goal is it didn't matter what I was doing. I just wanted projects on top of my GitHub. So even if you're doing one off contract gigs, then do one off contract gigs and build your portfolio to then get a job one day. the big thing is to always be building.

Ash (01:03:38.101)
You always have to be building. Because if you're not building, then you're not going to keep up with the pace of AI. AI is going to pass you every six months the moment you stop building. There's like this concept, I believe Michael Phelps said so. Like if he doesn't swim one day leading up to 2008 Olympics, it would take two days for him to catch up. I think if you stop building, if you're not coding every other day, then it's gonna take you about a month to catch up in AI. Any max age. I don't believe we have a max age. I mean we had somebody in her sixties that was a part of our program.

Ash (01:04:15.639)
So no max age. All right. With that, that concludes our night school for this week. I want to thank all of you for joining me. If you have any ideas for night school topics, please put them in the chat. Sunny and Tom on our team are always listening. They're always thinking about what they might be using for the next version of night school. So if there's something that you're curious about you want a class on, just put it inside of the chat. You you guys will get the recording and the slides after the session. With that, I wish you guys the best rest of your day. I will see you later in the week, maybe at a future night school session. And we'll see you guys again next Wednesday. Thank you again. Talk to you soon. And this is signing out of Austin, Texas. See ya. Bye.