Technology
AI Interview #6 - 1M context window in LLM?
Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A fast, whiteboard-style walkthrough of a basic RAG pipeline and the first fix for PDFs full of tables.
Gaurav Sen · Gaurav Sen · Transcript & summary by WhipScribe
A fast, whiteboard-style walkthrough of a basic RAG pipeline and the first fix for PDFs full of tables.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“in SQL you can only do it in one dimension, right?” Tanishka · at 1:36 — open in the transcript →
“a single parser might not be able to you know extract all of these things very efficiently” Tanishka · at 6:35 — open in the transcript →
In this seven-minute mock interview on Gaurav Sen's channel, a candidate named Tanishka is told that an internal support bot with about 100,000 users is failing as the company keeps adding documents. The data no longer fits in the model's context window, and Tanishka's answer is a retrieval-augmented generation (RAG) pipeline.
The design has an offline half and an online half. Offline, documents are cut into chunks of roughly 800 to 1,000 characters, turned into embeddings and stored in a vector database. Online, the user's question is embedded with the same model, a nearest-neighbour search returns the top five or ten chunks, and the LLM answers from that context, which Tanishka says also curbs hallucination. Pressed on why Postgres indexes are not enough, Tanishka points to vectors of 768 to 8,192 dimensions.
The interviewer then raises messy inputs: PDFs with tables, policy grids and screenshots of tables. Tanishka keeps the architecture and changes only ingestion, adding layout detection so each page region goes to a parser suited to text or to tables.
The candidate in this mock AI engineering interview, asked to scale a company's document-heavy support bot.
In the mock interview on Gaurav Sen's channel, candidate Tanishka splits RAG into two stages. An offline ingestion pipeline chunks company documents, converts the chunks into embeddings and stores them in a vector database. Online, the user's question is embedded with the same model, the database returns the most similar chunks, and the LLM answers using those chunks as context alongside the question.
When the interviewer says the company already runs Postgres, Tanishka argues that a traditional SQL index compares values along one dimension, while an embedding is a vector with many dimensions. Tanishka cites 768-dimension vectors per chunk, with some embedding models going up to 8,192, and says comparing those is what a vector database is built for.
Tanishka says the number of retrieved chunks, k, is decided empirically. People generally start with 5 or 10, but there is no fixed figure: it depends on the use case and on how the chunks were made. Chunk size itself is a choice too, with Tanishka giving 800 or 1,000 characters as typical starting points.
Tanishka says it does. Without retrieval, the model never sees the company's internal documents, so it can miss details or invent facts. Passing the relevant chunks with the query means the LLM answers from supplied context. Tanishka adds that checking whether the model sticks to that context is a separate job handled through evaluation.
Tanishka's answer in Gaurav Sen's mock interview is to change only the ingestion pipeline. Common parsers such as PyPDF and PyMuPDF handle tables poorly, so a layout detection step first labels each page region as a title, table, text or logo. Each element type is then sent to a parser built for it, since one parser rarely extracts everything well.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00Tanishka, we recently deployed an application which has around 100,000 users using it. But as our company is scaling and as the kind of queries are changing, we are adding more and more documents to our pipeline, customer support pipeline. So the question is, how do we actually support these various types of documents and scaling up the documents? So one solution can be a RAC architecture, which can be used so that you can chunk the larger documents. So let me first draw the complete architecture and then I'll start explaining. So let's say here is your company docs. Now this is where most of your problems are getting solved because your data was so huge that it was not getting fitted in the
0:36context window. So now what we are doing is we are chunking that data. Chunking is basically PDF breaking the data that you have into smaller pieces so that it can fit in inside a context window it can be 800 the size of one chunk can be 800 can be 1000. you mean 800 and 1000 tokens or something characters characters yeah so whatever data that you have we are basically dividing the data into smaller pieces of chunk now after the chunking what we are doing is we are converting it into embeddings. Embeddings are just basically numerical representation of the data. Why are you using a vector database? Why not? We currently use Postgres. So is it similar? No. So the reason why we are using a vector DB rather than, you know,
1:14a traditional transactional database like SQL or something like that, these are numeric values. We are going to compare two numeric values. So this is one reason why we are using a vector DB. But the SQL database also has indexes, right? If numerical representations are to be compared, then you can take an index and do some queries based on it. So when you're doing like index matching or when you're comparing two indexes, in SQL you can only do it in one dimension, right? One two, right? But here it is actually multiple dimensions. So if I talk about a single chunk, it is getting converted into a 768 dimension vector, right? So or it can also go up to 8192 dimensions, right? There are multiple embedding models which can go up to some more dimensions, right?
1:55Okay, that makes sense. Now there is a user query, right? And user query can be like, how many leaves do I have, right? So the bot that you're making is for your internal company and the user asked that how many leaves do I have now what is happening when the they press enter so that user query is getting converted into embeddings right now here one thing we have to focus is the embedding model which we are using for the chunk is the same embedding model we have to use it here is right so now after the user query is getting a wrote into embed it goes into the vector DB now what is happening inside the vector DB is just a search, right? We are basically searching for the relevant documents which can answer to this particular
2:32user. It can be, there are a lot of search algorithms which takes place inside a vector DB. There can be HNSW and all those things. They are generally called ANN, right? So after this search is happening, you can decide upon how many chunks you actually want right so this k number is actually empirically decided it can be either 5 it can be either 10 generally people start with this number there is no fixed number which I can suggest right now before watching this complete process that how are you making the chunks and all of that it depends upon what is your use case and how is how building. So we can decide upon K based on. Now after this top 5 chunks or top 10 chunks are extracted. Now this process
3:11might look very similar to you because previously what you were doing is you were sending all the documents to the LLM and you were giving a response. So this is what we are going to do. Now since we have the chunks, these are usually called context. To answer to this user query, what was missing in the first place was the context, right? LLM didn't have the context of your internal doc because they were so huge right now, it can hallucinate, it can miss something. So now through this process, through this pipeline, we are making it sure that it doesn't miss anything, right? So we are giving the relevant chunks, relevant context to the LLM along with the user query and the task of the LLM is just to give a response.
3:48So now as you can see through this whole pipeline, basically what I'm trying to do is giving relevant context to the LLM. Now what it also reduces is something which is called hallucination. Now if you were giving these kind of user query previously to the LLM, it might be hallucinating, it might be reinventing some facts, right? But right now we are making sure that LLM is basically answering based on the context which is given to the LLM now there are multiple caveats here that how can I ensure that LLM is only answering for the particular context that is handled in the eval but right now this is the whole process this is the offline process where I'm basically this is the data ingestion pipeline where I'm ingesting that your company documents and all of that storing the embeddings
4:26in the vector DB and when there is a user query online it basically gets converted into embeddings we search for the relevant documents we pass the relevant documents through the LLM along with the user query and it just formats the response okay then it's one problem though is that the data we have is not in a simple form it's usually PDF and those PDFs have tables. So sometimes those tables can be currency conversion that this country has this currency conversion or it could be a policy that in place we do this more than seven days have passed then we do something else and in fact those tables can sometimes be images also in the PDF. So how can use that also extract that context when answering queries. When you're saying that images, tables has images,
5:06is it that there is screenshots of tables or something like that? Yeah, sometimes just take a screenshot of an image and they post it. Sometimes it can be related to scanned PDFs. Scanned PDFs is closest. Yeah. Sometimes it's actually characters only, but somebody has taken a small screenshot and pasted it. Okay. So this is very closely related to how it happens in real time as well, right? So the only change in this whole architecture which you will have to focus more on is the data ingestion pipeline. I'm assuming the tables can not be just you know, simple row column table. It can also be a hierarchical one, right? Where you have multiple rows corresponding to a single row. It can also be some hierarchical table like this. Okay, yeah, we normally normally don't have that complex but luckily if you do it then good yeah so it can be even of this complexity as well so
5:49what we generally focus on more like if i'm using some parsers like pi pdf or pi mu pd they generally don't handle tables that way right so change some processes in the data ingestion pipeline one process can be introducing a layout detection algorithm sort of thing which basically what it does is it checks the layout of the data that you have and based on that you will decide upon the parsing technique. So let's say this is your particular page and here you have some logo, here you have the title, here you have table, text and all of that. What you do is basically the layout detection algorithm checks that okay this is the title, this is a table and this is a text right and you can basically you know design a parsing strategy based on this particular type of page so you can have
6:31one particular parser just for the text element you can have one particular parser just for the table elements because a single parser might not be able to you know extract all of these things very efficiently so this is one solution there are more caveats that we can talk about if you want i think this is great awesome tanishka i think this will really skip okay so are you ready for your unpaid internship now yes yes yes very unpaid and my professional experience is learning we don't consider it sorry i am not i am at a wrong place go to the lm only right and i have taken a 1 billion context window lm also but you are like sending everything all the I am at a wrong place.