The technology described here boosts exhaustivity and structuredness in LLM prompt results, efficiently exploiting the knowledge graph and contextual structure present in any professional or enterprise corpus. The case study deals with public financial reports from Nvidia, available as PDF documents.
In this article, I discuss the preprocessing steps used to turn a PDF repository into input suitable for LLMs. It includes contextual chunking, indexing text entities with hierarchical multi-index system, and retrieving contextual elements including lists, sub-lists, fonts (type, color, and size), images and tables – some not detected by standard Python libraries. I also discuss how to build additional contextual information such as agents, categories, or tags, to add to text entities to further improve any LLM architecture, and prompt results.
Methodology
I use the PyMuPDF Python library, with home-made algorithms to retrieve tables that it fails to detect. Another useful library is LlamaParse. Yet, a different approach consists of saving each PDF page (a slide in this case) as an image, then using computer vision technology to retrieve the various elements from the images and turn them into text when appropriate. However, I will not follow this approach. Instead, I convert the PDFs to JSON and then parse the JSON elements, including tables, diagrams, and images.
Figure 1 shows one input slide. Figure 2 shows the retrieved elements, including an undetected table embedded in the histogram (ID2 = TD0, TL0) and a bullet list and sub list. Note the multi-index consisting of ID1, ID2, ID3, Size (the size of the font) and other components not shown here: document ID, page number, font type (bold, italic) and color. Other examples show slides with multiples lists, images, or tables.
Get the full article
Includes high resolution images, the full input data, source code with output file, links to GitHub, and code documentation. The code does a lot more than the highlights discussed here, for instance dealing with special characters.
Download the free article, here. It contains the whole new chapter 10 added today to my book “Building Disruptive AI & LLM Technology from Scratch”. The relevant material is in section 10.1. There is also a subsection explaining how I build agents on the backend rather than the frontend (standard approach), leading to better design. Section 10.3 explains the differences between LLM 1.0 and 2.0.
In the full book, available here, all links are clickable. For GitHub, look for filenames starting with “PDF”, here. To not miss future articles on this topic and about GenAI in general, subscribe to my free newsletter, here. Subscribers get a 20% discount on all my books.
About the Author
Vincent Granville is a pioneering GenAI scientist and machine learning expert, co-founder of Data Science Central (acquired by a publicly traded company in 2020), Chief AI Scientist at MLTechniques.com and GenAItechLab.com, former VC-funded executive, author (Elsevier) and patent owner — one related to LLM. Vincent’s past corporate experience includes Visa, Wells Fargo, eBay, NBC, Microsoft, and CNET. Follow Vincent on LinkedIn.