Free lesson · AI Engineering
Working with Files and Data
Working with Files and Data
The Data Format That Determines Your Pipeline's First 100 Lines
At Meta, every AI project starts the same way: someone hands you data. A CSV of customer support tickets. A folder of JSON files containing API logs. A PDF of policy documents that need to be searchable. A directory of text files exported from a wiki. Before you write a single line of AI code - before prompts, embeddings, or model calls - you spend your first day reading, cleaning, and transforming data into a format your pipeline can consume.
This is not glamorous work. It is also not optional. The quality of your AI application is bounded by the quality of your data, and the quality of your data is determined by how well you handle the messy reality of files and formats. An AI engineer who cannot read a CSV with mixed encodings, parse nested JSON from an API response, or extract text from a PDF is an AI engineer who cannot build production applications.
This topic covers the file and data processing patterns you will use in the first week of every AI project.
Reading and Writing JSON
JSON is the lingua franca of AI engineering. API requests and responses are JSON. Configuration files are JSON. LLM structured outputs are JSON. Evaluation datasets are stored as JSON or JSONL (one JSON object per line). If you master one file format for AI engineering, make it JSON.
import json
# Reading JSON from a string (API response parsing)
api_response = '{"model": "gpt-4-turbo", "choices": [{"message": {"content": "RAG stands for..."}}]}'
data = json.loads(api_response)
content = data["choices"][0]["message"]["content"]
print(f"Response: {content}")
# Writing JSON (API request construction)
request_body = {
"model": "gpt-4-turbo",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is RAG?"}
],
"temperature": 0.7
}
json_string = json.dumps(request_body, indent=2)
print(json_string)
JSONL - JSON Lines for Large Datasets
JSONL (JSON Lines) stores one JSON object per line. This format is standard for evaluation datasets, training data, and batch processing inputs because it can be read line-by-line without loading the entire file into memory.
import json
# Writing JSONL
eval_examples = [
{"query": "What is RAG?", "expected": "Retrieval Augmented Generation", "category": "definition"},
{"query": "How do embeddings work?", "expected": "Neural network mapping to vectors", "category": "concept"},
{"query": "What is a vector database?", "expected": "Database for similarity search", "category": "definition"},
]
# Write to JSONL format (one JSON object per line)
jsonl_output = "\n".join(json.dumps(ex) for ex in eval_examples)
print("JSONL format:")
print(jsonl_output)
# Reading JSONL line-by-line (memory efficient for large files)
print("\nParsed back:")
for line in jsonl_output.strip().split("\n"):
example = json.loads(line)
print(f" [{example['category']}] {example['query']}")
Why JSONL matters for AI engineering: OpenAI's batch API accepts JSONL input. Fine-tuning datasets are formatted as JSONL. Evaluation datasets are commonly stored as JSONL. The line-by-line format means you can process files with millions of examples without loading the entire file into memory - which is essential for training data preparation and large-scale evaluation.
Reading and Writing CSV
CSV (Comma-Separated Values) is the format you receive when business stakeholders export data from spreadsheets, databases, or analytics tools. Customer support tickets, product catalogs, user feedback - all frequently arrive as CSVs.
import csv
import io
# Reading CSV from a string (simulating file read)
csv_data = """query,score,category
What is RAG?,0.94,definition
How do embeddings work?,0.87,concept
Explain vector databases,0.76,concept"""
reader = csv.DictReader(io.StringIO(csv_data))
results = list(reader)
for row in results:
print(f" {row['query']} → score: {row['score']}, category: {row['category']}")
# Converting CSV rows to the format your pipeline expects
processed = [
{"text": row["query"], "score": float(row["score"]), "metadata": {"category": row["category"]}}
for row in results
]
print(f"\nProcessed {len(processed)} rows into pipeline format")
CSV pitfalls in AI applications: CSV values are always strings - "0.94" is a string, not a float. You must explicitly convert types when processing CSV data for numerical operations. CSV files from different sources use different encodings (UTF-8, Latin-1, Windows-1252), different delimiters (comma, semicolon, tab), and different line endings (Unix LF vs Windows CRLF). The csv module handles delimiters and quoting, but encoding must be specified when opening the file. Always try encoding="utf-8" first, then fall back to encoding="latin-1" if you see garbled characters or UnicodeDecodeError. At Meta, approximately 15% of CSV files from external partners required encoding specification beyond the default UTF-8.
The file reading pattern for AI applications: Use Python's context manager (with statement) for all file operations. This ensures files are properly closed even if an exception occurs during processing. For large files, read line-by-line rather than loading the entire file into memory with .read():
# Memory-efficient file reading
def process_large_file(filepath: str) -> list[dict]:
"""Read a large JSONL file line by line without loading into memory."""
results = []
with open(filepath, "r", encoding="utf-8") as f:
for line_number, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
data = json.loads(line)
results.append(data)
except json.JSONDecodeError as e:
print(f" Skipping line {line_number}: {e}")
return results
Working with Text Files
Text files are the raw material for RAG applications. Documentation, articles, transcripts, code files, and plain-text exports all arrive as text that needs to be read, chunked, and processed.
# Simulating text file processing
document = """# Introduction to RAG
Retrieval Augmented Generation (RAG) is a technique that combines
a retrieval system with a language model. Instead of relying solely
on the model's training data, RAG retrieves relevant documents
and includes them in the model's context.
## Why RAG Matters
RAG reduces hallucination by grounding the model's responses
in actual source documents. This is critical for enterprise
applications where accuracy matters.
## How RAG Works
1. The user submits a query
2. The retrieval system finds relevant documents
3. The documents are added to the model's context
4. The model generates a response grounded in the documents"""
# Split into sections by heading
sections = []
current_section = {"title": "", "content": ""}
for line in document.strip().split("\n"):
if line.startswith("# "):
if current_section["content"]:
sections.append(current_section)
current_section = {"title": line.lstrip("# "), "content": ""}
else:
current_section["content"] += line + "\n"
if current_section["content"]:
sections.append(current_section)
for section in sections:
word_count = len(section["content"].split())
print(f" Section: '{section['title']}' - {word_count} words")
Extracting Text from PDFs
PDFs are the most common format for formal documents - research papers, contracts, policy documents, technical specifications. Extracting text from PDFs is a prerequisite for making these documents searchable via RAG.
# Using PyPDF2 for basic text extraction
from pypdf import PdfReader
def extract_text_from_pdf(pdf_path: str) -> list[dict]:
"""Extract text from each page of a PDF."""
reader = PdfReader(pdf_path)
pages = []
for i, page in enumerate(reader.pages):
text = page.extract_text()
if text and text.strip(): # Skip empty pages
pages.append({
"page_number": i + 1,
"text": text.strip(),
"word_count": len(text.split())
})
return pages
# Usage
# pages = extract_text_from_pdf("technical_spec.pdf")
# for page in pages:
# print(f" Page {page['page_number']}: {page['word_count']} words")
PDF extraction limitations: PyPDF2/pypdf handles text-based PDFs well but fails on scanned PDFs (which are images, not text). For scanned PDFs, use OCR tools like Tesseract or cloud services like AWS Textract. For complex PDFs with tables, columns, and mixed layouts, use specialised libraries like unstructured.io or Adobe PDF Extract API. The extraction quality varies dramatically by PDF structure - always inspect the extracted text before feeding it to your pipeline.
Data Cleaning Patterns
Raw data from files is never ready for AI processing. It contains encoding artefacts, HTML tags, excessive whitespace, special characters, and formatting that must be cleaned before embedding or prompting.
import re
def clean_text(text: str) -> str:
"""Clean text for AI pipeline consumption."""
# Remove HTML tags
text = re.sub(r'<[^>]+>', '', text)
# Replace multiple whitespace with single space
text = re.sub(r'\s+', ' ', text)
# Remove control characters (but keep newlines and tabs)
text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', text)
# Strip leading/trailing whitespace
text = text.strip()
return text
# Test with messy input
messy = " <p>This is some <b>messy</b> text \n\n\n with extra spaces. </p> "
clean = clean_text(messy)
print(f"Before: '{messy}'")
print(f"After: '{clean}'")
def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
"""Split text into overlapping chunks for embedding."""
words = text.split()
chunks = []
for i in range(0, len(words), chunk_size - overlap):
chunk = " ".join(words[i:i + chunk_size])
if chunk: # Skip empty chunks
chunks.append(chunk)
return chunks
# Test chunking
long_text = " ".join(["word"] * 1200)
chunks = chunk_text(long_text, chunk_size=500, overlap=50)
print(f"\nText of {len(long_text.split())} words → {len(chunks)} chunks")
for i, chunk in enumerate(chunks):
print(f" Chunk {i+1}: {len(chunk.split())} words")
Pandas Basics for AI Data Processing
Pandas is Python's standard library for tabular data manipulation. In AI engineering, Pandas handles evaluation dataset management, training data preparation, result analysis, and any task where you work with rows and columns of data.
import json
# Simulating Pandas-like operations with built-in Python
# (Pandas is typically imported as: import pandas as pd)
# Evaluation dataset as list of dicts (Pandas DataFrames work the same way)
eval_data = [
{"query": "What is RAG?", "category": "definition", "score": 0.94, "correct": True},
{"query": "How do embeddings work?", "category": "concept", "score": 0.87, "correct": True},
{"query": "Explain transformers", "category": "concept", "score": 0.72, "correct": False},
{"query": "What is FAISS?", "category": "tool", "score": 0.65, "correct": False},
{"query": "Define tokenization", "category": "definition", "score": 0.91, "correct": True},
]
# Group by category and compute accuracy
from collections import defaultdict
category_stats = defaultdict(lambda: {"total": 0, "correct": 0})
for row in eval_data:
cat = row["category"]
category_stats[cat]["total"] += 1
if row["correct"]:
category_stats[cat]["correct"] += 1
print("Per-category accuracy:")
for cat, stats in category_stats.items():
acc = stats["correct"] / stats["total"]
print(f" {cat}: {acc:.0%} ({stats['correct']}/{stats['total']})")
# Overall metrics
total_correct = sum(1 for r in eval_data if r["correct"])
avg_score = sum(r["score"] for r in eval_data) / len(eval_data)
print(f"\nOverall: {total_correct}/{len(eval_data)} correct, avg score: {avg_score:.2f}")
When to use Pandas vs built-in Python: Use built-in Python (lists, dicts, list comprehensions) for simple data processing with fewer than 10,000 rows. Use Pandas for larger datasets, complex aggregations, pivot tables, and data exploration. In AI engineering, Pandas is most useful for evaluation result analysis (computing per-category metrics across thousands of examples) and training data preparation (filtering, sampling, stratifying large datasets).
graph TD
subgraph Formats["Data Formats in AI Engineering"]
F1["JSON/JSONL\nAPI payloads, eval datasets\nmost common format"]
F2["CSV/TSV\nBusiness data exports\nspreadsheet data"]
F3["Text/Markdown\nDocuments for RAG\nknowledge base content"]
F4["PDF\nFormal documents\nrequires extraction"]
end
subgraph Processing["Processing Pipeline"]
P1["Read file\n(json.loads, csv.reader, open)"]
P2["Clean text\n(HTML removal, whitespace, encoding)"]
P3["Transform\n(type conversion, chunking, structuring)"]
P4["Validate\n(Pydantic models from A.4)"]
P5["Pipeline-ready data"]
end
Formats --> P1
P1 --> P2
P2 --> P3
P3 --> P4
P4 --> P5
style Formats fill:#1e1b4b,stroke:#818cf8,stroke-width:2px
style Processing fill:#166534,stroke:#86efac,stroke-width:2px
This diagram shows data formats and the processing pipeline in AI engineering. Four common formats feed the pipeline: JSON and JSONL for API payloads and evaluation datasets, CSV and TSV for business data exports, text and Markdown for documents in RAG knowledge bases, and PDF for formal documents requiring extraction. The processing pipeline has five stages: read the file using json.loads, csv.reader, or open; clean the text by removing HTML, fixing whitespace, and handling encoding; transform through type conversion, chunking, and structuring; validate using Pydantic models from topic A.4; and produce pipeline-ready data.
graph LR
subgraph Chunking["Text Chunking for RAG"]
C1["Raw document\n2,400 words"]
C2["Chunk 1\n500 words"]
C3["Chunk 2\n500 words\n(50-word overlap with 1)"]
C4["Chunk 3\n500 words\n(50-word overlap with 2)"]
C5["Chunk 4\n450 words\n(50-word overlap with 3)"]
end
style Chunking fill:#312e81,stroke:#a5b4fc,stroke-width:2px
This diagram shows how text chunking works for RAG. A raw document of 2,400 words is split into overlapping chunks. Chunk 1 is 500 words. Chunk 2 is 500 words with a 50-word overlap with Chunk 1. Chunk 3 is 500 words with a 50-word overlap with Chunk 2. Chunk 4 is 450 words (the remaining text) with a 50-word overlap with Chunk 3. The overlap ensures that information at chunk boundaries is preserved in at least two chunks.
Interview: AI engineering interview connection: Data processing is rarely the focus of an interview question, but it is the foundation of every take-home assignment and live coding exercise. When an interviewer gives you a dataset (evaluation results, user feedback, document corpus) and asks you to build something with it, the first 15 minutes are spent reading, cleaning, and transforming the data. Speed and fluency with JSON parsing, CSV reading, and text cleaning directly affect how much time you have for the AI-specific parts of the problem.
What to practice: Download a public dataset (a CSV from Kaggle, a JSONL from HuggingFace), read it into Python, clean it, and transform it into a format suitable for embedding or prompting. This exercise builds the muscle memory that saves time in interview settings.
Exercise: Starter: Write a function that reads a JSONL file (one JSON object per line), filters to rows where a "score" field exceeds a threshold, and returns the results as a list of dictionaries. Handle the case where a line contains invalid JSON.
Intermediate: Write a complete data processing pipeline that: reads a CSV of customer support tickets (columns: ticket_id, customer_message, category, resolution), cleans the customer_message text (remove HTML, normalise whitespace), chunks messages longer than 500 words, and outputs a JSONL file where each line is a dictionary with ticket_id, chunk_index, chunk_text, and category. Handle encoding errors gracefully.
Advanced: Write a document ingestion pipeline that processes a directory containing mixed file types: .txt, .json, .csv, and .pdf files. For each file, detect the format, extract text appropriately, clean the text, chunk it into 500-word segments with 50-word overlap, and output a unified JSONL file with source_file, chunk_index, chunk_text, and metadata. Include error handling for unreadable files and malformed data.
Key Takeaways
JSON is the primary data format in AI engineering - API requests, API responses, evaluation datasets, and configuration files are all JSON. Mastering json.loads() (string to dict) and json.dumps() (dict to string) covers 80% of the data serialisation you will do. JSONL (one JSON object per line) is the standard for large datasets because it enables line-by-line processing without loading the entire file into memory.
Data cleaning is the unglamorous first step of every AI project. Raw text contains HTML tags, encoding artefacts, excessive whitespace, and control characters that degrade embedding quality and confuse LLMs. A clean_text() function that handles these common issues is a utility you will copy into every project.
Text chunking - splitting documents into overlapping segments for embedding - is the bridge between raw documents and RAG-ready data. The chunk size (typically 200-1000 words) and overlap (typically 10-20% of chunk size) are tunable parameters that affect retrieval quality. Smaller chunks produce more precise retrieval. Larger chunks preserve more context. The optimal size depends on your retrieval evaluation results, not on a fixed rule.
PDF extraction varies dramatically by document structure. Text-based PDFs extract cleanly with pypdf. Scanned PDFs require OCR (Tesseract, AWS Textract, Google Document AI). Complex layouts with tables, multi-column formats, and embedded images require specialised tools like unstructured.io. Always inspect extracted text before feeding it to your pipeline - extraction quality is the silent quality bottleneck in many RAG applications. A RAG system that retrieves the right document but extracts garbled text from it produces the same result as a system that retrieves the wrong document entirely.