Batch Upload

​

Sample Request

curl --request POST \
  --url 'https://api.usecortex.ai/upload/batch_upload?tenant_id=tenant_1234&sub_tenant_id=sub_tenant_4567' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: multipart/form-data' \
  --form files=@example-file \
  --form 'file_ids=["doc_123"]' \
  --form 'tenant_metadata={}' \
  --form 'document_metadata={}'

​

Batch Processing Pipeline

​

1. Immediate Upload & Queue

• All documents are immediately accepted and stored securely
• Each document is added to our processing queue for background processing
• You receive a confirmation response with individual file_ids for tracking each file

​

2. Parallel Processing Phase

• Content Extraction: Extracting text from various supported formats (see Supported File Formats section below)
• Document Parsing: Understanding document structure, headers, and formatting
• Text Cleaning: Removing formatting artifacts and normalizing content

​

3. Intelligent Chunking

• Each document is split into semantically meaningful chunks
• Chunk size is optimized for both context preservation and search accuracy
• Overlapping boundaries ensure no information is lost between chunks
• Metadata is preserved and associated with each chunk

​

4. Embedding Generation

• Each chunk is converted into high-dimensional vector embeddings
• Embeddings capture semantic meaning and context
• Vectors are optimized for similarity search and retrieval

​

5. Indexing & Database Updates

• Embeddings are stored in our vector database for fast similarity search
• Full-text search indexes are created for keyword-based queries
• Metadata is indexed for filtering and faceted search
• Cross-references are established between related documents

​

6. Quality Assurance

• Automated quality checks ensure processing accuracy for each document
• Content validation verifies extracted text completeness
• Embedding quality is assessed for optimal retrieval performance

Recommended: For optimal performance, limit each batch to a maximum of 20 sources per request. Send multiple batch requests with an interval of 1 second between each request.

File ID Management: The system uses a priority-based approach for file ID assignment:

1. First Priority: If you provide a file_id as a direct body parameter, that specific ID will be used
2. Second Priority: If no direct file_id is provided, the system checks for a file_id in the document_metadata object
3. Auto-Generation: If neither source provides a file_id, the system will automatically generate a unique identifier

​

Duplicate File ID Behavior

• Overwrite Behavior: Each existing document with a matching file_id will be completely replaced with the new document
• Processing: Each new document will go through the full processing pipeline independently
• Search Results: Previous search results and embeddings from old documents will be replaced with the new documents’ content
• Idempotency: Uploading the same documents with the same file_ids multiple times is safe and will result in the same final state

{
  "message": "Batch upload successful. Documents with existing file_ids have been overwritten.",
  "document_ids": ["doc_123456", "doc_789012", "doc_345678", "doc_901234"],
  "overwritten_file_ids": ["doc_123456", "doc_789012"],
  "status": "success"
}

​

Supported File Formats

​

Best Practices

​

Document Preparation

• File Size: Documents up to 50MB are processed efficiently
• Content Quality: Clear, well-structured documents produce better embeddings
• Metadata: Include rich metadata for better filtering and organization

​

Processing Optimization

• Batch Size: Limit each batch to a maximum of 20 sources per request
• Request Intervals: Send multiple batch requests with an interval of 1 second between each request
• Metadata Consistency: Use consistent metadata schemas across your organization
• File Naming: Descriptive filenames help with document identification

​

Troubleshooting

• Wait 5-10 minutes for processing to complete
• Check if any document status is errored (rare occurrence)
• Verify your search query and filters

• Large documents (100+ pages) take longer to process
• Complex formatting may require additional processing time
• High system load may temporarily slow processing

• If status shows errored, ensure your documents aren’t corrupted or password-protected
• Check that the file format is supported (see Supported File Formats section above)
• Verify your API key has sufficient permissions
• For unsupported formats, you’ll receive a 400 error with the message: "Unsupported file format: [filename]. Please check our supported file formats documentation."

​

Error Responses

[
  {
    "file_id": "CortexDoc1234",
    "filename": "document1.pdf"
  },
  {
    "file_id": "CortexDoc4567",
    "filename": "document2.docx"
  }
]