I think you are running into 32-bit memory limitations while dealing with a large flat file (~4.5 million records), especially during command line or SQL Agent execution.
Even though your Server is 64-bit with 16 GB RAM. You have increased DefaultBufferMaxRows and DefaultBufferSize
SSIS may still be using the 32-bit runtime.
Try the below Steps:
- Make sure You are Using 64-bit Runtime
When executing the Command Line: Use dtexec from the 64-bit path
C:\Program Files\Microsoft SQL Server\130\DTS\Binn\dtexec.exe
For SQL Agent: Go to Job Step > Execution Options and uncheck 'Use 32-bit runtime'.
- Optimize SSIS Buffer Settings
update as below
DefaultBufferMaxRows = 100000
DefaultBufferSize = 10485760 (10 MB)
Avoid very large buffers (like 20 MB+) as they increase fragmentation risk.
- Check Flat File Source Settings
Ensure correct column delimiter, text qualifier, and row delimiter.
Consider using a Script Component to better handle parsing if data is messy.
- Split the File / Process in Chunks
If possible, split your 4.5 million row CSV into smaller batches and process them incrementally.
- Increase Virtual Memory on the Server
Sometimes increasing the paging file size manually helps SSIS buffer allocations.