MDF Tools Installer¶
Overview¶
The MDF Tools Installer provides an interactive setup wizard that automates the installation and configuration of prerequisites needed for processing SQL Server MDF (Master Database Files). This includes Docker Desktop installation, SQL Server Express container setup, and comprehensive container management tools.
Features¶
- ๐ Docker Desktop Integration: Automatic detection and installation across platforms
- ๐๏ธ SQL Server Express Setup: Containerized SQL Server 2019 Express with persistent storage
- โ๏ธ Interactive Configuration: Guided setup with customizable passwords and ports
- ๐ Real-time Status Monitoring: Comprehensive health checks for all components
- ๐ง Container Lifecycle Management: Complete control over SQL Server container
- ๐ Secure Configuration: Encrypted password storage and configurable security settings
Quick Start¶
# Install MDF processing tools
pyforge install mdf-tools
# Check installation status
pyforge mdf-tools status
# Test SQL Server connectivity
pyforge mdf-tools test
System Requirements¶
Minimum Requirements¶
- Operating System: Windows 10+, macOS 10.15+, or Ubuntu 18.04+
- Memory: 4GB RAM total (1.4GB for SQL Server + 2.6GB for host system)
- Storage: 4GB free space (2GB for Docker images + 2GB for SQL Server data)
- Network: Internet connection for downloading Docker images (~700MB)
- Docker: Docker Desktop 4.0+ with container support
Recommended Requirements¶
- Memory: 8GB RAM (for optimal performance with multiple databases)
- Storage: 20GB free space (for multiple MDF files and conversions)
- CPU: 4+ cores (though SQL Server Express limited to 4 cores max)
- Network: Broadband connection for faster image downloads
SQL Server Express Constraints¶
- Maximum Database Size: 10GB per attached MDF file
- Memory Limit: 1.4GB buffer pool (cannot be increased)
- CPU Utilization: 1 socket or 4 cores maximum
- Concurrent Connections: Practical limit of 5-10 users
- Query Parallelism: Disabled (DOP = 1)
Supported Platforms¶
- โ macOS (Intel and Apple Silicon)
- โ Windows (Windows 10/11 with WSL2)
- โ Linux (Ubuntu, CentOS, RHEL, Debian)
Architecture Overview¶
MDF Tools Installation Architecture¶
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ HOST SYSTEM โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ ๐ค User โ
โ โ โ
โ โโโบ ๐ง PyForge CLI โ
โ โ โ
โ โโโบ ๐ ~/.pyforge/mdf-config.json โ
โ โ โ
โ โโโบ ๐ณ Docker Desktop โ
โ โ โ
โ โโโบ ๐ฆ SQL Server Container (pyforge-sql-server) โ
โ โโโบ ๐๏ธ SQL Server Express 2019 โ
โ โโโบ ๐ง sqlcmd Tools โ
โ โโโบ ๐พ master database โ
โ โโโบ ๐พ Attached MDF Database โ
โ โ โ
โ โโโบ ๐ Docker Volumes โ
โ โโโบ pyforge-sql-data (/var/opt/mssql) โ
โ โโโบ pyforge-mdf-files (/mdf-files) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Installation Flow¶
Step-by-Step Installation Process:
1. User Command
๐ค User โ pyforge install mdf-tools
2. System Check
๐ง PyForge CLI โ Check OS compatibility
๐ง PyForge CLI โ Detect Docker installation
3. Docker Setup
๐ง PyForge CLI โ Start Docker Desktop
๐ง PyForge CLI โ Pull SQL Server image (700MB)
4. Container Creation
๐ง PyForge CLI โ Create pyforge-sql-server container
๐ง PyForge CLI โ Configure port mapping (1433)
๐ง PyForge CLI โ Mount persistent volumes
5. SQL Server Configuration
๐ง PyForge CLI โ Start SQL Server Express
๐ง PyForge CLI โ Test connectivity with sqlcmd
๐ง PyForge CLI โ Verify database engine
6. Finalization
๐ง PyForge CLI โ Save configuration file
๐ง PyForge CLI โ Display connection details
โ
Installation Complete!
Installation Workflow Components¶
1. Host System Components: - PyForge CLI: Main application orchestrating the installation - Configuration File: Persistent settings stored locally - Docker Desktop: Container runtime environment
2. Container Infrastructure: - SQL Server Express 2019: Database engine for MDF processing - Persistent Volumes: Data survival across container restarts - Network Mapping: Port 1433 exposed to host system
3. Data Flow: - Installation: CLI โ Docker โ SQL Server โ Configuration - MDF Processing: MDF File โ Volume Mount โ SQL Server โ Parquet Output - Management: CLI Commands โ Docker API โ Container Lifecycle
System Integration Points¶
Component | Purpose | Technology | Persistence |
---|---|---|---|
Docker Desktop | Container orchestration | Docker Engine | System service |
SQL Server Container | Database engine | SQL Server Express 2019 | Container lifecycle |
Data Volume | SQL Server system data | Docker volume | Persistent across restarts |
MDF Volume | User MDF files | Docker volume | Persistent across restarts |
Configuration | Connection settings | JSON file | Local filesystem |
Network Bridge | Host-container communication | Docker bridge | Dynamic port mapping |
MDF Processing Workflow¶
How MDF Files Are Processed (Future Feature):
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ MDF TO PARQUET CONVERSION โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Step 1: File Preparation
๐ค User โ pyforge convert database.mdf
๐ MDF File โ Copy to /mdf-files volume
Step 2: Database Attachment
๐ง PyForge CLI โ ATTACH DATABASE 'database.mdf'
๐๏ธ SQL Server โ Validate MDF structure
โ
SQL Server โ Database attached successfully
Step 3: Schema Discovery
๐ง PyForge CLI โ Query table metadata
๐๏ธ SQL Server โ Return table schemas & row counts
๐ PyForge CLI โ Display table overview
Step 4: Data Extraction
๐ง PyForge CLI โ Execute SELECT queries (chunked)
๐๏ธ SQL Server โ Return table data in batches
๐ฆ PyForge CLI โ Convert to string format
Step 5: Parquet Generation
๐ฆ PyForge CLI โ Generate .parquet files
๐ PyForge CLI โ Create Excel summary report
Step 6: Cleanup
๐ง PyForge CLI โ DETACH DATABASE
๐ MDF File โ Remains in volume (unchanged)
โ
Conversion Complete!
Output Structure:
database_parquet/
โโโ Users.parquet
โโโ Orders.parquet
โโโ Products.parquet
โโโ conversion_summary.xlsx
Supported MDF File Types¶
SQL Server Version | MDF Compatibility | Processing Status |
---|---|---|
SQL Server 2019 | โ Native | Optimal performance |
SQL Server 2017 | โ Compatible | Full support |
SQL Server 2016 | โ Compatible | Full support |
SQL Server 2014 | โ Compatible | Full support |
SQL Server 2012 | โ Compatible | Full support |
SQL Server 2008/R2 | โ ๏ธ Limited | May require upgrade |
SQL Server 2005 | โ Incompatible | Not supported |
Note: MDF files from newer SQL Server versions (2022+) may not be compatible with SQL Server Express 2019.
Installation Process¶
The installer follows a structured 5-stage process:
Stage 1: System Requirements Check¶
- Validates operating system compatibility
- Checks Docker Desktop installation status
- Verifies Docker SDK for Python availability
Stage 2: Docker Installation (if needed)¶
- macOS: Automatic installation via Homebrew
- Windows: Automatic installation via Winget
- Linux: Package manager instructions (apt/yum)
- Manual: Step-by-step installation guides
Stage 3: Docker Startup¶
- Connects to Docker daemon
- Waits for Docker Desktop to be fully operational
- Validates Docker API accessibility
Stage 4: SQL Server Express Setup¶
- Downloads Microsoft SQL Server 2019 Express image
- Creates and configures container with:
- Persistent data volume (
pyforge-sql-data
) - MDF files mount point (
pyforge-mdf-files
) - Default port mapping (1433)
- Secure password configuration
Stage 5: Configuration and Validation¶
- Saves configuration to
~/.pyforge/mdf-config.json
- Tests SQL Server connectivity using sqlcmd
- Displays connection details and next steps
macOS Installation Walkthrough¶
Scenario 1: Docker Already Installed¶
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ PyForge MDF Tools Setup Wizard โ
โ Setting up Docker Desktop and SQL Server Express for MDF file processing โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
[1/5] Checking system requirements...
โ Operating System: macOS 14.5.0 (supported)
โ Docker Desktop: Installed
โ Docker SDK for Python: Available
[3/5] Starting Docker Desktop...
โ Docker Desktop is running
[4/5] Setting up SQL Server Express...
๐ฅ Pulling SQL Server image: mcr.microsoft.com/mssql/server:2019-latest
โ ด โ SQL Server image downloaded
๐ Creating SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โ SQL Server is ready
[5/5] Installation Complete!
SQL Server Connection Details
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Property โ Value โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Host โ localhost โ
โ Port โ 1433 โ
โ Username โ sa โ
โ Password โ PyForge@2024! โ
โ Container โ pyforge-sql-server โ
โ Config File โ /Users/username/.pyforge/mdf-config.jsonโ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
๐ Setup completed successfully!
Scenario 2: Docker NOT Installed¶
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ PyForge MDF Tools Setup Wizard โ
โ Setting up Docker Desktop and SQL Server Express for MDF file processing โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
[1/5] Checking system requirements...
โ Operating System: macOS 14.5.0 (supported)
โ Docker Desktop: Not found
[2/5] Docker Installation Required
Docker Desktop is required for MDF file conversion.
Would you like to:
1. Install automatically using Homebrew (recommended)
2. Get installation instructions
3. Skip (I'll install manually)
4. Continue without Docker (installation will fail)
Choice [1]: 1
๐ฆ Installing Docker Desktop (this may take several minutes)...
โ
Docker Desktop installed successfully!
๐ Launching Docker Desktop...
โณ Waiting for Docker Desktop to start...
โ
Docker Desktop is running!
[4/5] Setting up SQL Server Express...
๐ฅ Pulling SQL Server image: mcr.microsoft.com/mssql/server:2019-latest
โ ด โ SQL Server image downloaded
๐ Creating SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โ SQL Server is ready
[5/5] Installation Complete!
Command Reference¶
Installation Commands¶
pyforge install mdf-tools
¶
Interactive installation wizard for MDF processing tools.
Usage:
Options:
- --password PASSWORD
: Custom SQL Server password (default: PyForge@2024!)
- --port PORT
: Custom SQL Server port (default: 1433)
- --non-interactive
: Run in non-interactive mode for automation
Examples:
# Default installation
pyforge install mdf-tools
# Custom password and port
pyforge install mdf-tools --password "MySecure123!" --port 1433
# Non-interactive mode (for scripts)
pyforge install mdf-tools --non-interactive
Container Management Commands¶
pyforge mdf-tools status
¶
Displays comprehensive status of all MDF tools components.
Usage:
Sample Output:
MDF Tools Status
โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Component โ Status โ Details โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Docker Installed โ โ OK โ Docker command available โ
โ Docker Running โ โ OK โ Docker daemon responsive โ
โ SQL Container Exists โ โ OK โ Container created โ
โ SQL Container Running โ โ OK โ Container active โ
โ SQL Server Responding โ โ OK โ Database accessible โ
โ Configuration File โ โ OK โ Settings saved โ
โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
All systems operational - ready for MDF processing!
pyforge mdf-tools start
¶
Starts the SQL Server Express container.
Usage:
Sample Output:
๐ Starting SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โ SQL Server is ready
pyforge mdf-tools stop
¶
Stops the SQL Server Express container.
Usage:
Sample Output:
pyforge mdf-tools restart
¶
Restarts the SQL Server Express container.
Usage:
Sample Output:
๐ Stopping SQL Server container...
โ SQL Server container stopped
๐ Starting SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โ SQL Server is ready
pyforge mdf-tools logs
¶
Displays SQL Server container logs.
Usage:
Options:
- --lines N
, -n N
: Number of log lines to show (default: 50)
Examples:
# Show last 50 lines (default)
pyforge mdf-tools logs
# Show last 100 lines
pyforge mdf-tools logs --lines 100
# Show last 10 lines
pyforge mdf-tools logs -n 10
pyforge mdf-tools config
¶
Displays current MDF tools configuration.
Usage:
Sample Output:
Configuration file: /Users/username/.pyforge/mdf-config.json
{
"sql_server": {
"container_name": "pyforge-sql-server",
"image": "mcr.microsoft.com/mssql/server:2019-latest",
"host": "localhost",
"port": 1433,
"username": "sa",
"password": "PyForge@2024!",
"data_volume": "pyforge-sql-data",
"mdf_volume": "pyforge-mdf-files"
},
"docker": {
"installed_version": "Docker version 20.10.17",
"installation_date": "2024-01-15T10:30:00Z"
},
"installer_version": "1.0.0"
}
pyforge mdf-tools test
¶
Tests SQL Server connectivity and responsiveness.
Usage:
Sample Output:
pyforge mdf-tools uninstall
¶
Removes SQL Server container and cleans up all data.
Usage:
Sample Output:
Are you sure you want to remove SQL Server and all data? [y/n]: y
๐ Stopping and removing container...
โ Container removed
โ Data volume removed
โ MDF files volume removed
โ Configuration file removed
โ
Uninstall completed successfully
Configuration¶
Configuration File Location¶
The installer saves configuration to ~/.pyforge/mdf-config.json
:
{
"sql_server": {
"container_name": "pyforge-sql-server",
"image": "mcr.microsoft.com/mssql/server:2019-latest",
"host": "localhost",
"port": 1433,
"username": "sa",
"password": "PyForge@2024!",
"data_volume": "pyforge-sql-data",
"mdf_volume": "pyforge-mdf-files"
},
"docker": {
"installed_version": "Docker version 20.10.17",
"installation_date": "2024-01-15T10:30:00Z"
},
"installer_version": "1.0.0"
}
Customizable Settings¶
Custom Password¶
Custom Port¶
Docker Volumes¶
The installer creates two persistent Docker volumes:
pyforge-sql-data
¶
- Mount Point:
/var/opt/mssql
- Purpose: SQL Server system databases and data files
- Persistence: Survives container restarts and recreations
pyforge-mdf-files
¶
- Mount Point:
/mdf-files
- Purpose: MDF files to be processed
- Access: Shared between host and container
SQL Server Express 2019 Technical Details¶
Database Engine Specifications¶
The MDF Tools Installer uses Microsoft SQL Server Express 2019, the free edition of SQL Server's enterprise database engine. This provides a robust, production-grade database environment for MDF file processing.
SQL Server Express 2019 Key Features: - Core Engine: Same database engine as Enterprise edition - T-SQL Support: Full Transact-SQL language support - Security: Enterprise-grade security features - Reliability: ACID compliance and transaction support - Performance: Query optimizer and execution engine - Backup/Restore: Full backup and restore capabilities
Edition Limitations and Constraints¶
โ ๏ธ Important Limitations to Consider:
Limitation | SQL Server Express 2019 | Impact on MDF Processing |
---|---|---|
Database Size | 10 GB per database maximum | Large MDF files (>10GB) cannot be processed |
Memory (RAM) | 1.4 GB buffer pool limit | Performance may be limited with large datasets |
CPU Cores | 1 socket or 4 cores maximum | Processing may be slower on high-core systems |
Concurrent Users | No enforced limit (practical ~5-10) | Multiple simultaneous conversions may impact performance |
Parallelism | Degree of Parallelism (DOP) = 1 | Queries cannot use parallel execution |
โ Features Not Available: - SQL Server Agent (automated jobs) - Advanced Services (Analysis Services, Reporting Services) - Advanced security features (Always Encrypted, Row-Level Security) - Advanced performance features (In-Memory OLTP, Columnstore) - Enterprise backup compression - Database mirroring and log shipping
โ Features Available for MDF Processing: - Full T-SQL query support - ATTACH DATABASE functionality - All standard data types - Backup and restore operations - Database schemas and relationships - Indexes and constraints
Performance Characteristics¶
Optimal MDF File Sizes: - Small MDF files: < 1 GB (Excellent performance) - Medium MDF files: 1-5 GB (Good performance) - Large MDF files: 5-10 GB (Acceptable performance, may require chunking) - Very Large MDF files: > 10 GB (โ Cannot be processed - requires SQL Server Standard/Enterprise)
Memory Usage Patterns: - Container Base: ~500 MB (SQL Server Express) - Available for Data: ~900 MB (after system overhead) - Recommended Host RAM: 4 GB minimum (for container + host OS)
Database Connection Details¶
Connection Parameters¶
- Server:
localhost
- Port:
1433
(default) or custom port - Database:
master
(default system database) - Authentication: SQL Server Authentication
- Username:
sa
(system administrator) - Password:
PyForge@2024!
(default) or custom password - Edition: SQL Server Express 2019
- Version: Microsoft SQL Server 2019 (RTM) - 15.0.4430.1
Connection String Examples¶
Python (pyodbc)¶
import pyodbc
connection_string = (
"DRIVER={ODBC Driver 17 for SQL Server};"
"SERVER=localhost,1433;"
"DATABASE=master;"
"UID=sa;"
"PWD=PyForge@2024!"
)
conn = pyodbc.connect(connection_string)
Command Line (sqlcmd)¶
# From host (requires SQL Server tools)
sqlcmd -S localhost,1433 -U sa -P "PyForge@2024!" -Q "SELECT 1"
# From container
docker exec pyforge-sql-server /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P "PyForge@2024!" -Q "SELECT 1" -C
Scaling Beyond SQL Server Express¶
When to Consider Upgrading¶
Upgrade to SQL Server Standard/Enterprise if you encounter: - MDF files larger than 10 GB - Need for high-performance parallel processing - Requirements for SQL Server Agent automation - Advanced security features (Always Encrypted, etc.) - Multiple concurrent users (>10) - Enterprise backup and restore features
Alternative Solutions¶
For Large MDF Files (>10 GB): 1. Split Processing: Break large tables into chunks using date ranges or ID ranges 2. SQL Server Standard: Upgrade to paid edition with higher limits 3. Cloud Solutions: Use Azure SQL Database or SQL Managed Instance 4. Alternative Tools: Consider specialized MDF extraction utilities
Migration Path Examples:
# Option 1: Cloud-based processing
# Upload MDF to Azure SQL Database
# Process using cloud resources
# Download results
# Option 2: Chunked processing (when converter supports it)
# pyforge convert large.mdf --chunk-size 1000000 --date-range "2020-2023"
# pyforge convert large.mdf --chunk-size 1000000 --date-range "2019-2020"
Cost Considerations¶
Edition | Cost | Database Size Limit | Memory Limit | Use Case |
---|---|---|---|---|
Express | Free | 10 GB | 1.4 GB | Development, small applications |
Standard | ~$1,500+ | 524 PB | OS limit | Medium applications |
Enterprise | ~$5,000+ | 524 PB | OS limit | Large enterprise applications |
Security Considerations¶
Password Security¶
- Default password meets SQL Server complexity requirements
- Custom passwords should be strong (8+ characters, mixed case, numbers, symbols)
- Passwords are stored in local configuration file (not transmitted)
Network Security¶
- SQL Server only accessible on localhost by default
- Container isolated in Docker bridge network
- No external network exposure unless explicitly configured
Container Security¶
- Runs SQL Server Express (free edition with limitations)
- Container uses official Microsoft SQL Server image
- Automatic security updates through image updates
Troubleshooting¶
Common Issues¶
Docker Desktop Not Starting¶
Symptoms: "Docker daemon is not responding" Solutions: 1. Manually launch Docker Desktop application 2. Restart Docker Desktop 3. Check system resources (memory, disk space) 4. Restart computer if needed
SQL Server Connection Failed¶
Symptoms: "SQL Server connection failed"
Solutions:
1. Check container status: pyforge mdf-tools status
2. View container logs: pyforge mdf-tools logs
3. Restart container: pyforge mdf-tools restart
4. Verify password in config: pyforge mdf-tools config
Port Already in Use¶
Symptoms: "Port 1433 is already allocated"
Solutions:
1. Stop other SQL Server instances
2. Use custom port: pyforge install mdf-tools --port 1434
3. Check for conflicting containers: docker ps
Insufficient Memory¶
Symptoms: Container exits with memory errors Solutions: 1. Increase Docker memory allocation (4GB minimum) 2. Close other applications to free memory 3. Check available system resources
Debug Commands¶
# Check Docker status
docker info
# List all containers
docker ps -a
# Check container logs
docker logs pyforge-sql-server
# Inspect container configuration
docker inspect pyforge-sql-server
# Check Docker volumes
docker volume ls
# Test SQL Server directly
docker exec pyforge-sql-server /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P "PyForge@2024!" -Q "SELECT @@VERSION" -C
Getting Help¶
If you encounter issues not covered in this guide:
- Check Status: Run
pyforge mdf-tools status
for diagnostic information - View Logs: Use
pyforge mdf-tools logs
to see SQL Server startup messages - Restart Services: Try
pyforge mdf-tools restart
to resolve temporary issues - Reinstall: Use
pyforge mdf-tools uninstall
followed bypyforge install mdf-tools
Next Steps¶
After successful installation:
- Verify Installation: Run
pyforge mdf-tools status
to confirm all components are operational - Test Connectivity: Use
pyforge mdf-tools test
to verify SQL Server is responding - Install MDF Converter: Install the MDF to Parquet converter (when available)
- Process MDF Files: Use PyForge to convert your MDF files to modern formats
Related Documentation¶
- MDF to Parquet Converter (coming soon)
- Database Files Overview
- CLI Reference
- Troubleshooting Guide