Skip to content

MDF Tools Installer

Overview

The MDF Tools Installer provides an interactive setup wizard that automates the installation and configuration of prerequisites needed for processing SQL Server MDF (Master Database Files). This includes Docker Desktop installation, SQL Server Express container setup, and comprehensive container management tools.

Features

  • ๐Ÿ‹ Docker Desktop Integration: Automatic detection and installation across platforms
  • ๐Ÿ—„๏ธ SQL Server Express Setup: Containerized SQL Server 2019 Express with persistent storage
  • โš™๏ธ Interactive Configuration: Guided setup with customizable passwords and ports
  • ๐Ÿ“Š Real-time Status Monitoring: Comprehensive health checks for all components
  • ๐Ÿ”ง Container Lifecycle Management: Complete control over SQL Server container
  • ๐Ÿ”’ Secure Configuration: Encrypted password storage and configurable security settings

Quick Start

# Install MDF processing tools
pyforge install mdf-tools

# Check installation status
pyforge mdf-tools status

# Test SQL Server connectivity
pyforge mdf-tools test

System Requirements

Minimum Requirements

  • Operating System: Windows 10+, macOS 10.15+, or Ubuntu 18.04+
  • Memory: 4GB RAM total (1.4GB for SQL Server + 2.6GB for host system)
  • Storage: 4GB free space (2GB for Docker images + 2GB for SQL Server data)
  • Network: Internet connection for downloading Docker images (~700MB)
  • Docker: Docker Desktop 4.0+ with container support
  • Memory: 8GB RAM (for optimal performance with multiple databases)
  • Storage: 20GB free space (for multiple MDF files and conversions)
  • CPU: 4+ cores (though SQL Server Express limited to 4 cores max)
  • Network: Broadband connection for faster image downloads

SQL Server Express Constraints

  • Maximum Database Size: 10GB per attached MDF file
  • Memory Limit: 1.4GB buffer pool (cannot be increased)
  • CPU Utilization: 1 socket or 4 cores maximum
  • Concurrent Connections: Practical limit of 5-10 users
  • Query Parallelism: Disabled (DOP = 1)

Supported Platforms

  • โœ… macOS (Intel and Apple Silicon)
  • โœ… Windows (Windows 10/11 with WSL2)
  • โœ… Linux (Ubuntu, CentOS, RHEL, Debian)

Architecture Overview

MDF Tools Installation Architecture

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                           HOST SYSTEM                               โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚  ๐Ÿ‘ค User                                                           โ”‚
โ”‚  โ”‚                                                                  โ”‚
โ”‚  โ””โ”€โ–บ ๐Ÿ”ง PyForge CLI                                                โ”‚
โ”‚      โ”‚                                                              โ”‚
โ”‚      โ”œโ”€โ–บ ๐Ÿ“„ ~/.pyforge/mdf-config.json                            โ”‚
โ”‚      โ”‚                                                              โ”‚
โ”‚      โ””โ”€โ–บ ๐Ÿณ Docker Desktop                                         โ”‚
โ”‚          โ”‚                                                          โ”‚
โ”‚          โ””โ”€โ–บ ๐Ÿ“ฆ SQL Server Container (pyforge-sql-server)          โ”‚
โ”‚              โ”œโ”€โ–บ ๐Ÿ—„๏ธ  SQL Server Express 2019                      โ”‚
โ”‚              โ”œโ”€โ–บ ๐Ÿ”ง sqlcmd Tools                                    โ”‚
โ”‚              โ”œโ”€โ–บ ๐Ÿ’พ master database                                โ”‚
โ”‚              โ”œโ”€โ–บ ๐Ÿ’พ Attached MDF Database                          โ”‚
โ”‚              โ”‚                                                      โ”‚
โ”‚              โ””โ”€โ–บ ๐Ÿ“ Docker Volumes                                 โ”‚
โ”‚                  โ”œโ”€โ–บ pyforge-sql-data (/var/opt/mssql)            โ”‚
โ”‚                  โ””โ”€โ–บ pyforge-mdf-files (/mdf-files)               โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Installation Flow

Step-by-Step Installation Process:

1. User Command
   ๐Ÿ‘ค User โ†’ pyforge install mdf-tools

2. System Check
   ๐Ÿ”ง PyForge CLI โ†’ Check OS compatibility
   ๐Ÿ”ง PyForge CLI โ†’ Detect Docker installation

3. Docker Setup
   ๐Ÿ”ง PyForge CLI โ†’ Start Docker Desktop
   ๐Ÿ”ง PyForge CLI โ†’ Pull SQL Server image (700MB)

4. Container Creation
   ๐Ÿ”ง PyForge CLI โ†’ Create pyforge-sql-server container
   ๐Ÿ”ง PyForge CLI โ†’ Configure port mapping (1433)
   ๐Ÿ”ง PyForge CLI โ†’ Mount persistent volumes

5. SQL Server Configuration
   ๐Ÿ”ง PyForge CLI โ†’ Start SQL Server Express
   ๐Ÿ”ง PyForge CLI โ†’ Test connectivity with sqlcmd
   ๐Ÿ”ง PyForge CLI โ†’ Verify database engine

6. Finalization
   ๐Ÿ”ง PyForge CLI โ†’ Save configuration file
   ๐Ÿ”ง PyForge CLI โ†’ Display connection details
   โœ… Installation Complete!

Installation Workflow Components

1. Host System Components: - PyForge CLI: Main application orchestrating the installation - Configuration File: Persistent settings stored locally - Docker Desktop: Container runtime environment

2. Container Infrastructure: - SQL Server Express 2019: Database engine for MDF processing - Persistent Volumes: Data survival across container restarts - Network Mapping: Port 1433 exposed to host system

3. Data Flow: - Installation: CLI โ†’ Docker โ†’ SQL Server โ†’ Configuration - MDF Processing: MDF File โ†’ Volume Mount โ†’ SQL Server โ†’ Parquet Output - Management: CLI Commands โ†’ Docker API โ†’ Container Lifecycle

System Integration Points

Component Purpose Technology Persistence
Docker Desktop Container orchestration Docker Engine System service
SQL Server Container Database engine SQL Server Express 2019 Container lifecycle
Data Volume SQL Server system data Docker volume Persistent across restarts
MDF Volume User MDF files Docker volume Persistent across restarts
Configuration Connection settings JSON file Local filesystem
Network Bridge Host-container communication Docker bridge Dynamic port mapping

MDF Processing Workflow

How MDF Files Are Processed (Future Feature):

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                     MDF TO PARQUET CONVERSION                       โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Step 1: File Preparation
   ๐Ÿ‘ค User โ†’ pyforge convert database.mdf
   ๐Ÿ“ MDF File โ†’ Copy to /mdf-files volume

Step 2: Database Attachment  
   ๐Ÿ”ง PyForge CLI โ†’ ATTACH DATABASE 'database.mdf'
   ๐Ÿ—„๏ธ  SQL Server โ†’ Validate MDF structure
   โœ… SQL Server โ†’ Database attached successfully

Step 3: Schema Discovery
   ๐Ÿ”ง PyForge CLI โ†’ Query table metadata
   ๐Ÿ—„๏ธ  SQL Server โ†’ Return table schemas & row counts
   ๐Ÿ“Š PyForge CLI โ†’ Display table overview

Step 4: Data Extraction
   ๐Ÿ”ง PyForge CLI โ†’ Execute SELECT queries (chunked)
   ๐Ÿ—„๏ธ  SQL Server โ†’ Return table data in batches
   ๐Ÿ“ฆ PyForge CLI โ†’ Convert to string format

Step 5: Parquet Generation
   ๐Ÿ“ฆ PyForge CLI โ†’ Generate .parquet files
   ๐Ÿ“Š PyForge CLI โ†’ Create Excel summary report

Step 6: Cleanup
   ๐Ÿ”ง PyForge CLI โ†’ DETACH DATABASE
   ๐Ÿ“ MDF File โ†’ Remains in volume (unchanged)
   โœ… Conversion Complete!

Output Structure:
   database_parquet/
   โ”œโ”€โ”€ Users.parquet
   โ”œโ”€โ”€ Orders.parquet  
   โ”œโ”€โ”€ Products.parquet
   โ””โ”€โ”€ conversion_summary.xlsx

Supported MDF File Types

SQL Server Version MDF Compatibility Processing Status
SQL Server 2019 โœ… Native Optimal performance
SQL Server 2017 โœ… Compatible Full support
SQL Server 2016 โœ… Compatible Full support
SQL Server 2014 โœ… Compatible Full support
SQL Server 2012 โœ… Compatible Full support
SQL Server 2008/R2 โš ๏ธ Limited May require upgrade
SQL Server 2005 โŒ Incompatible Not supported

Note: MDF files from newer SQL Server versions (2022+) may not be compatible with SQL Server Express 2019.

Installation Process

The installer follows a structured 5-stage process:

Stage 1: System Requirements Check

  • Validates operating system compatibility
  • Checks Docker Desktop installation status
  • Verifies Docker SDK for Python availability

Stage 2: Docker Installation (if needed)

  • macOS: Automatic installation via Homebrew
  • Windows: Automatic installation via Winget
  • Linux: Package manager instructions (apt/yum)
  • Manual: Step-by-step installation guides

Stage 3: Docker Startup

  • Connects to Docker daemon
  • Waits for Docker Desktop to be fully operational
  • Validates Docker API accessibility

Stage 4: SQL Server Express Setup

  • Downloads Microsoft SQL Server 2019 Express image
  • Creates and configures container with:
  • Persistent data volume (pyforge-sql-data)
  • MDF files mount point (pyforge-mdf-files)
  • Default port mapping (1433)
  • Secure password configuration

Stage 5: Configuration and Validation

  • Saves configuration to ~/.pyforge/mdf-config.json
  • Tests SQL Server connectivity using sqlcmd
  • Displays connection details and next steps

macOS Installation Walkthrough

Scenario 1: Docker Already Installed

$ pyforge install mdf-tools
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ PyForge MDF Tools Setup Wizard                                           โ”‚
โ”‚ Setting up Docker Desktop and SQL Server Express for MDF file processing โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

[1/5] Checking system requirements...
โœ“ Operating System: macOS 14.5.0 (supported)
โœ“ Docker Desktop: Installed
โœ“ Docker SDK for Python: Available

[3/5] Starting Docker Desktop...
โœ“ Docker Desktop is running

[4/5] Setting up SQL Server Express...
๐Ÿ“ฅ Pulling SQL Server image: mcr.microsoft.com/mssql/server:2019-latest
โ ด โœ“ SQL Server image downloaded
๐Ÿš€ Creating SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โœ“ SQL Server is ready

[5/5] Installation Complete!
              SQL Server Connection Details              
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Property    โ”ƒ Value                                   โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ Host        โ”‚ localhost                               โ”‚
โ”‚ Port        โ”‚ 1433                                    โ”‚
โ”‚ Username    โ”‚ sa                                      โ”‚
โ”‚ Password    โ”‚ PyForge@2024!                           โ”‚
โ”‚ Container   โ”‚ pyforge-sql-server                      โ”‚
โ”‚ Config File โ”‚ /Users/username/.pyforge/mdf-config.jsonโ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

๐ŸŽ‰ Setup completed successfully!

Scenario 2: Docker NOT Installed

$ pyforge install mdf-tools
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ PyForge MDF Tools Setup Wizard                                           โ”‚
โ”‚ Setting up Docker Desktop and SQL Server Express for MDF file processing โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

[1/5] Checking system requirements...
โœ“ Operating System: macOS 14.5.0 (supported)
โŒ Docker Desktop: Not found

[2/5] Docker Installation Required
Docker Desktop is required for MDF file conversion.

Would you like to:
  1. Install automatically using Homebrew (recommended)
  2. Get installation instructions
  3. Skip (I'll install manually)
  4. Continue without Docker (installation will fail)

Choice [1]: 1

๐Ÿ“ฆ Installing Docker Desktop (this may take several minutes)...
โœ… Docker Desktop installed successfully!
๐Ÿš€ Launching Docker Desktop...

โณ Waiting for Docker Desktop to start...
โœ… Docker Desktop is running!

[4/5] Setting up SQL Server Express...
๐Ÿ“ฅ Pulling SQL Server image: mcr.microsoft.com/mssql/server:2019-latest
โ ด โœ“ SQL Server image downloaded
๐Ÿš€ Creating SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โœ“ SQL Server is ready

[5/5] Installation Complete!

Command Reference

Installation Commands

pyforge install mdf-tools

Interactive installation wizard for MDF processing tools.

Usage:

pyforge install mdf-tools [OPTIONS]

Options: - --password PASSWORD: Custom SQL Server password (default: PyForge@2024!) - --port PORT: Custom SQL Server port (default: 1433) - --non-interactive: Run in non-interactive mode for automation

Examples:

# Default installation
pyforge install mdf-tools

# Custom password and port
pyforge install mdf-tools --password "MySecure123!" --port 1433

# Non-interactive mode (for scripts)
pyforge install mdf-tools --non-interactive

Container Management Commands

pyforge mdf-tools status

Displays comprehensive status of all MDF tools components.

Usage:

pyforge mdf-tools status

Sample Output:

                      MDF Tools Status                       
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Component             โ”ƒ Status โ”ƒ Details                  โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ Docker Installed      โ”‚ โœ“ OK   โ”‚ Docker command available โ”‚
โ”‚ Docker Running        โ”‚ โœ“ OK   โ”‚ Docker daemon responsive โ”‚
โ”‚ SQL Container Exists  โ”‚ โœ“ OK   โ”‚ Container created        โ”‚
โ”‚ SQL Container Running โ”‚ โœ“ OK   โ”‚ Container active         โ”‚
โ”‚ SQL Server Responding โ”‚ โœ“ OK   โ”‚ Database accessible      โ”‚
โ”‚ Configuration File    โ”‚ โœ“ OK   โ”‚ Settings saved           โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

โœ… All systems operational - ready for MDF processing!

pyforge mdf-tools start

Starts the SQL Server Express container.

Usage:

pyforge mdf-tools start

Sample Output:

๐Ÿš€ Starting SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โœ“ SQL Server is ready

pyforge mdf-tools stop

Stops the SQL Server Express container.

Usage:

pyforge mdf-tools stop

Sample Output:

๐Ÿ›‘ Stopping SQL Server container...
โœ“ SQL Server container stopped

pyforge mdf-tools restart

Restarts the SQL Server Express container.

Usage:

pyforge mdf-tools restart

Sample Output:

๐Ÿ›‘ Stopping SQL Server container...
โœ“ SQL Server container stopped
๐Ÿš€ Starting SQL Server container...
โณ Waiting for SQL Server to start (this may take a minute)...
โœ“ SQL Server is ready

pyforge mdf-tools logs

Displays SQL Server container logs.

Usage:

pyforge mdf-tools logs [OPTIONS]

Options: - --lines N, -n N: Number of log lines to show (default: 50)

Examples:

# Show last 50 lines (default)
pyforge mdf-tools logs

# Show last 100 lines
pyforge mdf-tools logs --lines 100

# Show last 10 lines
pyforge mdf-tools logs -n 10

pyforge mdf-tools config

Displays current MDF tools configuration.

Usage:

pyforge mdf-tools config

Sample Output:

Configuration file: /Users/username/.pyforge/mdf-config.json
{
  "sql_server": {
    "container_name": "pyforge-sql-server",
    "image": "mcr.microsoft.com/mssql/server:2019-latest",
    "host": "localhost",
    "port": 1433,
    "username": "sa",
    "password": "PyForge@2024!",
    "data_volume": "pyforge-sql-data",
    "mdf_volume": "pyforge-mdf-files"
  },
  "docker": {
    "installed_version": "Docker version 20.10.17",
    "installation_date": "2024-01-15T10:30:00Z"
  },
  "installer_version": "1.0.0"
}

pyforge mdf-tools test

Tests SQL Server connectivity and responsiveness.

Usage:

pyforge mdf-tools test

Sample Output:

๐Ÿ” Testing SQL Server connection...
โœ… SQL Server connection successful!

pyforge mdf-tools uninstall

Removes SQL Server container and cleans up all data.

Usage:

pyforge mdf-tools uninstall

Sample Output:

Are you sure you want to remove SQL Server and all data? [y/n]: y
๐Ÿ›‘ Stopping and removing container...
โœ“ Container removed
โœ“ Data volume removed
โœ“ MDF files volume removed
โœ“ Configuration file removed
โœ… Uninstall completed successfully

Configuration

Configuration File Location

The installer saves configuration to ~/.pyforge/mdf-config.json:

{
  "sql_server": {
    "container_name": "pyforge-sql-server",
    "image": "mcr.microsoft.com/mssql/server:2019-latest",
    "host": "localhost",
    "port": 1433,
    "username": "sa",
    "password": "PyForge@2024!",
    "data_volume": "pyforge-sql-data",
    "mdf_volume": "pyforge-mdf-files"
  },
  "docker": {
    "installed_version": "Docker version 20.10.17",
    "installation_date": "2024-01-15T10:30:00Z"
  },
  "installer_version": "1.0.0"
}

Customizable Settings

Custom Password

pyforge install mdf-tools --password "YourSecurePassword123!"

Custom Port

pyforge install mdf-tools --port 1434

Docker Volumes

The installer creates two persistent Docker volumes:

pyforge-sql-data

  • Mount Point: /var/opt/mssql
  • Purpose: SQL Server system databases and data files
  • Persistence: Survives container restarts and recreations

pyforge-mdf-files

  • Mount Point: /mdf-files
  • Purpose: MDF files to be processed
  • Access: Shared between host and container

SQL Server Express 2019 Technical Details

Database Engine Specifications

The MDF Tools Installer uses Microsoft SQL Server Express 2019, the free edition of SQL Server's enterprise database engine. This provides a robust, production-grade database environment for MDF file processing.

SQL Server Express 2019 Key Features: - Core Engine: Same database engine as Enterprise edition - T-SQL Support: Full Transact-SQL language support - Security: Enterprise-grade security features - Reliability: ACID compliance and transaction support - Performance: Query optimizer and execution engine - Backup/Restore: Full backup and restore capabilities

Edition Limitations and Constraints

โš ๏ธ Important Limitations to Consider:

Limitation SQL Server Express 2019 Impact on MDF Processing
Database Size 10 GB per database maximum Large MDF files (>10GB) cannot be processed
Memory (RAM) 1.4 GB buffer pool limit Performance may be limited with large datasets
CPU Cores 1 socket or 4 cores maximum Processing may be slower on high-core systems
Concurrent Users No enforced limit (practical ~5-10) Multiple simultaneous conversions may impact performance
Parallelism Degree of Parallelism (DOP) = 1 Queries cannot use parallel execution

โŒ Features Not Available: - SQL Server Agent (automated jobs) - Advanced Services (Analysis Services, Reporting Services) - Advanced security features (Always Encrypted, Row-Level Security) - Advanced performance features (In-Memory OLTP, Columnstore) - Enterprise backup compression - Database mirroring and log shipping

โœ… Features Available for MDF Processing: - Full T-SQL query support - ATTACH DATABASE functionality - All standard data types - Backup and restore operations - Database schemas and relationships - Indexes and constraints

Performance Characteristics

Optimal MDF File Sizes: - Small MDF files: < 1 GB (Excellent performance) - Medium MDF files: 1-5 GB (Good performance) - Large MDF files: 5-10 GB (Acceptable performance, may require chunking) - Very Large MDF files: > 10 GB (โŒ Cannot be processed - requires SQL Server Standard/Enterprise)

Memory Usage Patterns: - Container Base: ~500 MB (SQL Server Express) - Available for Data: ~900 MB (after system overhead) - Recommended Host RAM: 4 GB minimum (for container + host OS)

Database Connection Details

Connection Parameters

  • Server: localhost
  • Port: 1433 (default) or custom port
  • Database: master (default system database)
  • Authentication: SQL Server Authentication
  • Username: sa (system administrator)
  • Password: PyForge@2024! (default) or custom password
  • Edition: SQL Server Express 2019
  • Version: Microsoft SQL Server 2019 (RTM) - 15.0.4430.1

Connection String Examples

Python (pyodbc)

import pyodbc

connection_string = (
    "DRIVER={ODBC Driver 17 for SQL Server};"
    "SERVER=localhost,1433;"
    "DATABASE=master;"
    "UID=sa;"
    "PWD=PyForge@2024!"
)

conn = pyodbc.connect(connection_string)

Command Line (sqlcmd)

# From host (requires SQL Server tools)
sqlcmd -S localhost,1433 -U sa -P "PyForge@2024!" -Q "SELECT 1"

# From container
docker exec pyforge-sql-server /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P "PyForge@2024!" -Q "SELECT 1" -C

Scaling Beyond SQL Server Express

When to Consider Upgrading

Upgrade to SQL Server Standard/Enterprise if you encounter: - MDF files larger than 10 GB - Need for high-performance parallel processing - Requirements for SQL Server Agent automation - Advanced security features (Always Encrypted, etc.) - Multiple concurrent users (>10) - Enterprise backup and restore features

Alternative Solutions

For Large MDF Files (>10 GB): 1. Split Processing: Break large tables into chunks using date ranges or ID ranges 2. SQL Server Standard: Upgrade to paid edition with higher limits 3. Cloud Solutions: Use Azure SQL Database or SQL Managed Instance 4. Alternative Tools: Consider specialized MDF extraction utilities

Migration Path Examples:

# Option 1: Cloud-based processing
# Upload MDF to Azure SQL Database
# Process using cloud resources
# Download results

# Option 2: Chunked processing (when converter supports it)
# pyforge convert large.mdf --chunk-size 1000000 --date-range "2020-2023"
# pyforge convert large.mdf --chunk-size 1000000 --date-range "2019-2020"

Cost Considerations

Edition Cost Database Size Limit Memory Limit Use Case
Express Free 10 GB 1.4 GB Development, small applications
Standard ~$1,500+ 524 PB OS limit Medium applications
Enterprise ~$5,000+ 524 PB OS limit Large enterprise applications

Security Considerations

Password Security

  • Default password meets SQL Server complexity requirements
  • Custom passwords should be strong (8+ characters, mixed case, numbers, symbols)
  • Passwords are stored in local configuration file (not transmitted)

Network Security

  • SQL Server only accessible on localhost by default
  • Container isolated in Docker bridge network
  • No external network exposure unless explicitly configured

Container Security

  • Runs SQL Server Express (free edition with limitations)
  • Container uses official Microsoft SQL Server image
  • Automatic security updates through image updates

Troubleshooting

Common Issues

Docker Desktop Not Starting

Symptoms: "Docker daemon is not responding" Solutions: 1. Manually launch Docker Desktop application 2. Restart Docker Desktop 3. Check system resources (memory, disk space) 4. Restart computer if needed

SQL Server Connection Failed

Symptoms: "SQL Server connection failed" Solutions: 1. Check container status: pyforge mdf-tools status 2. View container logs: pyforge mdf-tools logs 3. Restart container: pyforge mdf-tools restart 4. Verify password in config: pyforge mdf-tools config

Port Already in Use

Symptoms: "Port 1433 is already allocated" Solutions: 1. Stop other SQL Server instances 2. Use custom port: pyforge install mdf-tools --port 1434 3. Check for conflicting containers: docker ps

Insufficient Memory

Symptoms: Container exits with memory errors Solutions: 1. Increase Docker memory allocation (4GB minimum) 2. Close other applications to free memory 3. Check available system resources

Debug Commands

# Check Docker status
docker info

# List all containers
docker ps -a

# Check container logs
docker logs pyforge-sql-server

# Inspect container configuration
docker inspect pyforge-sql-server

# Check Docker volumes
docker volume ls

# Test SQL Server directly
docker exec pyforge-sql-server /opt/mssql-tools18/bin/sqlcmd \
  -S localhost -U sa -P "PyForge@2024!" -Q "SELECT @@VERSION" -C

Getting Help

If you encounter issues not covered in this guide:

  1. Check Status: Run pyforge mdf-tools status for diagnostic information
  2. View Logs: Use pyforge mdf-tools logs to see SQL Server startup messages
  3. Restart Services: Try pyforge mdf-tools restart to resolve temporary issues
  4. Reinstall: Use pyforge mdf-tools uninstall followed by pyforge install mdf-tools

Next Steps

After successful installation:

  1. Verify Installation: Run pyforge mdf-tools status to confirm all components are operational
  2. Test Connectivity: Use pyforge mdf-tools test to verify SQL Server is responding
  3. Install MDF Converter: Install the MDF to Parquet converter (when available)
  4. Process MDF Files: Use PyForge to convert your MDF files to modern formats