cwicr-data-loader

# CWICR Data Loader ## Business Case ### Problem Statement DDC CWICR database is distributed in multiple formats: - Apache Parquet (optimized for analytics) - Excel workbooks (human-readable) - CSV files (universal exchange) - Qdrant snapshots (vector search) Applications need unified data access regardless of source format. ### Solution Universal data loader supporting all CWICR formats with automatic schema detection, validation, and pandas DataFrame conversion. ### Business Value - **Format agnostic** - Load from any CWICR distribution - **Validated data** - Automatic schema validation - **Memory efficient** - Lazy loading for large datasets - **Type-safe** - Proper data types preserved ## Technical Implementation ### Prerequisites ```bash pip install pandas pyarrow openpyxl qdrant-client ``` ### Python Implementation ```python import pandas as pd import pyarrow.parquet as pq from pathlib import Path from typing import Optional, Dict, Any, List, Union from dataclasses import dataclass, field from enum import Enum import json class CWICRFormat(Enum): """Supported CWICR data formats.""" PARQUET = "parquet" EXCEL = "excel" CSV = "csv" QDRANT = "qdrant" JSON = "json" class CWICRLanguage(Enum): """Supported languages in CWICR database.""" ARABIC = "ar" CHINESE = "zh" GERMAN = "de" ENGLISH = "en" SPANISH = "es" FRENCH = "fr" HINDI = "hi" PORTUGUESE = "pt" RUSSIAN = "ru" @dataclass class CWICRSchema: ...