doc/MANUAL_ALEGRO_EXPORT_INSTRUCTIONS.md

# Manual Alegro HVDC Outage Export Instructions

## Why Manual Export is Required

After extensive testing, the ENTSO-E Transparency Platform API **does not support programmatic access** to DC Link (HVDC) transmission outages:

1. **API Tested**: All combinations of border queries, asset-specific queries, and domain codes return 400/403 errors
2. **Scripts Created**:
   - `collect_alegro_outages.py` - Border query (400 Bad Request)
   - `collect_alegro_asset_outages.py` - Asset EIC query (400 Bad Request)
   - `download_alegro_outages_direct.py` - Direct export URL (403 Forbidden)
3. **Conclusion**: HVDC outages only accessible via web UI manual export

## Critical Importance

**Alegro outages are ESSENTIAL features**, not optional:
- Shadow prices up to €1,750/MW prove economic significance
- 93-98% availability means outages DO occur and impact flows
- Forward-looking planned outages are needed as future covariates for forecasting
- 8 Alegro CNECs in master list require outage data

## Step-by-Step Export Instructions

### Step 1: Navigate to ENTSO-E Transparency Platform

URL: https://transparency.entsoe.eu/outage-domain/r2/unavailabilityInTransmissionGrid/show

### Step 2: Set Filters

Apply the following filters in the web interface:

| Filter | Value |
|--------|-------|
| **Border** | CTA\|BE - CTA\|DE(Amprion) |
| **Asset Type** | DC Link |
| **Date From** | 01.10.2023 |
| **Date To** | 30.09.2025 |

**Important**:
- Asset Type MUST be "DC Link" - this is the HVDC filter
- Do NOT select "AC Link" or leave blank
- Border should specifically mention "Amprion" (Germany TSO operating Alegro)

### Step 3: Click Search/Apply

The table should populate with Alegro HVDC outage events.

**Expected Data**:
- Outages for the Alegro cable (1,000 MW HVDC Belgium-Germany)
- Mix of planned (A53) and forced (A54) outages
- Start and end timestamps
- Available/unavailable capacity

### Step 4: Export Data

Click the "Export" or "Download" button (usually top-right of results table).

**Export Format**: Choose CSV or Excel (both supported).

**Save As**: `alegro_manual_export.csv` (or `.xlsx`)

**Location**: Place in `data/raw/` directory

### Step 5: Convert to Standard Format

Run the conversion script:

```bash
python scripts/convert_alegro_manual_export.py data/raw/alegro_manual_export.csv
```

This will:
1. Auto-detect column names from ENTSO-E export
2. Map to standardized schema:
   - `asset_eic`: Transmission asset EIC code
   - `asset_name`: Alegro cable name
   - `start_time`: Outage start (UTC datetime)
   - `end_time`: Outage end (UTC datetime)
   - `businesstype`: A53 (planned) or A54 (forced)
   - `from_zone`: BE
   - `to_zone`: DE
   - `border`: BE_DE
3. Filter to future outages only (forward-looking for forecasting)
4. Save two outputs:
   - `alegro_hvdc_outages_24month.parquet` - All outages
   - `alegro_hvdc_outages_24month_future.parquet` - Future only

### Step 6: Verify Data

Check the converted data:

```bash
python -c "import polars as pl; df = pl.read_parquet('data/raw/alegro_hvdc_outages_24month.parquet'); print(f'Total outages: {len(df)}'); print(df.head())"
```

Expected output:
- At least 10-50 outages over 24-month period (based on 93-98% availability)
- Mix of planned and forced outages
- Timestamps in UTC
- Valid EIC codes

## Troubleshooting

**If no data appears after applying filters**:
1. Check "Asset Type" is set to "DC Link" (not "AC Link")
2. Try expanding date range
3. Try removing border filter (select "All Borders"), then manually filter results for Alegro
4. Check if login is required (some ENTSO-E data requires authentication)

**If export fails**:
1. Try different export format (CSV vs Excel)
2. Try smaller date ranges (e.g., 6-month chunks)
3. Check browser console for errors

**If conversion script fails**:
1. Check column names in exported file
2. Manually edit `column_mapping` in `convert_alegro_manual_export.py`
3. Ensure timestamps are in recognizable format

## Integration with Feature Pipeline

Once converted, the Alegro outages will be automatically integrated:

1. **Master CNEC List**: Already includes 8 Alegro CNECs with custom EIC codes
2. **Outage Feature Processing**: `process_entsoe_outage_features_master.py` will process Alegro outages
3. **Feature Output**: 8 Alegro CNECs × 4 features = 32 outage features:
   - `cnec_{EIC}_outage_binary`: Current outage indicator
   - `cnec_{EIC}_outage_planned_7d`: Planned outage next 7 days (FUTURE COVARIATE)
   - `cnec_{EIC}_outage_planned_14d`: Planned outage next 14 days (FUTURE COVARIATE)
   - `cnec_{EIC}_outage_capacity_mw`: MW offline

The planned outage indicators are **forward-looking** and serve as future covariates for forecasting.

## Expected Timeline

- Manual export: 5-10 minutes
- Conversion: <1 minute
- Integration: Automatic (already coded)

## Questions?

If you encounter issues, check:
1. ENTSO-E platform status: https://transparency.entsoe.eu
2. Alegro operator websites:
   - Elia (Belgium): https://www.elia.be
   - Amprion (Germany): https://www.amprion.net
3. ENTSO-E user guide for transmission outages

---

**Status**: Ready for manual export
**Created**: 2025-11-09
**Last Updated**: 2025-11-09