Anomalous column count

​

Definition

​

Taxonomy

• Task types: Tabular classification, tabular regression.
• Availability: monitoring only.

​

Why it matters

• Automated monitoring: Provides comprehensive data quality monitoring with minimal configuration required
• Early anomaly detection: Identifies unusual patterns across all columns simultaneously before they impact model performance
• Time series learning: Adapts to natural variations and trends in your data over time
• Comprehensive coverage: Monitors both numeric and categorical columns automatically
• Minimal setup: No need to manually configure thresholds for individual columns - the system learns appropriate bounds

​

How it works

1. Learning phase: Analyzes historical data to establish time series patterns for each column
   • Numeric columns: Tracks statistical measures (averages, medians, etc.) over time
   • Categorical columns: Monitors counts of each category over time
2. Prediction: Uses learned patterns to predict expected upper and lower bounds for each column’s current values
3. Anomaly detection: Compares current column values against predicted bounds
   • Values outside the confidence interval are flagged as anomalous
4. Counting: Counts the total number of columns exhibiting anomalous behavior
5. Threshold comparison: Compares the anomalous column count against your specified threshold

​

Configuration parameters

• interval_width: Confidence interval width (default: 0.95)
  • 0.95 = 95% confidence interval (stricter, detects more anomalies)
  • 0.99 = 99% confidence interval (more lenient, detects fewer anomalies)

​

Test configuration examples

[
  {
    "name": "No anomalous columns detected",
    "description": "Alerts when any column shows anomalous behavior based on learned patterns",
    "type": "integrity",
    "subtype": "anomalousColumnCount",
    "thresholds": [
      {
        "insightName": "anomalousColumnCount",
        "measurement": "anomalousColumnCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 86400,
    "delayWindow": 0,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  }
]

[
  {
    "name": "Anomalous columns with 99% confidence",
    "description": "Uses 99% confidence interval for more lenient anomaly detection",
    "type": "integrity",
    "subtype": "anomalousColumnCount",
    "thresholds": [
      {
        "insightName": "anomalousColumnCount",
        "insightParameters": [
          {
            "name": "interval_width",
            "value": 0.99
          }
        ],
        "measurement": "anomalousColumnCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 86400,
    "delayWindow": 0,
    "syncId": "96622fba-ea00-4e42-8f42-5e8f5f60805f"
  }
]