Skip to content

Implement Regular Expressions data extraction #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Shocker wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Implement Regular Expressions data extraction #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

# HTTP Agent for Home Assistant

HTTP Agent is a powerful Home Assistant custom integration that creates individual HTTP sensor instances with advanced data extraction capabilities supporting both JSON, XML and even HTML/CSS-Selectors. Unlike traditional hub-based integrations, each HTTP endpoint becomes its own integration instance and then you can define multiple sensors for that request.
HTTP Agent is a powerful Home Assistant custom integration that creates individual HTTP sensor instances with advanced data extraction capabilities supporting JSON, XML, HTML/CSS-Selectors and Regular Expressions. Unlike traditional hub-based integrations, each HTTP endpoint becomes its own integration instance and then you can define multiple sensors for that request.

## Installation

Expand Down Expand Up @@ -55,7 +55,7 @@ Configure request body with content type selection. The field supports templates
### Step 4: Sensors
Define sensors to extract data from the response:
- **Name**: Sensor name
- **Extraction Method**: JSON, XML or CSS selectors.
- **Extraction Method**: JSON, XML, CSS selectors or Regular Expression.
- **State**: Main sensor value selector/template
- **Icon**: Icon selector/template (auto-prefixed with `mdi:`)
- **Color**: Color selector/template
Expand Down Expand Up @@ -86,6 +86,14 @@ Use CSS selectors for HTML content:
div[data-sensor="temp"] # Attribute selector
```

### Regular Expressions
Use regular expressions for text content, use PCRE-format delimiters `/pattern/flags`:
```
/Some Text: ([0-9]+)/ # single group
/some text: ([0-9]+)/i # case insensitive
/Some Text: ([0-9]+) other text ([a-z]+) # multiple groups, result will be merged "group 1|group 2"
```

## Template Support

All configuration fields support Home Assistant templates:
Expand Down
45 changes: 45 additions & 0 deletions custom_components/http_agent/coordinator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

import aiohttp
from bs4 import BeautifulSoup
import re

from homeassistant.core import HomeAssistant
from homeassistant.helpers.template import Template
Expand Down Expand Up @@ -243,6 +244,9 @@ def _extract_value_auto(self, response: HTTPResponse, selector: str) -> Any:
if "html" in content_type or response.soup is not None:
methods_to_try.append(("css", self._extract_css_value, response.soup))

if response.text is not None:
methods_to_try.append(("regex", self._extract_regex_value, response.text))

# If no content type match, try all available methods
if not methods_to_try:
if response.json is not None:
Expand Down Expand Up @@ -333,6 +337,47 @@ def _extract_css_value(self, soup: BeautifulSoup, selector: str) -> Any:

return None

def _extract_regex_value(self, text: str, selector: str) -> Any:
"""Extract value from text using regular expressions."""
if not (selector.startswith("/") or selector.count("/") < 2):
return None

last_slash = selector.rfind("/")
pattern = selector[1:last_slash]
flags_str = selector[last_slash + 1:]

flag_map = {
"i": re.I,
"m": re.M,
"s": re.S,
"x": re.X,
"a": re.A,
"l": re.L,
}

re_flags = 0
for f in flags_str:
if f.lower() in flag_map:
re_flags |= flag_map[f.lower()]
else:
_LOGGER.debug(
"Unhandled regex flag '%s'",
f,
)
return None

try:
match = re.search(pattern, text, re_flags)
if match:
if match.lastindex == 1:
return match.group(1)
elif match.lastindex > 1:
return "|".join(match.group(i) for i in range(1, match.lastindex + 1))
except Exception:
pass

return None

async def async_close(self) -> None:
"""Close the HTTP session."""
if self.session:
Expand Down
42 changes: 21 additions & 21 deletions custom_components/http_agent/strings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,16 @@
},
"sensor_config": {
"title": "Sensor Configuration",
"description": "Configure the sensor settings. The extraction method (JSON/XPath/CSS) will be automatically detected based on the response content type.",
"description": "Configure the sensor settings. The extraction method (JSON/XPath/CSS/RegEx) will be automatically detected based on the response content type.",
"data": {
"sensor_state": "State Selector (JSON/XPath/CSS - auto-detected)",
"sensor_icon": "Icon Selector (JSON/XPath/CSS - auto-detected)",
"sensor_color": "Color Selector (JSON/XPath/CSS - auto-detected)",
"sensor_state": "State Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_icon": "Icon Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_color": "Color Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_device_class": "Device Class",
"sensor_unit": "Unit of Measurement",
"tracker_latitude": "Latitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON/XPath/CSS - auto-detected)",
"tracker_latitude": "Latitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_source_type": "Source Type"
}
}
Expand Down Expand Up @@ -159,17 +159,17 @@
},
"modify_sensor": {
"title": "Modify Sensor",
"description": "Edit sensor configuration. The extraction method (JSON/XPath/CSS) will be automatically detected based on the response content type.",
"description": "Edit sensor configuration. The extraction method (JSON/XPath/CSS/RegEx) will be automatically detected based on the response content type.",
"data": {
"sensor_name": "Sensor Name",
"sensor_state": "State Selector (JSON/XPath/CSS - auto-detected)",
"sensor_icon": "Icon Selector (JSON/XPath/CSS - auto-detected)",
"sensor_color": "Color Selector (JSON/XPath/CSS - auto-detected)",
"sensor_state": "State Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_icon": "Icon Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_color": "Color Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_device_class": "Device Class",
"sensor_unit": "Unit of Measurement",
"tracker_latitude": "Latitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON/XPath/CSS - auto-detected)",
"tracker_latitude": "Latitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_source_type": "Source Type"
}
},
Expand All @@ -183,16 +183,16 @@
},
"sensor_config": {
"title": "Sensor Configuration",
"description": "Configure the sensor settings. The extraction method (JSON/XPath/CSS) will be automatically detected based on the response content type.",
"description": "Configure the sensor settings. The extraction method (JSON/XPath/CSS/RegEx) will be automatically detected based on the response content type.",
"data": {
"sensor_state": "State Selector (JSON/XPath/CSS - auto-detected)",
"sensor_icon": "Icon Selector (JSON/XPath/CSS - auto-detected)",
"sensor_color": "Color Selector (JSON/XPath/CSS - auto-detected)",
"sensor_state": "State Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_icon": "Icon Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_color": "Color Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"sensor_device_class": "Device Class",
"sensor_unit": "Unit of Measurement",
"tracker_latitude": "Latitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON/XPath/CSS - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON/XPath/CSS - auto-detected)",
"tracker_latitude": "Latitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_longitude": "Longitude Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_location_name": "Location Name Selector (JSON\/XPath\/CSS\/RegEx - auto-detected)",
"tracker_source_type": "Source Type"
}
}
Expand Down
42 changes: 21 additions & 21 deletions custom_components/http_agent/translations/da.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,16 @@
},
"sensor_config": {
"title": "Sensor-konfiguration",
"description": "Konfigurer sensorindstillingerne. Udvindingsmetoden (JSON/XPath/CSS) vil automatisk blive registreret baseret på svarets indholdstype.",
"description": "Konfigurer sensorindstillingerne. Udvindingsmetoden (JSON/XPath/CSS/RegEx) vil automatisk blive registreret baseret på svarets indholdstype.",
"data": {
"sensor_state": "Tilstandsvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_color": "Farvevælger (JSON/XPath/CSS - auto-registreret)",
"sensor_state": "Tilstandsvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_color": "Farvevælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_device_class": "Enhedsklasse",
"sensor_unit": "Måleenhed",
"tracker_latitude": "Breddegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_latitude": "Breddegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_source_type": "Kildetype"
}
}
Expand Down Expand Up @@ -159,17 +159,17 @@
},
"modify_sensor": {
"title": "Modificer sensor",
"description": "Rediger sensor-konfiguration. Udvindingsmetoden (JSON/XPath/CSS) vil automatisk blive registreret baseret på svarets indholdstype.",
"description": "Rediger sensor-konfiguration. Udvindingsmetoden (JSON/XPath/CSS/RegEx) vil automatisk blive registreret baseret på svarets indholdstype.",
"data": {
"sensor_name": "Sensornavn",
"sensor_state": "Tilstandsvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_color": "Farvevælger (JSON/XPath/CSS - auto-registreret)",
"sensor_state": "Tilstandsvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_color": "Farvevælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_device_class": "Enhedsklasse",
"sensor_unit": "Måleenhed",
"tracker_latitude": "Breddegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_latitude": "Breddegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_source_type": "Kildetype"
}
},
Expand All @@ -183,16 +183,16 @@
},
"sensor_config": {
"title": "Sensor-konfiguration",
"description": "Konfigurer sensorindstillingerne. Udvindingsmetoden (JSON/XPath/CSS) vil automatisk blive registreret baseret på svarets indholdstype.",
"description": "Konfigurer sensorindstillingerne. Udvindingsmetoden (JSON/XPath/CSS/RegEx) vil automatisk blive registreret baseret på svarets indholdstype.",
"data": {
"sensor_state": "Tilstandsvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON/XPath/CSS - auto-registreret)",
"sensor_color": "Farvevælger (JSON/XPath/CSS - auto-registreret)",
"sensor_state": "Tilstandsvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_icon": "Ikonvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_color": "Farvevælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"sensor_device_class": "Enhedsklasse",
"sensor_unit": "Måleenhed",
"tracker_latitude": "Breddegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON/XPath/CSS - auto-registreret)",
"tracker_latitude": "Breddegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_longitude": "Længdegradvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_location_name": "Lokationsnavnvælger (JSON\/XPath\/CSS\/RegEx - auto-registreret)",
"tracker_source_type": "Kildetype"
}
}
Expand Down
42 changes: 21 additions & 21 deletions custom_components/http_agent/translations/de.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,16 @@
},
"sensor_config": {
"title": "Sensor-Konfiguration",
"description": "Sensor-Einstellungen konfigurieren. Die Extraktionsmethode (JSON/XPath/CSS) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"description": "Sensor-Einstellungen konfigurieren. Die Extraktionsmethode (JSON/XPath/CSS/RegEx) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"data": {
"sensor_state": "Zustandswähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_state": "Zustandswähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_device_class": "Geräteklasse",
"sensor_unit": "Maßeinheit",
"tracker_latitude": "Breitengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_latitude": "Breitengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_source_type": "Quellentyp"
}
}
Expand Down Expand Up @@ -159,17 +159,17 @@
},
"modify_sensor": {
"title": "Sensor ändern",
"description": "Sensor-Konfiguration bearbeiten. Die Extraktionsmethode (JSON/XPath/CSS) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"description": "Sensor-Konfiguration bearbeiten. Die Extraktionsmethode (JSON/XPath/CSS/RegEx) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"data": {
"sensor_name": "Sensor-Name",
"sensor_state": "Zustandswähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_state": "Zustandswähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_device_class": "Geräteklasse",
"sensor_unit": "Maßeinheit",
"tracker_latitude": "Breitengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_latitude": "Breitengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_source_type": "Quellentyp"
}
},
Expand All @@ -183,16 +183,16 @@
},
"sensor_config": {
"title": "Sensor-Konfiguration",
"description": "Sensor-Einstellungen konfigurieren. Die Extraktionsmethode (JSON/XPath/CSS) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"description": "Sensor-Einstellungen konfigurieren. Die Extraktionsmethode (JSON/XPath/CSS/RegEx) wird automatisch basierend auf dem Antwort-Inhaltstyp erkannt.",
"data": {
"sensor_state": "Zustandswähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"sensor_state": "Zustandswähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_icon": "Symbol-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_color": "Farb-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"sensor_device_class": "Geräteklasse",
"sensor_unit": "Maßeinheit",
"tracker_latitude": "Breitengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON/XPath/CSS - automatisch erkannt)",
"tracker_latitude": "Breitengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_longitude": "Längengrad-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_location_name": "Standortname-Wähler (JSON\/XPath\/CSS\/RegEx - automatisch erkannt)",
"tracker_source_type": "Quellentyp"
}
}
Expand Down
Loading