Commit Graph
184 Commits
Author SHA1 Message Date
Franck Nijhof 53545c984b Log lines do not end with a full stop (#37527) 2020-07-05 23:04:19 +02:00
J. Nick Koston 4b2ebf5487 Ensure removed entities are not displayed in logbook (#37395) 2020-07-03 23:08:46 -07:00
J. Nick Koston ccb77ba1e9 Handle index already existing on db migration with MySQLdb backend (#37384)
_create_index needed the same check as _add_columns since
the MySQLdb backend throws OperationalError instead
of InternalError in this case
2020-07-03 15:35:02 -07:00
Franck Nijhof 79f131066c Ensure recorder data integrity and MySQL lock error handling (#37228) 2020-06-29 16:23:11 -07:00
Franck Nijhof 7d74b74570 Fix recorder purging by batch processing purges (#37140) 2020-06-26 10:27:45 -07:00
J. Nick Koston a4501b93c4 Fix repack when using pymysql (#37142) 2020-06-26 11:45:40 -05:00
Franck Nijhof fe1a7f6d69 Upgrade sqlalchemy to 1.3.18 (#37123) 2020-06-26 08:15:54 -05:00
J. Nick Koston 255d706c24 Avoid creating a column in v8 schema that is removed in v9 schema (#37062) 2020-06-24 09:56:01 -07:00
J. Nick Koston cc8e0ef942 Handle mysql index/column already exists during migration (#37064) 2020-06-24 09:55:13 -07:00
mdegat01 6c7355785a Add support for glob matching to entity filters (#36913)
* Added GLOB capability to entityfilter and every place that uses it. All existing tests are passing

* added tests for components affected by glob change

* fixed flake8 error

* mocking the correct listener

* mocking correct bus method in azure test

* tests passing in 3.7 and 3.8

* fixed formatting issue from rebase/conflict

* Checking against glob patterns in more performant way

* perf improvments and reverted unnecessarily adjusted tests

* added new benchmark test around filters

* no longer using get with default in entityfilter

* changed filter name and removed logbook from filter benchmark

* simplified benchmark tests from feedback

* fixed apache tests and returned include exclude schemas to normal

* fixed azure event hub tests to properly go through component logic

* fixed azure test and clean up for other tests

* renaming test files to match standard

* merged mqtt statestream test changes with base

* removed dependency on recorder filter schema from history

* fixed recorder tests after merge and a bunch of lint errors
2020-06-23 20:02:29 -05:00
J. Nick Koston 91e0395c1c Optimize database indexes for existing queries (#37036)
Cleanup indexes as >50% of the db size was indexes,
many of them unused in any current query

Logbook search was having to filter event_types without
an index:

  Created ix_events_event_type_time_fired
  Dropped ix_events_event_type

States had a redundant keys on composite index:

  Dropped ix_states_entity_id
  Its unused since we have ix_states_entity_id_last_updated

De-duplicate storage of context in states as
its always stored in events and can be found
by joining the state on the event_id.

  Dropped ix_states_context_id
  Dropped ix_states_context_parent_id
  Dropped ix_states_context_user_id

After schema v9:

STATES............................................ 10186       40.9%
EVENTS............................................ 5502        22.1%
IX_STATES_ENTITY_ID_LAST_UPDATED.................. 2177         8.7%
IX_EVENTS_EVENT_TYPE_TIME_FIRED................... 1910         7.7%
IX_EVENTS_CONTEXT_ID.............................. 1592         6.4%
IX_EVENTS_TIME_FIRED.............................. 1383         5.6%
IX_STATES_LAST_UPDATED............................ 1079         4.3%
IX_STATES_EVENT_ID................................ 375          1.5%
IX_EVENTS_CONTEXT_PARENT_ID....................... 347          1.4%
IX_EVENTS_CONTEXT_USER_ID......................... 346          1.4%
IX_RECORDER_RUNS_START_END........................ 1            0.004%
RECORDER_RUNS..................................... 1            0.004%
SCHEMA_CHANGES.................................... 1            0.004%
SQLITE_MASTER..................................... 1            0.004%
2020-06-23 10:57:52 -07:00
J. Nick KostonandPaulus Schoutsen ad6315be5c Ensure recorder runs are cleaned up during purge (#36989)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2020-06-22 20:10:05 -07:00
Alex van den HoogenandPaulus Schoutsen ee816ed3dd Optimize recorder MySQL tables when repacking (#36762)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-06-22 17:28:03 -07:00
J. Nick Koston 53a91ece4e Improve isoformat timestamp performance (#36991)
* adj

* time_fired_isoformat

* remove unused code

* tests for processing timestamps

* restore missing import lost in merge conflict

* test for None case
2020-06-22 12:06:02 -05:00
J. Nick Koston edad387b12 Make recorder execute avoid native conversion by default (#36938) 2020-06-21 21:58:57 -07:00
J. Nick Koston 2b5e7c2611 Fix recorder stopping after unserializable state (#36937)
* Ensure unserializable states do not collapse recording

* augment test coverage

* fix wal mode being set every time
2020-06-19 12:03:06 -05:00
J. Nick Koston e7d982ee11 Improve db performance of state change events and reduce overall db size (#36883)
* Add old_state_id to states, remove old/new state data from events since it can now be found by a join

* remove state lookup on restart

* Ensure old_state is set for exisitng states
2020-06-17 22:26:41 -05:00
J. Nick Koston b0163b65c6 Use states to avoid decoding logbook state changed events. (#36768)
avg 4.43s -> 1.88s
2020-06-15 13:53:05 -05:00
J. Nick KostonandPaulus Schoutsen 0a6deeb49b Improve history api performance (#35822)
* Improve history api performance

A new option "minimal_response" reduces the amount of data
sent between the first and last history states to only the
"last_changed" and "state" fields.

Calling to_native is now avoided where possible and only
done at the end for rows that will be returned in the response.

When sending the `minimal_response` option, the history
api now returns a json response similar to the following
for an entity

Testing:

History API Response time for 1 day
Average of 10 runs with minimal_response

Before: 19.89s. (content length : 3427428)
After: 8.44s (content length: 592199)

```
[{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-18T23:20:03.213000+00:00",
	"last_updated": "2020-05-18T23:20:03.213000+00:00",
	"state": "on"
},
...
{
	"last_changed": "2020-05-19T00:41:08Z",
	"state": "unavailable"
},
...
{
	"attributes": {--TRUNCATED--},
	"context": {--TRUNCATED--},
	"entity_id": "binary_sensor.powerwall_status",
	"last_changed": "2020-05-19T00:42:08.069698+00:00",
	"last_updated": "2020-05-19T00:42:08.069698+00:00",
	"state": "on"
}]
```

* Remove impossible state check

* Remove another impossible state check

* Update homeassistant/components/history/__init__.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* Reorder to save some indent per review

* Make query response make sense with to_native=False

* Update test for 00:00 to Z change

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-05-26 21:53:56 -05:00
Odin Ugedal 6c4a6568f5 Fix timezone issues for db fields in recorder (#35719)
The database fields are timezoned via DateTime(timezone=True), so the
default value should be timezoned too. When using cockroachdb this is
fatal and results in the recorder crashing.
2020-05-19 13:13:27 -04:00
J. Nick Koston ebed1de581 Avoid creating multiple sqlalchemy sessions in a single history call (#35721)
* Avoid a context switch in the history api

The history api was creating a job to fetch the
states and another job to convert the states to
json. This can be done in a single job which
decreases the overhead of the operation.

* Ensure there is only one sqlalchemy session created per history
query.

Most queries created three sqlalchemy sessions which was
especially slow with sqlite since it opens and closes the
database.

In testing the UI is noticeably faster at generating history
graphs for entites.

* Add additional coverage

* pass hass first to _states_to_json and _get_significant_states
2020-05-19 07:52:38 +02:00
Josef Schlehofer 52a7a7175b Upgrade sqlalchemy to 1.3.17 (#35745) 2020-05-17 20:07:23 -05:00
J. Nick Koston 574d8d30a7 Make sqlalchemy engine connect listener recorder specific (#34908) 2020-04-30 00:08:40 -07:00
Anders Melchiorsen 6e2cf9663a Purge recorder database at night (#33646) 2020-04-16 15:11:36 -07:00
J. Nick Koston b09b5729a3 Accommodate mysql servers with a low wait_timeout (#33638)
Some providers have set their wait_timeout to 60s
in order to pack as many users as they can on a machine.
The mysql default is 28800 seconds (8 hours)

Since mysql connection build and tear down is relativity
expensive, we want to avoid being disconnected.

We now accommodate this scenario with the following:

1. Raise the mysql session wait_timeout 28800 when we connect
2. The event session now does a 30 second keep alive to
   ensure the connection stays open
2020-04-08 12:56:22 -07:00
Franck Nijhof 14f035a39b Upgrade sqlalchemy to 1.3.16 (#33810) 2020-04-08 13:12:07 +02:00
springstan 46bbe816f6 Remove None from dict.get(key, None) (#33794) 2020-04-07 21:06:05 +02:00
Franck Nijhof 187b6525b4 Removal of extraneous parenthesis (#33637) 2020-04-04 20:08:55 +02:00
Paulus Schoutsen f1d3c0d19b Remove unused manifest fields (#33595) 2020-04-03 12:58:19 -07:00
J. Nick Koston 6d311a31dd Ensure recorder event loop recovers if the database server dis… (#33253)
If the database server disconnects there were exceptions
that were not trapped which would cause the recorder event
loop to collapse.  As we never want the loop to end
we trap exceptions broadly.

Fix a bug in the new commit interval setting which caused
it to always commit after 1s
2020-03-25 13:08:20 -07:00
Franck Nijhof 11a25157c1 Upgrade sqlalchemy to 1.3.15 (#32747) 2020-03-13 00:28:26 +01:00
J. Nick Koston 3a680bf7b7 Add a commit interval setting to recorder (#32596)
* Add a commit interval setting to recorder

* Make the default every 1s instead of immediate

* See attached py-spy flamegraphs for why 1s

* This avoids disk thrashing during event storms

* Make Home Assistant significantly more responsive on busy systems

* remove debug

* Add commit forces for tests that expect commits to be immediate

* Add commit forces for tests that expect commits to be immediate

* make sure _trigger_db_commit is in the right place (all effective "wait_recording_done" calls)

* De-duplicate wait_recording_done code
2020-03-09 17:43:26 -07:00
Paulus Schoutsen 7d8da47309 Revert "Use orjson to parse json faster (#32153)" (#32185)
This reverts commit 2365e2e8cf.
2020-02-25 21:07:16 +01:00
Ruslan SayfutdinovandPaulus Schoutsen 2365e2e8cf Use orjson to parse json faster (#32153)
* [recorder] Use orjson to parse json faster

* Remove from http manifest

* Bump to orjson 2.5.1

* Empty commit to trigger CI

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-02-25 10:20:51 -08:00
Jonathan Østrup 150b376cf9 Add recorder vars db_max_retries and db_retry_wait (#31561)
* added recorder vars db_max_retries and db_retry_wait

* fixed test_recorder_setup_failure

I failed because it was missing the two new variables. I simply added these with default values.

* fixed syntax error in test_recorder_setup_failure

* fixed formatting error in test_init_py for recorder component

* fixed typo in test case

* Updated the way the default keys for db_,max_wait and db_retry_wait is set

Implemented based on suggestions from @springstan

* Updated config_schema call to adhere to Black

* changed conf.get to conf[dict] for var retrieval

* removed 2 blank lines
2020-02-09 17:43:09 +01:00
Franck Nijhof e9eaa6536d Upgrade sqlalchemy to 1.3.13 (#31101) 2020-01-23 17:10:40 +01:00
Franck Nijhof bfa0edaf92 Migrate Integration Quality Scale from docs to manifest: internal (#30551) 2020-01-07 20:58:49 +01:00
Ville Skyttä e6388e186c Remove unnecessary string literal concatenations (#30360) 2020-01-02 21:17:10 +02:00
Franck Nijhof b1bb2298e0 Bump sqlalchemy to 1.3.12 (#30142) 2019-12-22 19:52:39 +01:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder (#29652) 2019-12-08 18:48:18 +01:00
Ville Skyttä 99c7608fb4 Lint config cleanups (#28864)
* Remove bunch of unneeded lint exclusions

* Use symbolic names instead of identifiers in pylint disables

* Tighten scope of some pylint disables
2019-11-25 22:40:08 -08:00
Fabian Affolter d8ca2e9d71 Upgrade sqlalchemy to 1.3.11 (#28721) 2019-11-16 09:56:26 +01:00
bastshoes 0226b76e0a Add support SQL VACUUM for PostgeSQL (#28106)
* Add support SQL VACUUM for PostgeSQL

VACUUM PostgreSQL DB if repack is true

* Update tests
2019-10-22 08:39:26 -07:00
Malte Franken 83a709b768 Move imports in recorder component (#27859)
* move imports to top-level in recorder init

* move imports to top-level in recorder migration

* move imports to top-level in recorder models

* move imports to top-level in recorder purge

* move imports to top-level in recorder util

* fix pylint
2019-10-18 10:14:54 -07:00
Franck Nijhof ed3516186b Bump sqlalchemy to 1.3.10 (#27408) 2019-10-10 12:52:54 -07:00
Fabian Affolter 13956d3516 Upgrade sqlalchemy to 1.3.9 (#27322) 2019-10-08 09:30:18 -07:00
Franck Nijhof c7da781efc Update documentation link URL for integrations in all manifests (#27114) 2019-10-02 09:25:44 -07:00
Franck Nijhof b1118cb8ff Removes unnecessary else/elif blocks (#26884) 2019-09-24 13:53:03 -07:00
Ville Skyttä 33e1b44b3a Use PEP 526 type annotations, add some type hints (#26464)
* Add some more type hints to helpers.event

* Change most type comments to variable types

* Remove some superfluous type hints
2019-09-07 09:48:58 +03:00
Josef Schlehofer a80d26f0dc Upgrade sqlalchemy to 1.3.8 (#26331) 2019-09-01 14:10:58 +03:00