Snowflake query_history gets reset after warehouse suspension

Question

I am using the following query to retrieve query history from my Snowflake database.

SELECT *
FROM table(MY_DATABASE.information_schema.query_history(
  end_time_range_start => dateadd(HOUR, -4, current_timestamp()),
  current_timestamp()
)); 

Oddly, if the warehouse (size: XS) I am using gets suspended after a period of inactivity, the next time I attempt to retrieve query history- the history that was there prior to the warehouse's suspension is gone.

I could not find anything documented to explain this.

Anyone run into this issue or related documentation that could explain this?

Thank you!

Answer 1

I can't explain exactly the limitations of that information schema query you are running (some of them only return like 10,000 rows or like you said, once the warehouse turns off), but it's a limited view into the actual query history. You can use the snowflake database for all query history.

It's a massive table so make sure you put filters on it. Here's an example query to access it:

USE DATABASE snowflake;
USE SCHEMA account_usage;

SELECT *
FROM query_history
WHERE start_time BETWEEN '2020-01-01 00:00' AND '2020-01-03 00:00'
  AND DATABASE_NAME = 'DATABASE_NAME'
  AND USER_NAME = 'USERNAME'
ORDER BY START_TIME DESC;

Answer 2

1: Your question states that after a period of inactivity, does not specify what is the period of inactivity.

"after a period of inactivity, the next time I attempt to retrieve query history- the history that was there prior to the warehouse's suspension is gone."

If its beyond 7 days then the data can be found from account_usage table. Below is the link of difference between INFORMATION_SCHEMA and ACCOUNT_USAGE.

https://docs.snowflake.com/en/sql-reference/account-usage.html#differences-between-account-usage-and-information-schema

2: Your query does not specify USER_NAME or WAHREHOUSE_NAME in your query so it could be that before the output of your queries before suspension of warehouse may have moved beyond 4 hours period as in your predicate. If you can increase the time period and check if behaviour still exists.

3: In general its not advisable to query INFORMATION_SCHEMA to get query history unless your application requires data without any latency. If possible use ACCOUNT_USAGE table to get query history information.

Here is what I did.

1: Created an XS warehouse

2: Set auto_suspend to 5 minutes

3: Ran few queries

4: Ran your query (which does not specify user_name or warehouse_name) meaning you are searching for history from all users.

SELECT *
FROM table(MY_DATABASE.information_schema.query_history(
  end_time_range_start => dateadd(HOUR, -4, current_timestamp()),
  current_timestamp()
));

5: Returned output of few 100 records.

6: Used additional where clause to check for data of my user which ran few queries before auto_suspend of Warehouse and it returned few records.

SELECT *
FROM table(MY_DATABASE.information_schema.query_history(
  end_time_range_start => dateadd(HOUR, -4, current_timestamp()),
  current_timestamp()
))
WHERE USER_NAME = 'ADITYA';

7: Waited for 10 minutes so that my warehouse is auto_suspended.

8: Repeat point 5 and point 6 and again it returned records as expected.