Filtering and Grouping KQL by Hour of Day or Weekday

EDIT! A friend of mine pointed me to the use of the function HourOfDay() which is native to KQL. I will leave the post as is, but you know now that this exists. Thanks Morten.

Initially you will most likely use KQL for ad-hoc analysis if a customer calls you in panic that a system is slow. But it is much better to use the telemetry to prevent issues and predict that systems get slower and fix issues before users get their eyes all wet.

For this it is important to be able to do comparisons, and in most businesses you can compare business days (monday vs. tuesday) or weekdays (this monday vs. last monday) or hours (10am vs. 4pm).

This morning I logged in to a customers system and compared the last 4 weeks per hour.

You can immediately see that the system is not used on saturday or sunday and that the system gets busier during the day before end of day.

Also you can see the effect of Christmas, but that is irrelevant for this post. 😉

I am open to suggestions but this is what I came up with to render a chart showing the busiest hours of the day.

Hour

Event
| extend hour = tostring(toint(substring(tostring(bin(TimeGenerated, 1h)),11,2)) + 1)
| extend server = strcat(extract("Server instance:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend object = strcat(extract("AppObjectType:\s{1,}([^\ ]+)\s", 1, ParameterXml), extract("AppObjectId:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend executionTime = toint(executionTime = extract("Execution time:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend query = strcat(extract("SELECT\s.FROM\s.WHERE\s.", 0, ParameterXml), extract("DELETE\s.FROM\s.WHERE\s.", 0, ParameterXml), extract("UPDATE\s.SET\s.WHERE\s.*", 0, ParameterXml))
| where ParameterXml contains "Message: Long running SQL statement"
| summarize sum(executionTime) by hour
| render piechart

Note that this client is in CET so I have to manually convert for UTC.

Also, for some reason the Hour needs to be a string to be able to render as a label for the piechart.

Weekday

The WeekDay is easier if you accept that it is zero-based where sunday is zero.

Event
| extend weekday = substring(tostring(dayofweek(TimeGenerated)), 0, 1)
| extend server = strcat(extract("Server instance:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend object = strcat(extract("AppObjectType:\s{1,}([^\ ]+)\s", 1, ParameterXml), extract("AppObjectId:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend executionTime = toint(executionTime = extract("Execution time:\s{1,}([^\ ]+)\s", 1, ParameterXml))
| extend query = strcat(extract("SELECT\s.FROM\s.WHERE\s.", 0, ParameterXml), extract("DELETE\s.FROM\s.WHERE\s.", 0, ParameterXml), extract("UPDATE\s.SET\s.WHERE\s.*", 0, ParameterXml))
| where ParameterXml contains "Message: Long running SQL statement"
| summarize sum(executionTime) by weekday
| render piechart

Setting up Azure SQL Analytics (Preview) – Dynamics NAV

Telemetry is everything, you cannot have enough data when users start asking you why the system is behaving differently than yesterday or performance is changing over time.

This is where Azure SQL stands out from On Premises. You can get so much more data and in an easy way to analyse.

However, you need to know where to find it because not everyting is setup automatically after you create a database. Some is, some is not.

This blog is about how to connect Azure SQL Analytics to your Azure Monitor.

The steps how to do this are described in this docs entry and I don’t want to repeat existing documentation. I will add some screenshots of some results for a 220 GB Microsoft Dynamics NAV database with 80 concurrent users.

https://docs.microsoft.com/en-us/azure/azure-sql/database/metrics-diagnostic-telemetry-logging-streaming-export-configure?tabs=azure-portal

Step 1 – Patience!

After you have activated Azure SQL Analytics it will not be visible for a while. It takes time in the background to be generated and but together by the Microsoft Minions who control your tenant in the background. Remember that these Minions have labour contracts and a rights to have a break every now and then.

Step 2 – Azure Monitor & More…

When the Minions are finished the data will show up in Azure Monitor. Search for it in your environment

And then, at least in my case you have to click on More…

This should show a link to your Azure SQL Analysis. In my case with two databases. DEV and PROD.

Step 3 – The Dashboard

The first dashboard you’ll see is something like this, except for the fact that this shows data 24 hours after activation and we had a busy friday with a performance incident. I’ll get back to that.

There are some interesting statistics here already visible like wait stats, deadlocks and autotuning. I’ll handle wait stats in this blog and maybe I’ll get back to deadlocks and autotuning later. There is a “good” reason the autotuning is red and I’ll look at that tomorrow (sunday) when nobody is working on the system.

Step 4 – Drill Down | Database Waits

If we drill down into the Database Waits we see more details on what types of waits we are dealing with here.

It does not help looking at these waits without narrowing down into specific moments in time when “things go wrong” because specific events relate to specific wait stats and some waits are just there whether you like it or not. We all know CXPPACKET because NAV/Business Central fires a lot of simple queries to the Azure SQL engine resulting in CPU time wasted. There is not much you can do about that. (As far as I know).

Step 5 – Houston we have a problem!

It’s 3:51pm on friday afternoon when my teammate sends me a message on Skype that users are complaining about performance. Since we just turned on this great feature I decide to use it and see what goes wrong.

We drill down again one more time and click on the graph showing the waits.

Note that this screenshot was created a day after the incident but it clearly illustrates and confirms that “someting” is off around the time my teammate sent me a message. The wait time on LCK_M_U goes through the roof! We have a blocker in our company.

Hey, this is KQL again!

Now we are in a familiar screen, because this is the same logging that Business Central Application Insights is using. Drilling down into the graph actually generated a KQL query.

Step 6 – What is causing my block?

To see what query is causing my block I have to go back to the Azure Dashboard and click on Blocks like this

From here we have two options. If I click on the database graph I get taken into the KQL editor and if I click on a specific block event I get a more UI like information screen. Let’s click on the latter.

Step 7 – Get the Query Hash

This is where it get’s nerdy. The next screen shows the blocking victim and the blocking process.

It also shows a Query Hash.

This is where I had to use google, but I learned that each “Ad-Hoc” query targetted against SQL Server gets logged internally with a Query Hash.

Since NAV/Business Central only used Ad-Hoc queries we have a lot of them and it’s important to understand how to read them.

What worries me a bit here is the Blocking Process’ Status which is sleeping. I have to investigate this more, but I interpret this as a process that went silent and the user is not actively doing something.

Step 8 – Get the Query

Using Google I (DuckDuckGo actually) also found a way to get these queries as long as they still exist in the cache of your SQL Server. Simply use this query

SELECT deqs.query_hash ,
deqs.query_plan_hash ,
deqp.query_plan ,
dest.text
FROM sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_query_plan(deqs.plan_handle) AS deqp
CROSS APPLY sys.dm_exec_sql_text(deqs.sql_handle) AS dest
WHERE deqs.query_hash = 0xB569219D4B1BE79E

This will give you both the query and the execution plan. You have to use SQL Server Management studio to execute this against your Azure SQL Database

Step 9 – Restart the service tier

Unfortunately for me this journey resulted in having to restart the service tier. We could not identify the exact person/user who had executed the query that was locking. Maybe we will be able to do that in a future incident since I’m learning very fast how to use this stuff and time is off the most essence when incidents like this happen on production environments.

Needless to say that the NAV Database Locks screen was not showing anything. I would have used that otherwise off course.

Azure Application Insights 101

In my series around Application Insights for Microsoft Dynamics Business Central / NAV this is probably the most booring one. However it is quite important. In order to teach you folks about KQL and the Application Insights API etc.

Step 1 – Create Application Insights

In your Azure Tenant search for Application Insights and select Add.

There is not much to fill in here. The Resource Group is probably most important if you have a bigger Azure Tenant. You want to group your stuff together.

Step 2 – Grab the key!

After the resource is created grab the key to your clipboard and now leave the Azure Portal and move to the Business Central Admin Portal

Step 3 – Put the key in Business Central and Restart your system

Step 4 – Analyse the data

But that’s for the next blog, about KQL. This will be a language at least 1 person in your company needs to master. Definately.

Wait… is that all??

Essentially yes, but there is a caveat…

The million dollar question is probably whether or not to pot multiple customers into one Application Insights resource.

This probably depends on one question. Does your customer want to access the data? If they do, the data needs to be in it’s own AppInsights resource so you can grant your customer access.

The good news is, and we’ll get to that, is that you can query accross application insights instances.

Business Central Performance Tuning Series | Application Insights & More

If you ask a random Microsoft partner about their worries with Business Central there is a fair chance “Performance” is in their top three.

It probably depends if this partner has a background with NAV. If this is the case it’s a guarantee it’s in the list.

Continue reading “Business Central Performance Tuning Series | Application Insights & More”