Calculating incrementals with the LAG function

T-SQL Tuesday #25 is hosted by Allen White (blog). This month’s topic is “SQL Server Tips and Tricks”.

Transaction Log Reuse Wait

One question that keeps coming up in forums, at user group meetings and on twitter is: Why is my transaction log growing out of bounds?

To adhere to the ACID properties, SQL Server records all changes to the database first into the transaction log file. A transaction can only be committed after that write succeeds. The changes to the data itself actually get applied only to the data pages in the buffer pool. If and when they get written to the disk is influenced by several factors that are independent of the transaction itself. The records in the transaction log allow SQL Server to redo a change after a crash for example that prevented the data page changes to be written to disk.

That means in theory, that after the transaction is committed and after the data pages are preserved on disk, there is no need for SQL Server to hold on to the transaction log data anymore. That's why the transaction log usually does not grow unbounded, because SQL Server can reuse the parts of the file(s) that are not needed anymore.

In practice however, there can be several reasons, why such a reuse is not possible. The most common one is the requirement for log backups. If you have your database set to recovery mode FULL, SQL Server does not reuse any part of the log file until it is backed up with a transaction log backup. There are several other reasons why SQL Server might have to wait before it can reuse the transaction log, ranging from long running open transactions to transactional replication.

If you are in a situation that requires you to find out why the transaction log file keeps growing and growing, this simple query can give you the answer:

SELECT name,log_reuse_wait_desc FROM sys.databases;

The log_reuse_wait_desc column contains the reason why the SQL Server currently can't reuse the log file of that database. Explanations for each of the values you can find in BOL under Factors That Can Delay Log Truncation.

It is T-SQL Tuesday again, number 24 about procedures and functions, hosted by Brad Schulz (Blog|no twitter).

Performance Comparisons of different types of Functions

A lot has been written about the performanmce impact of functions. Amongst other effects, functions used in a WHERE clause will prevent the use of an index seek for the columns involved.

Sometimes however it makes sense to use functions to encapsulate a complicated calculation or to make code reuse possible. It is therefore important that to understand just how big the impact is, that functions have on queries.

I would like to use this blog post to look at the call overhead of different kinds of functions in SQL Server.

SQL Server knows three types of functions: Scalar Functions, Inline Table-Valued Functions and Multi Statement Table-Valued Functions. There are also Scalar and Table-Valued CLR Functions, but we will not be looking at those in this article.

To be able to look just at the overhead of the function call we will use a very simple calculation: 1 * value

All examples used in this post will select from a simple table:

CREATE TABLE dbo.tbl1
(
  id INT CONSTRAINT PK_tbl1 PRIMARY KEY CLUSTERED,
  data INT NOT NULL
);

This table has been initialized with 1000000 rows containing random values in the data column.

As a base line we will use the following SELECT statement:

SELECT * INTO #x FROM dbo.tbl1 WHERE (1*data) = 42;

Note the use of "(1*data)" instead of just "data". This is the calculation we will implement in the different functions and the use of the calculation in the baseline query has the same impact on index usage as a function call has: It effectively prevents a seek operation on any existing index. Also note the "INTO #x". It prevents that the results have to be send back to the client eliminating the chance for any ASYNCH_NETWORK_IO waits to affect the outcome of this performance test. Instead the results (if any) will be written to tempdb on the same machine.

Now lets look at the three functions:

CREATE FUNCTION dbo.simpleSVF(@parm INT)
RETURNS INT
AS
BEGIN
  RETURN 1*@parm;
END;
GO
CREATE FUNCTION dbo.simpleTVF(@parm INT)
RETURNS TABLE
AS
RETURN SELECT 1*@parm val;
GO
CREATE FUNCTION dbo.multiStmtTVF(@parm INT)
RETURNS @r TABLE(val INT)
AS
BEGIN
  INSERT INTO @r(val) VALUES(1*@parm);
  RETURN;
END

They all implement the same calculation we saw above. The Scalar Function (simpleSVF) just returns the result of that calculation directly, while the two Table-Valued functions return a resultset with one column and one row, containing the calculation result.

To use the Scalar Function we can just replace the calculation with a call to the function. The use of a Table-Values Function in this context requires a little more work. There are two patterns that you can use: A correlated subquery or a CROSS APPLY. We will compare both of them:

SELECT * INTO #x FROM dbo.tbl1 WHERE (SELECT val FROM dbo.simpleTVF(data)) = 42;
SELECT * INTO #x FROM dbo.tbl1 t CROSS APPLY dbo.simpleTVF(t.data) f WHERE f.val = 42;

The same two call patterns will also be used for the Multi Statement Table-Valued function.

To record the execution times I created a small stored procedure that uses the information in sys.dm_exec_requests together with the SYSDATETIME() function for it's measurements:

CREATE PROCEDURE dbo.RunTest 
  @cid INT,
  @cmd NVARCHAR(MAX)
AS 
BEGIN
  DECLARE @duration1 DATETIME2 ;
  DECLARE @duration2 DATETIME2 ;
  DECLARE @cpu1 INT ;
  DECLARE @cpu2 INT ;
  DECLARE @reads1 INT ;
  DECLARE @reads2 INT ;

  DBCC DROPCLEANBUFFERS ;
  DBCC FREEPROCCACHE ;
  SELECT  @duration1 = SYSDATETIME() ,
          @cpu1 = cpu_time ,
          @reads1 = logical_reads
  FROM    sys.dm_exec_requests
  WHERE   session_id = @@SPID ;

  EXEC(@cmd) ;

  SELECT  @duration2 = SYSDATETIME() ,
          @cpu2 = cpu_time ,
          @reads2 = logical_reads
  FROM    sys.dm_exec_requests
  WHERE   session_id = @@SPID ;
  
  INSERT INTO dbo.TestResults(cid,StartDTime,Duration,CPU,LogicalReads,Cmd)
  SELECT  @cid,
          @duration1,
          DATEDIFF(microsecond, @duration1, @duration2),
          @cpu2 - @cpu1,
          @reads2 - @reads1,
          @cmd;
END;

This procedure executes the passed in command and records the execution time in microseconds, the time spend working on any CPU in milliseconds(!) as well as the logical reads in pages. It also clears out the buffer cache as well as the procedure cache each time before executing the statement. (So don't run this in production!)

The first parameter passed in is the command id (cid) and it is not used by the procedure itself, it is just stored together with the results. The 6 different calls to this procedure are listed below:

EXEC dbo.RunTest 1,'SELECT * INTO #x FROM dbo.tbl1 WHERE (1*data) = 42;';
EXEC dbo.RunTest 2,'SELECT * INTO #x FROM dbo.tbl1 WHERE dbo.simpleSVF(data) = 42;';
EXEC dbo.RunTest 3,'SELECT * INTO #x FROM dbo.tbl1 WHERE (SELECT val FROM dbo.simpleTVF(data)) = 42;';
EXEC dbo.RunTest 4,'SELECT * INTO #x FROM dbo.tbl1 t CROSS APPLY dbo.simpleTVF(t.data) f WHERE f.val = 42;';
EXEC dbo.RunTest 5,'SELECT * INTO #x FROM dbo.tbl1 WHERE (SELECT val FROM dbo.multiStmtTVF(data)) = 42;';
EXEC dbo.RunTest 6,'SELECT * INTO #x FROM dbo.tbl1 t CROSS APPLY dbo.multiStmtTVF(t.data) f WHERE f.val = 42;';

After execution the first four 100 times each and the last two 10 times each on my system I got the following numbers (average per execution):

ExecCount	Duration	CPU	LogicalReads	Cmd
100	725,781	219	2,296	SELECT * INTO #x FROM dbo.tbl1 WHERE (1*data) = 42;
100	3,007,272	2,702	2,400	SELECT * INTO #x FROM dbo.tbl1 WHERE dbo.simpleSVF(data) = 42;
100	708,660	209	2,283	SELECT * INTO #x FROM dbo.tbl1 WHERE (SELECT val FROM dbo.simpleTVF(data)) = 42;
100	715,560	211	2,290	SELECT * INTO #x FROM dbo.tbl1 t CROSS APPLY dbo.simpleTVF(t.data) f WHERE f.val = 42
10	112,347,425	109,412	33,984,827	SELECT * INTO #x FROM dbo.tbl1 WHERE (SELECT val FROM dbo.multiStmtTVF(data)) = 42;
10	109,809,480	107,776	33,984,881	SELECT * INTO #x FROM dbo.tbl1 t CROSS APPLY dbo.multiStmtTVF(t.data) f WHERE f.val = 42

As you can see, there is almost no difference between the baseline and the two usages of the Inline Table-Valued Function. The Scalar Function on the other hand took more than four times as long. Most of that time (2.7 of 3 seconds) was spend actually using the processor. There were also about 100 more pages read.

Now lets look at the Multi Statement Table-Valued Function. I had to stop its execution after just 10 rounds to get the results in before Tuesday. This function slows the query by a factor of over 150.

This is caused by the fact that a Multi Statement Table-Valued Function uses a table variable to collect the rows. To create a table variable - similar to any other table - at least two pages need to be reserved as well as several changes to system tables and internal database pages need to be performed. That all needs to be undone once the function finished executing.

In both queries above, the function gets executed for every row — 1,000,000 times in this case. This is causing all that table setup and destroy work to be executed 1,000,000 times as well.

Conclusion

If you find yourself in a situation that lends itself to the use of a function, try to go with an Inlined Table-Valued Function as there is almost no performance impact. However, always remeber that this blog post did not look into the impact that a function has on index usage.

Unless you are writing a function that is going to be executed only rarely, stay away from Multi Statement Table-Valued Functions.

It is T-SQL Tuesday again, the 22nd incarnation and this time the topic is the presentation of data, hosted by Robert Pearl (Blog|Twitter).

Assume for example if you are trying to write a view that displays information about all the indexes in the database. You want this view to return one row per index. You also want to include the columns of the index. To make this happen you need to concatenate the information for the columns into one string.

This post is going to look at how to implement string concatenation in T-SQL.

In many other database management systems you have access to a function GROUP_CONCAT() that provides just this functionality. In T-SQL it is not that easy yet. While Denali offers a new CONCAT() function, a GROUP_CONCAT() is not available in SQL Server up to version Denali CTP3.

XML to the rescue

There are several ways to go about this problem. For example you could write a UDF or you could rely on undocumented behavior¹. Most of those solutions seem complicated and also do not really address a real concatenation in a GROUP BY context.

There is one solution that I have found to be the easiest to use: FOR XML.

Using the FOR XML clause, SQL Server allows the output of any select to be a single XML Document:

SELECT  name
FROM    master.sys.tables
FOR     XML PATH('row');

returns something like

<row>
  <name>spt_fallback_db</name>
</row>
<row>
  <name>spt_fallback_dev</name>
</row>
<row>
  <name>spt_fallback_usg</name>
</row>
<row>
  <name>spt_monitor</name>
</row>
<row>
  <name>spt_values</name>
</row>
<row>
  <name>MSreplication_options</name>
</row>

Every row gets wrapped into a <row> tag and every column in a tag that matches the columns name.

To eliminate the row tag we can pass in an empty string to the PATH() function. The column tags are eliminated by naming the column [text()].

So our example would look like this:

SELECT  name AS [text()]
FROM    master.sys.tables
FOR     XML PATH('');

It's output looks like this:

spt_fallback_dbspt_fallback_devspt_fallback_usgspt_monitorspt_valuesMSreplication_options

That looks already very promising. You probably would like to add separators in between the strings. How you can do that we will get to a little later. There are a few things that I would like to address first.

Special Characters

Let's look at characters that have a special meaning in XML like ‘<’:

SELECT '<&>' AS [text()]
FOR XML PATH('');

Output:

<&>

That is not what we wanted. To get those characters unescaped, we need to use one of the XML datatype functions:

SELECT (SELECT '<&>'
        FOR XML PATH(''), TYPE
       ).value('.','NVARCHAR(MAX)');

This has the expected output: ‘<&>’. You might have noticed the missing column name. If the column name is not specified at all (explicitly or otherwise), it has the same effect as if the [text()] name is given. The additional TYPE keyword causes the XML to be returned as an XML datatype result instead of as an NVARCHAR(MAX) datatype result. The .value() function is defined on the XML datatype and takes two parameters. The first one is using the XPath syntax and describes what we want to get back. The '.' here means everything. The second one specifies the datatype to which we want the result to be converted to.

Illegal Characters

The next problem is not that easy to solve. The SQL Server XML datatype cannot handle specific characters at all:

SELECT CHAR(1)
FOR XML PATH('');

SELECT CHAR(1)
FOR XML PATH(''), TYPE;

Msg 6841, Level 16, State 1, Line 1
FOR XML could not serialize the data for node 'NoName' because it contains a character (0x0001) which is not allowed in XML. To retrieve this data using FOR XML, convert it to binary, varbinary or image data type and use the BINARY BASE64 directive.

There is a total of 2079 single character values that will cause this error. How to figure out which ones exactly will have to wait for another blog post. For now, if you think you might run ito any of them, you can use the following code to replace the most common of them with a question mark:

SELECT REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( 
       N'the text that is to be cleaned here' 
,NCHAR(1),N'?'),NCHAR(2),N'?'),NCHAR(3),N'?'),NCHAR(4),N'?'),NCHAR(5),N'?'),NCHAR(6),N'?'),NCHAR(7),N'?'),NCHAR(8),N'?'),NCHAR(11),N'?'),NCHAR(12),N'?'),NCHAR(14),N'?'),NCHAR(15),N'?'),NCHAR(16),N'?'),NCHAR(17),N'?'),NCHAR(18),N'?'),NCHAR(19),N'?'),NCHAR(20),N'?'),NCHAR(21),N'?'),NCHAR(22),N'?'),NCHAR(23),N'?'),NCHAR(24),N'?'),NCHAR(25),N'?'),NCHAR(26),N'?'),NCHAR(27),N'?'),NCHAR(28),N'?'),NCHAR(29),N'?'),NCHAR(30),N'?'),NCHAR(31),N'?');

This replacement has to happen before the text is converted to XML. The following examples are going to skip this step for readability.

Separators

Now we know how to concatenate string values together. To make our initial example of a select statement (or view) that returns one row per index and includes a comma separated column list work, two more steps are missing. First, the separator needs to be inserted between the values but not in front of the first one. This is not possible. But it is fairly simple to remove the additional separator after adding one in front of every item:

SELECT  STUFF((SELECT   ', ' + name
               FROM     master.sys.tables
               FOR XML PATH('') ,TYPE
              ).value('.', 'NVARCHAR(MAX)'), 
              1, 2, ''
             );

The STUFF function helps us with this problem. The first three parameters work just like the ones in the SUBSTRING function. But instead of returning that substring, the STUFF function replaces it with the value passed in as fourth parameter and then returns the complete string.

Putting it all together

The last step is to do this all inside a GROUP BY query. For that we have to use the CROSS APPLY functionality to create the column list for each group value in a correlated sub-query. Putting this all together looks like this:

SELECT  t.name AS TblName ,
        i.name AS IdxName ,
        Columns.List AS Columns
FROM    sys.tables t
        JOIN sys.indexes i ON i.object_id = t.object_id
        CROSS APPLY ( SELECT    STUFF((SELECT   ', ' + c.name
                                                + CASE WHEN ic.is_descending_key = 1
                                                       THEN ' DESC'
                                                       ELSE ''
                                                  END
                                       FROM     sys.index_columns ic
                                                JOIN sys.columns c ON ic.object_id = c.object_id
                                                              AND ic.column_id = c.column_id
                                       WHERE    i.object_id = ic.object_id
                                                AND i.index_id = ic.index_id
                                       ORDER BY ic.index_column_id
                                       FOR   XML PATH('') ,
                                          TYPE
                                      ).value('.', 'NVARCHAR(MAX)')
                                      , 1, 2, ''
                                     )
                    ) Columns ( List ) ;

This query returns the table name, the index name and a comma separated list of all columns for each index in the database. It also marks descending columns with the DESC keyword. What it does not do is separate out included columns. I'll leave that as an exercise for the reader.

1) One method that relies on undocumented behavior would be to continually add to a single variable within a single multirow select.