sqlity.net » SQL Server Internals

The Mysterious “sp_” System Procedure Prefix

Sebastian — Sun, 13 May 2012 21:04:46 +0000

Introduction

If you have been working with SQL Server for a while you probably know that you should not select names for your stored procedures that start with the three characters "sp_".

Today I would like to take a closer look at all the myths and facts around this prefix. This will include a series of examples, all of which have been tested on SQL 2008R2 and on SQL 2012.

Special Objects

The first myth I would like to bust is that the "sp_" prefix stands for "system procedure".

While the BOL entry for Creating Stored Procedures uses language that could easily be interpreted like that, it actually never says anything about the etymology of this prefix. In fact, the "sp_" prefix causes a special path to be taken in the resolution of the object name. This works for stored procedures as well as for other object types. Later on this article contains an example with a table.

As "sp_" designates more objects than just stored procedures as special, it clearly does not mean "System Procedure", even though it currently is only used for system procedures. Instead it just stands for "special".

Resource Database

The idea for this article came from a tweet in a discussion about this prefix started by @ericstephani. In the tweet in question @SirSQL suggested to check if master was indeed the database checked first, as everyone in the discussion had assumed before, or if it rather is the resource database. This makes a lot of sense, as most of the system objects live in the resource database. So I decided to check this out by running a series of tests.

If you want to follow along with these tests you will need two databases: One with the name test and one that is called OtherDb.

For the first test I created a procedure with the name sp_executesql in master as well as in test and then executed it from both places:

USE master;
GO
CREATE PROCEDURE dbo.sp_executesql
@statement NVARCHAR(MAX)
AS
BEGIN
  SELECT DB_NAME() [called from], 'master' [sp_executesql];
END
GO
EXEC dbo.sp_executesql @statement = N'SELECT DB_NAME() [called from], ''system'' [sp_executesql];';
GO
DROP PROCEDURE dbo.sp_executesql;
GO
------------------------------------------------
GO
USE test;
GO
CREATE PROCEDURE dbo.sp_executesql
@statement NVARCHAR(MAX)
AS
BEGIN
  SELECT DB_NAME() [called from], 'test' [sp_executesql];
END
GO
EXEC dbo.sp_executesql @statement = N'SELECT DB_NAME() [called from], ''system'' [sp_executesql];';
GO
DROP PROCEDURE dbo.sp_executesql;
GO

The sp_executesql version that gets shipped with SQL Server lives in the resource database and so we would expect this version to get executed in all cases. This is indeed correct, as you can see in the result:

The above test shows that the resource database is checked first when an object name with the "sp_" prefix is encountered. That means that if you create any object with that name prefix you are running the risk that it will not be accessible anymore after the next service pack install if Microsoft decided to create an object with the same name in the resource database. This is even true when the resource database object is of a different type than your object. This you can quickly confirm by creating a table with the name sp_executesql in any database and trying to insert a row into it.

Master Piece

The fact that the resource database is the one checked first to resolve "sp_" object names raises the question where the master database fits in. Read on, as the results might surprise you.

The second test involves a procedure that is not (yet) part of the resource database: sp_MyOwnProc

I again created this procedure in master and in test:

GO
USE master;
GO
CREATE PROCEDURE sp_MyOwnProc
AS
BEGIN
  SELECT DB_NAME() [called from], 'master' [sp_MyOwnProc];
END
GO
USE test;
GO
CREATE PROCEDURE sp_MyOwnProc
AS
BEGIN
  SELECT DB_NAME() [called from], 'test' [sp_MyOwnProc];
END
GO
EXEC sp_MyOwnProc;
GO
USE OtherDb;
GO
EXEC sp_MyOwnProc;
GO
USE test;
GO
DROP PROCEDURE sp_MyOwnProc;
GO
USE master;
GO
DROP PROCEDURE sp_MyOwnProc;
GO

The above code creates the procedure in both places. It then executes the EXEC sp_MyOwnProc; statement, first from the test database and afterwards from the OtherDb database. The result is shown below:

As you can see, master is not checked first for an "sp_" object. Instead the object in the current database is used. Only if the current database does not contain the object in question master is looked at to see if the object exists in there.

The same behavior can be observed when all references to the sp_MyOwnProc in the above script are schema-qualified with dbo.

Other Object Types

You also get the same behavior if you go through this exercise with a table:

USE master;
GO
CREATE TABLE sp_MyOwnTable(
sp_MyOwnTable NVARCHAR(MAX)
);
INSERT INTO sp_MyOwnTable SELECT DB_NAME();
GO
USE test;
GO
CREATE TABLE sp_MyOwnTable(
sp_MyOwnTable NVARCHAR(MAX)
);
INSERT INTO sp_MyOwnTable SELECT DB_NAME();
GO
SELECT DB_NAME() [called from], * FROM sp_MyOwnTable;
GO
USE OtherDb;
GO
SELECT DB_NAME() [called from], * FROM sp_MyOwnTable;
GO
USE test;
GO
DROP TABLE sp_MyOwnTable;
GO
USE master;
GO
DROP TABLE sp_MyOwnTable;
GO

This script creates a table with the name sp_MyOwnTable in the master as well as in the test database and then executes a select against this name executing in test as well as in OtherDb. The result is shown here:

This shows that the name resolution works the same for tables as it does for stored procedures. It also works views, but not for functions or user defined types.

Performance Impact

Now we know that SQL Server tries to find an "sp_" object in the resource database first and only if it does not exist there the search is continued in the current database. So there should be a measurable performance impact showing this extra work.

To measure the impact I used this script:

USE test;
GO
IF OBJECT_ID('dbo.sp_MyOwnProc2') IS NOT NULL DROP PROCEDURE dbo.sp_MyOwnProc2;
GO
CREATE PROCEDURE dbo.sp_MyOwnProc2
AS
RETURN 0;
GO
IF OBJECT_ID('dbo.MyOwnProc2') IS NOT NULL DROP PROCEDURE dbo.MyOwnProc2;
GO
CREATE PROCEDURE dbo.MyOwnProc2
AS
RETURN 0;
GO
------------------------------------
GO
DECLARE @StartTime DATETIME2 = SYSDATETIME();
DECLARE @EndTime DATETIME2 = SYSDATETIME();
DECLARE @Counter INT = 0;
DECLARE @CmdA NVARCHAR(100) = 'EXEC dbo.MyOwnProc2;--';
DECLARE @CmdB NVARCHAR(100) = 'EXEC dbo.sp_MyOwnProc2;--';
DECLARE @TimeA BIGINT=0;
DECLARE @TimeB BIGINT=0;
DECLARE @Cmd2 NVARCHAR(100);
WHILE(@Counter<10000000)
BEGIN
  SET @Cmd2 = @CmdA + CAST(@Counter AS NVARCHAR(20));
  SET @StartTime = SYSDATETIME();
  EXEC(@Cmd2);
  SET @EndTime = SYSDATETIME();
  SET @TimeA += DATEDIFF(microsecond,@StartTime,@EndTime)
  SET @Cmd2 = @CmdB + CAST(@Counter AS NVARCHAR(20));
  SET @StartTime = SYSDATETIME();
  EXEC(@Cmd2);
  SET @EndTime = SYSDATETIME();
  SET @TimeB += DATEDIFF(microsecond,@StartTime,@EndTime)
  SET @Counter += 1;
END
SELECT @Counter Counter,@TimeA [MyOwnProc2], @TimeB [sp_MyOwnProc2];
GO

It first creates two identical stored procedures that do nothing but return a 0. The first one is called sp_MyOwnProc2, the second one carries the name MyOwnProc2 without the "sp_" prefix. To measure the performance impact, the script calls both procedures alternating in a loop and records their execution times. Each call gets executed as dynamic sql with the current loop count being part of the sql string. This prevents any possible attempts to cache the plan for the batch. It does however not prevent caching the plan for the procedure itself. That is okay, as we are after the name resolution piece of the execution. Because both procedures get called alternatingly, any background noise caused by other processes on the test system should affect both evenly.

On my system I got these results:

After 10 million executions of each of the two procedures you can see, that there is a performance impact. With less than 2 percent, however, it is very small compared to the time it takes to just call the procedure. Remember, that the procedures in this test did not actually do any work, so all the time recorded by this test was spent on identifying the procedure and the overhead of calling it.

Conclusion

The "sp_" object name prefix causes SQL Server to take a special route when resolving the name of this object: First SQL Server checks if the object exists in the hidden resource database. Second it tries to find the object in the current database and if it does not exist there SQL Server goes on to check if the object exists in the master database.

If you name your own objects using this prefix, you have to be aware of two possible consequences: First there is a small but measurable impact on performance. Second it can cause your application to suddenly break, if Microsoft decides to add an object with the same name to the resource database.

Because of that it is a best practice to not use the "sp_" name prefix anywhere in your code.

There is one exception however: If you are creating an object that you want to be accessible from all databases, you can use this prefix and place the object in the master database. However, because of the database precedence there are now two possibilities for those objects to get eclipsed by another object. So, if you go this route, make sure to regularly check that the object you are trying to execute is actually the one executing.

The Unloved Backward Scan

Sebastian — Sun, 29 Apr 2012 20:31:29 +0000

Introduction

When a query requires rows to be sorted, either directly requested with an ORDER BY clause or because one of the iterators requires it, SQL Server has two options to guarantee that order. The obvious one is to utilize a sort iterator. If the data comes from an index (clustered or covering), SQL Server can also use an "Ordered Scan" of the data. Depending on the requested sort direction, such an ordered scan can be either forward or backward.

This would hardly be worth an article, if there wasn't the peculiarity that SQL Server obviously does not like the idea of having to execute an ordered scan that is directed backwards.

Backward Scan Dislike

Let's look at an example. First we need some tables with data:

IF OBJECT_ID('dbo.T1') IS NOT NULL DROP TABLE dbo.T1;
IF OBJECT_ID('dbo.T2') IS NOT NULL DROP TABLE dbo.T2;

CREATE TABLE dbo.T1(
Id INT CONSTRAINT T1_PK PRIMARY KEY CLUSTERED,
v1 INT,
c1 VARCHAR(8000)
);

CREATE TABLE dbo.T2(
Id INT CONSTRAINT T2_PK PRIMARY KEY CLUSTERED,
v1 INT,
c1 VARCHAR(8000)
);

INSERT INTO dbo.T1(Id,v1,c1)
SELECT n,CHECKSUM(NEWID()),'*'
FROM dbo.GetNums(1000000);

INSERT INTO dbo.T2(Id,v1,c1)
SELECT n,CHECKSUM(NEWID()),'*'
FROM dbo.GetNums(1000000);
--Get dbo.GetNums here: http://www.sqlmag.com/article/sql-server/virtual-auxiliary-table-of-numbers

This script is creating two tables containing one million rows each. The v1 column is filled with random values; the c1 column just contains a single constant character. The Id column is the clustered primary key and it is providing the ordering on disk that we are going to use.

The query is a little made up. It joins T2 to itself bringing back only one set of columns. It also restricts the result of this join to the first one million rows, sorted by T2.Id. The one million values in T2.v1 are random from a set of 4 billion making them "mostly" unique. That means the self-join is expected to return about one million rows anyway. Those rows are then joined to T1. In the end the ORDER BY requests that all rows be returned sorted by the Id column of the T1 table:

SELECT  *
FROM    dbo.T1 A
        INNER JOIN ( SELECT TOP ( 1000000 )
                            B.*
                     FROM   dbo.T2 B
                            INNER JOIN dbo.T2 C ON B.v1 = C.v1
                     ORDER BY B.Id
                   ) BC ON A.Id = BC.Id
ORDER BY A.Id;

The execution plan of this query looks like this:

There is nothing unexpected in this plan. The T2 self-join is done with a hash join operator, as there is no index on the v1 column. The data is then sorted by a sort operator and passed through a serial zone for the top operator. After that there is no additional sort operator for the data to pass through. As the merge join requires both streams to be sorted the same way on the join column, this means that the data must be produced sorted by the scan of the T1 table. A quick look at its properties confirms that:

This scan is a forward scan, as both ORDER BY statements are asking to sort by Id ascending. To make this scan go backwards we should just have to change the query to sort by Id descending in both places:

SELECT  *
FROM    dbo.T1 A
        INNER JOIN ( SELECT TOP ( 1000000 )
                            B.*
                     FROM   dbo.T2 B
                            INNER JOIN dbo.T2 C ON B.v1 = C.v1
                     ORDER BY B.Id DESC
                   ) BC ON A.Id = BC.Id
ORDER BY A.Id DESC;

However, this query produces this rather unexpected plan:

A loop join with an index seek against the T1 table to retrieve 1,000,000 single rows – that cannot be good.

Before we look at execution statistics let's force the merge join back by specifying a join hint:

SELECT  *
FROM    dbo.T1 A
        INNER MERGE JOIN ( SELECT TOP ( 1000000 )
                                  B.*
                           FROM   dbo.T2 B
                                  INNER JOIN dbo.T2 C ON B.v1 = C.v1
                           ORDER BY B.Id DESC
                         ) BC ON A.Id = BC.Id
ORDER BY A.Id DESC;

Now we get the expected plan again:

If you compare the properties for the iterators in the forward scan and the backward scan plan you will notice, that the cost of the two scan directions is not that much different:

The estimated cost of the scan iterator is 0% in both cases. The bulk of the work is done in the hash self-join and the following sort of the T2 table.

So, let us take a look at the execution statistics of all three queries:

The execution statistics clearly show that the decision to go with a loop join was not necessarily the best. While the estimated cost for the backward scan is about 150% of the estimated cost for the loop join, the actual reads for the loop join version are about 3 million, which is more than 250 times higher than the 8000 reads for either scan direction.

Disk Access

To understand were this dislike is coming from we need to look at the access pattern necessary to retrieve the data from disk. A forward scan of a table (or index), that is not heavily fragmented, looks pretty much like doing a single contiguous read of a big blob of data. SQL Server has a lot of performance optimizations build in that make this type of access as fast as possible. The Read-Ahead mechanism is a good example for that.

A backward scan on the other hand looks quite the opposite: After reading a page, the disk has to do almost a complete turn to get to the previous page, which is the next one in line to be read. This is about the worst kind of random access you can come by.

As SQL Server always assumes that none of the requested pages are in cache, the decision to not get into that backward spinning game seems quite understandable.

Also, if you closely compare the two "scanning" execution plans you will notice that while the forward scan iterator is parallelized, the backward scan operator is executed single threaded. This is a general rule: SQL Server cannot execute a backward scan in parallel. That makes sense as with all that waiting for the disk to do another spin to get to the "next" page, it is unlikely for the data to come in quickly enough to keep multiple threads busy.

Estimates?

But wait, there is one more thing: If you look at the image above and compare the properties of all iterators in the two scanning plans, you will notice something odd. The estimated CPU cost of the "hash match inner join" iterator goes from 11 in the "forward" case to 1357 in the "backward" case; the estimated IO cost makes a similar jump from 0 to 1654. That seems to not make a lot of sense, as the next iterator, the sort, is a blocking iterator. "Blocking" means that the iterator reads in all input rows into a holding area before producing any output rows. This implies that anything on the left side of a sort should not be able to influence the cost of operators on the right side of it. That is, unless it affects the number of rows significantly, as this is a TOP N Sort. See TOP N Sort – A Little Bit of Sorting for an explanation. But as in this example all columns are "mostly" unique, the optimizer guesses correctly that about one million rows get passed into the hash join on each input and also about one million rows come out the other end. So row count estimates should not matter here.

If you look at the estimates when executing just that inner query sorting backwards, you will see that they exactly match the ones in the imbedded forward sorting case:

That means that the scan direction change of the T1 table causes the estimates on the other side of the merge join to get completely thrown off.

Conclusion

SQL Servers dislike of backward scan operations is understandable when looking at the work necessary to retrieve the records from disk in opposite index order. Additionally, a backward scan cannot be executed in parallel. That adds to the list of reasons, why the backward scan seems unloved.

However, the main reason why SQL Server avoids backward scans seems to be that they throw its cost estimations off, making following iterators appear a lot more expensive than they really are.

TOP N Sort – A Little Bit of Sorting

Sebastian — Mon, 23 Apr 2012 00:39:06 +0000

Introduction

Sorting is one of the most often used functions in SQL Server. It is used when creating indexes. It allows us to retrieve data in a particular order. It also is used to filter out repeated rows in the context of a DISTINCT request. The SQL Server Team spent a lot of effort optimizing the sort algorithms to give the best possible performance in each context. They even implemented sort avoidance where the expensive sort gets removed from the query plan if it is known that the input is sorted already or will contain only a single distinct value.

One of those optimizations is the TOP N Sort that I would like to look at in a little more detail today.

Memory Requirements for Sorting

Sorting requires memory – potentially a lot of memory. SQL Server estimates the size of the data using statistics and data size estimates and then requests about 150 percent of that in memory for the sort operation.

To see how much memory a query is using, you can use the sys.dm_exec_query_memory_grants DMV. It shows the current memory usage for each SPID that requested additional memory for one (or more) of the memory consuming operators.

The problem with this DMV is that its values change quickly, so it is difficult to get meaningful information for a particular query. There is however a trick you can use: Every memory consuming iterator has a build phase where most of the memory is used, and a delivery phase in which the rows get send to the parent operator. The memory consumption during the build phase is usually a lot higher. SQL Server recognizes this and reuses the additional memory during the delivery phase for other memory consuming iterators.

The sys.dm_exec_query_memory_grants DMV contains the columns used_memory_kb and columns max_used_memory_kb that report current and maximum memory usage. If we can ensure that we get the data from this DMV during the delivery phase of the operator in question, the max_used_memory_kb column will tell us exactly how much memory was used during the build phase. This however only works for queries with a single memory consuming operator that also needs to be blocking.

Sort is a blocking operator. That means that during the build phase no rows get transmitted. If we include the request for the memory data in the query and place it before ("left of" in the graphical plan) the sort, we know that the sort will have finished its build phase when the data gets retrieved from the DMV.

Let us look at an example:

IF OBJECT_ID('dbo.FixedWidth') IS NOT NULL DROP TABLE dbo.FixedWidth;

CREATE TABLE dbo.FixedWidth(
Id INT PRIMARY KEY CLUSTERED,
c1 INT NOT NULL,
fill VARCHAR(1000) NOT NULL
);

INSERT INTO dbo.FixedWidth(Id,c1,fill)
SELECT n,CHECKSUM(NEWID()),REPLICATE('X',192)
FROM dbo.GetNums(10000);

SELECT SUM(DATALENGTH(Id)+DATALENGTH(c1)+DATALENGTH(fill))
FROM dbo.FixedWidth;

This snippet creates the table dbo.FixedWidth and inserts 10000 rows into it. It then sums up the DATALENGTH of all columns over all rows. The size of all the data totals 2,000,000 bytes. It is using Itzik Ben-Gan's GetNums function.

The following query will sort the data in this table and report its memory usage:

SELECT  ( SELECT  *
          FROM    sys.dm_exec_query_memory_grants
          WHERE   session_id = @@SPID
        FOR
          XML PATH('') ,
              TYPE
        ) Mem ,
        *
FROM    dbo.FixedWidth
ORDER BY c1;

This query has below execution plan:

The access to the sys.dm_exec_query_memory_grants DMV happens after the sort has finished. Therefore the max_used_memory_kb will tell us the amount actually used for the sort of the entire 2MB table. The information from the memory DMV is returned as a single XML value. As the DMV result is spooled in this query, all rows will show the same XML value:

66
0
1
1
2012-04-22T19:09:40.303
2012-04-22T19:09:40.307
7352
7352
512
2456
2456
3.460044653270174e+000
86
0
BgA6AB2ceA5AIZDfAAAAAAAAAAAAAAAA
AgAAAB2ceA7MMroMHW//TQYN1cSbf5KW
2
2
0
7352

It shows that 7,352KB is the ideal memory grant size the optimizer came up with. This was also the amount that was granted. However, only 2,456KB where actually used during the sort.

TOP N Sort

If you are interested only in the first few rows based on a sort order, you do not need to complete sort the rest of the rows. To find the top most row it is enough to take the first row of the data set and compare it to each remaining row, keeping each time the one row that comes first according to the sort order. If you have more than one row you can follow the same principle, you just have to keep the first n rows after each step.

SQL Server implements this with the TOP N Sort operator:

SELECT TOP ( 10 )
        *
FROM    dbo.FixedWidth
ORDER BY c1;

This simple query has the following execution plan:

As you can see, the sort operator is of type "TOP N Sort".

You can use the same trick as shown above to get its memory consumption, but you have to add an outer layer, otherwise SQL Server removes the TOP N Sort optimization:

SELECT  ( SELECT  *
          FROM    sys.dm_exec_query_memory_grants
          WHERE   session_id = @@SPID
        FOR
          XML PATH('') ,
              TYPE
        ) Mem ,
        *
FROM    ( SELECT TOP ( 10 )
                  *
          FROM    dbo.FixedWidth
          ORDER BY c1
        ) X;

66
0
1
1
2012-04-22T19:25:18.023
2012-04-22T19:25:18.023
1024
1024
24
24
24
8.388694532701750e-001
25
1
BgA6AM04yitAYc7jAAAAAAAAAAAAAAAA
AgAAAM04yivnP5lrZVMtEy5tLK/6dHjX
2
2
1
32

This query used only 24KB, clearly not enough to hold the entire table, so the TOP N Sort optimization was used.

The algorithms SQL Server uses for sorting are not documented. However, it is clear that an algorithm to find the TOP N rows cannot be as efficient as a normal sort when sorting the same number of rows. For that reason there is a maximum number of rows for which SQL Server uses the TOP N Sort optimization. That number was set to 100.

If you run above query with TOP(100) you will get a memory usage of 128KB:

66
0
1
1
2012-04-22T19:32:38.127
2012-04-22T19:32:38.127
1024
1024
128
128
128
8.624926532701749e-001
25
1
BgA6AKCsNDBAIVXkAAAAAAAAAAAAAAAA
AgAAAKCsNDB6vVhQtTSg6/3xIflWWU1M
2
2
1
136

If you use TOP(101) the memory usage goes up to 2,456KB:

66
0
1
1
2012-04-22T19:32:11.993
2012-04-22T19:32:11.993
7352
7352
512
2456
2456
8.627551332701751e-001
25
0
BgA6AMCfYwlA4WnkAAAAAAAAAAAAAAAA
AgAAAMCfYwlb3oB4gW5K1oLplwLKxsA+
2
2
0
7352

All the memory numbers match the ones from the whole table sort, so SQL Server decided to execute a standard sort followed by a standard TOP operation instead of using the TOP N Sort optimization.

This boundary is fixed and cannot be configured. There is also no hint to force this optimization. That is rather unfortunate as 100 seems to be a quite arbitrary boundary value and a TOP N Sort would often perform a lot better that a full table sort followed by a TOP.

Conclusion

We all know that while the optimizer in general is very good a finding a (close to) optimal plan, sometimes it does not.

The TOP N Sort optimization is a great enhancement, if you are dealing with 100 or less rows. However at 101 rows you will see a sudden drop in performance. To deal with this, you can either restrict the value of rows to return to 100, if the business allows. Or you can help SQL Server to deal with the "all rows" sort. One of the most obvious options here is to add an index to the table.

Optimizer, what if I had more CPUs?

Sebastian — Tue, 27 Mar 2012 14:00:45 +0000

In the March issue of SQL Server Pro Itzik Ben-Gan wrote about an undocumented DBCC command that was recently discovered in the wild by Eladio Rincon (http://www.sqlmag.com/article/sql-server/options-affecting-parallelism-141505). Itzik closed the article by expressing his hope to get this statement documented as it can be very helpfull to database professionals trying to performance tune a query on a none production system, that does not completely match the production environment.

I agree with this opinion, and so I decided to spend some time investigating this functionality further.

Optimizer_WhatIf?

The DBCC command is the OPTIMIZER_WHATIF cammand. DBCC HELP gives you the following information about it:

dbcc OPTIMIZER_WHATIF ({property/cost_number | property_name} [, {integer_value | string_value} ])

This is – according to Google – the only publicly available information about this command. Eladio discovered it in use at a customer site. The customer was told by Microsoft that with the first parameter equal to one, the second would influence parallelism decisions of the optimizer.

After reading Itziks article it seemed clear that the "1" meant "number of schedulers" or "number of CPUs". The above documentation snippet says that you can either specify a number or a name for the first parameter, so knowing the possible non-numeric values for it would be a start in understanding what else this command does.

Finding Strings

Every windows programm has to somehow contain the strings it is using. SQL Server is no exception. A well designed program will have related strings close by each other.

You can use a program like Mark Russinovich's Process Explorer to get a list of all strings in the memory space of a running process.

As all statements (including DBCC commands) are strings, you can expect to find the possible values for the first parameter next to each other. With that thought in mind I went ahead and searched for strings containing "CPU" or "scheduler". I found this list of possible property_name values for SQL Server 2008 and SQL Server 2008 R2:

Status
ResetAll
CPUs
MemoryMBs
Bits
ParallelCardThreshold
ParallelCostThreshold
ParallelCardCrossProd

In SQL Server 2012 these properties were added.

LowCEThresholdFactorBy10
HighCEThresholdFactorBy10
CEThresholdFactorCrossover
DMLChangeThreshold

DBCC OPTIMIZER_WHATIF was available in SQL Server 2005 as well, but "ResetAll" and "Status" are not working, so I did not investigate further.

Warning

All property names are case sensitive, even in a case insensitive installation! You also will not get an error message if an invalid name or value is passed in, so be careful.

Many More CPUs

The third one, "CPUs" causes the functionality described by Itzik: If you run

DBCC OPTIMIZER_WHATIF(CPUs, 16) WITH NO_INFOMSGS;

the optimizer will subsequently optimizer all statements submitted in the same session as if the system had 16 CPUs.

Administrative Help

If you run

DBCC OPTIMIZER_WHATIF(ResetAll) WITH NO_INFOMSGS;

all changes caused by this DBCC command to the current session are undone.

DBCC OPTIMIZER_WHATIF(Status) WITH NO_INFOMSGS;

finally gives you the currently set values and their defaults. It also returns the integer representation of each property:

---------------------------------------------------------

Optimizer what-if status

---------------------------------------------------------

property_number current_value default_value property_name

---------------------------------------------------------

         1                  0             0 CPUs

         2                  0             0 MemoryMBs

         3                  0             0 Bits

         4               1000          1000 ParallelCardThreshold

         5                  1             1 ParallelCostThreshold

         6                200           200 ParallelCardCrossProd

         7                 50            50 LowCEThresholdFactorBy10

         8                 12            12 HighCEThresholdFactorBy10

         9             100000        100000 CEThresholdFactorCrossover

        10                 10            10 DMLChangeThreshold

(As with all undocumented DBCC commands, you first need to run "DBCC TRACEON (3604) WITH NO_INFOMSGS;" in your session to get any output.)

Conclusion

This DBCC command is a great tool, when you are trying to answer questions like: What if the optimizer had more (or less) CPUs available than there are in my current system.

As with all undocumented functionality however, don't use this in production unless advised to do so by Microsoft support.

This post covered only three of the possible values for the first parameter. The other ones will be addressed in future posts.

Key Lookup Operator in Update Statements

Sebastian — Tue, 20 Mar 2012 14:00:39 +0000

When thinking about the indexes for update statements it is easy to overlook an important peculiarity in the way SQL Server finds the rows to update before updating them.

Those pesky Lookups

If you have an UPDATE statement that filters on two columns and you have an index on one of them, you might be tempted to think that the necessary lookup for the second column is not going to cost a lot - particular when that second column is not selective at all - as SQL Server has to access the storage place of each row anyway when executing the actual update. This however is not true, as SQL Server executes the update in two separate phases: One to identify the rows and one to update those rows. That means a lookup in the find phase can be very expensive.

Let us look at an example. Use the following script to create two identical tables and insert 200,000 rows into each of them.


CREATE TABLE dbo.tst1(
 Id INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,
 Val INT,
 Filter INT,
 IdxKey INT,
 Fill CHAR(1000) DEFAULT 'Fill'
);

CREATE TABLE dbo.tst2(
 Id INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,
 Val INT,
 Filter INT,
 IdxKey INT,
 Fill CHAR(1000) DEFAULT 'Fill'
);

INSERT INTO dbo.tst1(Val,Filter,IdxKey)
OUTPUT INSERTED.Val,INSERTED.Filter,INSERTED.IdxKey INTO dbo.tst2(Val,Filter,IdxKey)
SELECT 0,0, n%1000
FROM dbo.GetNums(200000);

The Filter column will be used to force the lookup later on by adding "Filter = 0" to the where clause. It is valued 0 in all rows, so no row will be excluded by this filter.

After creating the tables, create the following indexes:

CREATE INDEX dbo_tst1_IdxKey ON dbo.tst1(IdxKey);
CREATE INDEX dbo_tst2_IdxKey ON dbo.tst2(IdxKey) INCLUDE (Filter);

Both indexes are on the IdxKey column, but the index on the dbo.tst2 table also includes the Filter column.

Now run the following two identical update statements against the two tables:

UPDATE dbo.tst1
  SET Val = 17
WHERE IdxKey <30 AND Filter = 0;

UPDATE dbo.tst2
  SET Val = 17
WHERE IdxKey <30 AND Filter = 0;

The first one requires a lookup operator as the Filter column is not included in the index on dbo.tst1. Below are the two execution plans:

As you can clearly see, the update operator is separated from the data retrieval operator(s) by several other operators. It looks like all the filtering and value preparations happen before the update operator which is then just updating all rows passed to it without any further filtering.

Statistics

I ran the above statements in a loop, each executing 2000 times and recorded time and logical reads. The averaged results are below:

cpu_time	total_elapsed_time	logical_reads	exec_count	command
98	175	37212	2000	UPDATE dbo.tst1 SET Val = 17 WHERE IdxKey <30 AND Filter = 0;
47	72	19211	2000	UPDATE dbo.tst2 SET Val = 17 WHERE IdxKey <30 AND Filter = 0;

The cpu time, the elapsed time and the logical reads were each about twice as high in the query against dbo.tst1, the one requiring the lookup operator. This was the case even though the lookup did not actually filter any rows - every row accessed by the lookup had to be updated as well.

Conclusion

The above statistics prove what the execution plan already suggested: The finding of rows for an update and the update itself are separate steps in SQL Server update queries. That means the "find" portion has to be optimized as if it were a standalone SELECT. In particular, lookup operations can not be ignored as SQL Server accesses the rows twice in this case: Once for the lookup and once for the update.

The Gap in the Identity Value Sequence

Sebastian — Tue, 13 Mar 2012 14:00:41 +0000

With this blog post I am going to dive a little deeper into how SQL Server generates and handles identity values and what the implications for your code are.

Your New Identity

SQL Server manages identity values on a per table basis. Every time a new row gets inserted into a table, the current identity value for that table gets incremented and then used for the new row. You can see this in action by running the following code:

IF OBJECT_ID('dbo.TableWithIdentity') IS NOT NULL DROP TABLE dbo.TableWithIdentity;

CREATE TABLE dbo.TableWithIdentity
(
  Id INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,
  ExecId UNIQUEIDENTIFIER,
  RowId INT
);

SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
INSERT INTO dbo.TableWithIdentity OUTPUT INSERTED.IDENTITYCOL DEFAULT VALUES;
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
INSERT INTO dbo.TableWithIdentity OUTPUT INSERTED.IDENTITYCOL DEFAULT VALUES;
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
INSERT INTO dbo.TableWithIdentity OUTPUT INSERTED.IDENTITYCOL DEFAULT VALUES;
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];

The output of this query looks like this:

For rows two and three you can clearly see that the value got increased first and than used, so IDENT_CURRENT returns the last used identity value for a table. When the table was just created or truncated however, IDENT_CURRENT returns the next value, not the one that was used last. That it actually is the next value you can easily test by setting the start value of the identity column to any number other than one.

Instead of the next value, I would have expected to see a NULL returned to indicate that no identity value had been used so far. However, that is not the case, so you need to be aware of this inconsistency of the IDENT_CURRENT function in you code.

Identity Transactions

Identity values do not participate in any transaction in SQL Server. When a row gets inserted, an identity value gets consumed, even if that insert fails as the following demo shows.

SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
GO
INSERT INTO dbo.TableWithIdentity(RowId)VALUES('NotANumber');
GO
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
GO
INSERT INTO dbo.TableWithIdentity(RowId)VALUES('NotANumber');
GO
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];
GO
INSERT INTO dbo.TableWithIdentity(RowId)VALUES('NotANumber');
GO
SELECT IDENT_CURRENT('dbo.TableWithIdentity') AS [IDENT_CURRENT];

The output will look something like this.

No new row made it into the table because of the conversion errors, but the identity value got increased anyway. The reason for this is simple: To implement the identity value management transactional, a session would need to take a lock on the identity value itself for the duration of the transaction. That would force all inserts to be executed consecutively and concurrency would be practically non-existent.

There are many reasons for which an insert could fail. Examples are an illegal value as in the above example, a deadlock, a full disk drive and so on. Some of the reasons you can control by cleaning your values prior to using them, others cannot be controlled in the inserting code at all. So you really cannot assume, that you will have a gap free list of identity values in your table.

Sequence Chasms

SQL Server does not even guarantee that the identity values used by a single insert are contiguous. To prove that run the following piece of SQL code in multiple connections at the same time.

DECLARE @ExecId UNIQUEIDENTIFIER = NEWID();
INSERT INTO dbo.TableWithIdentity(ExecId,RowId)
SELECT @ExecId,n FROM dbo.GetNums(10000);

As most of my examples, this one is also using Itzik's GetNums function that you can get here: Virtual Auxiliary Table of Numbers.

The easiest way to run multiple executions of this query at the same time is to use SQL Query Stress by Adam Machanic as shown below.

After SQL Query Stress finishes execution, run the following query.

SELECT  B.Id B_Id ,
        A.Id A_Id ,
        A.Id - B.Id Gap ,
        B.RowId B_RowId ,
        A.RowId A_RowId ,
        A.ExecId
FROM    dbo.TableWithIdentity A
        JOIN dbo.TableWithIdentity B ON A.ExecId = B.ExecId
                                        AND A.RowId = B.RowId + 1
WHERE   A.Id - B.Id <> 1
ORDER BY A.ExecId , B_Id;

This query uses a self-join on the TableWithIdentity table to return a list of all rows that have a gap between their identity value and the identity value of the preceding row of the same execution. On my system it returned about 20,000 gaps (for 200,000 inserted rows in total) with sizes from 2 to 16,000.

This example has shown that even in single inserts you cannot rely on getting contiguous identity values

Replication

One final way to get gaps in your identity stream that I would like to bring up is replication. In all replication scenarios where data could flow in more that one direction, SQL Server has to manage identity ranges to prevent conflicts (see http://technet.microsoft.com/en-us/library/ms152543.aspx). For this, SQL Server assigns a fixed interval (per article) of identity values to each participant in the replication setup. If a participant runs out of values, a new range is automatically assigned to that participant. That new range will not be adjacent to the previous one in almost all cases, causing gaps in the list of identity values.

That means that even if you have a system that only ever inserts s single row at a time, has only a single connection, and never runs into an issue like a bug or a hard drive failure you still can end up with gaps in your identity value stream, if someone decides to add replication into the mix.

Conclusion

The way identity values are handled by SQL Server, there is no guarantee that the stream of identity values for a single table is free of gaps. Even if you think you have covered all you bases, you still might end up with gaps later on, if the business suddenly requires replication to be added into the mix. Therefor it is a best practice to never rely on contiguous identity values anywhere in your code.

The Story of the Evil Assert Operator

Sebastian — Tue, 06 Mar 2012 15:00:24 +0000

Today I am going to talk about the evil side of the Assert Operator. To start out, let me introduce this operator to you:

The Assert Operator

The Assert Operator is used by SQL Server in execution plans that require some condition to be satisfied to be valid. A common scenario in which an Assert Operator is used is an insert into a table that has a check constraint defined. Take for example the dbo.TableWithCheckConstraint table:

CREATE TABLE dbo.TableWithCheckConstraint(
  Id INT PRIMARY KEY CLUSTERED,
  Even INT CONSTRAINT TableWithCheckConstraint_Even_Check CHECK (Even % 2 = 0)
);

If you run this simple insert against that table

INSERT INTO dbo.TableWithCheckConstraint(Id,Even)
SELECT n, 2*n FROM (VALUES(1),(2))X(n);

you will get an execution plan that looks like this:

The properties of the highlighted Assert Operator are shown below:

Asserts in general are very cheap operators as you can see when you look at the Estimated Operator Cost in above image. An Assert operator works by calculating a given expression. If that expression results in anything other than NULL, the assert fails and stops the execution of the current statement. The expression in above example is the formula we used for the check constraint itself, wrapped in a case statement to return either NULL or 0.

The Dark Side of the Assert

So, how can such an innocent little operator be evil?

Sometimes SQL Server requires a certain condition to be satisfied in situations where you do not necessarily expect it. The most common example is a sub-query in the column list or the WHERE clause of a select statement as shown in the following example:

SELECT  ( SELECT  m
          FROM    (VALUES (1), (2) ) X(m)
        );

If you execute this query you get this error:

Msg 512, Level 16, State 1, Line 1
Subquery returned more than 1 value. This is not permitted when the subquery follows =, !=, <, <= , >, >= or when the subquery is used as an expression.

To enforce that rule, SQL Server is using an Assert Operator again:

In above example the Stream Aggregate Operator is tasked with counting the rows returned:

[Expr1005] = Scalar Operator(Count(*))

The Assert then makes sure that this count is at max one with the following predicate:

CASE WHEN [Expr1005]>(1) THEN (0) ELSE NULL END

(If you look more closely you will find that the Stream Aggregate actually calculates two aggregates in this query and that the second one is of a surprising type. Check out this article by Paul White for an explanation.)

In this example the Stream Aggregate and the Assert make up together 50% of the (estimated) query cost, but as this is a very made-up example I would not pay to much attention to that. So let us look at a real life example.

The Forgotten Uniqueness

First we need a few tables:

CREATE TABLE dbo.AssertBigFact(
 Id INT PRIMARY KEY CLUSTERED,
 Sort1 INT UNIQUE,
 Luv INT,
 Data CHAR(1000)
);

DECLARE @BigSize INT = 1000000;
DECLARE @LookupSize INT = 1000;

INSERT INTO dbo.AssertBigFact(Id,Sort1,Luv,Data)
SELECT n,(n*1123)%@BigSize+1,n%@LookupSize+1,'' FROM dbo.GetNums(@BigSize)


CREATE TABLE dbo.AssertLookupNonUnique(
 Id INT ,
 Val INT
);
CREATE CLUSTERED INDEX AssertLookupNonUnique_CI ON dbo.AssertLookupNonUnique(Id);

CREATE TABLE dbo.AssertLookupUnique(
 Id INT ,
 Val INT
);
CREATE UNIQUE CLUSTERED INDEX AssertLookupUnique_CI ON dbo.AssertLookupUnique(Id);

INSERT 
  INTO dbo.AssertLookupNonUnique(Id,Val)
OUTPUT INSERTED.Id,INSERTED.Val
  INTO dbo.AssertLookupUnique(Id,Val)
SELECT n,n%37 FROM dbo.GetNums(@LookupSize);

As most of my examples, this one is also using Itzik's GetNums function that you can get here: Virtual Auxiliary Table of Numbers.

It creates three tables, the table dbo.AssertBigFact that contains 1,000,000 rows and the two small 1,000 row lookup tables dbo.AssertLookupUnique and dbo.AssertLookupNonUnique. The insert statement uses the OUTPUT clause to insert the same rows in both tables, so with the exception of the clustered index on the dbo.AssertLookupNonUnique table that got "accidentally" not declared as unique, these two tables are identical.

Now let us compare the following two almost identical queries. They only differ in the lookup table they go against.

SET STATISTICS IO ON;
SET STATISTICS TIME ON;
GO
SELECT b.*,(SELECT Val FROM dbo.AssertLookupNonUnique l WHERE l.Id = b.Luv) Val
FROM dbo.AssertBigFact b
ORDER BY b.Sort1,Val;
GO
SELECT b.*,(SELECT Val FROM dbo.AssertLookupUnique l WHERE l.Id = b.Luv) Val
FROM dbo.AssertBigFact b
ORDER BY b.Sort1,Val;
GO
SET STATISTICS TIME OFF;
SET STATISTICS IO OFF;

The execution plans for above queries look like this:

The top version, going against the non-unique table, requires the use of an Assert Operator. The second query on the other end can get away without in, as SQL Server knows that the sub-query can return only a single row because the Id column was declared unique in the definition of the clustered index.

In a scenario like this where a rather large table gets joined with a small table, using a hash join is usually the fastest way for SQL Server to execute the query. A hash join requires the entire first table to be stored in a hash table in memory before any row can be returned. A hash can spill to tempdb if there is not enough free memory available, but such a spill is expensive. So hash joins are perfect in a scenario where one of the two join partners is small.

In our case SQL Server chose the hash join for the second query, as expected. However the first query uses a loop join. The reason for this is quite simple: SQL Server does not know that the Id column in the dbo.AssertLookupNonUnique table is unique, so it needs to assert that for each Luv value in the dbo.AssertBigFact table there is only one row in the dbo.AssertLookupNonUnique table with a matching Id value. For that it needs to read the dbo.AssertBigFact table first, as there might be id values in the dbo.AssertLookupNonUnique table that do not have a matching Luv value in the dbo.AssertBigFact table. If those Id values were duplicated it would not matter for this query, so the Assert can run only against Id values with a matching Luv value. That however means, that SQL Server can't build the hash table first, at least not with the small dbo.AssertLookupNonUnique table. But it doesn't make sense to build the hash table using the dbo.AssertBigFact table as that would take to much time and memory and would therefore be rather ineffective.

A merge join is not a good choice either, as it requires both inputs to be sorted on the join column which is not the case. SQL Server could insert a sort before the join, but that would mean that the full one million rows had to be sorted twice as a different sort order is requested by the query – a scenario worth avoiding.

That means that SQL Server has to use the loop join algorithm. That means it has to assert that there is only a single row in the dbo.AssertLookupNonUnique for each row in the dbo.AssertBigFact table which in turn means that the Stream Aggregate and Assert Operator pair gets executed 1,000,000 times. While each execution is small and cheap, doing a million of them is not. But this is not even the worst problem with this query.

Wasted Logical Reads

If we look at the logical reads (returned by the SET STATISTICS IO ON statement in above batch) for the query against the dbo.AssertLookupUnique we get these numbers:

Table 'AssertLookupUnique'. Scan count 3, logical reads 10, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'AssertBigFact'. Scan count 3, logical reads 143712, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

The dbo.AssertLookupUnique has 3 pages as a quick check against sys.dm_db_index_physical_stats confirms:

SELECT  OBJECT_SCHEMA_NAME(i.object_id) + '.' + OBJECT_NAME(i.object_id) Tbl ,
        i.name ,
        ips.index_depth ,
        ips.page_count
FROM    sys.indexes i
        JOIN sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) ips 
        ON ips.object_id = i.object_id AND ips.index_id = i.index_id
WHERE   i.index_id < 2
        AND i.object_id IN ( OBJECT_ID('dbo.AssertLookupNonUnique'),
                             OBJECT_ID('dbo.AssertLookupUnique'),
                             OBJECT_ID('dbo.AssertBigFact') );

This query returns these numbers:

The 10 logical reads are caused by the parallelism of this query. The different threads of a single query often read pages that another thread has looked at before. The same behavior you can see with the dbo.AssertBigFact table where also a few of the 142858 pages got read more than once.

If we now look at the numbers for the query against the dbo.AssertLookupNonUnique table we get a quite different picture:

Table 'AssertLookupNonUnique'. Scan count 1000000, logical reads 2004000, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'AssertBigFact'. Scan count 3, logical reads 143760, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

While the number of reads against the dbo.AssertBigFact table is about the same as before, there are over 2 million logical reads recorded for the dbo.AssertLookupNonUnique table. — What a difference...

The seek against the dbo.AssertLookupNonUnique was executed one million times, once for each row in the dbo.AssertBigFact table. As the index depth is 2 levels, each seek had to look at two pages – two million in total.

Conclusion

As you have seen in the above example, requiring an assert can be quite expensive. While the Assert Operator is not that expensive in it self, SQL Server is severely restricted in the plan choices it can make when an assert is logically required.

To avoid situations like this you should always try to give the Optimizer as much information as possible. That includes keeping statistics up to date and using proper indexes. But, as this article has shown, creating an index on the right columns is not necessarily enough. So, if you have an index on a column with unique values, make sure to declare it as unique.

Why CXPACKET Waits are not your Performance Problem

Sebastian — Tue, 28 Feb 2012 15:00:13 +0000

There is a lot of information out there recommending to do this or that to deal with the performance problems that you must have if you see CXPACKET waits. All these recommendations build on the notion that the CXPACKET wait is a bad thing. Today I am going to explain why this is not the case. I am going to show you that the CXPACKETs are just one part of how the SQL Server engine works and that they do not indicate any problem.

Parallel Processing

First we need to dive into, how parallel processing of requests in SQL Server works. Every time SQL Server executes a query it does so using its execution plan. You can think of an execution plan as the compiled executable for the query. In this post I am not going to explain how SQL Server produces this plan, but if you want more detail I can recommend the book Inside the SQL Server Query Optimizer by Benjamin Nevarez (blog|twitter).

Under specific circumstances SQL Server can decide to produce a parallel plan. You can recognize a parallel plan by the yellow circles with two arrows on them that show up on some of the icons in the graphical plan:

You will also find at least one of the parallelism iterators:

They are called "Gather Streams", "Repartition Streams" and "Distribute Steams", and as they are always executed in parallel mode, they always appear with the yellow circle on them.

A parallelism iterator acts as thread control point. It takes a request from one ore more threads and passes it on, again to one ore more new threads. When the data is coming back, the iterator collects it and distributes it to the requesting threads. I am again not going to go into the details here. A good explanation can be found in Paul Whites(blog|twitter) article on simple talk: Understanding and Using Parallelism in SQL Server

I would like to call your attention to the word "new" in the above paragraph. Each icon in an execution plan represents an iterator and execution always starts with the left most iterator. It is the "SELECT" iterator in the above example. Each iterator requests rows from it's direct child iterators and then does some work with those rows. When a parallelism iterator is invoked it spawns n new threads and passes control to all of them. The number n is based on the type of the parallelism iterator and the current Degree Of Parallelism settings.

The original threads, after passing control to the newly spawned threads wait to receive rows back. This waiting is not done by spinning or polling. Instead the thread(s) enter a wait state and wake bake up once a row is available for processing. This allows the processor to be used for other worker threads.

The wait type that SQL Server reports for those threads waiting at a parallelism iterator is - you guessed it - the CXPACKET wait.

An Example

Let's look at an example:

IF OBJECT_ID('dbo.t1') IS NOT NULL 
  DROP TABLE dbo.t1 ;

CREATE TABLE dbo.t1
  (
    i INT ,
    t TEXT ,
    c1 INT DEFAULT CHECKSUM(NEWID()) ,
    c2 INT DEFAULT CHECKSUM(NEWID())
  ) ;

INSERT  INTO dbo.t1
        ( i ,
          t
        )
        SELECT  n ,
                REPLICATE('X', 200 + ( CHECKSUM(NEWID()) % 10 ))
        FROM    dbo.GetNums(1000000) ;

This creates the table "t1" and fills it with 1.000.000 strings of random length between 200 and 210 characters. The code uses Itzik Ben-Gan's GetNum function to quickly create the one million rows.

Now run this select statement against that table:

SELECT ln, COUNT(1) cnt FROM(SELECT LEN(CAST(t AS NVARCHAR(MAX))) ln FROM dbo.t1) X GROUP BY ln;

Let's call this the "Count Query". It has the execution plan that was introduced earlier.

While this query runs, execute the following "Monitor Query" in a new window. Make sure to change the session_id variable to the SPID of the window executing the Count Query.

DECLARE @session_id INT = 75;
SELECT  t.task_address ,
        t.task_state ,
        t.session_id ,
        t.exec_context_id ,
        wt.wait_duration_ms ,
        wt.wait_type ,
        wt.blocking_session_id ,
        wt.blocking_exec_context_id ,
        wt.resource_description ,
        t.scheduler_id
FROM    sys.dm_os_tasks t
        LEFT JOIN sys.dm_os_waiting_tasks wt ON t.task_address = wt.waiting_task_address
WHERE   t.session_id = @session_id
ORDER BY t.exec_context_id,wt.blocking_exec_context_id;

This query returns one row for every worker thread currently assigned to this query. A thread that is currently blocked might get more than one row, if it is blocked by more than one other thread. For each waiting thread it also shows what it is waiting for. After about 10 seconds of execution of the Count Query, the Monitor Query returned this result on my machine:

As you can see, there are five different execution_context_ids. Each thread on a parallel query gets its own execution context. The first one, the thread the the query execution started on has always the id 0. The other ones get increasing consecutive numbers.

Thread 0 in above query covers everything up to the "gather Streams" iterator. From there to the "Repartition Streams" iterator threads 1 and 2 get to work in parallel. After that up on to the table scans the query is executed by threads 3 and 4. Those two are busily shoveling rows into the Hash iterator. Because a Hash is a blocking iterator (see Properties of Iterators by Craig Freedman(blog) for an explanation of this term), it will not produce any rows until it has consumed all incoming rows. This takes a little over 12 seconds on my machine. During this time the threads 0,1 and 2 are suspended, not taking up any processor resources and therefore not hurting any other thread or query. But they each crank up close to 12 seconds of CXPACKET wait time, for a total of 36 seconds.

Conclusion

The above example showed that any parallel query will show potentially very high CXPACKET waits. This is not an indication of a problem, but rather it is an indication that SQL Server is doing it's job properly, making sure that threads that do not have anything to do at the moment make room for other threads on the processors. So, if your system shows high CXPACKET waits, the only conclusion you can draw from that is, that it is using parallelism and that SQL Server's multitasking system is working as designed.

However, as the sys.dm_exec_requests (as well as the compatibility view sysprocesses) show only the wait type of thread 0, CXPACKET waits can potentially hide a real problem. If you know that you have a problem, use appropriate techniques to narrow down the source. If you don't know if you have a problem, CXPACKET will not help you to change that.

That means, that in almost all situations it is safe to ignore CXPACKET waits. I therefore recommend to add it to the list of waits to ignore in your monitoring queries.

Deferred Deallocations of Pages in TempDb

Sebastian — Sun, 19 Feb 2012 15:00:53 +0000

The other day I ran into a question on msdn:Tempdb Deallocation (dm_db_session_space_usage). A person with the pseudonym DontPageMeBro noticed an odd behavior in his SQL Server installation: The sum of the reported usage of tempdb across all sessions was often bigger than tempdb itself. Something is clearly off here, so I figured it was worth digging a little deeper.

TempDb Usage

First let us look at how to get information about tempdb usage. SQL Server provides two DMVs that return information about the space used in tempdb:

sys.dm_db_session_space_usage

and

sys.dm_db_task_space_usage

sys.dm_db_task_space_usage reports the amount of pages taken and released by a task currently running. If a request uses parallelism you might see more than one row for that request returned by this DMV. After each task finishes, it's numbers get rolled up into the sys.dm_db_session_space_usage. So here you can see how much space in tempdb a request used in total, but this information is only available after the request is completed.

Both DMVs report the number of pages in four columns:

user_objects_alloc_page_count, 
user_objects_dealloc_page_count, 
internal_objects_alloc_page_count, 
internal_objects_dealloc_page_count

The two "user" columns report on spaced used by user created objects like #temptables. The two "internal" columns report on objects SQL Server decided to create to support the execution of a query. Both pair are made up of a "alloc" and a "dealloc" page count column. The former is the total count of all pages that where allocated in the scope of the DMV whereas the latter contains the total count of all deallocated pages.

So, to get the current user object space allocation in tempdb for a given session, you need to subtract the "user_objects_dealloc_page_count" from the "user_objects_alloc_page_count" in both DMVs and then SUM all those values together. The following query shows how to do that for user as well as for internal space used:

SELECT  session_id ,
        SUM(user_objects_alloc_page_count - user_objects_dealloc_page_count) CurrentUserPages,
        SUM(internal_objects_alloc_page_count - internal_objects_dealloc_page_count) CurrentInternalPages
FROM    ( SELECT  session_id ,
                  user_objects_alloc_page_count ,
                  user_objects_dealloc_page_count ,
                  internal_objects_alloc_page_count ,
                  internal_objects_dealloc_page_count
          FROM    sys.dm_db_session_space_usage
          UNION ALL
          SELECT  session_id ,
                  user_objects_alloc_page_count ,
                  user_objects_dealloc_page_count ,
                  internal_objects_alloc_page_count ,
                  internal_objects_dealloc_page_count
          FROM    sys.dm_db_task_space_usage
        ) X
GROUP BY X.session_id
ORDER BY X.session_id ASC ;

DontPageMeBro is dealing with the problem that some users create large #temptables and then leave the session open without dropping those tables. If you have a lot of users like that your tempdb might grow to a significant size. Because it is always advisable to have tempdb on very fast storage, this problem can mean a significant cost to the business, so it makes sense to follow up and see how big the problem really is. The sys.dm_db_session_space_usage can give us that information. Every time there is a session that has more allocated than deallocated pages, that session is holding on to a temporary object. If the last request for that session has finished a while ago, that means, that space is not actually in use and those objects should have been dropped.

Counting Inaccuracies

So far the theory. However, as DontPageMeBro noticed, if a #temptable is big it seems that SQL Server sometimes forgets to count the page deallocation. So, even though the #temptable is gone, the pages are still being counted as in use by the session that created the table.

To understand what is going on here, we need to look at how SQL Server handles page deallocations after dropping an object.

Each Table consists of at least one allocation_unit. An allocation_unit contains for example all the data pages for a given table or index. There might also be additional allocation_units holding special pages for e.g. row overflow data. Every index in turn has it's own set of allocation_units.

When dropping an object, all allocation_units belonging to that object are deallocated and all their pages are marked as free. This can take quite some time for big objects and usually this cleanup work is done synchronously, which means that the session issuing the drop request has to wait. However if the allocation unit contains more than 1024 pages, this cleanup work is deferred. The allocation_unit is marked as "DROPPED" and the link to the table is removed at the time the drop statement executes. The actual work of freeing up all those pages is then done by a cleanup process later on. This process is the same in all databases, not only tempdb.

Lets look at an example to demonstrate this:

CREATE TABLE #temp
  (
    id INT IDENTITY ,
    name VARCHAR(100) 
  );
GO
SELECT  allocation_unit_id
INTO    #aus
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN tempdb.sys.system_internals_partitions p ON u.container_id = p.partition_id
WHERE   p.object_id = OBJECT_ID('tempdb..#temp');
GO
INSERT  INTO #temp
        SELECT TOP ( 328168 )
                'FillText'
        FROM    sys.system_internals_partition_columns a ,
                sys.system_internals_partition_columns;
GO
SELECT  'A' A , *
FROM    sys.dm_db_session_space_usage
WHERE   session_id = @@SPID;

SELECT  'A' A , u.* , p.rows
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN tempdb.sys.system_internals_partitions p ON u.container_id = p.partition_id
        JOIN #aus t ON u.allocation_unit_id = t.allocation_unit_id;
GO
DROP TABLE #temp;
GO
SELECT  'B' B , *
FROM    sys.dm_db_session_space_usage
WHERE   session_id = @@SPID;

SELECT  'B' B , u.*
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN #aus t ON u.allocation_unit_id = t.allocation_unit_id;
IF ( @@ROWCOUNT > 0 ) 
  BEGIN
    WAITFOR DELAY '00:00:06' ;
    SELECT  'C' C , *
    FROM    sys.dm_db_session_space_usage
    WHERE   session_id = @@SPID;
    
    SELECT  'C' C , *
    FROM    tempdb.sys.system_internals_allocation_units u
            JOIN #aus t ON u.allocation_unit_id = t.allocation_unit_id;
  END ;
GO

DROP TABLE #aus;

This script creates the table #temp, fills it with a few thousand rows and then drops it again. If you run this script you will get the following output back:

The first result-set shows that the current session has 1019 pages currently reserved in tempdb. The second result-set tells us that the table #temp contains 328168 rows that are stored in a single allocation unit of type "IN_ROW_DATA" with 1018 pages.

The third and forth result-set were captured after the table was dropped. Here we can see that 1017 pages were deallocated in tempdb for the current connection and that the allocation_unit is now gone. (That not all pages of #temp show as being dropped in the count is due to another optimization that is not part of today's discussion.)

If we change the number of rows inserted (line 14 in above script) from 328168 to 328169 we get this output instead:

This time the allocation_unit has 1026 pages, so we would expect the deferred drop to take place. Result-set 4 now shows, directly after the drop table took place, that the allocation_unit is still there, but it's type changed to "DROPPED". The link to the table it once belonged to is also removed, as is visible in the container_id column.

If the above script sees that the allocation_unit still exists after the drop (see line 37) it waits for a few seconds and then looks again. In result-set 6 you can now see that the allocation_unit got removed. However the current session still shows 1027 allocated and no deallocated pages (result-set 5).

Cleaning Up

To see the cleanup process in action you can use the following script:

CREATE TABLE #temp
  (
    id INT IDENTITY ,
    name VARCHAR(100) 
  );
GO
SELECT  allocation_unit_id
INTO    #aus
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN tempdb.sys.system_internals_partitions p ON u.container_id = p.partition_id
WHERE   p.object_id = OBJECT_ID('tempdb..#temp');
GO
INSERT  INTO #temp
        SELECT TOP ( 328169 )
                'FillText'
        FROM    sys.system_internals_partition_columns a ,
                sys.system_internals_partition_columns;
GO
SELECT  'A' A , *
FROM    sys.dm_db_session_space_usage
WHERE   session_id = @@SPID;

SELECT  'A' A , u.* , p.rows
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN tempdb.sys.system_internals_partitions p ON u.container_id = p.partition_id
        JOIN #aus t ON u.allocation_unit_id = t.allocation_unit_id;
GO
DROP TABLE #temp;

SELECT  IDENTITY(INT,1,1) Id,SYSDATETIME()now, u.allocation_unit_id , data_pages , type_desc
INTO    #siau
FROM    tempdb.sys.system_internals_allocation_units u
        JOIN #aus t ON u.allocation_unit_id = t.allocation_unit_id ;

SELECT IDENTITY(INT,1,1) Id,SYSDATETIME()now, *
INTO #ssu
 FROM  sys.dm_db_session_space_usage 
WHERE user_objects_dealloc_page_count>0 OR session_id = @@SPID
ORDER BY session_id;

SELECT IDENTITY(INT,1,1) Id,SYSDATETIME()now, tsu.*,r.status,r.command
INTO #tsu
 FROM  sys.dm_db_task_space_usage tsu
 JOIN sys.dm_exec_requests r
 ON tsu.session_id = r.session_id
WHERE user_objects_dealloc_page_count>0
ORDER BY session_id;

DECLARE @AuId BIGINT = (SELECT TOP(1) allocation_unit_id FROM #aus);
DECLARE @Stop DATETIME2 = DATEADD(SECOND,10,SYSDATETIME());
WHILE (SYSDATETIME()<@Stop)
BEGIN
  INSERT   INTO    #siau
  SELECT  SYSDATETIME()now, u.allocation_unit_id , data_pages , type_desc
  FROM    tempdb.sys.system_internals_allocation_units u
  WHERE u.allocation_unit_id = @AuId;

  INSERT INTO #ssu
  SELECT SYSDATETIME()now, *
   FROM  sys.dm_db_session_space_usage 
  WHERE user_objects_dealloc_page_count>0 OR session_id = @@SPID
  ORDER BY session_id;

  INSERT INTO #tsu
  SELECT SYSDATETIME()now, tsu.*,r.status,r.command
   FROM  sys.dm_db_task_space_usage tsu
   JOIN sys.dm_exec_requests r
   ON tsu.session_id = r.session_id
   AND tsu.request_id = r.request_id
  WHERE user_objects_dealloc_page_count>0
  ORDER BY session_id;
END

SELECT MIN(now) MinNow,MAX(now) MaxNow,type_desc,data_pages,allocation_unit_id
FROM #siau
GROUP BY type_desc,data_pages,allocation_unit_id
ORDER BY MIN(Id) ASC;

SELECT MIN(now) MinNow,MAX(now) MaxNow,session_id,database_id,user_objects_alloc_page_count,user_objects_dealloc_page_count,internal_objects_alloc_page_count,internal_objects_dealloc_page_count 
FROM #ssu 
GROUP BY session_id,database_id,user_objects_alloc_page_count,user_objects_dealloc_page_count,internal_objects_alloc_page_count,internal_objects_dealloc_page_count
ORDER BY session_id ASC, MIN(Id) ASC;

SELECT MIN(now) MinNow,MAX(now) MaxNow,status,command,session_id,request_id,exec_context_id,database_id,user_objects_alloc_page_count,user_objects_dealloc_page_count,internal_objects_alloc_page_count,internal_objects_dealloc_page_count
FROM #tsu 
GROUP BY status,command,session_id,request_id,exec_context_id,database_id,user_objects_alloc_page_count,user_objects_dealloc_page_count,internal_objects_alloc_page_count,internal_objects_dealloc_page_count
ORDER BY session_id ASC, MIN(Id) ASC;

GO
DROP TABLE #ssu;
DROP TABLE #tsu;
DROP TABLE #aus;
DROP TABLE #siau;

This script again creates the same table #temp that was used by the previous script, fills it and drops it. After dropping the table it records for ten seconds information from the allocation_unit DMV and the two space_usage DMVs. After those ten seconds it produces this output:

The first two result-sets show us the same information about the table that we have seen before. The other three result-sets contain grouped information of the three recorded DMVs. Each is grouped by all columns but the time of capture. That means, each row shows when those values where visible first and last, so you can see the progression of events.

The third result-set tells us that right after dropping the table the allocation_unit was marked as dropped. It at this point also shows a size of 0 pages. Just a little under a second later, at 17:18:06.1417576, the allocation_unit was seen last. At 17:18:06.1387573, just barely a millisecond after dropping the table, session 8 suddenly appears and starts to deallocate pages – pages it clearly did not allocate – until it, at 17:18:06.1417576, has deallocated all 1025 pages that belonged to our allocation_unit. That is exactly the same time when the allocation_unit itself was seen last. That this process started right after dropping the table is a coincidence and you might see several seconds pass before it starts, when you try this out yourself.

During the whole time of the recording from the table drop to ten seconds later, the information returned by sys.dm_db_session_space_usage DMV did not change at all. So, while the cleanup process can be seen to deallocate pages, that information never gets rolled up into the session space DMV. The session that originally created the table also never gets updated.

Versions

I have tested this in the following versions of SQL Server:

Microsoft SQL Server 2005 - 9.00.5057.00 (X64)
Microsoft SQL Server 2008 (SP3) - 10.0.5500.0 (X64)
Microsoft SQL Server 2008 R2 (SP1) - 10.50.2500.0 (X64)
Microsoft SQL Server 2012 RC0 - 11.0.1750.32 (X64)

All SQL Server versions listed above show the same behavior.

Conclusion

I consider this behavior a bug in SQL Server. As the last example showed, the allocation_unit gets cleaned up by a background process. At that point the link to the original table and with it the originating session is gone, so the background process cannot update the session space usage information. However, right at the time of the drop, when the allocation_unit is marked as DROPPED, it also shows a size of 0 pages. At this point the task space usage information for the request executing the drop should have been updated to also reflect the drop. The pages could at the same time appear in the allocated column of one of the "TASK MANAGER" requests that execute the background cleanup tasks to make sure that the total of all allocated and deallocated pages in those two DMVs always matches the current state of tempdb.

I created a Connect item with Microsoft for this behavior here:. If you agree that this should be addressed, please go there and vote for it.

The Performance Impact of Forwarded Records on Table Scans

Sebastian — Tue, 31 Jan 2012 16:00:18 +0000

Forwarded records are a mechanism that SQL Server employs to reduce the amount of maintenance work for indexes on a table without a clustered index. During an update of a record in a table that does not have a clustered index, an increase in size might prevent the record to fit on its current page. If that happens, a forwarding pointer is created in the current page and the record is moved to a new location.

It is well known that such forwarded records will cause each RID-Lookup operation that encounters one to have to read a second page. If you are doing a filtered index scan followed by an RID-Lookup this can potentially double the amount of work necessary to retrieve all rows.

What is less well known is the fact, that such forwarded records can significantly impact the performance of a simple table scan as well. In this article I will show you why this happens.

Terminology

To explain the way SQL Server handles forwarded records during a table scan, I am first going to rehash some storage basics.

SQL Server knows two ways to organize the data in a table on disk. One is the Clustered Index and the other one the Heap. In a Clustered Index data is organized in a B+Tree structure allowing for fast access based on the Clustered Index Key. In a Heap on the other hand the data is not sorted or linked in any way. Any new row is inserted into the table in any free space big enough to hold the row.

In both cases the smallest unit of storage and access is a Page. Every piece of data SQL Server needs to store on disk is organized in blocks called pages. Each page is exactly 8192 Bytes in size. The only exception to this rule are the log files. Log data is stored in a structure called virtual log file and those are not part of today's discussion.

Indexes created on a Heap use a physical address to point from the index record to the underlying data record. This address consists of the file number, the number of the page that contains the record within that file and the slot number inside that page.

Forwarded Records

When a record with variable length data type columns is updated, its size might increase. If the page it is stored in is filled already, the updated record might need to be moved to a new page. If that happens, all pointers in any index on that table addressing the moved row now point to the wrong (old) place. There are two ways to handle this situation: Update all indexes with the new location or put a special record in the old place that "forwards" to the new location.

Going through all existing indexes can be extensively resource intensive. For that reason SQL Server is using the second option to put a forwarding record in the old place.

Such a forwarding record links to the new physical address of the actual data. The new record also contains a pointer back to the forwarding record. This back-pointer allows the forwarding record to get updated in case the data record has to move again or in case it gets deleted which causes the forwarding record to be deleted as well. Updating the forwarding records in cases of repeated data record moves prevents long forwarding chains. Instead there is always only at max on hop.

In the case of an RID-Lookup SQL Server has to follow those forwarding records to get to the actual data record. This causes an additional (logical) page read for each forwarding record encountered.

During a table scan on the other hand each page will be read anyway so SQL Server could just ignore all forwarding records. That however could cause reading inconsistencies. Rows that are updated after the scan started and before it finishes could be missed entirely by the scan, if the record gets moved to a page that was scanned already. For that reason SQL Server follows each forwarding record immediately. This not only causes an additional read right there, is also turns the sequential read of all pages in allocation order into a succession of random reads, hurting performance potentially even more.

Demonstration

To demonstrate this behavior let us set up two tables dbo.ForwardSmall and dbo.NoForwardSmall as well as a Procedure dbo.ChangeRow to help with the creation of forwarded records:

IF OBJECT_ID('dbo.ForwardSmall') IS NOT NULL DROP TABLE dbo.ForwardSmall;
CREATE TABLE dbo.ForwardSmall(Id INT ,F VARCHAR(8000))
GO
IF OBJECT_ID('dbo.NoForwardSmall') IS NOT NULL DROP TABLE dbo.NoForwardSmall;
CREATE TABLE dbo.NoForwardSmall(Id INT ,F VARCHAR(8000))
GO

IF OBJECT_ID('dbo.ChangeRow') IS NOT NULL DROP PROCEDURE dbo.ChangeRow;
GO
CREATE PROCEDURE dbo.ChangeRow
@Id INT,
@Size INT
AS
BEGIN
  SET NOCOUNT ON;
  MERGE dbo.ForwardSmall fs
  USING (SELECT @Id,REPLICATE('X',@Size)) x(Id,F)
  ON fs.Id = x.Id
  WHEN MATCHED THEN
    UPDATE SET fs.F = x.F
  WHEN NOT MATCHED THEN
    INSERT (Id,F)VALUES(x.Id,x.F);
END
GO

The procedure will take an Id parameter as well as a size parameter. It uses to merge command to either insert or update the record with the given Id value. The F column will be valued with a string of length @Size.

The following script will create 16 rows in the table dbo.ForwardSmall and then update those records to create forwarded records.

EXEC dbo.ChangeRow 11,900;
EXEC dbo.ChangeRow 12,900;
EXEC dbo.ChangeRow 13,900;
EXEC dbo.ChangeRow 14,900;
EXEC dbo.ChangeRow 15,900;
EXEC dbo.ChangeRow 16,900;
EXEC dbo.ChangeRow 17,900;
EXEC dbo.ChangeRow 18,900;

EXEC dbo.ChangeRow 21,900;
EXEC dbo.ChangeRow 22,900;
EXEC dbo.ChangeRow 23,900;
EXEC dbo.ChangeRow 24,900;
EXEC dbo.ChangeRow 25,900;
EXEC dbo.ChangeRow 26,900;
EXEC dbo.ChangeRow 27,900;
EXEC dbo.ChangeRow 28,900;
--------------------------
EXEC dbo.ChangeRow 27,0;
EXEC dbo.ChangeRow 28,0;
EXEC dbo.ChangeRow 18,1800;
EXEC dbo.ChangeRow 28,900;
EXEC dbo.ChangeRow 18,900;

EXEC dbo.ChangeRow 26,0;
EXEC dbo.ChangeRow 17,1800;
EXEC dbo.ChangeRow 27,900;
EXEC dbo.ChangeRow 17,900;

EXEC dbo.ChangeRow 25,0;
EXEC dbo.ChangeRow 16,1800;
EXEC dbo.ChangeRow 26,900;
EXEC dbo.ChangeRow 16,900;

EXEC dbo.ChangeRow 24,0;
EXEC dbo.ChangeRow 15,1800;
EXEC dbo.ChangeRow 25,900;
EXEC dbo.ChangeRow 15,900;


EXEC dbo.ChangeRow 23,0;
EXEC dbo.ChangeRow 14,1800;
EXEC dbo.ChangeRow 24,900;
EXEC dbo.ChangeRow 14,900;

EXEC dbo.ChangeRow 22,0;
EXEC dbo.ChangeRow 13,1800;
EXEC dbo.ChangeRow 23,900;
EXEC dbo.ChangeRow 13,900;

EXEC dbo.ChangeRow 21,0;
EXEC dbo.ChangeRow 12,1800;
EXEC dbo.ChangeRow 22,1499;
EXEC dbo.ChangeRow 12,780;

EXEC dbo.ChangeRow 11,910;

EXEC dbo.ChangeRow 22,900;
EXEC dbo.ChangeRow 21,910;

Each record starts out with ~950 bytes, so the 16 records will fit into two pages. From there the records are updated in a way so that each of them ends up on the other page with a forwarding record in its old place.

Now lets put the exact same data into the dbo.NoForwardSmall table:

INSERT INTO dbo.NoForwardSmall
SELECT * FROM dbo.ForwardSmall;

This again requires two pages, but this time no forwarded records are created.

To confirm, we can use the following query using the sys.dm_db_index_physical_stats DMF:

SELECT OBJECT_SCHEMA_NAME(object_id)+'.'+OBJECT_NAME(object_id) Tbl,index_type_desc,alloc_unit_type_desc,page_count,record_count,forwarded_record_count
FROM sys.dm_db_index_physical_stats(DB_ID(),NULL,NULL,NULL,'detailed') ps
WHERE object_id IN (OBJECT_ID('dbo.ForwardSmall'),OBJECT_ID('dbo.NoForwardSmall'));

This query returns the number of pages, the total number of records and the number of forwarded records for each table:

Tbl	index_type_desc	alloc_unit_type_desc	page_count	record_count	forwarded_record_count
dbo.ForwardSmall	HEAP	IN_ROW_DATA	2	32	16
dbo.NoForwardSmall	HEAP	IN_ROW_DATA	2	16	0

Now let us look at what happens during a table scan:

SET STATISTICS IO ON;
SELECT * FROM dbo.NoForwardSmall;
GO
SELECT * FROM dbo.ForwardSmall;
SET STATISTICS IO OFF;

This produces the following output:

(16 row(s) affected)
Table 'NoForwardSmall'. Scan count 1, logical reads 2, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

(16 row(s) affected)
Table 'ForwardSmall'. Scan count 1, logical reads 18, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

Scanning the dbo.NoForwardSmall table caused 2 logical reads - one for each page of the table. Scanning the dbo.ForwardSmall table however caused 18 logical reads. That is almost ten times more. Each of the 16 records caused an additional page read.

Conclusion

This article showed that SQL Server follows each forwarding record it encounters right away. This not only happens in the case of an RID-Lookup but also in the case of a table scan. These additional reads can present a significant performance impact. If you have a decently sized table with variable length columns and with regular update activity, add this behavior to the list of reasons why the table really should have a clustered index.