Introduction

If you have been working with SQL Server for a while you probably know that you should not select names for your stored procedures that start with the three characters "sp_".

Today I would like to take a closer look at all the myths and facts around this prefix. This will include a series of examples, all of which have been tested on SQL 2008R2 and on SQL 2012.

Special Objects

The first myth I would like to bust is that the "sp_" prefix stands for "system procedure".

While the BOL entry for Creating Stored Procedures uses language that could easily be interpreted like that, it actually never says anything about the etymology of this prefix. In fact, the "sp_" prefix causes a special path to be taken in the resolution of the object name. This works for stored procedures as well as for other object types. Later on this article contains an example with a table.

As "sp_" designates more objects than just stored procedures as special, it clearly does not mean "System Procedure", even though it currently is only used for system procedures. Instead it just stands for "special".

Resource Database

The idea for this article came from a tweet in a discussion about this prefix started by @ericstephani. In the tweet in question @SirSQL suggested to check if master was indeed the database checked first, as everyone in the discussion had assumed before, or if it rather is the resource database. This makes a lot of sense, as most of the system objects live in the resource database. So I decided to check this out by running a series of tests.

If you want to follow along with these tests you will need two databases: One with the name test and one that is called OtherDb.

For the first test I created a procedure with the name sp_executesql in master as well as in test and then executed it from both places:

USE master;
GO
CREATE PROCEDURE dbo.sp_executesql
@statement NVARCHAR(MAX)
AS
BEGIN
  SELECT DB_NAME() [called from], 'master' [sp_executesql];
END
GO
EXEC dbo.sp_executesql @statement = N'SELECT DB_NAME() [called from], ''system'' [sp_executesql];';
GO
DROP PROCEDURE dbo.sp_executesql;
GO
------------------------------------------------
GO
USE test;
GO
CREATE PROCEDURE dbo.sp_executesql
@statement NVARCHAR(MAX)
AS
BEGIN
  SELECT DB_NAME() [called from], 'test' [sp_executesql];
END
GO
EXEC dbo.sp_executesql @statement = N'SELECT DB_NAME() [called from], ''system'' [sp_executesql];';
GO
DROP PROCEDURE dbo.sp_executesql;
GO

The sp_executesql version that gets shipped with SQL Server lives in the resource database and so we would expect this version to get executed in all cases. This is indeed correct, as you can see in the result:

The above test shows that the resource database is checked first when an object name with the "sp_" prefix is encountered. That means that if you create any object with that name prefix you are running the risk that it will not be accessible anymore after the next service pack install if Microsoft decided to create an object with the same name in the resource database. This is even true when the resource database object is of a different type than your object. This you can quickly confirm by creating a table with the name sp_executesql in any database and trying to insert a row into it.

Master Piece

The fact that the resource database is the one checked first to resolve "sp_" object names raises the question where the master database fits in. Read on, as the results might surprise you.

The second test involves a procedure that is not (yet) part of the resource database: sp_MyOwnProc

I again created this procedure in master and in test:

GO
USE master;
GO
CREATE PROCEDURE sp_MyOwnProc
AS
BEGIN
  SELECT DB_NAME() [called from], 'master' [sp_MyOwnProc];
END
GO
USE test;
GO
CREATE PROCEDURE sp_MyOwnProc
AS
BEGIN
  SELECT DB_NAME() [called from], 'test' [sp_MyOwnProc];
END
GO
EXEC sp_MyOwnProc;
GO
USE OtherDb;
GO
EXEC sp_MyOwnProc;
GO
USE test;
GO
DROP PROCEDURE sp_MyOwnProc;
GO
USE master;
GO
DROP PROCEDURE sp_MyOwnProc;
GO

The above code creates the procedure in both places. It then executes the EXEC sp_MyOwnProc; statement, first from the test database and afterwards from the OtherDb database. The result is shown below:

As you can see, master is not checked first for an "sp_" object. Instead the object in the current database is used. Only if the current database does not contain the object in question master is looked at to see if the object exists in there.

The same behavior can be observed when all references to the sp_MyOwnProc in the above script are schema-qualified with dbo.

Other Object Types

You also get the same behavior if you go through this exercise with a table:

USE master;
GO
CREATE TABLE sp_MyOwnTable(
sp_MyOwnTable NVARCHAR(MAX)
);
INSERT INTO sp_MyOwnTable SELECT DB_NAME();
GO
USE test;
GO
CREATE TABLE sp_MyOwnTable(
sp_MyOwnTable NVARCHAR(MAX)
);
INSERT INTO sp_MyOwnTable SELECT DB_NAME();
GO
SELECT DB_NAME() [called from], * FROM sp_MyOwnTable;
GO
USE OtherDb;
GO
SELECT DB_NAME() [called from], * FROM sp_MyOwnTable;
GO
USE test;
GO
DROP TABLE sp_MyOwnTable;
GO
USE master;
GO
DROP TABLE sp_MyOwnTable;
GO

This script creates a table with the name sp_MyOwnTable in the master as well as in the test database and then executes a select against this name executing in test as well as in OtherDb. The result is shown here:

This shows that the name resolution works the same for tables as it does for stored procedures. It also works views, but not for functions or user defined types.

Performance Impact

Now we know that SQL Server tries to find an "sp_" object in the resource database first and only if it does not exist there the search is continued in the current database. So there should be a measurable performance impact showing this extra work.

To measure the impact I used this script:

USE test;
GO
IF OBJECT_ID('dbo.sp_MyOwnProc2') IS NOT NULL DROP PROCEDURE dbo.sp_MyOwnProc2;
GO
CREATE PROCEDURE dbo.sp_MyOwnProc2
AS
RETURN 0;
GO
IF OBJECT_ID('dbo.MyOwnProc2') IS NOT NULL DROP PROCEDURE dbo.MyOwnProc2;
GO
CREATE PROCEDURE dbo.MyOwnProc2
AS
RETURN 0;
GO
------------------------------------
GO
DECLARE @StartTime DATETIME2 = SYSDATETIME();
DECLARE @EndTime DATETIME2 = SYSDATETIME();
DECLARE @Counter INT = 0;
DECLARE @CmdA NVARCHAR(100) = 'EXEC dbo.MyOwnProc2;--';
DECLARE @CmdB NVARCHAR(100) = 'EXEC dbo.sp_MyOwnProc2;--';
DECLARE @TimeA BIGINT=0;
DECLARE @TimeB BIGINT=0;
DECLARE @Cmd2 NVARCHAR(100);
WHILE(@Counter<10000000)
BEGIN
  SET @Cmd2 = @CmdA + CAST(@Counter AS NVARCHAR(20));
  SET @StartTime = SYSDATETIME();
  EXEC(@Cmd2);
  SET @EndTime = SYSDATETIME();
  SET @TimeA += DATEDIFF(microsecond,@StartTime,@EndTime)
  SET @Cmd2 = @CmdB + CAST(@Counter AS NVARCHAR(20));
  SET @StartTime = SYSDATETIME();
  EXEC(@Cmd2);
  SET @EndTime = SYSDATETIME();
  SET @TimeB += DATEDIFF(microsecond,@StartTime,@EndTime)
  SET @Counter += 1;
END
SELECT @Counter Counter,@TimeA [MyOwnProc2], @TimeB [sp_MyOwnProc2];
GO

It first creates two identical stored procedures that do nothing but return a 0. The first one is called sp_MyOwnProc2, the second one carries the name MyOwnProc2 without the "sp_" prefix. To measure the performance impact, the script calls both procedures alternating in a loop and records their execution times. Each call gets executed as dynamic sql with the current loop count being part of the sql string. This prevents any possible attempts to cache the plan for the batch. It does however not prevent caching the plan for the procedure itself. That is okay, as we are after the name resolution piece of the execution. Because both procedures get called alternatingly, any background noise caused by other processes on the test system should affect both evenly.

On my system I got these results:

After 10 million executions of each of the two procedures you can see, that there is a performance impact. With less than 2 percent, however, it is very small compared to the time it takes to just call the procedure. Remember, that the procedures in this test did not actually do any work, so all the time recorded by this test was spent on identifying the procedure and the overhead of calling it.

Conclusion

The "sp_" object name prefix causes SQL Server to take a special route when resolving the name of this object: First SQL Server checks if the object exists in the hidden resource database. Second it tries to find the object in the current database and if it does not exist there SQL Server goes on to check if the object exists in the master database.

If you name your own objects using this prefix, you have to be aware of two possible consequences: First there is a small but measurable impact on performance. Second it can cause your application to suddenly break, if Microsoft decides to add an object with the same name to the resource database.

Because of that it is a best practice to not use the "sp_" name prefix anywhere in your code.

There is one exception however: If you are creating an object that you want to be accessible from all databases, you can use this prefix and place the object in the master database. However, because of the database precedence there are now two possibilities for those objects to get eclipsed by another object. So, if you go this route, make sure to regularly check that the object you are trying to execute is actually the one executing.

Introduction

SQL Server offers several ways to get to the execution plan for a particular query. Most of them however only provide the estimated execution plan without actual counts and statistics. In this article we are going to look at a way to get to the actual execution plan of a particular query.

Tracing

The certainly easiest way to get the actual plan of a query is to actually execute it in SSMS while the "collect actual execution plan" option is turned on. However, sometimes you want to collect the actual execution plan for a query every time it gets executed in your production environment. The only way to do this in SQL Server 2008R2 and earlier was to run a trace and collect the "Showplan XML Statistics Profile" event.

The problem with this approach is that it is not trivial to restrict the collected information to execution plans of one query. That means that on a busy system thousands of these events will fire in a very short time. With the included XML execution plan the amount of data to be collected will be quickly overloading most production systems.

Filtering Options

If the statement you are interested in is inside of a stored procedure you are somewhat in luck, as it is simple to filter by the name of the procedure which is returned by the event in the ObjectName column.

However, if the statement is an ad hoc or a prepared statement, this simple option does not exist anymore. There are a few other columns you can filter by like the login name, but most of the time you won't have the option to single out a specific query to use different connection settings.

The next option would be to filter on the text of the query. However, the query text is unfortunately not included in this event – neither as separate column nor within the execution plan XML. That leaves only one column that we can try to filter on, the execution plan itself.

While the execution plan is stripped of most of the query text, there are two types of names that are included in the execution plan: Parameter names and table alias names. Parameter names are only included if the statement is a prepared statement. Table alias names will be included in all types of statements that access a table. Be aware, that column alias names are not included in the execution plan.

Random Alias

The easiest way to come up with such an identifying name is to just use a random character string like "ir83n476s9d". Make sure however that the name of your choice begins with a letter. It also should not contain any special characters.

Once your query is prepared like this, you can easily restrict a "Showplan XML Statistics Profile" trace to only include this query by using a "Like" filter on the TextData column and setting it to "%ir83n476s9d%".

Conclusion

This simple trick allows you to filter the "Showplan XML Statistics Profile" event in a trace to fire only for a particular query. While it is not always possible to change a query to accommodate this, in many cases it is easy to do.

On final word of caution: When running traces in a production environment you should never use the SQL Profiler, especially when dealing with high-volume events like the "Showplan XML Statistics Profile" event. Instead setup your trace as a server-side trace to write the collected data to a fast, preferably dedicated drive. That way the performance impact on the system will be kept as small as possible.

Introduction

When a query requires rows to be sorted, either directly requested with an ORDER BY clause or because one of the iterators requires it, SQL Server has two options to guarantee that order. The obvious one is to utilize a sort iterator. If the data comes from an index (clustered or covering), SQL Server can also use an "Ordered Scan" of the data. Depending on the requested sort direction, such an ordered scan can be either forward or backward.

This would hardly be worth an article, if there wasn't the peculiarity that SQL Server obviously does not like the idea of having to execute an ordered scan that is directed backwards.

Backward Scan Dislike

Let's look at an example. First we need some tables with data:

IF OBJECT_ID('dbo.T1') IS NOT NULL DROP TABLE dbo.T1;
IF OBJECT_ID('dbo.T2') IS NOT NULL DROP TABLE dbo.T2;

CREATE TABLE dbo.T1(
Id INT CONSTRAINT T1_PK PRIMARY KEY CLUSTERED,
v1 INT,
c1 VARCHAR(8000)
);

CREATE TABLE dbo.T2(
Id INT CONSTRAINT T2_PK PRIMARY KEY CLUSTERED,
v1 INT,
c1 VARCHAR(8000)
);

INSERT INTO dbo.T1(Id,v1,c1)
SELECT n,CHECKSUM(NEWID()),'*'
FROM dbo.GetNums(1000000);

INSERT INTO dbo.T2(Id,v1,c1)
SELECT n,CHECKSUM(NEWID()),'*'
FROM dbo.GetNums(1000000);
--Get dbo.GetNums here: http://www.sqlmag.com/article/sql-server/virtual-auxiliary-table-of-numbers 

This script is creating two tables containing one million rows each. The v1 column is filled with random values; the c1 column just contains a single constant character. The Id column is the clustered primary key and it is providing the ordering on disk that we are going to use.

The query is a little made up. It joins T2 to itself bringing back only one set of columns. It also restricts the result of this join to the first one million rows, sorted by T2.Id. The one million values in T2.v1 are random from a set of 4 billion making them "mostly" unique. That means the self-join is expected to return about one million rows anyway. Those rows are then joined to T1. In the end the ORDER BY requests that all rows be returned sorted by the Id column of the T1 table:

SELECT  *
FROM    dbo.T1 A
        INNER JOIN ( SELECT TOP ( 1000000 )
                            B.*
                     FROM   dbo.T2 B
                            INNER JOIN dbo.T2 C ON B.v1 = C.v1
                     ORDER BY B.Id
                   ) BC ON A.Id = BC.Id
ORDER BY A.Id;

The execution plan of this query looks like this:

There is nothing unexpected in this plan. The T2 self-join is done with a hash join operator, as there is no index on the v1 column. The data is then sorted by a sort operator and passed through a serial zone for the top operator. After that there is no additional sort operator for the data to pass through. As the merge join requires both streams to be sorted the same way on the join column, this means that the data must be produced sorted by the scan of the T1 table. A quick look at its properties confirms that:

This scan is a forward scan, as both ORDER BY statements are asking to sort by Id ascending. To make this scan go backwards we should just have to change the query to sort by Id descending in both places:

SELECT  *
FROM    dbo.T1 A
        INNER JOIN ( SELECT TOP ( 1000000 )
                            B.*
                     FROM   dbo.T2 B
                            INNER JOIN dbo.T2 C ON B.v1 = C.v1
                     ORDER BY B.Id DESC
                   ) BC ON A.Id = BC.Id
ORDER BY A.Id DESC;

However, this query produces this rather unexpected plan:

A loop join with an index seek against the T1 table to retrieve 1,000,000 single rows – that cannot be good.

Before we look at execution statistics let's force the merge join back by specifying a join hint:

SELECT  *
FROM    dbo.T1 A
        INNER MERGE JOIN ( SELECT TOP ( 1000000 )
                                  B.*
                           FROM   dbo.T2 B
                                  INNER JOIN dbo.T2 C ON B.v1 = C.v1
                           ORDER BY B.Id DESC
                         ) BC ON A.Id = BC.Id
ORDER BY A.Id DESC;

Now we get the expected plan again:

If you compare the properties for the iterators in the forward scan and the backward scan plan you will notice, that the cost of the two scan directions is not that much different:

The estimated cost of the scan iterator is 0% in both cases. The bulk of the work is done in the hash self-join and the following sort of the T2 table.

So, let us take a look at the execution statistics of all three queries:

The execution statistics clearly show that the decision to go with a loop join was not necessarily the best. While the estimated cost for the backward scan is about 150% of the estimated cost for the loop join, the actual reads for the loop join version are about 3 million, which is more than 250 times higher than the 8000 reads for either scan direction.

Disk Access

To understand were this dislike is coming from we need to look at the access pattern necessary to retrieve the data from disk. A forward scan of a table (or index), that is not heavily fragmented, looks pretty much like doing a single contiguous read of a big blob of data. SQL Server has a lot of performance optimizations build in that make this type of access as fast as possible. The Read-Ahead mechanism is a good example for that.

A backward scan on the other hand looks quite the opposite: After reading a page, the disk has to do almost a complete turn to get to the previous page, which is the next one in line to be read. This is about the worst kind of random access you can come by.

As SQL Server always assumes that none of the requested pages are in cache, the decision to not get into that backward spinning game seems quite understandable.

Also, if you closely compare the two "scanning" execution plans you will notice that while the forward scan iterator is parallelized, the backward scan operator is executed single threaded. This is a general rule: SQL Server cannot execute a backward scan in parallel. That makes sense as with all that waiting for the disk to do another spin to get to the "next" page, it is unlikely for the data to come in quickly enough to keep multiple threads busy.

Estimates?

But wait, there is one more thing: If you look at the image above and compare the properties of all iterators in the two scanning plans, you will notice something odd. The estimated CPU cost of the "hash match inner join" iterator goes from 11 in the "forward" case to 1357 in the "backward" case; the estimated IO cost makes a similar jump from 0 to 1654. That seems to not make a lot of sense, as the next iterator, the sort, is a blocking iterator. "Blocking" means that the iterator reads in all input rows into a holding area before producing any output rows. This implies that anything on the left side of a sort should not be able to influence the cost of operators on the right side of it. That is, unless it affects the number of rows significantly, as this is a TOP N Sort. See TOP N Sort – A Little Bit of Sorting for an explanation. But as in this example all columns are "mostly" unique, the optimizer guesses correctly that about one million rows get passed into the hash join on each input and also about one million rows come out the other end. So row count estimates should not matter here.

If you look at the estimates when executing just that inner query sorting backwards, you will see that they exactly match the ones in the imbedded forward sorting case:

That means that the scan direction change of the T1 table causes the estimates on the other side of the merge join to get completely thrown off.

Conclusion

SQL Servers dislike of backward scan operations is understandable when looking at the work necessary to retrieve the records from disk in opposite index order. Additionally, a backward scan cannot be executed in parallel. That adds to the list of reasons, why the backward scan seems unloved.

However, the main reason why SQL Server avoids backward scans seems to be that they throw its cost estimations off, making following iterators appear a lot more expensive than they really are.

Introduction

Sorting is one of the most often used functions in SQL Server. It is used when creating indexes. It allows us to retrieve data in a particular order. It also is used to filter out repeated rows in the context of a DISTINCT request. The SQL Server Team spent a lot of effort optimizing the sort algorithms to give the best possible performance in each context. They even implemented sort avoidance where the expensive sort gets removed from the query plan if it is known that the input is sorted already or will contain only a single distinct value.

One of those optimizations is the TOP N Sort that I would like to look at in a little more detail today.

Memory Requirements for Sorting

Sorting requires memory – potentially a lot of memory. SQL Server estimates the size of the data using statistics and data size estimates and then requests about 150 percent of that in memory for the sort operation.

To see how much memory a query is using, you can use the sys.dm_exec_query_memory_grants DMV. It shows the current memory usage for each SPID that requested additional memory for one (or more) of the memory consuming operators.

The problem with this DMV is that its values change quickly, so it is difficult to get meaningful information for a particular query. There is however a trick you can use: Every memory consuming iterator has a build phase where most of the memory is used, and a delivery phase in which the rows get send to the parent operator. The memory consumption during the build phase is usually a lot higher. SQL Server recognizes this and reuses the additional memory during the delivery phase for other memory consuming iterators.

The sys.dm_exec_query_memory_grants DMV contains the columns used_memory_kb and columns max_used_memory_kb that report current and maximum memory usage. If we can ensure that we get the data from this DMV during the delivery phase of the operator in question, the max_used_memory_kb column will tell us exactly how much memory was used during the build phase. This however only works for queries with a single memory consuming operator that also needs to be blocking.

Sort is a blocking operator. That means that during the build phase no rows get transmitted. If we include the request for the memory data in the query and place it before ("left of" in the graphical plan) the sort, we know that the sort will have finished its build phase when the data gets retrieved from the DMV.

Let us look at an example:

IF OBJECT_ID('dbo.FixedWidth') IS NOT NULL DROP TABLE dbo.FixedWidth;

CREATE TABLE dbo.FixedWidth(
Id INT PRIMARY KEY CLUSTERED,
c1 INT NOT NULL,
fill VARCHAR(1000) NOT NULL
);

INSERT INTO dbo.FixedWidth(Id,c1,fill)
SELECT n,CHECKSUM(NEWID()),REPLICATE('X',192)
FROM dbo.GetNums(10000);

SELECT SUM(DATALENGTH(Id)+DATALENGTH(c1)+DATALENGTH(fill))
FROM dbo.FixedWidth;
 

This snippet creates the table dbo.FixedWidth and inserts 10000 rows into it. It then sums up the DATALENGTH of all columns over all rows. The size of all the data totals 2,000,000 bytes. It is using Itzik Ben-Gan's GetNums function.

The following query will sort the data in this table and report its memory usage:

SELECT  ( SELECT  *
          FROM    sys.dm_exec_query_memory_grants
          WHERE   session_id = @@SPID
        FOR
          XML PATH('') ,
              TYPE
        ) Mem ,
        *
FROM    dbo.FixedWidth
ORDER BY c1;

This query has below execution plan:

The access to the sys.dm_exec_query_memory_grants DMV happens after the sort has finished. Therefore the max_used_memory_kb will tell us the amount actually used for the sort of the entire 2MB table. The information from the memory DMV is returned as a single XML value. As the DMV result is spooled in this query, all rows will show the same XML value:

<session_id>66</session_id>
<request_id>0</request_id>
<scheduler_id>1</scheduler_id>
<dop>1</dop>
<request_time>2012-04-22T19:09:40.303</request_time>
<grant_time>2012-04-22T19:09:40.307</grant_time>
<requested_memory_kb>7352</requested_memory_kb>
<granted_memory_kb>7352</granted_memory_kb>
<required_memory_kb>512</required_memory_kb>
<used_memory_kb>2456</used_memory_kb>
<max_used_memory_kb>2456</max_used_memory_kb>
<query_cost>3.460044653270174e+000</query_cost>
<timeout_sec>86</timeout_sec>
<resource_semaphore_id>0</resource_semaphore_id>
<plan_handle>BgA6AB2ceA5AIZDfAAAAAAAAAAAAAAAA</plan_handle>
<sql_handle>AgAAAB2ceA7MMroMHW//TQYN1cSbf5KW</sql_handle>
<group_id>2</group_id>
<pool_id>2</pool_id>
<is_small>0</is_small>
<ideal_memory_kb>7352</ideal_memory_kb> 

It shows that 7,352KB is the ideal memory grant size the optimizer came up with. This was also the amount that was granted. However, only 2,456KB where actually used during the sort.

TOP N Sort

If you are interested only in the first few rows based on a sort order, you do not need to complete sort the rest of the rows. To find the top most row it is enough to take the first row of the data set and compare it to each remaining row, keeping each time the one row that comes first according to the sort order. If you have more than one row you can follow the same principle, you just have to keep the first n rows after each step.

SQL Server implements this with the TOP N Sort operator:

SELECT TOP ( 10 )
        *
FROM    dbo.FixedWidth
ORDER BY c1; 

This simple query has the following execution plan:

As you can see, the sort operator is of type "TOP N Sort".

You can use the same trick as shown above to get its memory consumption, but you have to add an outer layer, otherwise SQL Server removes the TOP N Sort optimization:

SELECT  ( SELECT  *
          FROM    sys.dm_exec_query_memory_grants
          WHERE   session_id = @@SPID
        FOR
          XML PATH('') ,
              TYPE
        ) Mem ,
        *
FROM    ( SELECT TOP ( 10 )
                  *
          FROM    dbo.FixedWidth
          ORDER BY c1
        ) X;

<session_id>66</session_id>
<request_id>0</request_id>
<scheduler_id>1</scheduler_id>
<dop>1</dop>
<request_time>2012-04-22T19:25:18.023</request_time>
<grant_time>2012-04-22T19:25:18.023</grant_time>
<requested_memory_kb>1024</requested_memory_kb>
<granted_memory_kb>1024</granted_memory_kb>
<required_memory_kb>24</required_memory_kb>
<used_memory_kb>24</used_memory_kb>
<max_used_memory_kb>24</max_used_memory_kb>
<query_cost>8.388694532701750e-001</query_cost>
<timeout_sec>25</timeout_sec>
<resource_semaphore_id>1</resource_semaphore_id>
<plan_handle>BgA6AM04yitAYc7jAAAAAAAAAAAAAAAA</plan_handle>
<sql_handle>AgAAAM04yivnP5lrZVMtEy5tLK/6dHjX</sql_handle>
<group_id>2</group_id>
<pool_id>2</pool_id>
<is_small>1</is_small>
<ideal_memory_kb>32</ideal_memory_kb> 

This query used only 24KB, clearly not enough to hold the entire table, so the TOP N Sort optimization was used.

The algorithms SQL Server uses for sorting are not documented. However, it is clear that an algorithm to find the TOP N rows cannot be as efficient as a normal sort when sorting the same number of rows. For that reason there is a maximum number of rows for which SQL Server uses the TOP N Sort optimization. That number was set to 100.

If you run above query with TOP(100) you will get a memory usage of 128KB:

<session_id>66</session_id>
<request_id>0</request_id>
<scheduler_id>1</scheduler_id>
<dop>1</dop>
<request_time>2012-04-22T19:32:38.127</request_time>
<grant_time>2012-04-22T19:32:38.127</grant_time>
<requested_memory_kb>1024</requested_memory_kb>
<granted_memory_kb>1024</granted_memory_kb>
<required_memory_kb>128</required_memory_kb>
<used_memory_kb>128</used_memory_kb>
<max_used_memory_kb>128</max_used_memory_kb>
<query_cost>8.624926532701749e-001</query_cost>
<timeout_sec>25</timeout_sec>
<resource_semaphore_id>1</resource_semaphore_id>
<plan_handle>BgA6AKCsNDBAIVXkAAAAAAAAAAAAAAAA</plan_handle>
<sql_handle>AgAAAKCsNDB6vVhQtTSg6/3xIflWWU1M</sql_handle>
<group_id>2</group_id>
<pool_id>2</pool_id>
<is_small>1</is_small>
<ideal_memory_kb>136</ideal_memory_kb> 

If you use TOP(101) the memory usage goes up to 2,456KB:

<session_id>66</session_id>
<request_id>0</request_id>
<scheduler_id>1</scheduler_id>
<dop>1</dop>
<request_time>2012-04-22T19:32:11.993</request_time>
<grant_time>2012-04-22T19:32:11.993</grant_time>
<requested_memory_kb>7352</requested_memory_kb>
<granted_memory_kb>7352</granted_memory_kb>
<required_memory_kb>512</required_memory_kb>
<used_memory_kb>2456</used_memory_kb>
<max_used_memory_kb>2456</max_used_memory_kb>
<query_cost>8.627551332701751e-001</query_cost>
<timeout_sec>25</timeout_sec>
<resource_semaphore_id>0</resource_semaphore_id>
<plan_handle>BgA6AMCfYwlA4WnkAAAAAAAAAAAAAAAA</plan_handle>
<sql_handle>AgAAAMCfYwlb3oB4gW5K1oLplwLKxsA+</sql_handle>
<group_id>2</group_id>
<pool_id>2</pool_id>
<is_small>0</is_small>
<ideal_memory_kb>7352</ideal_memory_kb> 

All the memory numbers match the ones from the whole table sort, so SQL Server decided to execute a standard sort followed by a standard TOP operation instead of using the TOP N Sort optimization.

This boundary is fixed and cannot be configured. There is also no hint to force this optimization. That is rather unfortunate as 100 seems to be a quite arbitrary boundary value and a TOP N Sort would often perform a lot better that a full table sort followed by a TOP.

Conclusion

We all know that while the optimizer in general is very good a finding a (close to) optimal plan, sometimes it does not.

The TOP N Sort optimization is a great enhancement, if you are dealing with 100 or less rows. However at 101 rows you will see a sudden drop in performance. To deal with this, you can either restrict the value of rows to return to 100, if the business allows. Or you can help SQL Server to deal with the "all rows" sort. One of the most obvious options here is to add an index to the table.

In the March issue of SQL Server Pro Itzik Ben-Gan wrote about an undocumented DBCC command that was recently discovered in the wild by Eladio Rincon (http://www.sqlmag.com/article/sql-server/options-affecting-parallelism-141505). Itzik closed the article by expressing his hope to get this statement documented as it can be very helpfull to database professionals trying to performance tune a query on a none production system, that does not completely match the production environment.

I agree with this opinion, and so I decided to spend some time investigating this functionality further.

Optimizer_WhatIf?

The DBCC command is the OPTIMIZER_WHATIF cammand. DBCC HELP gives you the following information about it:

dbcc OPTIMIZER_WHATIF ({property/cost_number | property_name} [, {integer_value | string_value} ])

This is – according to Google – the only publicly available information about this command. Eladio discovered it in use at a customer site. The customer was told by Microsoft that with the first parameter equal to one, the second would influence parallelism decisions of the optimizer.

After reading Itziks article it seemed clear that the "1" meant "number of schedulers" or "number of CPUs". The above documentation snippet says that you can either specify a number or a name for the first parameter, so knowing the possible non-numeric values for it would be a start in understanding what else this command does.

Finding Strings

Every windows programm has to somehow contain the strings it is using. SQL Server is no exception. A well designed program will have related strings close by each other.

You can use a program like Mark Russinovich's Process Explorer to get a list of all strings in the memory space of a running process.

As all statements (including DBCC commands) are strings, you can expect to find the possible values for the first parameter next to each other. With that thought in mind I went ahead and searched for strings containing "CPU" or "scheduler". I found this list of possible property_name values for SQL Server 2008 and SQL Server 2008 R2:

Status
ResetAll
CPUs
MemoryMBs
Bits
ParallelCardThreshold
ParallelCostThreshold
ParallelCardCrossProd

LowCEThresholdFactorBy10
HighCEThresholdFactorBy10
CEThresholdFactorCrossover
DMLChangeThreshold

Warning

All property names are case sensitive, even in a case insensitive installation! You also will not get an error message if an invalid name or value is passed in, so be careful.

Many More CPUs

The third one, "CPUs" causes the functionality described by Itzik: If you run

DBCC OPTIMIZER_WHATIF(CPUs, 16) WITH NO_INFOMSGS;

Administrative Help

If you run

DBCC OPTIMIZER_WHATIF(ResetAll) WITH NO_INFOMSGS;

DBCC OPTIMIZER_WHATIF(Status) WITH NO_INFOMSGS;

---------------------------------------------------------

Optimizer what-if status

---------------------------------------------------------

property_number current_value default_value property_name

---------------------------------------------------------

         1                  0             0 CPUs

         2                  0             0 MemoryMBs

         3                  0             0 Bits

         4               1000          1000 ParallelCardThreshold

         5                  1             1 ParallelCostThreshold

         6                200           200 ParallelCardCrossProd

         7                 50            50 LowCEThresholdFactorBy10

         8                 12            12 HighCEThresholdFactorBy10

         9             100000        100000 CEThresholdFactorCrossover

        10                 10            10 DMLChangeThreshold

Conclusion

This DBCC command is a great tool, when you are trying to answer questions like: What if the optimizer had more (or less) CPUs available than there are in my current system.

As with all undocumented functionality however, don't use this in production unless advised to do so by Microsoft support.

This post covered only three of the possible values for the first parameter. The other ones will be addressed in future posts.

When thinking about the indexes for update statements it is easy to overlook an important peculiarity in the way SQL Server finds the rows to update before updating them.

Those pesky Lookups

If you have an UPDATE statement that filters on two columns and you have an index on one of them, you might be tempted to think that the necessary lookup for the second column is not going to cost a lot - particular when that second column is not selective at all - as SQL Server has to access the storage place of each row anyway when executing the actual update. This however is not true, as SQL Server executes the update in two separate phases: One to identify the rows and one to update those rows. That means a lookup in the find phase can be very expensive.

Let us look at an example. Use the following script to create two identical tables and insert 200,000 rows into each of them.

CREATE TABLE dbo.tst1(
 Id INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,
 Val INT,
 Filter INT,
 IdxKey INT,
 Fill CHAR(1000) DEFAULT 'Fill'
);

CREATE TABLE dbo.tst2(
 Id INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,
 Val INT,
 Filter INT,
 IdxKey INT,
 Fill CHAR(1000) DEFAULT 'Fill'
);

INSERT INTO dbo.tst1(Val,Filter,IdxKey)
OUTPUT INSERTED.Val,INSERTED.Filter,INSERTED.IdxKey INTO dbo.tst2(Val,Filter,IdxKey)
SELECT 0,0, n%1000
FROM dbo.GetNums(200000);

The Filter column will be used to force the lookup later on by adding "Filter = 0" to the where clause. It is valued 0 in all rows, so no row will be excluded by this filter.

After creating the tables, create the following indexes:

CREATE INDEX dbo_tst1_IdxKey ON dbo.tst1(IdxKey);
CREATE INDEX dbo_tst2_IdxKey ON dbo.tst2(IdxKey) INCLUDE (Filter);

Both indexes are on the IdxKey column, but the index on the dbo.tst2 table also includes the Filter column.

Now run the following two identical update statements against the two tables:

UPDATE dbo.tst1
  SET Val = 17
WHERE IdxKey <30 AND Filter = 0;

UPDATE dbo.tst2
  SET Val = 17
WHERE IdxKey <30 AND Filter = 0;

The first one requires a lookup operator as the Filter column is not included in the index on dbo.tst1. Below are the two execution plans:

As you can clearly see, the update operator is separated from the data retrieval operator(s) by several other operators. It looks like all the filtering and value preparations happen before the update operator which is then just updating all rows passed to it without any further filtering.

Statistics

I ran the above statements in a loop, each executing 2000 times and recorded time and logical reads. The averaged results are below:

cpu_time	total_elapsed_time	logical_reads	exec_count	command
98	175	37212	2000	UPDATE dbo.tst1 SET Val = 17 WHERE IdxKey <30 AND Filter = 0;
47	72	19211	2000	UPDATE dbo.tst2 SET Val = 17 WHERE IdxKey <30 AND Filter = 0;

The cpu time, the elapsed time and the logical reads were each about twice as high in the query against dbo.tst1, the one requiring the lookup operator. This was the case even though the lookup did not actually filter any rows - every row accessed by the lookup had to be updated as well.

Conclusion

The above statistics prove what the execution plan already suggested: The finding of rows for an update and the update itself are separate steps in SQL Server update queries. That means the "find" portion has to be optimized as if it were a standalone SELECT. In particular, lookup operations can not be ignored as SQL Server accesses the rows twice in this case: Once for the lookup and once for the update.

Today I am going to talk about the evil side of the Assert Operator. To start out, let me introduce this operator to you:

The Assert Operator

The Assert Operator is used by SQL Server in execution plans that require some condition to be satisfied to be valid. A common scenario in which an Assert Operator is used is an insert into a table that has a check constraint defined. Take for example the dbo.TableWithCheckConstraint table:

CREATE TABLE dbo.TableWithCheckConstraint(
  Id INT PRIMARY KEY CLUSTERED,
  Even INT CONSTRAINT TableWithCheckConstraint_Even_Check CHECK (Even % 2 = 0)
);

If you run this simple insert against that table

INSERT INTO dbo.TableWithCheckConstraint(Id,Even)
SELECT n, 2*n FROM (VALUES(1),(2))X(n);

you will get an execution plan that looks like this:

The properties of the highlighted Assert Operator are shown below:

Asserts in general are very cheap operators as you can see when you look at the Estimated Operator Cost in above image. An Assert operator works by calculating a given expression. If that expression results in anything other than NULL, the assert fails and stops the execution of the current statement. The expression in above example is the formula we used for the check constraint itself, wrapped in a case statement to return either NULL or 0.

The Dark Side of the Assert

So, how can such an innocent little operator be evil?

Sometimes SQL Server requires a certain condition to be satisfied in situations where you do not necessarily expect it. The most common example is a sub-query in the column list or the WHERE clause of a select statement as shown in the following example:

SELECT  ( SELECT  m
          FROM    (VALUES (1), (2) ) X(m)
        );

If you execute this query you get this error:

Msg 512, Level 16, State 1, Line 1
Subquery returned more than 1 value. This is not permitted when the subquery follows =, !=, <, <= , >, >= or when the subquery is used as an expression.

To enforce that rule, SQL Server is using an Assert Operator again:

In above example the Stream Aggregate Operator is tasked with counting the rows returned:

[Expr1005] = Scalar Operator(Count(*))

CASE WHEN [Expr1005]>(1) THEN (0) ELSE NULL END

(If you look more closely you will find that the Stream Aggregate actually calculates two aggregates in this query and that the second one is of a surprising type. Check out this article by Paul White for an explanation.)

In this example the Stream Aggregate and the Assert make up together 50% of the (estimated) query cost, but as this is a very made-up example I would not pay to much attention to that. So let us look at a real life example.

The Forgotten Uniqueness

First we need a few tables:

CREATE TABLE dbo.AssertBigFact(
 Id INT PRIMARY KEY CLUSTERED,
 Sort1 INT UNIQUE,
 Luv INT,
 Data CHAR(1000)
);

DECLARE @BigSize INT = 1000000;
DECLARE @LookupSize INT = 1000;

INSERT INTO dbo.AssertBigFact(Id,Sort1,Luv,Data)
SELECT n,(n*1123)%@BigSize+1,n%@LookupSize+1,'' FROM dbo.GetNums(@BigSize)


CREATE TABLE dbo.AssertLookupNonUnique(
 Id INT ,
 Val INT
);
CREATE CLUSTERED INDEX AssertLookupNonUnique_CI ON dbo.AssertLookupNonUnique(Id);

CREATE TABLE dbo.AssertLookupUnique(
 Id INT ,
 Val INT
);
CREATE UNIQUE CLUSTERED INDEX AssertLookupUnique_CI ON dbo.AssertLookupUnique(Id);

INSERT 
  INTO dbo.AssertLookupNonUnique(Id,Val)
OUTPUT INSERTED.Id,INSERTED.Val
  INTO dbo.AssertLookupUnique(Id,Val)
SELECT n,n%37 FROM dbo.GetNums(@LookupSize);

As most of my examples, this one is also using Itzik's GetNums function that you can get here: Virtual Auxiliary Table of Numbers.

It creates three tables, the table dbo.AssertBigFact that contains 1,000,000 rows and the two small 1,000 row lookup tables dbo.AssertLookupUnique and dbo.AssertLookupNonUnique. The insert statement uses the OUTPUT clause to insert the same rows in both tables, so with the exception of the clustered index on the dbo.AssertLookupNonUnique table that got "accidentally" not declared as unique, these two tables are identical.

Now let us compare the following two almost identical queries. They only differ in the lookup table they go against.

SET STATISTICS IO ON;
SET STATISTICS TIME ON;
GO
SELECT b.*,(SELECT Val FROM dbo.AssertLookupNonUnique l WHERE l.Id = b.Luv) Val
FROM dbo.AssertBigFact b
ORDER BY b.Sort1,Val;
GO
SELECT b.*,(SELECT Val FROM dbo.AssertLookupUnique l WHERE l.Id = b.Luv) Val
FROM dbo.AssertBigFact b
ORDER BY b.Sort1,Val;
GO
SET STATISTICS TIME OFF;
SET STATISTICS IO OFF;

The execution plans for above queries look like this:

The top version, going against the non-unique table, requires the use of an Assert Operator. The second query on the other end can get away without in, as SQL Server knows that the sub-query can return only a single row because the Id column was declared unique in the definition of the clustered index.

In a scenario like this where a rather large table gets joined with a small table, using a hash join is usually the fastest way for SQL Server to execute the query. A hash join requires the entire first table to be stored in a hash table in memory before any row can be returned. A hash can spill to tempdb if there is not enough free memory available, but such a spill is expensive. So hash joins are perfect in a scenario where one of the two join partners is small.

In our case SQL Server chose the hash join for the second query, as expected. However the first query uses a loop join. The reason for this is quite simple: SQL Server does not know that the Id column in the dbo.AssertLookupNonUnique table is unique, so it needs to assert that for each Luv value in the dbo.AssertBigFact table there is only one row in the dbo.AssertLookupNonUnique table with a matching Id value. For that it needs to read the dbo.AssertBigFact table first, as there might be id values in the dbo.AssertLookupNonUnique table that do not have a matching Luv value in the dbo.AssertBigFact table. If those Id values were duplicated it would not matter for this query, so the Assert can run only against Id values with a matching Luv value. That however means, that SQL Server can't build the hash table first, at least not with the small dbo.AssertLookupNonUnique table. But it doesn't make sense to build the hash table using the dbo.AssertBigFact table as that would take to much time and memory and would therefore be rather ineffective.

A merge join is not a good choice either, as it requires both inputs to be sorted on the join column which is not the case. SQL Server could insert a sort before the join, but that would mean that the full one million rows had to be sorted twice as a different sort order is requested by the query – a scenario worth avoiding.

That means that SQL Server has to use the loop join algorithm. That means it has to assert that there is only a single row in the dbo.AssertLookupNonUnique for each row in the dbo.AssertBigFact table which in turn means that the Stream Aggregate and Assert Operator pair gets executed 1,000,000 times. While each execution is small and cheap, doing a million of them is not. But this is not even the worst problem with this query.

Wasted Logical Reads

If we look at the logical reads (returned by the SET STATISTICS IO ON statement in above batch) for the query against the dbo.AssertLookupUnique we get these numbers:

Table 'AssertLookupUnique'. Scan count 3, logical reads 10, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'AssertBigFact'. Scan count 3, logical reads 143712, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

The dbo.AssertLookupUnique has 3 pages as a quick check against sys.dm_db_index_physical_stats confirms:

SELECT  OBJECT_SCHEMA_NAME(i.object_id) + '.' + OBJECT_NAME(i.object_id) Tbl ,
        i.name ,
        ips.index_depth ,
        ips.page_count
FROM    sys.indexes i
        JOIN sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) ips 
        ON ips.object_id = i.object_id AND ips.index_id = i.index_id
WHERE   i.index_id < 2
        AND i.object_id IN ( OBJECT_ID('dbo.AssertLookupNonUnique'),
                             OBJECT_ID('dbo.AssertLookupUnique'),
                             OBJECT_ID('dbo.AssertBigFact') );

This query returns these numbers:

The 10 logical reads are caused by the parallelism of this query. The different threads of a single query often read pages that another thread has looked at before. The same behavior you can see with the dbo.AssertBigFact table where also a few of the 142858 pages got read more than once.

If we now look at the numbers for the query against the dbo.AssertLookupNonUnique table we get a quite different picture:

Table 'AssertLookupNonUnique'. Scan count 1000000, logical reads 2004000, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'AssertBigFact'. Scan count 3, logical reads 143760, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

While the number of reads against the dbo.AssertBigFact table is about the same as before, there are over 2 million logical reads recorded for the dbo.AssertLookupNonUnique table. — What a difference...

The seek against the dbo.AssertLookupNonUnique was executed one million times, once for each row in the dbo.AssertBigFact table. As the index depth is 2 levels, each seek had to look at two pages – two million in total.

Conclusion

As you have seen in the above example, requiring an assert can be quite expensive. While the Assert Operator is not that expensive in it self, SQL Server is severely restricted in the plan choices it can make when an assert is logically required.

To avoid situations like this you should always try to give the Optimizer as much information as possible. That includes keeping statistics up to date and using proper indexes. But, as this article has shown, creating an index on the right columns is not necessarily enough. So, if you have an index on a column with unique values, make sure to declare it as unique.

There is a lot of information out there recommending to do this or that to deal with the performance problems that you must have if you see CXPACKET waits. All these recommendations build on the notion that the CXPACKET wait is a bad thing. Today I am going to explain why this is not the case. I am going to show you that the CXPACKETs are just one part of how the SQL Server engine works and that they do not indicate any problem.

Parallel Processing

First we need to dive into, how parallel processing of requests in SQL Server works. Every time SQL Server executes a query it does so using its execution plan. You can think of an execution plan as the compiled executable for the query. In this post I am not going to explain how SQL Server produces this plan, but if you want more detail I can recommend the book Inside the SQL Server Query Optimizer by Benjamin Nevarez (blog|twitter).

Under specific circumstances SQL Server can decide to produce  a parallel plan. You can recognize a parallel plan by the yellow circles with two arrows on them that show up on some of the icons in the graphical plan:

You will also find at least one of the parallelism iterators:

They are called "Gather Streams", "Repartition Streams" and "Distribute Steams", and as they are always executed in parallel mode, they always appear with the yellow circle on them.

A parallelism iterator acts as thread control point. It takes a request from one ore more threads and passes it on, again to one ore more new threads. When the data is coming back, the iterator collects it and distributes it to the requesting threads. I am again not going to go into the details here. A good explanation can be found in Paul Whites(blog|twitter) article on simple talk: Understanding and Using Parallelism in SQL Server

I would like to call your attention to the word "new" in the above paragraph. Each icon in an execution plan represents an iterator and execution always starts with the left most iterator. It is the "SELECT" iterator in the above example. Each iterator requests rows from it's direct child iterators and then does some work with those rows. When a parallelism iterator is invoked it spawns n new threads and passes control to all of them. The number n is based on the type of the parallelism iterator and the current Degree Of Parallelism settings.

The original threads, after passing control to the newly spawned threads wait to receive rows back. This waiting is not done by spinning or polling. Instead the thread(s) enter a wait state and wake bake up once a row is available for processing. This allows the processor to be used for other worker threads.

The wait type that SQL Server reports for those threads waiting at a parallelism iterator is - you guessed it - the CXPACKET wait.

An Example

Let's look at an example:

IF OBJECT_ID('dbo.t1') IS NOT NULL 
  DROP TABLE dbo.t1 ;

CREATE TABLE dbo.t1
  (
    i INT ,
    t TEXT ,
    c1 INT DEFAULT CHECKSUM(NEWID()) ,
    c2 INT DEFAULT CHECKSUM(NEWID())
  ) ;

INSERT  INTO dbo.t1
        ( i ,
          t
        )
        SELECT  n ,
                REPLICATE('X', 200 + ( CHECKSUM(NEWID()) % 10 ))
        FROM    dbo.GetNums(1000000) ;

This creates the table "t1" and fills it with 1.000.000 strings of random length between 200 and 210 characters. The code uses Itzik Ben-Gan's GetNum function to quickly create the one million rows.

Now run this select statement against that table:

SELECT ln, COUNT(1) cnt FROM(SELECT LEN(CAST(t AS NVARCHAR(MAX))) ln FROM dbo.t1) X GROUP BY ln;

Let's call this the "Count Query". It has the execution plan that was introduced earlier.

While this query runs, execute the following "Monitor Query" in a new window. Make sure to change the session_id variable to the SPID of the window executing the Count Query.

DECLARE @session_id INT = 75;
SELECT  t.task_address ,
        t.task_state ,
        t.session_id ,
        t.exec_context_id ,
        wt.wait_duration_ms ,
        wt.wait_type ,
        wt.blocking_session_id ,
        wt.blocking_exec_context_id ,
        wt.resource_description ,
        t.scheduler_id
FROM    sys.dm_os_tasks t
        LEFT JOIN sys.dm_os_waiting_tasks wt ON t.task_address = wt.waiting_task_address
WHERE   t.session_id = @session_id
ORDER BY t.exec_context_id,wt.blocking_exec_context_id;

This query returns one row for every worker thread currently assigned to this query. A thread that is currently blocked might get more than one row, if it is blocked by more than one other thread. For each waiting thread it also shows what it is waiting for. After about 10 seconds of execution of the Count Query, the Monitor Query returned this result on my machine:

As you can see, there are five different execution_context_ids. Each thread on a parallel query gets its own execution context. The first one, the thread the the query execution started on has always the id 0. The other ones get increasing consecutive numbers.

Thread 0 in above query covers everything up to the "gather Streams" iterator. From there to the "Repartition Streams" iterator threads 1 and 2 get to work in parallel. After that up on to the table scans the query is executed by threads 3 and 4. Those two are busily shoveling rows into the Hash iterator. Because a Hash is a blocking iterator (see Properties of Iterators by Craig Freedman(blog) for an explanation of this term), it will not produce any rows until it has consumed all incoming rows. This takes a little over 12 seconds on my machine. During this time the threads 0,1 and 2 are suspended, not taking up any processor resources and therefore not hurting any other thread or query. But they each crank up close to 12 seconds of CXPACKET wait time, for a total of 36 seconds.

Conclusion

The above example showed that any parallel query will show potentially very high CXPACKET waits. This is not an indication of a problem, but rather it is an indication that SQL Server is doing it's job properly, making sure that threads that do not have anything to do at the moment make room for other threads on the processors. So, if your system shows high CXPACKET waits, the only conclusion you can draw from that is, that it is using parallelism and that SQL Server's multitasking system is working as designed.

However, as the sys.dm_exec_requests (as well as the compatibility view sysprocesses) show only the wait type of thread 0, CXPACKET waits can potentially hide a real problem. If you know that you have a problem, use appropriate techniques to narrow down the source. If you don't know if you have a problem, CXPACKET will not help you to change that.

That means, that in almost all situations it is safe to ignore CXPACKET waits. I therefore recommend to add it to the list of waits to ignore in your monitoring queries.