How To Identify The Source Of Page Splits

In my last post I talked about how one could go about troubleshooting an issue in SQL Server, and today I would like to talk about one of the common issues that most DBA’s come across their professional career, Page Splits. When one page becomes full and a new page is needed, this is reported as a page split

Page splits can slow down your queries and increase the amount of I/O needed to retrieve your data. So, how do you monitor and identify the source of page splits, so you can avoid them in your databases?

The short answer is that you can monitor page splits with extended events. There’s an event called page_split. The problem is that there are many kinds of page split. You only need to catch the problematic ones, but the page_split event doesn’t identify this.  A page split is a regular operation – it doesn’t necessarily have bad consequences for your queries. The problem occurs with updates and non-sequential inserts, when a row needs to be inserted in the middle of an object’s pages and there’s no space.

SQL Server creates a new page, transfers half of the page data to the new page and writes the row data. This creates page fragmentation, which is very bad for performance. This is also reported as page split. You can find these problematic page splits using the event sql_server.transaction_log. This event monitors all the activities in the transaction log, so we need to use it with caution. Specifically, you need to filter the operation field, looking for the value 11, which records LOP_DELETE_SPLIT. This is the deletion of rows that happens when SQL Server moves rows from one page to another in a ‘bad’ page split.

This event isn’t visible in the Extended Event interface, so we need to create a session using T-SQL

CREATE EVENT SESSION [BadSplits] ON SERVER

ADD EVENT sqlserver.transaction_log (
    WHERE operation = 11 -- LOP_DELETE_SPLIT 
)
ADD TARGET package0.event_file

-- You need to customize the path
    ( SET filename = N'C:\xelogs\badsplits.xel' )
GO

-- Start the session
ALTER EVENT SESSION [Blocked]
ON SERVER
STATE = START;
GO

After some time, we can query this session to find the source of a page split. The first challenge is to find the database where the page splits are happening. To read the information in the files, you need to use the DMF sys.fn_xe_file_target_read_file. The information will be returned in XML, so we need to use XML methods to retrieve the information:

WITH    qry    AS 
( SELECT 
-- Retrieve the database_id from inside the XML document
theNodes.event_data.value
    ('(data[@name="database_id"]/value)[1]','int') 
AS database_id FROM
(SELECT CONVERT(XML, event_data) 
    event_data -- convert the text field to XML
FROM	-- reads the information in the event files
sys.fn_xe_file_target_read_file
   ('c:\xelogs\badsplits*.xel',NULL, NULL, NULL)
) theData
CROSS APPLY theData.event_data.nodes
   ('//event') theNodes ( event_data ))
SELECT	DB_NAME(database_id) ,COUNT(*) AS total
FROM	qry
GROUP BY DB_NAME(database_id) 
   -- group the result by database
ORDER BY total DESC

Once you know the database with the most page splits, you need to identify the objects that are causing the problem. This is a bit tricky, but here’s how it works (and there’s a code example at the end).

The event only returns the allocation_unit_id, so you need to convert the allocation_unit_id to the object name that owns these pages. The query to convert the allocation_unit_id to an object name needs to run on the database that contains the allocation units, so we’ll filter the results for this specific database. We now need to join the result with the DMV sys.allocation_units using the allocation_unit_id field. This DMV has three kinds of allocation units but we are interested only in two of them: type 1 and type 3, which represent data row pages and row overflow pages.

Next, we need to add sys.partitions to the query, and join the container_id field of sys.allocation_units with the hobt_id field in sys.partitions. Finally, we add sys.objects to the query, joining with the object_id field from sys.partitions, to get the name from sys.objects:

WITH qry AS
	(SELECT
theNodes.event_data.value
   ('(data[@name="database_id"]/value)[1]','int') 
     AS database_id,
theNodes.event_data.value
   ('(data[@name="alloc_unit_id"]/value)[1]','varchar(30)') 
     AS alloc_unit_id,
theNodes.event_data.value
   ('(data[@name="context"]/text)[1]','varchar(30)') 
     AS context
	FROM 
		(SELECT CONVERT(XML,event_data) event_data
		FROM
	sys.fn_xe_file_target_read_file
		('c:\xelogs\badsplits*.xel', NULL, NULL, NULL)) theData
   CROSS APPLY theData.event_data.nodes('//event') 
               theNodes(event_data) )
SELECT name,context,COUNT(*) AS total 
  -- The count of splits by objects
FROM qry,sys.allocation_units au, sys.partitions p, sys.objects ob
WHERE qry.alloc_unit_id=au.allocation_unit_id 
  AND au.container_id=p.hobt_id AND p.object_id=ob.object_id
  AND (au.type=1 or au.type=3) AND
	db_name(database_id)='MDW' -- Filter by the database
GROUP BY name,context -- group by object name and context
ORDER BY name

Now that we know which table is causing the most page splits, we can analyze its indexes to solve the problem.

Please note that although this solution can identify page split problems, you can’t let the session run for a long time, because you’re capturing transaction log activities, which can be too intensive for the server.

Posted in ms sql, Performance Tuning, Troubleshooting | Tagged , , , , , , , , , , | Leave a comment

Troubleshoot Your SQL Server

Hi Everyone. I wanted to ask a question before I begin my post today. And the question is:

When you’re trying to troubleshoot SQL Server, Where do you start looking?

This is one of the most hardest question to answer when you are to troubleshoot your SQL Server in terms of performance tuning. And if you don’t know where to focus your efforts, it’s easy to jump to the wrong conclusion.

Then how do you go about troubleshooting your SQL Server. Well, the key is to look at multiple sources of information, and not to take your data as multiple separate entities. A troubleshooting framework at such scenario, helps enormously. For me this basically trims down to the following items:

1. Wait statistics: A great starting point is to query sys.dm_os_wait_stats. You’re looking for significant system waits, such as an I/O bottleneck. This won’t tell you the root cause, but it’s a great start.

2. Virtual file statistics: Next, you should look at the virtual stats in the sys.dm_io_virtual_file_stats. These, combined with your wait stats, provide further evidence for the source of your problem. From the above mentioned statistics, you’ll be able to see the level of I/O activity and where it’s happening.

3. Performance counters: sys.dm_os_performance_counters provides you with performance counters that specifically relate to SQL Server. For example, if you’re seeing a problem in the disk I/O subsystem, high Full Scans/Sec may indicate a missing index or inefficient code.

4. Plan cache usage: Finally, query the execution information in sys.dm_exec_query_stats, and look for problems related to your findings in the previous stages of analysis. Find statements that have taken the most CPU, the longest execution time, and more.

Combining information from multiple statistics as shown above can help you pinpoint the source of your issue and come up with a solution that can improve the performance of your SQL Server.

Let me know what other statistics do you look at when you have to troubleshoot your SQL Servers and what kind of approach do you take while you’re at it?

Posted in Best Practices, ms sql, Performance Tuning, Troubleshooting, Uncategorized | Tagged , , , , , , , , | Leave a comment

You’re a DBA, you should know these

  • How many databases are you responsible for?
  • How many different version do you support?
  • How much data have we handled this month?
  • How much capacity will we need next year?
  • What new IT techniques will help you get there?

These are some of the questions people from business will tend to ask a DBA, and as soon as such questions are thrown at you, all that comes to ones mind is metrics. These metrics help you answer a lot of questions for the business people and help you get ready for any actions to be taken. For example, when you know about the different versions that you support, you can easily identify when you have to upgrade a server. Another example would be if you know how much data you handled this month, or How much capacity will you need next year, then you can easily ask for a specific amount in the budget that you might need to handle next year’s capacity. Remember that most business folks know what you do, but they don’t necessarily know how you do it. All they understand is numbers that relate to them and you need to be ready with a set of answers for questions as above when thrown at you. You don’t want to be searching for a way to get those answers when asked, but be ready with the answers.

Is every database being backed up?

The answer to this question must be a definitive yes, and nothing but yes. But saying yes isn’t enough, and it’s best to prove it with information coming from MSDB.DBO.BACKUPSET. From here, you can get all the information that you require to answer the above question. Even so, that is not all. For example, Do you know, if you are backing up to a folder on disk, are your backups making it to the tape? It’s not enough to know that you’re backups are going fine. You need to know that their backups are going fine as well. And by saying their , I mean the sysadmins. Are they completing the tape backups properly or not. There is a neat and easy way in windows to do this and almost all tape backup software supports this feature. All you have to do is switch to the details view in the file explorer and select the attributes flag for the files to be seen. You might see the files having the flags ‘A’. This means that these files are ready to be archived and sent to tape. Now whenever a tape backup software checks through these files and moves it to the tapes, most of them have a way to clear this flag and unset them. So, a file that has been moved to the tape, will not have this flag setup as ‘A’. If you are not seeing the flags unset, then its best to talk to the sysadmins and configure the tape backup software to unset the flags once the files are moved to the tape.

Will SQL Email us when something goes horribly wrong?

Again, there should be no other answer to this question than a definite and a confident YES. To do this all one needs to do is to setup the following:

  • SQL Server Agent
  • Database Mail
  • Operator
  • Failsafe operator
  • Alerts

If you have all these configured, then at times when something goes wrong with the SQL Server, then you will get an email. But what if, the SQL Server Agent goes down, as SQL Server Agent is the one to check for anything going wrong and send an email. For this, you can use any 3rd party freeware tool available, to healthcheck on the SQL Server agent. Or you can use the healthcheck tool that you currently use to check in on the status of SQL Server agent. Again, when configuring the above, its best to use distribution lists and not individuals, so that in the absence of one person, someone else can respond to the issue at hand. Also, know that these SQL Server agent alerts are not your best option, but they are free so always include them as a part of your strategy.

Find out if any of your databases have a different collation before it’s too late

Collation can be a headache, if they are unknown to you, or if you know about it too late. This is because, in SQL Server, collation can be server level, database level, schema level and even objects level. So if there are multiple collation in a single server. It would make your work a lot easier if you knew about all those different collations, before doing any work in production machines. Imagine a scenario where your database is in collation A and your server is in collation B. Now you go ahead and create a temporary table and insert data from your database. Which collation do you think the data in the temporary table will be?

The answer is, it will inherit whatever collation the server is in, unless the temporary table that was created has been explicitly assigned a collation in its DDL. And this messes up a lot of things especially sorting and storage in SQL Server.

Are you sure all of our servers are set up right?

There is no second answer to this question. It’s always the same, and must remain that way. And that answer is a big overwhelming YES. This is something that you as a DBA must answer with a yes and to prove it, you must be ready to provide supporting answers in a jiffy. Now you must be thinking, I have tons of servers that I handle, how can I query all of them at once and provide answers to the stakeholders. Well, there are a lot of ways to do that, but the most easiest and convenient way to do it would be through server registration and central management server in the SSMS. You can group all your servers into categories like production, DR, development, etc and even into versions, as there are usually differences in the DMV query output in the version when you do a SELECT * in any of them. Having said so, once you have this setup, you can easily fire a multi-server query and get your results, that can help you know if your servers are all setup right or not.

Finally, can you get answers to all these questions really fast?

When business people come to you with such questions, they want answers, fast. They don’t have the time for you to query the results and make them wait. When they say now, they mean it. So how do you go about getting all these answers in a moment notice, so that you are ready to splash out the metrics to the business people and support your requirements or requests for budget or hardware or anything that you need to support proper working of the system.

Here is something I personally use and have been using for a long time and find it ever so more an arsenal in my code repository to get my metrics right.

Brent Ozar’s sp_Blitz

This was a life saver for me when I began my career as a DBA, and gave me information about my servers that I wasn’t even aware of. This way I avoided a lot of scenarios that could have gotten my servers in a whole lot of trouble and now it’s currently helping me in providing the business people the metrics they ask for and it’s also helping me to fine tune my servers for better operation.

What other things do you think a DBA should know, other than the ones that I listed? Let me know your thoughts on this one in your comments below. Tune in until next time.

Posted in Best Practices, ms sql, Uncategorized | Tagged , , , , , , , , , | Leave a comment

Better…..Faster….SQL Server Backups

Ever since I phased out the legacy backup system in one of my recent projects with a centralized SQL Server backup solution, I have been asked by many people regarding the results that I got from the transition. To put it simply, the backup size reduced by over 400 percent (from 3.4 TB to 700 GB) and the time spent on backup tasks went from 6 hours to 1.5 hours. Now these are some good numbers to put up for a case study scenario, but how exactly did I achieve these results.

My answers usually begin by explaining the process using some of the terms that are relevant to the SQL Server backup command, and if people are not using SQL Server backup command to backup their database, then they might not know some of them. Today, I just want to talk about some of these essential backup options provided in SQL Server that can help any DBA make SQL Server backups go faster and better.

COMPRESSION

Backup compression was added in SQL Server 2008 in Enterprise Edition, but is now available in standard edition as well from SQL Server 2008 R2 onwards. This makes your backups smaller, and can also speed up your backup operation as well as recovery operations. The only trade-off is that compressing and decompressing uses more CPU, but this is usually not a problem.

COPY_ONLY

I have come across many situations where people have gone ahead and made a one-off full backup of databases that are scheduled for backups. In doing so, the one-off full backup becomes the basis for subsequent differential backups and so the backup chain disrupts. Especially, if the one-off full backup is not recorded properly. In case of a disaster recovery operation, this single one-off full backup can disrupt the disaster recover process. To avoid having one-off backups affecting the backup chain, you can use the COPY_ONLY option, as this option makes the backup NOT become the basis for subsequent differential backups, and for a log backup, this option makes the backup NOT become the part of the log backup chain.

CHECKSUM

This option does two things:

  1. As the data file pages are being read from the data files, their page checksums are validated. If an invalid checksum is found, by default the backup will fail and report a message indicating which page is corrupt.
  2. Calculate a checksum over the entire backup and store it in the backup header

From above we can conclude that this option is a way for SQL Server to determine that the I/O subsystem has corrupted a page. So, if you have backups setup with this option and your backups are failing then that could mean corruption in your database. This would make, usage of this option an absolute requirement and should be used on all backups, manual or automated.

The next and last of the BACKUP options are something that not many people are used to work with but can help in speeding up the backup process. There are no fixed values for these options to follow, but rather need a trial and measure approach. I suggest enabling the following trace flags prior to the tests to know what values are best for the following BACKUP options in your particular server setup

DBCC TRACEON (3605, 1)
DBCC TRACEON (3213, 1)

BLOCKSIZE

This specifies the physical block size, in bytes. The supported sizes are 512, 1024, 2048, 4096, 8192, 16384, 32768, and 65536 (64 KB) bytes. The default is 65536 for tape devices and 512 otherwise. From my tests, I have found that by specifying a large 64 KB sector size instead of the default 512 bytes typically shows a 5-6% improvement in backup throughput.

BUFFERCOUNT

This specifies the total number of I/O buffers to be used for the backup operation. The total space that will be used by the buffers is determined by buffercount * maxtransfersize.

MAXTRANSFERSIZE

This specifies the largest unit of transfer in bytes to be used between SQL Server and the backup media. The possible values are multiple of 64 KB ranging up to 4194304 bytes (4 MB). The default is 1 MB.

Once you have enabled the trace flags, you can play around these values and find a combination that gives you the maximum buffer space usage. Once that is reached it is not fruitful to increase the values of the parameters.

These are the parameters that I played with in the central backup system deployed by me making use of the above mentioned BACKUP options to get a better and faster SQL Server backup. To summarize, Backup compression is a great feature to use, as it will save you disk capacity and reduce the time needed to backup your data. You can also add multiple backup destination files which I have not mentioned here, but is another efficient way to increase backup throughput. Then, by specifying the BUFFERCOUNT, BLOCKSIZE and MAXTRANSFERSIZE parameters you can typically double the backup throughput, which means reducing the backup time-taken by half. So, the next time you need to a better and faster backup option, don’t just ask for better hardware, look for these options and see if you can achieve your task using these BACKUP options.

That’s all for today folks. Let me know if you have come across such backup performance improvement task and what exactly did you do to resolve it.

Posted in backup, Best Practices, ms sql | Tagged , , , , , , , | 1 Comment

Find the locks

There are many blog posts on the internet that have explained time and again regarding this topic. Many of them are pretty basic stuff while some of them are quite advanced. A while back I myself wrote a blog post explaining how to find blocking sessions in Oracle database. These posts usually talk about activity monitor, sp_who2, sysprocesses, etc.

A simple query to provide us information regarding locks in SQL Server would look like this

SELECT * FROM sys.dm_os_waiting_tasks
WHERE blocking_session_id IS NOT NULL
GO

This code block is quite helpful because looking carefully at the query output tells us who is waiting for whom which lets us know exactly what is to be done for the lock to be removed.

Although this query is already sufficient in itself, I wanted a query output which would help to understand the locks visually for all sorts of stakeholders, when I as a DBA, am requested for such information. And even before I could write such query I found a blog post from Pinal Dave having such a query which is shown below

SET NOCOUNT ON
GO
SELECT SPID, BLOCKED, REPLACE (REPLACE(T.TEXT, CHAR(10), ‘ ‘), CHAR (13), ‘ ‘ ) AS BATCH
INTO #T
FROM sys.sysprocesses R CROSS APPLY sys.dm_exec_sql_text(R.SQL_HANDLE) T
GO
WITH BLOCKERS (SPID, BLOCKED, LEVEL, BATCH)
AS
(
SELECT SPID,
BLOCKED,
CAST (REPLICATE (‘0′, 4-LEN (CAST (SPID AS VARCHAR))) + CAST (SPID AS VARCHAR) AS VARCHAR (1000)) AS LEVEL,
BATCH FROM #T R
WHERE (BLOCKED = 0 OR BLOCKED = SPID)
AND EXISTS (SELECT * FROM #T R2 WHERE R2.BLOCKED = R.SPID AND R2.BLOCKED = R2.SPID)
UNION ALL
SELECT R.SPID,
R.BLOCKED,
CAST (BLOCKERS.LEVEL + RIGHT (CAST ((1000 +R.SPID) AS VARCHAR (100)), 4) AS VARCHAR(1000)) AS LEVEL,
R.BATCH FROM #T AS R
INNER JOIN BLOCKERS ON R.BLOCKED =BLOCKERS.SPID WHERE R.BLOCKED > 0 AND R.BLOCKED = R.SPID
)
SELECT N’ ‘ + REPLICATE (N’| ‘,LEN (LEVEL)/4 – 1) +
CASE WHEN (LEN(LEVEL)/4 – 1) = 0
THEN ‘HEAD – ‘
ELSE ‘|—— ‘ END
+ CAST (SPID AS NVARCHAR (10)) + N’ ‘ + BATCH AS BLOCKING_TREE
FROM BLOCKERS ORDER BY LEVEL ASC
GO
DROP TABLE #T
GO

The query output from the above code block gives a visual representation of the block that is present in the database which can be called a blocking tree. This piece of code is my daily go to code for finding locks and their respective locking sessions in the database.

Let me know what codes or tools do you use to find locks in your database.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

TempDB configuration, the way I prefer

TempDB configuration is one of the few things, that I as a SQL Server DBA had to handle as soon as I got to work with production machines. Considering there was no SQL Server DBA prior to my appointment, there were a lot of things that were incorrectly configured or were left with default configurations. There was a lot of things I had to do, but I singled out this particular configuration as it was easier to do and would bring about drastic performance improvement.

As the TempDB configuration were left as default when installed, the production machines I was working with had only 1 physical data file for the TempDB, which is wrong in so many aspects, that it cannot be included in this post. I shall leave it for later. Now, to adjust the TempDB configuration I usually follow a simple rule as below:

  1. If you have less than or equal to 8 logical cores, i.e. one CPU server with 4 physical cores and hyperthreading enabled has 8 logical cores, then use number of TempDB data files = logical cores
  2. If you have number of logical cores greater than 8 but less than or equal to 24, then use number of TempDB data files = 1/2 * (logical cores)
  3. And finally, if you have number of logical cores greater than 24, then use number of TempDB data files = 1/4 * (logical cores)

To find the number of logical processors we can use the following query:

SELECT cpu_count AS logicalCPUs FROM sys.dm_os_sys_info

One thing to mention here though is that the additional data files that you add, has to be of similar configuration to the already present data file for the tempDB. This means that the initial size, and increment of the data files has to be similar. But this can be done only when you are configuring a fresh server. Then what about a server, which is already being used? In such case, I’d advise to add additional data files with initial size greater than that of the existing data file of the TempDB and then increase the size of the existing data file to match the size of the new files.

To know the configuration of the existing TempDB we can use the following query:

selectDB_NAME(mf.database_id) database_name
, mf.namelogical_name, mf.file_id
, CONVERT(DECIMAL(20,2)
, (CONVERT(DECIMAL, size)/128)) as[file_size_MB]
, CASEmf.is_percent_growth
WHEN1 THEN'Yes'
ELSE'No'
ENDAS[is_percent_growth]
, CASEmf.is_percent_growth
WHEN1 THENCONVERT(VARCHAR, mf.growth) + '%'
WHEN0 THENCONVERT(VARCHAR, mf.growth/128) + ' MB'
ENDAS[growth_in_increment_of]
, CASEmf.is_percent_growth
WHEN1 THENCONVERT(DECIMAL(20,2)
,(((CONVERT(DECIMAL, size)*growth)/100)*8)/1024)
WHEN0 THENCONVERT(DECIMAL(20,2)
, (CONVERT(DECIMAL, growth)/128))
ENDAS[next_auto_growth_size_MB]
, physical_name fromsys.master_files mf
wheredatabase_id =2 andtype_desc= 'rows'

I prefer this way of adding data files to the TempDB because I have seen cases when someone just goes ahead and adds an additional data file to the TempDB without considering what exactly is going wrong. They just find that allocation contention is being seen with the TempDB and just add an extra TempDB. Now although, the idea to solve the issue is correct, but the process is completely wrong. Let me tell you why.

When there is one tempdb data file, allocation bitmap contention will occur. And after the server has been using tempdb for a while, it may become full. What might happen the allocations, if we add more data file? Basically, allocations use 2 algorithms and it will try to allocate from each file in the filegroup in turn, but proportionally more frequently from files that have more space. Now when one file is very full and the other is empty, the majority of the allocations will be from the new file. Which means that almost all of the contentions moves from the existing tempdb data file to the new one, which again does not solve the problem of overall contention. In such cases, adding some more data files will help so that the allocation system has multiple files, spreading the contention over those files and leading to an overall drop in contention and increase in transaction throughput.

Let me know which way do you prefer when configuring TempDB in your machines.

Posted in Uncategorized | Leave a comment

Corruption for the Accidental DBA’S

I apologize to all the readers for the lay-off that I had been in which resulted in me discontinuing my blog posts. I hope that with this post I can catch up to my intended speed of blogging useful information for all my colleagues and enthusiasts in the database domain.

With that I want to shed some light on the different verticals of database domain I have had the privilege of working on. Currently I work on Oracle, MS-SQL Server and Vertica technologies but since I began blogging I feel I have not shared information other than Oracle DB. So with this blog post I want to share about corruption in MS-SQL database and how to handle it, which I think is a major conundrum among professionals in the MS-SQL DB domain. I intend to keep this post and the subsequent follow-up to this post very simple and focused on the DBCC CHECKDB command, how to interpret it and how to resolve corruption.

To begin with let me say that corruption does happen, most of the MS-SQL professionals haven’t seen corruption or tend to ignore them claiming that they have the best hardware and system set-up. But at some point in time you are going to see corruption so it is best to be prepared for such situation to avoid data loss and downtime. Corruption happens mostly due to IO subsystem problems and most people don’t realize that they have corruption until its too late mostly because either they don’t know how to check for corruption or they miss the warning signs. On the other hand, people don’t know what to do when they do have corruption which leads to more data loss and downtime than necessary, monetary and even job loss, overall lowered perception of SQL server which makes it harder to convince the management that SQL server is Enterprise capable.

So, for these DBA’s who have never seen corruption, one of the common reaction when faced with corruption is to panic, and they tend to do things which make matters worse such as dropping a database or dropping a suspect database or running repair or rebuilding log causing more corruption.
Now let us come to the target of the subsequent posts that I will be blogging about. I shall talk about tools and methods to discover corruption and how to handle them. I will focus primarily on the CHECKDB command output, how to interpret the output and find what and where the corruption is. From this interpretation of the CHECKDB command output we shall than be able to choose between a restore or a repair, whatever might be the case. And finally we shall look at recovering from last resort, and that is our backups.
Ok, so how can we know when something has gone wrong? Well, we can assert the corruption from various ways, primarily from application/user connection betting broken frequently which happens when SQL server tries to process data that is corrupt or if the SQL server hits 823 or 824 errors which hints at IO subsystem problem. Another way to catch corruption is if we have regular backup jobs configured with checksum commands, and if these start to fail then we will be seeing error like 3043, which is the backup has found a checksum and it’s not going to finish the backup due to corrupt database. We can also catch corruption if we have alerts configures in our system. We should have alerts for any error with severity >=19 which will catch errors like 823, 824, etc.. And if such alerts start firing, means we have corruption. We have another error 825 which is a bit tedious to catch. As of SQL server 2005, if it gets 823, 824 error in few attempts, it will retry few times and if after few attempts if it succeeds, then everything carries on as far as the user is concerned. All the DBA knows about this is that an error with severity 10 and error message 825 is logged. So we should have alerts for this error 825 as well. Another sure shot way of finding the corruption is that if the regular maintenance jobs start failing. So it should be made customary to regularly check the maintenance jobs.

Now then as we have suspected corruption via the methods mentioned above, what do we do next. The first and foremost thing to do is, to not panic. The ideal thing is to have a certain set of steps or a certain algorithm to be followed which should be designed by the most experienced DBA and tested by the most junior DBA. This is what a proper DR plan should be as no one knows when a corruption is going to happen and who is going to be on support to handle the corruption. Next in process is to determine the extent of the corruption before resolving it. As it turns out in most of the cases the corrupted database might be easily repaired rather than have to be restored. I have seen cases when people have gone ahead and done things randomly to fix small meta-data issues causing even more corruption. One way to identify the extent of corruption is to run the CHECKDB command, and my subsequent posts are going to rely heavily on this command. We shall talk about it sooner. Again, we should also look at the SQL server error logs because the corruption might have been detected today but it might be an accumulated result of a corruption that began several days ago. This way we can easily find the extent of the corruption and deduce the best contingency plan. Similarly, it is also a good practice to check maintenance job history which will tell us the same thing as to from when did the database find corruption.

Another item in the check-list is to check for what backups are available. In some scenarios we might only need to restore in some point in time and as such might not need to do a full restore. So it is advised to have backups as per required and check for what kind of backups we have to ascertain the contingency plan to recover from the corruption. And finally, I will always stress on to wait for the CHECKDB command to finish before doing anything else as you may not need to do anything intrusive/destructive.

That’s it for today’s post. I shall follow-up this post with another one where we will be looking at the DBCC CHECKDB command and look at some scenarios and how to handle them.

P.S. Comments/suggestions/feedbacks are always welcomed

Posted in ms sql, Uncategorized | Tagged , , , , , , | Leave a comment

REDO……How do you do?????

Its been quite a while since my last post, and I am finally breaking the silence streak. The reason it took me this long would be because I was in a dilemma regarding do I continue the Tuning series from where I left or do I follow up my last post regarding TEMP tablespace configuration with another important database configuration which plays a vital role in performance.
Well I made up my mind and as my blog post headline suggests, this post is going to be about REDO. Quite recently I was in a discussion with one of my LinkedIn friends regarding a problem he had come up with his REDO log files. We shared few information regarding how he had his REDO log files configured like how many groups, how man logs per groups, size of each log and their location, etc. Eventually, the problem was zeroed in and we found that his REDO log files were being switched every 20 minutes. He has his REDO log files configured with 1GB per REDO log file. WOW, that is some good amount of REDO being generated.
From this scenario, we can only conclude that we need to create enough REDO log files with ample space to minimize such rapid log file switches, as it causes the database to wait for the log to switch before completing any transaction that are currently in queue. But how do we know how much REDO is enough? This is not something we can predict beforehand especially for a new system being placed. If we know how much data bandwidth our database uses then we might have some idea but for most of the cases this work post oracle installation/configuration and only once when we are working with the database.
Now then that we have our database in place, we need to have some sort of weapon with us that can tell us how much REDO is being generated, how much log file switches are being occurred. So that we can configure our REDO log files to handle such scenarios and avoid database to wait as much as possible. Below is a sample script that again uses AWR data mining to display information as how much REDO is being generated, by what size should the REDO log files be increased and how much time is it taking to switch the REDO log files whenever it has occurred in the system.
with t_interval as
(
select /*+ inline */ sysdate-30 begin, sysdate as end from dual
)
select
stats.dbid dbid,
stats.instance_number instance_number,
to_char(stats.begin_interval_time, 'YYYYMMDD HH24MI') tim,
stats.cpu_used / 100 cpu_sec,
stats.phy_reads phy_reads,
stats.phy_writes phy_writes,
stats.cr_served cr_served,
stats.current_served current_served,
stats.redo_size / 1024 / 1024 redo_mb,
procs.current_utilization processes,
--
waits.df_seq_micro / 1000 / nullif(waits.df_seq_waits,0) avg_df_seq,
waits.df_scat_micro / 1000 / nullif(waits.df_scat_waits,0) avg_df_scat,
(stats.redo_size - stats.md_redo_size) * 100 / stats.md_redo_size redo_diff_to_md_pct,
stats.redo_write_time*10/stats.redo_writes avg_lfpw,
waits.log_sync_micro/nullif(waits.log_sync_waits, 0) / 1000 avg_log_sync,
waits.log_ckpt_micro/1e6 log_ckpt_sec,
( stats.redo_size /
( waits.snap_interval * 86400 ) ) *
( waits.log_ckpt_micro/1e6 ) / 1024 / 1024 redo_needed,
stats.is_restart
from
(
select
snap_id,
begin_interval_time,
snap_interval,
instance_number,
dbid,
log_sync_micro,
log_sync_waits,
log_ckpt_micro,
log_ckpt_waits,
df_seq_micro,
df_seq_waits,
df_scat_micro,
df_scat_waits,
direct_micro,
direct_waits,
median(log_sync_micro/nullif(log_sync_waits, 0)) over (partition by dbid, instance_number) md_log_sync_micro
from
(
select
snap_id,
begin_interval_time,
instance_number,
dbid,
max(snap_interval) snap_interval,
max(decode(event_name, 'log file sync', wait_micro)) log_sync_micro,
max(decode(event_name, 'log file sync', total_waits)) log_sync_waits,
max(decode(event_name, 'log file switch (checkpoint incomplete)', wait_micro)) log_ckpt_micro,
max(decode(event_name, 'log file switch (checkpoint incomplete)', total_waits)) log_ckpt_waits,
max(decode(event_name, 'db file sequential read', wait_micro)) df_seq_micro,
max(decode(event_name, 'db file sequential read', total_waits)) df_seq_waits,
max(decode(event_name, 'db file scattered read', wait_micro)) df_scat_micro,
max(decode(event_name, 'db file scattered read', total_waits)) df_scat_waits,
max(decode(event_name, 'direct path read', wait_micro)) direct_micro,
max(decode(event_name, 'direct path read', total_waits)) direct_waits
from
(
select
e.snap_id,
e.instance_number,
e.dbid,
sn.begin_interval_time,
cast(begin_interval_time as date) - cast(lag(begin_interval_time) over (partition by e.dbid, e.instance_number, e.event_name order by sn.begin_interval_time) as date) snap_interval,
sn.startup_time,
e.event_name,
case when (sn.begin_interval_time >= sn.startup_time and lag(sn.begin_interval_time) over (partition by e.dbid, e.instance_number, e.event_name order by sn.begin_interval_time) = sn.startup_time and lag(sn.begin_interval_time) over (partition by e.dbid, e.instance_number, e.event_name order by sn.begin_interval_time) = sn.startup_time and lag(sn.begin_interval_time) over (partition by stats.dbid, stats.instance_number, stats.stat_id order by sn.begin_interval_time) = sn.startup_time and lag(sn.begin_interval_time) over (partition by stats.dbid, stats.instance_number, stats.stat_id order by sn.begin_interval_time) < sn.startup_time)
then 'Yes'
end is_restart
from
dba_hist_sysstat stats,
dba_hist_snapshot sn,
t_interval t
where
sn.snap_id = stats.snap_id and
sn.dbid = stats.dbid and
sn.instance_number = stats.instance_number and
sn.begin_interval_time between t.begin and t.end and
stats.stat_name in (
'redo size',
'redo write time',
'redo writes',
'CPU used by this session',
'physical read total IO requests',
'physical reads cache',
'physical write total IO requests',
'physical writes from cache',
'gc cr blocks served',
'gc current blocks served'
)
)
group by dbid, instance_number, begin_interval_time, snap_id
)
) stats,
(
select
stats.snap_id,
stats.instance_number,
stats.dbid,
stats.resource_name,
stats.current_utilization
from
dba_hist_resource_limit stats,
dba_hist_snapshot sn,
t_interval t
where
sn.snap_id = stats.snap_id and
sn.dbid = stats.dbid and
sn.instance_number = stats.instance_number and
sn.begin_interval_time between t.begin and t.end and
stats.resource_name = 'processes'
) procs
where
waits.dbid = stats.dbid and
waits.instance_number = stats.instance_number and
waits.snap_id = stats.snap_id and
waits.dbid = procs.dbid and
waits.instance_number = procs.instance_number and
waits.snap_id = procs.snap_id
order by
stats.dbid, stats.instance_number, stats.begin_interval_time
;

In the output, the column REDO_MB is the amount of redo log being generated, the column REDO_NEEDED is the amount of additional REDO size needed to avoid log switch and AVG_LOG_SYNC is the average time taken to switch the REDO logs. The query also provides additional information such as CPU utilization, physical I/O, etc which all serves in finding the right combination of REDO log file side, number and groups to create optimal database system to perform in the most optimize way possible.
That is all for this time. Hope to see you all soon with my next post. Feel free to drop in comments or let me know if you want me to post on something.

Posted in Best Practices, Oracle | Tagged , , , , , , | Leave a comment

Manage Your Temporary…….Permanently

Taking a break from the tuning trio-logy, this time I wanted to share my views on addressing something in Oracle database, that I have come across multiple times in my professional career. And every time I came across this question I always had the same answer. In-fact, when one is trying to enhance or troubleshoot performance in the database realm, they have only 2 options:

  1. Increase available hardware resources
  2. configure and maintain the database as per the best practice for the business

Looking at the options available, the 1st option is always tempting one because it is easy. But it may not be affordable always, and in some point of time can backfire as well. That is why going about the 2nd option should always be the 1st line of action to be taken. In the same context, this time I wanted to talk about managing the Temporary Tablespace in Oracle as per the best practice.

So, what is this temporary tablespace exactly? To be simple, A temporary tablespace contains transient data that persists only for the duration of the session. Within a temporary tablespace, all sort operations for a given instance and tablespace share a single sort segment. Sort segments exist for every instance that performs sort operations within a given tablespace. The sort segment is created by the first statement that uses a temporary tablespace for sorting, after startup, and is released only at shutdown. Hmmmmmmm…….is released only at shutdown, so this would mean that the temporary tablespace keeps growing. And if it keeps growing then we might run out of space if we do not configure the tablespace correctly. For example, while creating the temporary tablespace, if the clause tempfile is not used then a datafile is created for the use of temporary tablespace which is not a good way to start the database configuration. So even though the tablespace is called temporary tablespace, because of the presence of a datafile it would actually be a permanent tablespace marked as temporary.

So how does one configure a temporary tablespace for optimum performance? Lets look at some best practices that can be followed:

  • Use locally managed temporary tablespace
    Locally managed temporary tablespaces use tempfiles, which do not modify data outside of the temporary tablespace or generate any redo for temporary tablespace data. Because of this, they enable you to perform on-disk sorting operations in a read-only or standby database. Locally managed temporary tablespaces are uniform extent tablespaces
  • Guidelines for choosing extent size while creating the temporary tablespace
    For DSS, OLAP applications involving huge sorts, hash joins or when large temporary LOBs are predominant then choose between 1M-10M
    When,Global temporary tables are predominant and amount of data loaded is small or when the application is predominantly OLTP then choose between 64K or multiple
  • For a RAC, use single temp tablespace for entire database
  • Create temporary tablespace group
    One or more temp tablespace can be assigned to a tablespace group, and a user or a database can be assigned to an entire group
  • Place the tempfile for the temp tablespace on a separate disk device other than the root partition and the disk where our buisness data resides.
  • Specify a TEMPORARY TABLESPACE for each user (schema). If one is not specified, Oracle defaults to SYSTEM, which is almost guaranteed to have a negative impact on performance.
  • Finally, V$TEMPSEG_USAGE can be used to monitor space usage and workload distribution, which gives us a clear picture of how the temporary tablespace is being used and we can pinpoint the culprit queries that are extensively using the temp tablespace. This way, we can tune those queries and enhance database performance

And that was a shorthand on how to mange the temporary tablespace……..permanently. Again, these are just guidelines to be followed. And before implementing it in the production environment, it is always best to test the configuration as per one’s business needs and then roll-out into the production.

Posted in Best Practices, Oracle | Tagged , , , , , , | Leave a comment

Tuning 101: Automatic Memory Management (AMM)

In my previous posts we came across few parameters that can affect the performance of Oracle database, such as SHARE POOL, BLOCK BUFFERS, SORT AREA, etc. Adding to these lists there can be other parameters too, which can affect the database as per the database design. Now tuning is an ongoing process, as no database system can be tuned once and let it be that way. Performance issues are bound to happen, and the database has to be tuned to address such issues. Going by what I have shared till now, for every such issues, changing several of these parameters can be a headache. But fortunately, Oracle has made great strides in simplifying memory management over the last few versions of the database. Oracle 9i automated PGA management by introducing PGA_AGGREGATE_TARGET parameter. Oracle 10g continued this trend by automating SGA management using the SGA_TARGET parameter. Oracle 11g takes this one step further by allowing you to allocate one chunk of memory, which Oracle uses to dynamically manage both the SGA and PGA.

Automatic memory management is configured using two new initialization parameters:

  • MEMORY_TARGET : The amount of shared memory available for Oracle to use when dynamically controlling the SGA and PGA. This parameter is dynamic, so the total amount of memory available to Oracle can be increased or decreased, provided it does not exceed the MEMORY_MAX_TARGET limit. The default value is “0”.
  • MEMORY_MAX_TARGET : This defines the maximum size the MEMORY_TARGET can be increased to without an instance restart. If the MEMORY_MAX_TARGET is not specified, it defaults to MEMORY_TARGET setting.

Enabling automatic memory management on a system that didn’t previously use it, is a simple task. Assuming one want to use a similar amount of memory to your current settings you will need to use the following calculation.

MEMORY_TARGET = SGA_TARGET + GREATEST(PGA_AGGREGATE_TARGET, "maximum PGA allocated")

The following queries show you how to display the relevant information and how to combine it in a single statement to calculate the required value.

-- Individual values.
SELECT name, value
FROM   v$parameter
WHERE  name IN ('pga_aggregate_target', 'sga_target')
UNION
SELECT 'maximum PGA allocated' AS name, TO_CHAR(value) AS value
FROM   v$pgastat
WHERE  name = 'maximum PGA allocated';

-- Calculate MEMORY_TARGET
SELECT sga.value + GREATEST(pga.value, max_pga.value) AS memory_target
FROM (SELECT TO_NUMBER(value) AS value FROM v$parameter WHERE name = 'sga_target') sga,
     (SELECT TO_NUMBER(value) AS value FROM v$parameter WHERE name = 'pga_aggregate_target') pga,
     (SELECT value FROM v$pgastat WHERE name = 'maximum PGA allocated') max_pga;

Assuming our required setting was 10G, we might issue the following statements.

CONN / AS SYSDBA
-- Set the static parameter. Leave some room for possible future growth without restart.
ALTER SYSTEM SET MEMORY_MAX_TARGET=11G SCOPE=SPFILE;

-- Set the dynamic parameters. Assuming Oracle has full control.
ALTER SYSTEM SET MEMORY_TARGET=10G SCOPE=SPFILE;
ALTER SYSTEM SET PGA_AGGREGATE_TARGET=0 SCOPE=SPFILE;
ALTER SYSTEM SET SGA_TARGET=0 SCOPE=SPFILE;

-- Restart instance.
SHUTDOWN IMMEDIATE;
STARTUP;

Once the database is restarted the MEMORY_TARGET parameter can be amended as required without an instance restart.

ALTER SYSTEM SET MEMORY_TARGET=9G SCOPE=SPFILE;

Once the AMM settings are in place, tuning the memory parameters can be done very easily. The V$MEMORY_TARGET_ADVICE view provides information to help tune the MEMORY_TARGET parameter. It displays a range of possible MEMORY_TARGET settings, as factors of the current setting, and estimates the potential DB Time to complete the current workload based on these memory sizes.

When you have large SGA sizes you can get considerable benefits from using HugePages. Automatic Memory Management and HugePages on Linux are not compatible, which means AMM is probably not a sensible option for any large systems. Instead, AutomaticSharedMemoryManagement and AutomaticPGAManagement should be used as they are compatible with HugePages.

We shall know about AutomaticSharedMemoryManagement and AutomaticPGAManagement along with HugePages in the next post. See you all soon.

Posted in Oracle | Tagged , , , , , , | Leave a comment