Managing Indexes on large tables

Hi I wonder if anyone could give some advice around managing a non clustered
Index on a large table:
rows reserved data
index_size unused
229324002 27215584 KB 20200600 KB 6940968 KB 74016
KB
We keep getting problems with this index as it is on a large & busy table -
Ideally I'd rebuild the index every day but it is a very busy production
system & that just isn't practical.
Would Index defragmentation help with Invalid key/corruption issues & the
general health of this index? or has anyone any advice on mangement
strategies for big tables/indexes'
Many Thanks for any input on this!
--
Mike Knee
Attenda Monitoring & ManagementDefragmentation is done for performance purposes. I don't know what you mean by "invalid key", but
if you have corruption issues, you need to get to the root cause of this.
--
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
http://www.solidqualitylearning.com/
Blog: http://solidqualitylearning.com/blogs/tibor/
"Sinister China Penguin" <SinisterChinaPenguin@.discussions.microsoft.com> wrote in message
news:6195F440-E5C7-44E7-B162-70E5AA99A8CE@.microsoft.com...
> Hi I wonder if anyone could give some advice around managing a non clustered
> Index on a large table:
> rows reserved data
> index_size unused
> 229324002 27215584 KB 20200600 KB 6940968 KB 74016
> KB
> We keep getting problems with this index as it is on a large & busy table -
> Ideally I'd rebuild the index every day but it is a very busy production
> system & that just isn't practical.
> Would Index defragmentation help with Invalid key/corruption issues & the
> general health of this index? or has anyone any advice on mangement
> strategies for big tables/indexes'
> Many Thanks for any input on this!
> --
> Mike Knee
> Attenda Monitoring & Management|||If you are getting errors sucha s that you most likely have corruption in
the index. Since it is a nonclustered index I suggest you drop and recreate
it or run DBCC DBREINDEX on just that index to be sure you get a clean
index. It might be busy but you need to fix it. It's unlikely that you need
to rebuild it every night. If you do then you may want to adjust your fill
factor to avoid the pagesplits.
--
Andrew J. Kelly SQL MVP
"Sinister China Penguin" <SinisterChinaPenguin@.discussions.microsoft.com>
wrote in message news:6195F440-E5C7-44E7-B162-70E5AA99A8CE@.microsoft.com...
> Hi I wonder if anyone could give some advice around managing a non
> clustered
> Index on a large table:
> rows reserved data
> index_size unused
> 229324002 27215584 KB 20200600 KB 6940968 KB
> 74016
> KB
> We keep getting problems with this index as it is on a large & busy
> table -
> Ideally I'd rebuild the index every day but it is a very busy production
> system & that just isn't practical.
> Would Index defragmentation help with Invalid key/corruption issues & the
> general health of this index? or has anyone any advice on mangement
> strategies for big tables/indexes'
> Many Thanks for any input on this!
> --
> Mike Knee
> Attenda Monitoring & Management|||On Fri, 10 Feb 2006 11:03:29 -0800, "Sinister China Penguin"
<SinisterChinaPenguin@.discussions.microsoft.com> wrote:
>Hi I wonder if anyone could give some advice around managing a non clustered
>Index on a large table:
> rows reserved data index_size unused
> 229324002 27215584 KB 20200600 KB 6940968 KB 74016 KB
Well, that's reasonably large, alrighty!
>We keep getting problems with this index as it is on a large & busy table -
>Ideally I'd rebuild the index every day but it is a very busy production
>system & that just isn't practical.
>Would Index defragmentation help with Invalid key/corruption issues & the
>general health of this index? or has anyone any advice on mangement
>strategies for big tables/indexes'
That is a lot of rows on one table. Have you looked at partitioned
views (SQL2000) or partitioned tables (SQL2005)?
Only situation I've had with that many rows was with them split into
ten tables and joined with a partitioned view, had no problems with
that.
Can you say what is going on with that table when the problems occur,
I presume not all select's?
Also, what the PK is, and what the clustered key is, if different?
For that matter, what is the key in this index - single field int, or
multiple varchars, or what?
J.|||Thanks for the replies - some more details: (System is SQL2000 clustered on
Win2K)
- The table collects data points for I.T. systems performance (perfmons etc)
on 800 odd servers so there are hundreds of Inserts going on 24/7 & no
updates.
- Once a day a maintenance job runs to delete any data points > 200 days old
so the size of the table stays roughly the same...
- This table has just the one Non-Clustered Index on DataID(int - non
unique) , Time(Int) & Value (float)
- The DataID is non unique as it id defined in a "Header" table which
contains all the definitions for the performance data being collected (name
of perfmon, system name etc), the V large table I originally posted about
holds the actual performance data so there are thousands of entries for each
dataID.
- The Table is part of a commercial product (NetIQ's AppManager) so I have
no control over the schema/Indexes etc.
I'm not sure when the problems occur - I'm gussing during a flurry of Inserts?
I hope this all makes sense - I can't help but think this table could be
managed better - should I force a Full table lock for example when the
deletes are taking place? or can I manage the Index better (hence the
questions about defragging/rebuilding)
Thanks again for any advice or ideas.
--
Mike Knee
Attenda Monitoring & Management
"JXStern" wrote:
> On Fri, 10 Feb 2006 11:03:29 -0800, "Sinister China Penguin"
> <SinisterChinaPenguin@.discussions.microsoft.com> wrote:
> >Hi I wonder if anyone could give some advice around managing a non clustered
> >Index on a large table:
> >
> > rows reserved data index_size unused
> > 229324002 27215584 KB 20200600 KB 6940968 KB 74016 KB
> Well, that's reasonably large, alrighty!
> >We keep getting problems with this index as it is on a large & busy table -
> >Ideally I'd rebuild the index every day but it is a very busy production
> >system & that just isn't practical.
> >
> >Would Index defragmentation help with Invalid key/corruption issues & the
> >general health of this index? or has anyone any advice on mangement
> >strategies for big tables/indexes'
> That is a lot of rows on one table. Have you looked at partitioned
> views (SQL2000) or partitioned tables (SQL2005)?
> Only situation I've had with that many rows was with them split into
> ten tables and joined with a partitioned view, had no problems with
> that.
> Can you say what is going on with that table when the problems occur,
> I presume not all select's?
> Also, what the PK is, and what the clustered key is, if different?
> For that matter, what is the key in this index - single field int, or
> multiple varchars, or what?
> J.
>|||Sinister China Penguin wrote:
> Thanks for the replies - some more details: (System is SQL2000
> clustered on Win2K)
> - The table collects data points for I.T. systems performance
> (perfmons etc) on 800 odd servers so there are hundreds of Inserts
> going on 24/7 & no updates.
> - Once a day a maintenance job runs to delete any data points > 200
> days old so the size of the table stays roughly the same...
> - This table has just the one Non-Clustered Index on DataID(int - non
> unique) , Time(Int) & Value (float)
> - The DataID is non unique as it id defined in a "Header" table which
> contains all the definitions for the performance data being collected
> (name of perfmon, system name etc), the V large table I originally
> posted about holds the actual performance data so there are thousands
> of entries for each dataID.
> - The Table is part of a commercial product (NetIQ's AppManager) so I
> have
> no control over the schema/Indexes etc.
> I'm not sure when the problems occur - I'm gussing during a flurry of
> Inserts?
> I hope this all makes sense - I can't help but think this table could
> be managed better - should I force a Full table lock for example when
> the deletes are taking place? or can I manage the Index better (hence
> the questions about defragging/rebuilding)
> Thanks again for any advice or ideas.
It sounds as if having a clustered index on the timestamp could be a good
idea.I'm guessing that you query and delete data based on timestamp so
this index probably would help both. But mind you it'll take considerable
time and space to create it - it's likely that it'll be even too resource
intensive in your case.
I still do not understand what problems you have. Did you get any error
messages about a bad index or IO errors? If so then something seems to be
seriously wrong with your db. Are queries slow? Inserts?
Regards
robert|||Really I'm just afterany general advice around looking after this big, busy
table - the Integrity checks do seem to come up with Index problems quite
often & I wanted to make sure I was doing all the right things to keep it
working well.
For Example I don't really fully understand fragmentattion of Non Clustered
Indexes should I be defragging often? or is it not worth it? would it help to
defrag/rebuild indexes before or after this big daily delete from the table?
or doesn't it matter?
Sorry to be so vauge - my DBA knowledge is a bit patchy & I just wanna make
sure i'm doing things properly!
Cheers
--
Mike Knee
Attenda Monitoring & Management
"Robert Klemme" wrote:
> Sinister China Penguin wrote:
> > Thanks for the replies - some more details: (System is SQL2000
> > clustered on Win2K)
> >
> > - The table collects data points for I.T. systems performance
> > (perfmons etc) on 800 odd servers so there are hundreds of Inserts
> > going on 24/7 & no updates.
> > - Once a day a maintenance job runs to delete any data points > 200
> > days old so the size of the table stays roughly the same...
> > - This table has just the one Non-Clustered Index on DataID(int - non
> > unique) , Time(Int) & Value (float)
> > - The DataID is non unique as it id defined in a "Header" table which
> > contains all the definitions for the performance data being collected
> > (name of perfmon, system name etc), the V large table I originally
> > posted about holds the actual performance data so there are thousands
> > of entries for each dataID.
> > - The Table is part of a commercial product (NetIQ's AppManager) so I
> > have
> > no control over the schema/Indexes etc.
> >
> > I'm not sure when the problems occur - I'm gussing during a flurry of
> > Inserts?
> >
> > I hope this all makes sense - I can't help but think this table could
> > be managed better - should I force a Full table lock for example when
> > the deletes are taking place? or can I manage the Index better (hence
> > the questions about defragging/rebuilding)
> >
> > Thanks again for any advice or ideas.
> It sounds as if having a clustered index on the timestamp could be a good
> idea.I'm guessing that you query and delete data based on timestamp so
> this index probably would help both. But mind you it'll take considerable
> time and space to create it - it's likely that it'll be even too resource
> intensive in your case.
> I still do not understand what problems you have. Did you get any error
> messages about a bad index or IO errors? If so then something seems to be
> seriously wrong with your db. Are queries slow? Inserts?
> Regards
> robert
>|||Robert is correct in that the table should have a clustered index on
datetime. It should make the inserts and especially the deletes smoother
and faster. Does the nonclustered index cover all the queries on the table
properly? How are the deletes being done now? Are they in small batches or
one big delete statement each day?
--
Andrew J. Kelly SQL MVP
"Sinister China Penguin" <SinisterChinaPenguin@.discussions.microsoft.com>
wrote in message news:EE3A13E0-7D0D-4E30-90FF-6CF5AE5E7501@.microsoft.com...
> Really I'm just afterany general advice around looking after this big,
> busy
> table - the Integrity checks do seem to come up with Index problems quite
> often & I wanted to make sure I was doing all the right things to keep it
> working well.
> For Example I don't really fully understand fragmentattion of Non
> Clustered
> Indexes should I be defragging often? or is it not worth it? would it help
> to
> defrag/rebuild indexes before or after this big daily delete from the
> table?
> or doesn't it matter?
> Sorry to be so vauge - my DBA knowledge is a bit patchy & I just wanna
> make
> sure i'm doing things properly!
> Cheers
> --
> Mike Knee
> Attenda Monitoring & Management
>
> "Robert Klemme" wrote:
>> Sinister China Penguin wrote:
>> > Thanks for the replies - some more details: (System is SQL2000
>> > clustered on Win2K)
>> >
>> > - The table collects data points for I.T. systems performance
>> > (perfmons etc) on 800 odd servers so there are hundreds of Inserts
>> > going on 24/7 & no updates.
>> > - Once a day a maintenance job runs to delete any data points > 200
>> > days old so the size of the table stays roughly the same...
>> > - This table has just the one Non-Clustered Index on DataID(int - non
>> > unique) , Time(Int) & Value (float)
>> > - The DataID is non unique as it id defined in a "Header" table which
>> > contains all the definitions for the performance data being collected
>> > (name of perfmon, system name etc), the V large table I originally
>> > posted about holds the actual performance data so there are thousands
>> > of entries for each dataID.
>> > - The Table is part of a commercial product (NetIQ's AppManager) so I
>> > have
>> > no control over the schema/Indexes etc.
>> >
>> > I'm not sure when the problems occur - I'm gussing during a flurry of
>> > Inserts?
>> >
>> > I hope this all makes sense - I can't help but think this table could
>> > be managed better - should I force a Full table lock for example when
>> > the deletes are taking place? or can I manage the Index better (hence
>> > the questions about defragging/rebuilding)
>> >
>> > Thanks again for any advice or ideas.
>> It sounds as if having a clustered index on the timestamp could be a good
>> idea.I'm guessing that you query and delete data based on timestamp so
>> this index probably would help both. But mind you it'll take
>> considerable
>> time and space to create it - it's likely that it'll be even too resource
>> intensive in your case.
>> I still do not understand what problems you have. Did you get any error
>> messages about a bad index or IO errors? If so then something seems to
>> be
>> seriously wrong with your db. Are queries slow? Inserts?
>> Regards
>> robert
>>