Building the Perfect SharePoint Farm

SharePoint Server Roles: Web Front end, Server

Single Server Farm (simplest, everything on one server, only for dev or similar use, can run on Win7) – he does use for his laptop due to resources. Can put ISO into your machine, choose defaults & fine (just for dev).

Small Farm – a little more robust – break out sQL server, WFE & Svc App layer running on same box, recommended for small workgroups.

Medium Farm – break out sQL server, WFE & Svc app layer running on indiv servers, can handle large amoutns of users & content, no redundancy. Option 2: WFE & Svc App layer running on same servers (but two of them) – can handle larger amts of users but App Layer impacts perf (would need to crawl overnight) - gives some redundancy – 2 WFEs can handle up to 100K users.

["SQL server is the water for our fish"]

Large Redundant Farm: Multiple WFEs, multiple svc app layers, clustered (highly available) SQL Server (should find someone who really knows how to do it right), no single point of failure.

The “Perfect” SharePoint Farm? How can one, or even four, farm architectures satisfy every need? Need to define perfection – different for every implementation. Issues to determine biz reqts for: Capability, performance, reliability, scalability, user experience (ie like Tron, fight for the users – mobile? [must turn on basic auth to access via Android]), security, cost (will help determine what they can get).

Capability – what are we trying to accomplish? Doc Mgt (need to consider search & versioning – estimate # of docs x avg size x # of versions + 25%), Records Mgt, Web content Mgt (.com site – most are relatively small – usually 10-20G – but then reliability is ultra important), Social collab, BI platform, enterprise messaging (intranet portal).

Performance – how fast does it need to be? Is there an established SLA, and should there be? Page load vs transaction time (what can we directly control). Avg user load (estimate total users x % concurrent (actively clicking) users / time per request = requests per second – can get 50 from one WFE).

Reliability: (are you sure you can afford it?) Expensive. No really… it’s expensive (to achieve 99.999%) - hdwe more than doubled, support staff, staff competency must be higher. Challenge early and often (vs cost). Most co’s need weekly mnte window (5 9′s means 6 sec allowed per week – 99% is still under 2 hrs per wk for maint – could be Sat 2am – might mean not requiring SQL cluster – but probably not OK in middle of EOM close).

Scalability – What is my growth curve? Establish metrics on your farm & stay on top of them (eg how are versions really working – use reports to monitor svc constantly). Know scaling options: server size/numbers/role – WFE (users), Svc app (usage), SQL (size/use of content). Should be part of initial plan – if x grows to x, here’s what we’ll need.

User Experience (the mythical user): Needs & wants; Goals, motivations & triggers; obstacles/limitations; tasks, activities & behaviors; geography/language; envir/gear, work life/experience.

Security (info wants to be free!): Easiest would be give everyone admin access, but obv can’t. Encrypt data (even internal communication – most info theft happens inside org – see diagram on his slide – sometimes some data must be both in transit & at rest; maybe use custom code to mask b4 & after storage), use SSL (but offload certifs to TMG [Threat Mgt Gateway]/F5 etc), segment SP on dedicated VLAN, use reverse proxy servers (TMG), use dedicated admin accounts (if I am admin of boxes, do not use normal user acct – each admin should have their own admin acct - more of a policy). Co might decided to accept certain amt of risk (vs cost). Point of vulnerability is always SP admins.

Cost – plan for now & future. Know licensing options (Foundation free, Std vs Enterprise, Internet sites). Don’t forget to include all other server costs (Windows/SQL ($12,500 per processor)/TMG/etc). Hardware costs (servers, routers). When you are building SP, you’re building a home; pour foundation first & never again – need someone who really knows what they’re doing. Changing org structure etc is like knocking down & rearranging walls, can be done tho difficult. Rearranging furniture is easy (things users can do – build lists).

Secret sauce (how do we put it all together): Planning. Roles (WFE server – very sim to MOSS, primary scale component is # of users connecting, 8GB RAM, 4 “cores”, 80GB HD (his rec); SQL Server – clustered or other HA tech, can have multiple instances to share load, perf critical to farm, 16-64G RAM, 8+ cores, 80G HD + data [on C drive]).

SQL considerations: alias your server (SQL alias or DNS [usually OK]). Size of content DB determines min RAM (med (<4TB content) -> 32GB RAM, lg -> 64DB RAM. Disk perf critical (tune SAN for optimal perf by use – RAID 10 best (more efficient but eats more disk space), RAID 5 OK – data/log/temp DB all on sep LUNS (logical unit ____) & logical disks – ensure SAN NIC isn’t saturated).

Service App Servers: Automatic scale & load balance. Can partition svc to specific servers. Search very scalable, but very complex – crawl servers (can have mult servers crawling data, query data sent directly to query servers not on crawl) – Query Servers (can create mult partitions, 10 mill items upper limit – about 5% query, about 25% for search – can have active/passive failover query partitions (can have dup on another box to split up load to cut time in half – see slide diagram – significantly more complex – FAST is something different) - query size est at 10% of crawled content (usual smaller)).

Virtualization – Problem or Solution? He says we can virtualize everything (WFE/Svc App. SQL maybe – you can, but most ppl don’t). Allows us to dynamically manage farm as needed by reqts (so resources aren’t being wasted & can be used for elsewhat). Snapshots are your friend (always b4 hotfix).

High Availability, DR, Virtualization: Plan for emergencies (plan for worst, hope for best – Recovery time Objective & Recovery Point Objective (about 2x size of log shipment[??]). Prevent single point of failure. Consider Virtual DR. Also plan for effort to recover from failover (how much work involved).

Can the Cloud save us? Transfers risk to cloud provider (servers/network/patches/backup) – you make sure my servers are running, etc. Reduces “dark” hardware costs. Can be cheaper (shared), or more expensive (private).

Define, then Design (server roles, server size/#, virtualized components, availability, DR, test farm – “everyone has a test farm, some have production farm”)

Develop: Stick to plan, ensure all pre-reqs and patches are installed, script install, snapshot everything b4 & after, take lots of screen shots.

SharePoint 2010 Physical Topology

•SharePoint 2010 Service Architecture
•SharePoint 2010 Logical Architecture
•SharePoint 2010 Architecture and Design Models
•SharePoint 2010 Performance and Capacity Case Studies
•SharePoint 2010 Capacity Planning
•SharePoint 2010 Cache Strategies
•SharePoint 2010 Databases
•SharePoint 2010 Physical Topology

Introduction

In this blog I am going to build off what we has been discussed thus far in this series. Whenever I start working with a new SharePoint client, the first thing that comes up is what is needed to install to get going. Well that answer is always based on business requirements however I am rather quickly forced into a corner to show them how it works J

To be able to define the physical topology you need to know:

• SharePoint service architecture
• SharePoint logical architecture
• Having a strong understanding of availability, capacity, and continuity

Once you have a good idea of that, you will be in a good position to draw out sketches of the Physical Topology for SharePoint you need. For SharePoint 2010, things have changed a lot and the logical architecture is significantly more flexible and scalable. I would say the game has changed in lots of regards.

In this blog I am going to focus on understanding some of the decision logic I would go through when architecting a new SharePoint 2010 farm.

Getting Started

I am going to keep this simple as possible. The first place to get started is to understand your business requirements so you can come up with a list of services that you need. You can read my previous post on SharePoint service architecture.

Next once you have understanding of what services are needed, you will need to come up with a logical architecture for SharePoint. In this series I went over a lot of the design decisions for coming up with that logical architecture for SharePoint.

Here is a refresh on some of those logical architectures. You may come up with a diagram as simple as this one below which is for a Single Farm Single Service Group where services are shared across farms.

You may come up with something like below which is a Single Farm with Multiple Service Groups. The service groups provide dedicated services to site collections.

You may come up with something more complicated which is a Multi Farm environment with centrally hosted services and localized dedicated services.

You come up with a Multi-farm with application services that are partitioned so services are centralized but data is not shared.

Or you may come up with a hosted partitioned farm where farm subscribe to centralized services.

To get more details I recommend you read the Logical Architecture posting of this series.

Selection of a logical architecture should driven by business requirements. The logical architectures I presented above are by no means the only ones; you can come up with tons of different permutations. What I did want to demonstrate is that I made no assumptions yet on how many machines would be needed to support the logical architectures. I will say in a perfect world this would always be true however sometimes the physical architecture affects the logical architecture.

Physical Topology

When we start talking physical topology we need to identify how we are going to configure the environment. There are three types of machines you will have in your farm:

1. Web tier
2. Application tier
3. Database tier

There were a couple things I left out where going over the logical architecture:

• I did not discuss web or database tiers. I focused only on understanding the application services.
• I did not discuss how many services would be configured one a single application server or across multiple application servers. Understanding how many servers are needed will be driven by capacity and availability planning which I have also discussed in this series. So it is really important to understand the demographics of the users, their profile, their expectations and how they plan to use SharePoint for both the short and long term.
• The service groups that I defined in the logical architecture do not drive what services will be configured onto what machines. What we need to focus on is identifying services that have common performance characteristics and configure them onto dedicated server resources (physical or virtual).

There is some basic guidance that Microsoft provides that will help you get started which I am going to go over.

Limited Deployments

This is described as either:

• One-server farm with all the tiers installed on one server.
• Two-server where the web and application tiers are installed one server and the database would be installed on an existing SQL Machine.

Small Farm Topology Deployments

This farm would be scaled out a little more to support more users or provide dedicated servers for services. Web servers, search service and other application services will be distributed across multiple machines. Some potential configurations are:

• Three-server configuration where there are two web servers that are load balanced. We web server will have the query server configured (which searches an index) and one will be dedicated for all other application services. The third machine is the SQL machine.
• Four-server configuration where there are two load balanced web servers with the query service running on each. The third server is dedicated to hosting application services and the fourth is used for SQL Server.
• Five-server configuration is similar to the four-server farm but we separate the index service onto a separate application server.
• Six-server configuration which is the same as the five-server farm except we create dedicated SQL database for search service to better support crawling.





Microsoft states that two WFEs can support 10,000 to 20,000 users. As well this configuration can support searching up to 10M documents.

The five and six-server farm configurations are very common configurations out there today. It is highly recommended starting place for a company division or for a dedicated solution farm because:

• There is redundancy for the WFEs – if one goes down users can still access the site.
• There is performance improvement for users because there are two load balanced WFEs.
• The application services are dedicated to a machine.
• Search indexing which can be an expensive operation is dedicated onto a separate application server.
• Separating the search query and index services provide both performance improves in search and redundancy (i.e. users can still search of the search server is down).

The three and four-server farms I really do not see as viable configurations. At minimum you need to consider a five or six-server farm for production environments.

Medium/Large Farm Topology Deployments

There is no definition which says exactly that X amount servers means you have a medium or large farm. Really I always start with a five farm configuration and from there I will start scaling out servers based on capacity requirements and the logical architecture that I created earlier.

Some observations:

• Adding more web servers will support more users. A rule of thumb is every web server will support an additional 10,000 users.
• Based on capacity and availability requirements, add more application servers and spread your services across those servers. These servers can be dedicated to supporting specific service applications. The logical architecture you created will not have a one-to-one mapping of services to server. In some cases, you may have to create multiple servers to support a single service from the logical architecture. Review the blog in this series on capacity planning for more information.
• There is a potential that you may want to scale content databases onto their own dedicated database server. You may have to consider adding more databases to better support search.
• You may consider creating dedicated WFEs for search, so when content is crawled, the WFEs that users use is not affected.
• Scaling out search is very important. SharePoint Search is more scalable and if you are considering using FAST, even more servers will have to be considered. I am not going to dive deeply into scaling search in this blog.
• It is highly recommended to plan out server groups and with service applications that have similar requirements. You can consider centrally hosting these servers and making them available to other farms.

Extranet/Internet Farm Topology Deployments

Extranets and Internets will require you to have an understanding of how you are going to secure SharePoint with a DMZ. This model by Microsoft is the best place to start. The diagrams for SharePoint 2010 are identical to the ones for SharePoint 2007. I usually recommend implementing the Split Back-to-Back server configuration. Even though the configuration may be more complex, all of the application and database servers will reside behind the firewall and only the WFEs will be placed onto the DMZ.

References
Hopefully this will help you get started with architecting your SharePoint 2010 environments.

• SharePoint 2010 Products Deployment
• Topologies for SharePoint Server 2010
• Extranet Topologies for SharePoint 2010 Products
• SharePoint 2010 Products: Virtualization Process

SharePoint 2010 Databases

•SharePoint 2010 Service Architecture
•SharePoint 2010 Logical Architecture
•SharePoint 2010 Architecture and Design Models
•SharePoint 2010 Performance and Capacity Case Studies
•SharePoint 2010 Capacity Planning
•SharePoint 2010 Cache Strategies
•SharePoint 2010 Databases
•SharePoint 2010 Physical Topology

Introduction

 One of the first things that will come up with any SharePoint 2010 is what databases are needed to support SharePoint 2010. With every SharePoint deployment there are two people who you need to become your best friends, the system admin and database admin. Now both these guys/gals have very distinctive views of the world – and at times we can find them down right annoying J However they do what they do because they have gone through lessons we as solution developers do not.

In this blog I am going to go over the databases that are needed for SharePoint 2010. The number and types of databases that are needed to support SharePoint 2010 have changed from SharePoint 2007. As you are about to see when I say more databases, I really mean more databases. Many of the maintenance, sustainment, governance, etc. challenges a SharePoint engagement suffer from tend to take this for granted or think that is can be resolved later – and by then it is too late.

I would highly recommend you understand this along with the new service architecture and logical topology of SharePoint 2010.

External Databases

 I figured I put this first because it is an important. I have always said that understanding the databases of SharePoint is not good enough. Once you bring in data from an external line of business systems, the databases become part of SharePoint from a user's perspective. So capacity planning, continuity management, etc. need to be part of your SharePoint governance plan.

Configuration Database (SharePoint 2010 Foundation)This database is responsible for managing data associated to the all the SharePoint databases in the farm, all IIS web sites, trusted solutions, WSP packages, site templates, all web application and farm settings.

From a size perspective this database will be small and there can only be one per farm. This database has a strong dependency to the Central Administration Content Database and they must reside on the same database instance.

Central Administration Content Database (SharePoint 2010 Foundation)

 This is the content database from for the Central Administration web site. It will not grove very large and has a strong dependency on the Configuration Database (i.e. they must be located on the same instance). Only one of these databases will be created per farm.


Content Database (SharePoint 2010 Foundation)

 This is the database(s) that is responsible for storing all content stored in SharePoint websites. This would include lists, documents, web part properties, audit logs, sandboxed solutions, etc. It will also store data for Office Web Applications (Excel, Access, OneNote, InfoPath, etc.)


A content database can store data for multiple site collections however data within a specific site can only be store in on content database. There will potentially be numerous content databases based on the design of your SharePoint environment.

Microsoft strongly recommends that content databases size should be limited to 200GB. Supporting content databases with terabytes of data is supported for large single repositories of data like a Records Center. If you have gone over 200GB of data in a content database, you have not done your planning nor put the governance in to manage your environment. I personally would recommend making dedicated content databases per site collection and for an enterprise deployment of SharePoint there should be multiple site collections, not just one big one.
Content databases can be located anywhere; there are no dependencies other than being accessible to the SharePoint farm. For very large sites, you may even created dedicated instances to support performance.

Usage Database (SharePoint 2010 Foundation)

 This is a new database which is dedicated to supporting the new Usage and Health Data Collection Service Application service. This database stores all of the health monitoring and usage data collected and the data within it is temporary. This database needs to support heavy write operations because data will be continually written to it. The health monitoring service will later take this data, aggregate it and then store it in the Web Analytics Reporting Database.

This database can get very large relative to the size of the amount of content you have stored in SharePoint as well as how many reports you have running. It will never be as large as the content database(s) because the actual data will not be store in it but it will store information about all data in all content databases across the entire farm. There can only be one of these databases per farm.
Business Data Connectivity (SharePoint 2010 Foundation)

 This is the database that is used to support BCS services. All it stores is external content types and associated metadata. This database will remain small because it does not store any data from the external systems. The only thing this database will need to support if heavy read operations because on the usage of BCS within SharePoint.

Application Registry Database (SharePoint 2010 Foundation)This database stores data required to support backwards compatibility for Business Data Connectivity (BDC) from SharePoint 2007. This database is only used during the upgrade process and can actually be deleted after the upgrade is complete.

Subscription Settings Database (SharePoint 2010 Foundation)

 This is a new database for SharePoint 2010 and supports the Subscription Settings Service. This database is used to support the new partitioning feature for SharePoint 2010. If you did not know, SharePoint data can now be partitioned by service subscription. This is will be used if you are providing hosted, centrally managed services and you want to make sure one service subscriber cannot access data of another service subscriber. This way services can be shared in a farm but the data can be protected. This database needs to support heavy read operations for hosted services that are highly utilized.

This database is not big and will not be created by default. The SharePoint administrator will create this database using PowerShell
Search Administration Database (SharePoint 2010 Standard)

 This database is used to support SharePoint 2010 Search service. It contains all the configuration information associated to search and Access Control List (ACL) which is used for securing content that is indexed. This data bases is neither small nor big. An instance of this database can be created for each search service that is running.
Crawl Database (SharePoint 2010 Standard)

 This is another database that is used to support SharePoint 2010 Search service. This database will store the state of the crawled data and the crawl history.

This database can grow to be very large based on the amount of content that you are indexing. More crawl databases can always be added into the farm to scale out. This database must support heavy read operations and it is recommended to run on SQL Server Enterprise Edition.

Property Database (SharePoint 2010 Standard)


This is the third database that is used to support SharePoint 2010 Search service. This database will store information associated to crawled data (i.e. properties, history, and crawl queries). This database can become large but not as big as the Crawl Database. It recommended for very large SharePoint environments that this database be put in a different database server; separate from the crawl database. This database must support heavy write operations and it recommended to run on SQL Server Enterprise Edition.


Web Analytics Staging Database (SharePoint 2010 Standard)


This database store temporary usage data collected from the Usage Database. The data comes to this database in an un-aggregated format and web analytics service will take this data, process it, aggregate it and then sent it to the Web Analytics Reporting Database. This database will be cleaned out every 24 hours but then refilled with new data that has been collected.

Web Analytics Reporting Database (SharePoint 2010 Standard)


This is new database for SharePoint 2010 used to support the Web Analytics Service. This database stores all the aggregated analytics data collected across the SharePoint 2010 farm. This is the database the usage reports run against and there will only be one of these databases per farm.

This database can grow to become very large relative to the amount of data stored in the entire farm. This database will only have analytics data; it will not have any actual data from the content databases. By default, data will be stored in here for up to 25 months
State Database (SharePoint 2010 Standard)


The state service is used to support storing temporary data across HTTP request. This database is utilized by InfoPath Form Services, Visio Services, Exchange, Chart Web Part, etc. (). The space required for this database is driven by the usage of the services that utilize of this database. Multiple state databases can be added through PowerShell commands.


Profile Database (SharePoint 2010 Standard)


This is a database used by the User Profile service and is used to store profile data. This database will not become very big and the size will be based on amount data be stored about each user. The database needs to support heavy read operations to get user data which is access commonly (user permissions are not store here; they would be in the content database).

Synchronization Database (SharePoint 2010 Standard)
This is another database used by the User Profile service. Its purpose is to store the configuration of the service that brings user profile data into SharePoint. It is also used to stage data that is being synchronized from directory services like Active Directory. The size of this database will be relative to the number of users and groups that are being synchronized. This database needs to support both heavy reading and writing when the synchronization service is running.

Social Tagging Database (SharePoint 2010 Standard)
This is the third database used by the User Profile service. It is used for storing social tags and notes created by users for content in SharePoint. The size of this database is completely based on the utilization of social networking services. This database will experience mostly heavy read operations.
Managed Metadata Service Database (SharePoint 2010 Standard)
This is the database used to support the new Managed Metadata Service will stored centralized content types that can be used across the farm. This database will not get very big. If managed metadata is used a lot, this database will need to support heavy read operations.

Secure Store Database (SharePoint 2010 Standard)
This is used by the secure store service which is the new SharePoint 2010 service to support Single Sign-On. It stores user credentials and passwords. This database will be small. It is recommended that this database have limited access and potentially even in a different location from the other databases.

Word Automation Services Database (SharePoint 2010 Enterprise)
This database is used by the Word Automation service and stores all pending and completed document conversions. The database will not get very large and has processes to ensure that it does not get too large.

PerformancePoint Database (SharePoint 2010 Enterprise)
This is another small database used to support PerformancePoint. It will store temporary objects and settings needed to support dashboards.

FAST Search Administration Database (SharePoint 2010 FAST)
This stores all configurations associated to groups, keywords, synonyms, term entity, inclusions, exclusions, spell check, best bets, search schema, etc. This will be a small database but must support heavy read operations to support both indexing and querying of data.

References

• Databases That Support SharePoint 2010 Products
• Manage the State Service (SharePoint Server 2010)
• User Profile service overview (SharePoint Server 2010)

SharePoint 2010 Cache Strategies

•SharePoint 2010 Service Architecture
•SharePoint 2010 Logical Architecture
•SharePoint 2010 Architecture and Design Models
•SharePoint 2010 Performance and Capacity Case Studies
•SharePoint 2010 Capacity Planning
•SharePoint 2010 Cache Strategies
•SharePoint 2010 Databases
•SharePoint 2010 Physical Topology

Introduction
When architecting your SharePoint 2010 solution need consider how you will leverage cache to make your applications more fast and scalable. In think a lot people (myself included) forget about caching strategies and how they can benefit from it. Each one has different pros/cons and correctly picking one based on the business requirements is important for success.

I found a detailed whitepaper called SharePoint Server Caches Overview, Advanced details on the SharePoint BLOB, Output, and Object Caches which goes over the topic. You will need to download SharePointServerCachesPerformance.docx.

The following are my summary notes from the whitepaper.


Really the purpose of caching for SharePoint is to reduce the amount of calls to SQL Server such that you can quickly return results to users while lowering SQL Server utilization. The negative is there can be a lag in showing the user the latest and greatest content. Once the cache is created, it is maintained locally on the SharePoint WFEs. There are three caching strategies you need to be aware of for SharePoint 2010: BLOB cache, output cache and object cache.

BLOB Cache

 BLOB cache help improve performance by storing requested files on the WFE Server such that they do not need to retrieved every time from SQL Server. There are two basic ways files can be store in SharePoint, they can be placed directly on the server (like in the layouts directory in WFE) or they can be stored in SharePoint library. Placing the files on the SharePoint server is quicker than retrieving a file out of SharePoint however only administrators can update the files. BLOB caching specifically solves the issue of retrieving files from the document library by caching them on the WFE. This gives you the best of both worlds, centrally managed in SharePoint and improved load time of files.

• BLOB cache should be used when pages that are accessed frequently have Javascript, CSS, images files, and large rich media files that can be cached on the WFEs. However BLOB caching is not useful if the files are not frequently accessed or if the files are modified on a frequent basis.
• Another advantage of BLOB cache is that it reduces the time to reload web pages. This is because cache control headers can be added to the HTTP responses for the cached files on the WFE. What this will do is push cached files on the WFE down to the user's browser's cache. This will reduced in even less HTTP requests to the WFE itself.
• BLOB cache is particularly helpful for cache large files out of the SQL server. The while paper goes into the details but there is no disk buffering for serving up larger files (5MB) which results in low latency. SharePoint is optimized for server up smaller chunk sizes (100KB).
• When using BLOB cache, HTTP range request is supported which allows the browser to request pieces of the file to cache locally instead of the entire file. Media players that run on the client benefit when this is supported.

Let's take a deeper look at BLOB caching:

• There is performance overhead to initially build the BLOB cache, which is around five times more expensive. One reason why permissions and metadata associated to the file is needs to be brought over to the cache to ensure security is still maintained.
• BLOB cache is created by each web application on each WFE machine on the farm. This translates to each virtual site in IIS (which maps to a WFE) will have its own BLOB cache. BLOB cache cannot be reused across web gardens (or zones).
• It is possible to configure the files that can be placed into the BLOB cache. There is a file with a list of extensions which can be modified based on your business requirements.
• BLOB cache can handle multiple concurrent requests even when the requested file has not been cached yet. The example given was if a link to a large video file is mailed out you want to make sure when everyone starts clicking that link, the server does not get flooded requesting the same large file. With BLOB cache on, even if the file has not been fully cached, the video file will only be retrieved once per WFE from the SQL Server.
• An interesting thing to know is BLOB files are stored on the WFE in folders that match the location in SharePoint. There is a 260 character limitation on file paths in Windows, so if you URLs are larger then there will be problems building your cache. It is recommended to keep relative URLs smaller than 160 characters.
• You will need to plan for RAM utilization when using BLOB caching. The BLOB cache index will use 800 bytes of REM per entry.
• BLOB cache is persistent cache because the cache is periodically written to file on the WFE. This means an IIS recycle or shut down will not lose the built cache. If the BLOB cache is very larger, there is a lag on when the cache will become available again once the IIS operation is complete.
• Items in the cache are invalidated based on a polling mechanism that checks SQL Server every 5 seconds. This interval can be changed. They will not be added back to the cache until it is requested again.
• BLOB cache also has a configurable size limit to keep the cache from growing at an uncontrolled rate. If the max is exceeded, files used the least will be removed until the cache is 70% below the max size. If this threshold is exceeded a lot, there will be performance overhead incurred, and it would be recommended to increase the max size.
• It is possible to manually flush the BLOB cache forcing it to be rebuilt.
• BLOB cache is optimized for returning files anonymously because the file can be immediately returned without making any SQL Server round trips. This can be done by marking the site as anonymous or storing the files in a library that has AllowEveryoneViewItems set to true.

Configuring and Managing BLOB Cache:

• A mentioned earlier BLOB Cache is enabled by IIS site or SharePoint Web Applicatio. All you need to do is go to the web.config and modify the BlobCache node as enabled (Reference). There are several other configurations that are available for tuning the BLOB cache, I recommend reading the whitepaper for those details.
• For the application pool you will need to increase the startup and shutdown time limits. It is recommended to set it to 300 seconds which will allow enough time initialize or serialize on startup or shutdown. Note this does not mean it takes 300 seconds to perform the operation, however it prevents IIS from terminating the application until 300 has elapsed so the cache is not lost (great reference).
• It is recommended to keep all content to be cached in a specified list and sure the site containing that list is stable. This is because frequent changes to the site or list will invalidate all the cached files.
• To flush cache a simple IIS Reset can be performed, the SharePoint API can be used or you can finally disable the cache, delete the folder that contains the cache and then re-enable the cache in the web.cong.

In summary use BLOB Cache:

• If there is a high read to write ratio BLOB caching should be used. For instance you would want to cache a site logo that is used on every page request versus a collaboration word document that is actively updated.
• It is optimized for supporting large files which can significantly reduce bandwidth between the WFEs and SQL Server.
• It is optimized to support cache control headers so that clients can cache small files which can reduce overall number of hits to the WFEs.
• If there is anonymous access, there can be dramatic improvements because permissions do not have to be validated for cache files.
• Client applications that use range requests can optimize load times to access large files.

Output Cache
The second caching option you have with SharePoint 2010 is ASP.net Output Cache. This is an in-memory cache that saves rendered ASPX pages. Using Output cache improves performance in two ways first it reduces the amount of SQL calls. Second it reduces workload on the WFE because pages do not need to be re-rendered. Along those lines if the pages are anonymous, then no SQL check needs to be done at all present the cached pages. Microsoft testing concluded a ninefold improvement in throughput when compared to having to render the page every time it was rendered.

The only catch for using Output Cache is that it can only be used in conjunction with Publishing pages. It cannot be used with a collaboration site. Output cache is configured on a per site collection basis using cache profiles. A cache profile is the settings and parameters used to control how pages will be cached. Some examples of rules that can be capture in a cache profile would be to not cache if the requestor is a user who can edit pages to ensure they see the latest version of content. The cache profile also specifies rules for when a page is invalidated so that when the next request is made, it comes from the database.
There are two options for cache invalidation:

• Time to live (TTL) – Is a basic rule that will retain the page until a length of time has been exceeded. Microsoft testing results found that TTL cache invalidation did perform well when the site content changes frequently.
• Check for Changed – Is a rule that states all pages using the profile are invalidated when if there is any site change or TTL has passed. This is best used for sites where changes do not occur often.

One of the main considerations for Output cache is the memory needed to support it. For each rendered page, 2(size of the page) + 32KB is needed to store the rendered page in memory. Depending on your cache profile, you may store multiple different versions of the cache. You may create different cache versions based on what the users role is, what type of browser they use, or page layout type. For each version a different cache entry will be made for the same page. So it would actually be possible to create a rule that says specific types of publishing pages may become invalid every 10 minutes while other types would become invalidated every 24 hours.
To configure Output Cache:

• To configure and set up Output cache and profiles read here.
• Next you need to go to the site collection where publishing has been turned on and in the collection settings page turn on the output cache.
• On the page you can set up the Output cache profiles – read the whitepaper for details on those configurations.

In summary Output Cache should not be used with sites using a low read to write ratio because frequent changes to content make it hard to keep the cache fresh. So understanding how important it is to have the most current content available to the user is important. Another consideration to know is how dynamic the content is and if per-user content has to be supported. Having to support per user cache, more space it is needed to store the cache. As well having to support lots of variations of cache will again require memory to support the cache.
Object Cache

 Object cache is the third caching option we have for SharePoint 2010. What Object cache does is stores metadata about SharePoint Server objects (like SPWeb, SPSite, SPList, etc.) on the WFEs. When a page is rendered, if there is data that needs to be retrieved through these objects, the SQL Server will not be hit. Features of SharePoint that use Object cache are publishing, content query web part, navigation, search query box and metadata navigation. These features are specifically written to use the Object cache API instead of the SharePoint API directly. Developers writing custom functionality can also tap into the Object cache API.

The Output cache algorithm for how in determines what to cache is complicated because the user permissions have to be accounted for. Obviously we want to make sure security trimming is respected but it would be completely inefficient to create an Object cache of each and every user who comes to the web site. At a high-level there accounts (Portal Super Accounts) which can be created that can have standard permission levels assigned to them. Cache will be created based on these accounts and then the ACL of the current user will be applied to show the user data from the Object cache they have permission to. Along with this, there is the Object cache multiplier which is set at in a site collection. This multiplier controls the number of rows that will be cached. Increasing the multiplier will increase the number items returned from cache at the expense of utilizing more RAM to store more data.
Object cache cannot be disabled. Object cache configuration is very similar to Output cache. Object cache can check the site for changes every time there is a request or check for updates periodically. The only difference is Object cache will not become completely invalidated when any change is made to the site. For instance, if a list item were to change, only that list item would be invalidated and all of the other list items would remain in Object cache. Also, dependencies between cached items are maintained and if the list itself were deleted, the all the list items for the list in the cache would as well become invalidated.

The first step in configuration of Object Cache is the set up of the Portal Super Accounts which control how the cache is built. Please read the whitepaper for more information about how to set up these accounts. Some configurations can be easily access in the site collection administration page. As well, there are some configurations that can be applied to the web.config which are new to SharePoint 2010.

Finally you need to plan for sizing of the Object Cache. According to Microsoft, a small number of site collections (fewer than 50) the object cache should have a little to no memory issues but more than that you may need to do some planning. Microsoft's recommendations is to plan 500KB of RAM per site collection that has Object Caching turned on.

SharePoint 2010 Capacity Planning

•SharePoint 2010 Service Architecture
•SharePoint 2010 Logical Architecture
•SharePoint 2010 Architecture and Design Models
•SharePoint 2010 Performance and Capacity Case Studies
•SharePoint 2010 Capacity Planning
•SharePoint 2010 Cache Strategies
•SharePoint 2010 Databases
•SharePoint 2010 Physical Topology

Introduction

 One of the most important things for planning, managing and governing a SharePoint 2010 environment is to understand the boundaries and thresholds that affect performance and maintenance. Capacity Planning is extremely important when architecting a new SharePoint 2010 environment. A lot of times people designing SharePoint environments or creating SharePoint solutions completely forget about how Capacity Planning is directly tied to use cases and business scenarios. Knowing where your limits are and how you plan to grow will affect both your physical topology and logical topology.

Microsoft has actually put out a ton of information that you can use for SharePoint 2010 Capacity Planning.

In this blog I will touch on:

• Web Site and Site Collection Capacity Planning
• Content Database Capacity Planning
• List and Library Capacity Planning
• Search Capacity Planning
• Security Capacity Planning
• Social Networking Capacity Planning
• User Profile Service and Social Networking Capacity Planning
• Web Analytics Capacity Planning
• Workflow Capacity Planning
• Excel Services Capacity Planning
• PerformancePoint Service Capacity Planning
• InfoPath Services Capacity Planning
• Visio Services Capacity Planning
• Word Automation Services Capacity Planning
• Office Web Application Services Capacity Planning
• Access Services Capacity Planning

Resources
The following are the resources that I used for this analysis

• SharePoint 2010 Capacity Planning (Boundaries and Limits) Whitepaper (View or Download). This is a really good whitepaper which should be used as a baseline for architecting your SharePoint 2010 environments. I know there will be creative solutions over the long run that will allow you to exceed these however this is the best place to start. I actually enjoyed reading through this.
• SharePoint Server 2010 performance and capacity test results and recommendations (View or Download). This is a set of several detailed whitepapers that go into the nuts and bolts of how SharePoint 2010 was tested and provides tons of very detailed recommendations.

All this information in this blog comes from these whitepapers and these are basically my notes.
Web Application Limits

• Try to limit to 300 content databases per web application. This is an important one because a well managed SharePoint environment will not have one big content database; but instead have many to facilitate back-up and recovery procedures. If you plan to have lots of content databases, the recommendation is to use Powershell to manage the web application instead of using the management interface.
• Zones have a hard limit of five (default, intranet, extranet, internet and custom); nothing new there.
• It is recommended that you do not exceed 10 managed paths per web application without testing.

Thoughts: Not much new here other than the most important one everyone should know – do not have one big massive content database for your web application.

Web Server and Application Pools

• It is recommended to have no more than 10 Application Pools per web server. This is mostly driven by amount of RAM and usage by the users.

Content Database Limits

• Content databases should not exceed 200GB of data. Again SharePoint 2010 does scale up to 1 TB but that is only recommended for large, single site repositories.
• Remote BLOB Storage (RBS) cannot exceed 20 milliseconds to access the first byte of data.

Thoughts: I can say this over and over but you need to plan to keep your content database sizes down and do not let them exceed recommended limits. RBS is good for performance but you still think about not letting your content databases store TBs of data. That is a business requirements issue.

Site Collection Limits

• It is recommended to not exceed 250,000 sites per Site Collection. Basically the deal is that there is performance overhead when both creating and deleting sites. If you have a highly dynamic SharePoint web site, where this many sites are created and deleted on a regular basis, you will experience performance issues across the board.
• Site Collections should not exceed more than 100 GB. Again the issue is to make backup and restore procedures run quickly.

Thoughts: If this becomes an issue you probably have too many people with rights to create sites. Another way I can see this happening is if you create some sort of automated site provisioning process that just creates too many sites. Your SharePoint Topology should plan for not exceeding these limits.

SQL Service Column Limits

• As we mentioned before, we need to limit the number of row wrapping to 6 rows. According to SQL Server the threshold is 64 columns per row, so you should not exceed more than 384 columns in your SharePoint list or library definition.
• As mentioned before, the amount data stored SharePoint item per row cannot be more that 8,000 bytes.
• This whitepaper goes into specifics the size limits by SharePoint column type.

Thoughts: I had never really thought about this in the past. It will become a more important factor in your SharePoint design given the amount that can now be stored in SharePoint.


Web Page Limits

• It is not recommended to exceed more than 25 web parts per SharePoint web page.

Thoughts: If you have than many on your screen you are trying to display too much or your have made your web parts way too granular.

List and Library LimitsUnderstanding SharePoint list and library capacity is extremely important, especially with the new improvements to SharePoint 2010.
Here are some specifics:

• For each row in a SharePoint List or Document Library List Item you have a hard maximum of 8,000 bytes. What this basically means is the amount of data stored in the columns of a list item cannot total more than 8,000 bytes.
• The maximum File Size is 2GB. The default is 50 MB.
• It is recommended to not exceed 30,000,000 documents per document library.
• It is recommended to not exceed 30,000,000 items per SharePoint list.
• Should not exceed more than 6 table rows in the Content database to store a list or document library item. What this basically means is that if your list or library definition has lots of columns, SharePoint will break that storage of that data across rows. There is fixed amount of columns in a SQL table and if there more list columns that SQL table columns the data for the list item will be stored across rows (referred to as row wrapping). So this will only become an issue for you if you have an extreme amount of columns in your SharePoint list definitions.
• The SharePoint list user interface only allows 100 items to be processed in bulk at any one time.
• SharePoint List queries have a maximum of 8 joins allowed per query. If there are more than 8 joins in the query, it will be blocked.
• List view threshold by default will show 5,000 items. Going beyond this you will experience poor performing queries and overall throughput will decrease. List view threshold for auditors/administrators is 20,000 items.
• It is not recommended to exceed 2,000 subsites for each website. Performance will degrade on several of the out of the box pages, controls and web parts when 2,000 subsites are exceeded in a single website.
• It is recommended to not exceed 10 users simultaneously editing the same document. If there are more than 10, you will receive performance issues when committing the document to SharePoint. The hard boundary is 99 users.
• When extracting datasheet view into Excel, there is a hard limit of 50,000 items that can be exported.
• A SharePoint Workspace has a hard limit of not containing more than 30,000 items for synchronization.

Now are some other recommendations that you should be made aware of:

• A basic way for calculating the potential size of the list is the following. First estimate the average size of a document to be stored. Multiply that against the number of versions of that document. They multiple that by 20% for the data stored associated to the content database. It is also recommended to add in an appropriate buffer.
• When designing how content will be stored in lists, you need to consider if storing data in a single list, across multiple lists or even across lists in different site collections. There is a detailed discussion of this in "Single list, multiple lists, or multiple site collections" section in DesigningLargeListsMaximizingListPerformance.docx.
• Even through list and libraries can store large amount of content, we still need to get it back out. Microsoft recommends that when retrieving data from large lists using lists views or CAML queries, they be partitioned across folders, indexes or both. If not, the best and most efficient way to retrieve items is through search.
• Lots of permissions applied to a library will degrade performance. It is worse if you are doing item level permissions on large amounts of documents. There is a configuration will limit 50,000 unique permissions to a list. However it is recommended to lower this to 5,000.
• I mentioned row wrapping earlier and why it is used. It is recommended on for lists with large amount of data, to not exceed 1 or 2 additional rows of wrapping.
• Lookup columns can again be another source for performance issues. Lookup columns in a list view will result in a join being applied to the query which increases the complexity. By default each list view will support up to 8 lookup columns. It is strongly recommended to not exceed this number as it will cause significant throughput decrease for queries in a view and use lots of SQL resources.
• List indexes have little effect to list performance but can insert and update operations if there are lots of them. Up to 20 indexes can be applied per list. It is recommended to only use indexes when they are needed.
• There are three basic ways to access list data: list views, content query web parts and search. List views access data directly off SQL Server which incur slower query performance and higher load on the SQL Server. They also generate the most HTML which can slow down page rendering. However list views provide the best user experience when it comes to managing data in a list. Content query web parts display a configured view of data that is cached in the portal site map provider. It will generate the least amount of HTML and is cached, so overall performance will be better. Search web parts offer the best performance as it is optimized for queries large amounts of data. However the data returned in search is only as recent as the last crawl.
• Microsoft has several types of lists they have identified and when designing your SharePoint site, it may be good to classify in this manner so you can plan and govern better of the long run.
• There are unstructured document libraries which are basically team libraries with tens to hundreds of documents. They are highly utilized with expensive operations but the volume is low. It is important to make sure they libraries stay under 5,000 items.
• There are collaborative large lists which can have thousands of documents that can be edited, like in a knowledge management solution. Typically these lists grow significantly faster than anticipated and requiring lots of administration.
• There are structured large repositories which may have thousands to hundreds of thousands of documents. Typically these are departmental archives with automated processes to publish the documentation in a controlled manner. Almost all of the interactions with the list are read only.
• Finally there are large scale archives which contain millions of documents spread across multiple lists. Typically there are is a low amount of reads and updates and the purpose is for long term storage for compliance requirements.
• Content Organizers are a new feature that will route content to configured libraries, folders or sites. Users can submit data and not be concerned about the rules for storing content. It can support evenly storing data within a library to better manage performance.
• Metadata Navigation is a new feature that allows users to search for content in a tree based on the data associated to the document. This feature will use the most efficient query possible to return items instead of allowing the user to freely search and filter for content.
• Throttling of SharePoint list is also new. This can be configured at the web application so that inefficient operations by a user will not affect performance of the entire farm.
• Another new feature is compound indexes so indexes can be built across multiple columns instead of just one.
• The developer dashboard is another new feature that can help when understanding potential performance issues with a page. When it is turned on, tons of statistics are available, including database queries and load times.
• Allow object model is a new configuration that allows for object model to override the list view thresholds. This allows for developers to create web parts that query for content that do not adhere to list view thresholds, which is a good thing.
• Daily time window is another new configuration that when turned on, allows queries to be run that do not adhere to list view thresholds.
• You need to be aware that if the list will grow very large and exceed thresholds, several operations will be blocked, prevented or limited. For the details read the "Differences between large lists and regular lists" section in DesigningLargeListsMaximizingListPerformance.docx.
• Microsoft testing of extremely large document libraries (millions of documents) generally concluded that most of the bottlenecks would occur at the SQL Server level. To get around this bottleneck you should split content across multiple instances of SQL Server.

Thoughts: I am a broken record on this topic. Yes SharePoint 2010 resolved the challenges of storing mass amounts of data in SharePoint lists. However I still believe quantities of list data, especially "highly relational data", should be stored in a relational database and then exposed to SharePoint through the appropriate means.

Security Limits

• It is recommended to not exceed having a user belong to more than 5,000 SharePoint groups. There are no severe performance penalties if you go over, just a recommendation. If you do go over the user's token is bigger, it takes longer to cache access control list (ACLs) for search, and increases the amount of time check security.
• It is recommended to not exceed 2,000,000 users per site collection. Again you can exceed this number however there are long term management issues with that many users and it is recommended that you use PowerShell.
• It is recommended to not exceed 5,000 users or external groups (AD group) per SharePoint Group. The more users, the longer it takes to validate permissions or render a screen.
• It is not recommended to exceed more than 10,000 SharePoint Groups per Site Collection.
• It is not recommended to exceed 5,000 permission items in an access control list (ACL).

Thoughts: It is important to keep in mind for very large organizations and when planning to pull in data from external directory services (which may not always be Active Directory).

SharePoint Search Limits (not FAST)When it comes to search, this is just such a big topic. I highly recommend reading all of "Estimate performance and capacity requirements for SharePoint Server 2010 Search" (SearchforSPServer2010CapacityPlanningDoc.docx). Understanding the search architecture is critical for sizing and capacity management. I am going to go over some high level concepts to get you started:

• First when planning you need to understand what data needs to be searchable and how much of it is there. You need to know how available the data must be. You need to know how fresh the data must be kept and many people will be searching for data simultaneously. This actually goes a lot deeper and I have blogged about scaling FAST 5.3 here. The concepts are the same.
• For SharePoint 2010 there is a search life-cycle that you need to understand. There is Index Acquisition which is where full crawls for data are performed which is dependent on the size and access to the content is being crawled. Next there is Index Maintenance which is incremental crawls of all content. Finally there is Index Cleanup is basically when content sources change or removed from the crawler.
• Next there is the query service which is responsible for querying for data. It is important to know both the query latency and query throughput. You want to decrease latency which is the amount of time it takes for data to become searchable. You want to increase throughput which is the amount of data which can be searched.
• There are really just so many strategies for configuration your SharePoint logical and physical topology based on your requirements. Here are some simple things to think about. First never run both the query and search services on the same machine. Second, if you have the hardware, it is also recommended to run your query service on a separate machine which will decrease latency and increase throughput. Third it is good to have multiple search servers which build the indexes so there is redundancy when there is a failure or if you need to dedicate resources to search specific content sources.
• There some calculations which you can do to estimate the amount of space you need to indexes that will be built. Again, read this whitepaper for further details.

Some general thresholds and boundaries are:

• It is recommended to not have more than 20 SharePoint search service applications deployed to a single farm.
• For each SharePoint search service, do not exceed 10 crawl databases which store crawl data. The optimal configuration is 4 crawl databases per Search service. Each crawl databases should not exceed 25 million items.
• It is recommended to not exceed 16 total crawl components per service application.
• It is recommended to not exceed 20 index partitions for a search service. The limit is 128 index partitions. Partitioning an index allows for smaller indexes to created which can be searched faster, however having too many can have opposite effects.
• It is recommended to not exceed 10 million items per search index. The overall recommended limit for all indexes in a search service is 100 million items.
• It is recommended to not exceed 100 million crawl log items per search service.
• For each search service, it is not recommended to exceed 10 property databases. The property database contains metadata for items in each index partition. There is a hard limit of 128 property databases.
• There is a hard limit of 128 query components per search application.
• For search scopes, you should not exceed 100 scope rules per scope. As well you should not exceed 600 scope rules per search service. Exceeding these thresholds will affect performance. As well, you should not exceed more than 200 scopes per web site, which again can affect performance.
• It is recommended to not exceed 25 display groups per site. Display groups are used for grouped display of scopes through the user interface. Exceeding this number will affect performance.
• For alerts, there is a limit of 1M per search application.
• For search content sources it is not recommended to exceed 50 data sources but the top threshold is 500 data sources.
• There is a threshold for running 20 concurrent crawls within the same search service; exceeding this will cause performance issues.
• During crawling, each search application will support up to 500,000 properties.
• Recommended not to exceed 100 crawl impact rules per farm.
• Recommended not to exceed 100 crawl rules per search service. This is because the administrative screen performance will degrade.
• There is a threshold of 100,000 managed properties per search services. Managed properties are used in search queries. Basically the crawled properties are mapped to the managed properties.
• It is not recommended to exceed 100 mappings per managed properties. Exceeding this will degrade crawl speed and query performance.
• Each search service will support 100 URL removals per operations.
• It is recommended to have only 1 top level authoritative page and minimize second and third level pages.
• It is recommended not to exceed 200 keywords per site collection but the maximum is 5,000 per site collection. The only real effect of exceeding the recommendation is performance of the administration page.
• There is a limit of 10,000 metadata properties that can be crawled per item crawled.

Thoughts: Microsoft has put a lot of emphasis into this. Even though FAST is a very compelling, the out of the box SharePoint search has been dramatically improved.

User Profile Service and Social Networking limits

• The user profiles service supports up to 2 million user profiles with full social networking features. This basically means this is the limit of profiles that can be imported into the people profiles repository.
• There is a limit of 500 million total social tags, notes and ratings in the social database. Exceeding this runs the risk of significant performance issues.
• Microsoft's testing found that there was linear increase in throughput up to 8 WFEs. Past that there was no improvement. Scaling can be further achieved when separating the content and services database onto two separate database services.
• Most of the bottlenecks that will be experienced are typically due to WFEs. They found no bottlenecks associated to any of the application servers when evaluating social networking features.
• It is expected that separate web site with dedicated resources is created for MySites and the new social networking features. When planning to support performance this is not just maintaining a user's profile information, connections, newsfeeds, etc. It is also storage of documentation, collaboration, etc.
• User Profile service does have caching. The user profile service job is to take data from whatever location where you are loading from, and it will save that data in SharePoint profile database. Once the user profile cache is loaded, WFEs request for profile data is returned from the application server without requiring a call to the SQL database. I guess Microsoft's strategy is that profile information is not really dynamic data and they want to keep transactions down.
• I am not going to go in the details of the Outlook Social Connector however there is very detailed information about the performance of this in the Social Computing Capacity Planning whitepaper.

Thoughts: Much of this is associated to the new social networking features of SharePoint 2010.

Business Connectivity Services limits

• There is a hard limit of 5,000 external content type definitions that can be loaded into memory at any given time.
• There is a limit of 500 external system connections that can either be active or open at any given time. The default is set to 200 and it does not matter what type of connection is being used either.
• The default is 2,000 database items that can be returned in a request when using the database connector. The boundary is 1 million.
• There are three variables that you must always consider: the number of items in the external list, the number of columns per item and the size of each item.
• Profile pages display data from external content type data. The performance of these pages is driven by the complexity of the associations to external systems.
• The internal process for bringing external data into SharePoint via BCS is pretty simple. There is load (queries external source and loads into SharePoint), process (applies sort, filter, group processing) and render (display data onto page). BCS does not have in-memory cache for external items. Data has to go through load, process and render each time an external list is refreshed. Knowing this you need to make sure you control the amount of data that is processed at any given time.
• Microsoft's recommendation is to keep the number of items to be processed as possible by reducing the amount of items returned from external systems. It is recommended to keep the number of returned between 100 to 500 rows and it is recommended to not exceed 2.000 rows. It is recommended to use filters to ensure you work within these guidelines. More can be returned if needed but this needs to be done by an administrator.
• Rendering external list can be intensive for both the WFE and the application server. It is recommended to keep the number of items being displayed at any time to around 30. Note that the number of items that are rendered is not the same amount of items that were processed. The number of items rendered is controlled by the external list view that is on a SharePoint page.
• It is recommended to reduce the number of columns from external list. Obviously a large number of columns will affect performance.
• When rendering it is recommended not to use large-sized columns in list views. Columns larger that 1KB should not be utilized in a view. However performance is more affected by the number of items and not the size. So it is recommended always try to keep the number of items lower for better results.
• When designing an external list which is using BCS, make sure the default view will be the view the user needs to see the most. If the user needs to sort or filter the view, that will require data go back through the load, process and render process.
• For a profile page, the number of associations is the key for good performance. It is recommended to not exceed two associations. There will be lower performance of both throughput and latency when large number of items in an association.
• Diagnostic logging of BCS can become a factor for performance and it is recommended to lower when not doing testing.
• Performance of the external system will have performance implications to BCS and you need to sure those systems perform well.

Thoughts: I personally think is an area that has not been explored enough. Understand that BCS is another layer that the data must pass through. In Microsoft's detailed testing results for BCS they only tested WCF Web Services using .NET data types and SQL Server 2008 databases. As we know in an enterprise architecture not everything will be SQL Server or web service based. Data resides in formats all over the organization. Performance will be driven by factors out of scope of this blog.

Web Analytics limits

• There are three basic categories of analytics – traffic, search and inventory. Reports can be aggregated at the site, site collection, and web application level. The web analytics service mostly utilizes the application servers and SQL server, so that is where most of the capacity planning needs to be.
• At a high-level this is how the data is gathered. On each web server usage data is gathered and there is a Usage time job then calls the Logging Web Service to submit the data. The data is stored in a staging database for seven days. The web analytics components clean and process the submitted data and then every 24 hours the data is aggregated from the staging database and written to a reporting database. The aggregated data is stored for 25 months (default).
• Performance of the logging web service is directly related to the number of application servers on the farm. However the performance of the web analytics components is dependent on the performance of the analytics and reporting databases.
• Other areas of performance challenges are number of queries run each day, number of unique users each day, total number of unique assets clicked each day and existing data size in the reporting database. Basically the point is, the more data and more interactive your SharePoint farm is, the more performance overhead there will be to process that data for reporting purposes.

Workflow limits

• Workflow throughput can be affected by numerous factors such as the number of users, the type of workflow, the complexity of the workflow, dependencies on external calls and frequency on the number of users accessing the workflow. Note that workflows that use data from the content database or are registered for lots of events will run slower. So if you reference a large SharePoint list or you call out to an external database obviously that will take a little more time.
• Microsoft's testing found that throughput for workflows topped out at three to four WFEs added.
• There are some administrative settings that can be set which will affect the performance of workflows.
• Postpone Threshold is the number of maximum workflows that can run concurrently against a single content database. Subsequent ones will be placed into a queue. The default max is 15 workflows. It is important to understand this is the number of workflows that can be in progress at one time and it not the total number workflows that can be in progress.
• Batch Size is an exception to the Postpone Threshold limit and can pick up larger amounts of workflow items from the queue. The default batch size is 100 and this can be set per service instance.
• Another configuration is the Timer Job Frequency which is the frequency that the workflow time service runs. This service is responsible for picking up items off the workflow queue based on the interval. The default is every 5 minutes.
• I found this interesting but you should consider how all three of these configurations are used together to affect workflow performance. Workflow throughput is impacted based on how quickly operations get out of the queue and then processed. In the example I read, if there are 1,000 work items in the queue, it will take the time job 10 runs to execute all of them (Batch Size), which would take close to 50 minutes to execute (Timer Job Frequency). Modifying the Batch Size would increase that at the expense of taking up lots of processing resources.
• Another thing to make sure you do not do is use the same list for workflow history and task lists. Lots of items will be written to these lists as time goes on. Allowing these lists to get very large will affect performance.
• To keep the task lists from not getting too large there is a workflow job (autoclean) that will delete workflow instances and associated tasks that have been in a completed state for greater than 60 days. It is recommended that if historical information is needed past that, to write the data to other locations as part of the workflow. Workflow history items are not part of this autoclean job and a script should be written to purge those items.
• Another maintenance trick I learned which can affect performance is removing workflow columns. If you have a list with like 50,000 items in it, and you remove the workflow status column will cause database operation proportional to each item in the list, which can flood your SharePoint environment with transactions. It is recommended to just turn off the workflow so no new instances can be created and may do the operation during off hours.
• Another performance consideration when building workflows is make sure you are not going to violate SharePoint's database locking for modifying data at the same time. You need to make sure users and/or other workflows do not try to access the same thing at the same time.

Thoughts: I found the discussion on how to configure the workflow service very interesting because I had really never thought about that when I had built workflows in MOSS 2007. I can see how you may want to configure this very differently based on the site where the workflow is running. They had a creative example that was discussed where if you want to dedicate running workflows to a specific machine in the topology, you could try to lower the postpone threshold to force all workflows onto the queue so the batch job would pick them up on a specific machine. You would have to set the batch size to be high so that it picks up the work items before another machine. If you want to make sure the processing is shared across all the machines, you could try increasing the postpone threshold so items never sit around for a long time.
Excel Services limits

• Excel Services performance is directly correlated to the size and complexity workbooks that are hosted within SharePoint.
• Excel Services is stateful meaning that workbooks will remain in memory between user interactions.
• Excel Services have most effect on both the application servers and the WFEs. Excel Services will not highly utilize SQL Server because the workbook is read as a binary blob and used within the Excel Service. Microsoft's finding was that bottlenecks only happened on either the application or WFE servers, which can be scaled out by upping CPU memory or adding new Excel Services machines to the topology.
• Within Central Admin there are several configurations which can be used to better manage performance.
• Maximum Private Bytes is a percentage value Excel Services can use of memory on the machine. If the machine is dedicated to Excel Services, you can up this percentage. This is one of the main bottlenecks identified by Microsoft.
• Memory Cache Threshold is another configuration available. Excel Services will cache unused objects in memory and by default it will use 90% of the Maximum Private Bytes. Increasing this number gives a better chance that the requested workbook would already be in memory when the user accesses it. However lowering this number will improve performance of other services if the application server is not just hosting Excel Services.
• Maximum Unused Object Age is a configuration that helps keep objects in memory as long as possible but you will want to again modify this if other services are hosted on the application server.
• There are also settings which are not global to Excel Services however they are specific to the trusted locations where Excel Services are being used. For instance there is max workbook size, max chart/image size, session timeout, volatile function cache lifetime and allow external data all of which can be used to better manager performance.

Thoughts: For SharePoint 2010 Excel Services I would say there is more anticipation for a more feature rich solution than what was available for SharePoint 2007. It is very hard in my opinion to come up with a one size fits all sort of assessment for performance when it comes to Excel Services. This is because each hosted workbook can be dramatically different than the other.
InfoPath Service limits

• There are numerous factors that affect the throughput of InfoPath Services, some of which are: number of users, type/complexity/frequency of user operations, number of postbacks per operation, and performance of data connections. When building and deploying forms, this should always be accounted for.
• In the design of the form, try to optimize the first request to the form template. Try to exclude custom onLoad events, queries or business logic that occur when opening the form. You should try to make these operations as user driven as possible. What we want to do is delay the creation of session state data in the database until an actual post occurs. If the form only have one post, meaning when it is submitted, no session state for the form will ever be created.
• If the InfoPath form needs to be saved to a document library, it is better to submit the form into the document library rather than saving it. The reason is a submit operation triggers only one post but a Save will trigger two posts.
• Document library forms can support more throughput that InfoPath list forms.
• Form complexity such as the number of controls, rules, encapsulated data, etc. dramatically affects performance. The more complex, the more pressure will be put on the WFEs to support greater throughput.
• It is recommended to minimize the amount of controls displayed at any given time. If you do have a lot of controls that have to be displayed, it is recommended to put them into a secondary view so that you reduce the first page loading we talked about earlier. This is not just to reduce the amount of HTML generated; there can be a lot of presentation rules which can slow performance down.
• Large amounts of complex XML can cause more performance issues. Be careful of using attachment controls as the binary of the attachment will actually be embedded into the XML of the InfoPath form, making it very large.
• Database locks prevent multiple users from making changes to the same set of data. Since an InfoPath form is more than a piece of data in SharePoint, a lock is placed on it while it is edited. So it will not support concurrent usage of the same form well.
• It is recommended to return filtered data directly to the InfoPath form instead of filtering it within SharePoint.
• When reviewing the results of Microsoft's test, there was always more performance overhead when the form has web service submits.

Thoughts: From a performance perspective, not much has changed from a performance perspective. Yes - InfoPath 2010 form services will be faster than SharePoint 2007 – but the design concepts to create forms that perform well has not really changed. Your design of the forms drives performance and you should not lean on adding more WFEs to solve your problems.

Additional References:

• http://blogs.msdn.com/b/infopath/archive/2009/10/27/infopath-2010-performance-highlights.aspx
• http://blogs.msdn.com/b/infopath/archive/2008/05/09/designing-browser-enabled-forms-for-performance-in-infopath-forms-services.aspx

Visio Services limits

• There are three basic factors that affect performance: drawing size, complexity and data connectivity.
• Typically Visio Services are tough on the WFEs which will require you to scale up or out.
• The initial file size of Visio web drawings is 50MB. This can be changed by the administrator however larger files will increase memory required for Visio services, increase in CPU usage, reduction in service requests per second, increased latency, and increased SharePoint farm network load.
• There is a configuration called Visio Web drawing recalculation timeout which changes the maximum amount of time it will spend recalculating a drawing after a data refresh. The default it 120 seconds. Increasing the timeouts will increase CPU and memory, reduce application requests per second and increase latency across all documents.
• For Visio services there is a Minimum cache age setting which is used for data connected diagrams. This setting is between 0 to 24 hours. Setting this to a lower value will reduce throughput and increase latency because recalculations have to occur more often. This subsequently reduces CPU and memory.
• As well there is a Maximum cache age for non-data connected diagrams that determines how long to keep the diagram in memory. Increasing this value will decrease latency for diagrams that are used a lot. However increasing the maximum cache age will increase latency and slow down throughput for all items that have not been cached because the cached items consume available memory.

Thoughts: I found this section interesting to read because it gave me insight into how Visio Services is actually working. I had heard the Visio services can be data driven. When reading this you need to focus on caching as part of your design and understand the types of documents that you plan to post.

PerformancePoint Service limits

• The factors that most affect throughput are number of users, type of user interaction, frequency of use, number of postbacks in an operation and data connection performance.
• When a scorecard uses an Excel data sources, there is a limit 1 million cells per query.
• In a dashboard that is using Excel as a data source, the max number of columns is 15 and 60,000 rows.
• It is recommended that if you do have business logic or complex controls, the demand on WFEs will increase and adding more WFEs will alleviate those issues.

Additional Resource

• http://technet.microsoft.com/en-us/library/bb660539(office.12).aspx

Word Automation Services limits

• The input file size cannot exceed 512MB.
• The frequency of the conversions can be configured by minutes. A lower number will allow the timer job to run faster. The default is every 15 minutes but the recommendation is every 1 minute.
• There is a threshold for the number of conversions to start per conversion process. Obviously if the value is high, this can cause intermittent failures.
• The conversion job size can support 100,000 items at one time. A large number of conversion items increase the amount of time it takes to execute.
• The total number of conversion processes is directly related to the number of core processors. You should have not more conversion processors running than you have of processors. The reason why you should know this is because word automation services can at times full leverage a processing core and cause issues if other services are hosted on the application server. There is a property of word automation services called Total Active Conversion Processes which can be used to "throttle down" word automation services such that it does not use too many processing cores per application server.
• The work automation service database size cannot exceed 2 million conversion items. Items are not deleted from the database automatically after processing; so administrators will have to write jobs to remove the history. Exceeding 2 million items can cause performance challenges the longer the word automation service runs.
• When scaling out word automation services it is recommended to add more application servers and potentially add a new SQL Server which has the work automated services database dedicated to it. Another scaling recommendation would be to create dedicated word automation application servers.
• There is a fundamental performance limitation with converting PDF and XPS file types. Adding servers will not always solve the challenge.

Office Web Application Services limits

• This includes the Word Web App, PowerPoint Web App and OneNote Web App. There are several different perspectives that you have to look at each of these. Specifically you should focus on viewing versus editing because the resources required are quite different.
• The drivers for performance are expected concurrent users and what type of operations are going to be done. Microsoft's initial recommendations are 100 daily users with average of 10 concurrent can be supported by 1 WFE and 1 App Server. Going to 1000 daily and 30 concurrent would require 2 WFE and 2 App servers. Going to 10,000 daily and 300 concurrent would require 4 WFE and 3 app servers.
• Heavy Word Web App viewing will typically require more CPUs to the application servers.
• Heavy Word Web App editing and OneNote Web App viewing/editing require more CPU to the WFEs.
• Heavy PowerPoint Web App viewing requires more CPUs to the app servers and more memory to the WFEs.
• Heavy PowerPoint Web App editing requires more memory to be added to the application services.
• If you did not know what this is this feature enables presenters to broadcast a slide show from Microsoft PowerPoint 2010 to remote viewers over the web through SharePoint. It is positioned as a low-infrastructure presentation capability. Heaving PowerPoint Broadcast will require the WFEs to have more CPU. It is recommended that if you have extremely PowerPoint Broadcast feature to create an environment to support that. The reason being is web browsers will hit the server every second to get the latest changes to the slidedeck.
• Much of the limits associated to OneNote Services are directly tied to the limits for list and libraries. This is because each section in OneNote is stored as folders and documents in a library.
• The maximum size for each section of an OneNote section is again driven by the file size limits for lists and libraries.
• If there are embedded images, files, etc. in OneNote, that are greater than 100KB, they will be split out into their own binary files within the SharePoint library.

Access Services limits

• The performance of Access Services is dependent on the other applications that are hosted within the service. It would be recommended that Access Services to be dedicated to a service if there is lots of data to be managed.
• The amount of data in the tables and the size of the queries will have the most impact to performance of Access Services. It is recommended to limit the size and complexity of queries and well as control the amount of data that flows through. This can be done in Central admin. First the query can be configured by controlling the max sources per query and max records per table. Second results can be configured by max columns per query, max rows per query and allowing out joints. Third processing complexity can be configured by max calculation columns per query and max order by clauses per query.
• There is another configuration called All Non-Remotable Queries. Way Access Services work under the hood is that is augments SharePoint query operations support Access Service features, like working with large amounts of data. If there are performance challenges, you can change this configuration so that Access Services just uses the SharePoint query operations – subsequently making data fetches less complex (at the same time not as robust).
• Access Services is stateful. It will maintain in memory cursors and record sets between user interactions. Microsoft's testing did not determine that this would be a bottleneck.
• There are no special hard requirements for WFE or application servers to support Access Services. It is recommended to increase capacity by scaling up your existing servers or scale out by adding more servers into the SharePoint topology.
• To support more users it is suggested to add more CPUs to the servers and more Access Services computers if needed. Microsoft's testing found out that if you add too many Access Service machines the WFEs can become a bottleneck to the Access Services. This tends to happen when the four Access Service machines have been added.
• Reporting services are utilized with Access Services. It is recommended to create a dedicated machine to support it if processing of reports is taking a long time.
• There are several performance counters that can be used to determine performance that are in the whitepaper.

Thoughts: Again reading over this gave me some great initial insight into the architecture of Access Services. I actually intrigued to see how this will be utilized with organizations.