AdobeDispatcherHacks ".statfile"

-
AEM DISPATCHER STATFILE UNDERSTANDING & CACHE INVALIDATION:-

AEM Developers, Infrastructure Engineers regularly come across a challenge on decoding the statfile and using it efficiently especially statfile becomes highly relevant in a multi-tenanted environment with different project teams controlling different sites. The article addresses in a simple way on how to understand the mechanisms of stat file and gives a detailed explanation of how it can be used in a multi-tenant environment model. 
The image for your reference as a quick overview of the data flow, before we take a deep dive. 
This article covers -
1 - When dispatcher serves the old version of the content. How to avoid it.
2- Cache invalidation mechanism.

Assumption - If you are reading this article, I believe you would have a basic understanding of Dispatcher and it's configuration.
Firstly let’s set the initial configuration for the cache invalidation section of the dispatcher. The configuration file then study how invalidation works at the low level and finally return to tools for invalidation. 

Invalidation section initial settings:-

Open the dispatcher.any files or any other *farm.any file's 

Inside /cache section there is /invalidate block which determines cached files that may be automatically invalidated when content is updated. For example, the following configuration invalidates all HTML pages and the .hpeg images. 

/cache
{
    /invalidate
    {
        /0000  { /glob "*" /type "deny" }
        /0001  { /glob "*.html" /type "allow" }
        /0002  { /glob "*.jpeg" /type "allow" }
    }
}

With automatic invalidation, the dispatcher doesn’t delete cached files after updating content but checks their validity when they are next requested. Documents in the cache that are not auto-invalidate will remain in the cache until a content update explicitly deletes them. 
Do not forget to reload or restart the apache/IIS server post configuration changes to take the effect.  

My Dispatcher is serving the old Content - 

The following points should be noted:

  • Content Updates are typically used in conjunction with an authoring system which "knows" what must be replaced.
  • Files that are affected by a content update are removed, but not replaced immediately. The next time such a file is requested, the Dispatcher fetches the new file from the AEM instance and places it in the cache.
  • You may have several statfiles, for example one per language folder. If a page is updated, AEM looks for the next parent folder containing a statfile, and touches that file. 

Now we will understand the stat file in detail, as it is a vital part of cache invalidation. 

Cache invalidation mechanism & Understanding of stat file – 

At the low-level dispatcher uses empty files which are named by default “.stat”. By default setting is used /statfileslevel "0" which means that there is only one stat-file is used and is placed at the root of htdocs directory or the cache root directory. 

The dispatcher module checks, If modification time of stat file is newer than the modification time of the resource then dispatcher consider such resources are obsolete or are invalidated.

For example, we have the cached resources after requesting the page "http://myhost:port/content/geometrixx/en/products.html". 
Let’s invalidate them by the low level mechanism of the stat-files. Create an empty file with name “.stat” at the root of your htdocs directory:

You may see that stat-file modification time is newer than cached resource modification time. That means for the dispatcher that all resources are obsolete. This is an invalidation mechanism at the low level in depth. After creating such stat-file if you will visit again the page http://host:port/content/geometrixx/en/products.html then requested cached resources will be updated:
In this example demonstrates the default invalidation scheme with /statfileslevel "0". 

Let’s study how we may configure invalidation more detailed with help of /statfileslevel setting.

/statfileslevel

You may use /statfileslevel property of the dispatcher configuration file to selectively invalidate cached files according to their path. There are some rules for /statfileslevel property mechanism: 
  1. Dispatcher creates .stat files in each folder from the docroot folder down to the level that you specify. The docroot folder is level 0.
  2. When a file is updated dispatcher locates the folder on the file path that is at the statfileslevel and invalidates all files below that folder.
  3. If level of the updated file is less than statfileslevel then all files in such folder are invalidated. Files below that folder are not invalidated.
  4. When a file is updated then all files from file folder up to the root level inclusive will be invalidated.
  For a better understanding of the /statfileslevel rules let’s consider a couple of examples. 
-   statefile lelvel 0 - A single .stat file will be created at the root of the cache directory.
        /.stat
- statefile lelvel 1 - .stat file at root of the cache and under each direct chile directory.
       /.stat
       /content/.stat
- statefile level 2 - .stat file at root of cache and under each direct child directory and each of their  direct  child directories:
      /.stat
      /content/.stat
      /content/foo/.stat
      /content/bar/.stat

let's see the default  /statfileslevel “0” looks like -  

- There is only one stat-file at the root folder of our docroot/cache root.


- The scope of responsibility of that stat-file is all file-tree under htdocs. 

- If any file from this tree has older modification time then stat-file modification time then dispatcher consider such file is invalidated.

- statfileslevel = 0 is only for learning demo. It is not recommended to use 0 in the live production environment as it may end up flushing all cache every time any file gets modified. 

Now let's  set /statfileslevel “4”  more realistic scenario, and see how the  invalidation works - :

- There will be stat files at all levels from 0 (root) to 4 inclusive.

 - There will be stat files at 5 levels (0 to 4) of the folder structure. 
Stat-files at levels less than 4 have the scope of responsibility with the only directory with this stat-file.
- That means if stat-file inside content/geometrixx/en folder is newer than any file from this folder then such file is invalidated but the validation of all files from all other folders is determined by other stat-files. 

- Stat-files at the level with a value of statfileslevel  property (level 4 in our case) only have the scope of responsibility with all underlying tree which begins from a folder with this stat-file and expands down to lower levels of the file tree. 
That means that if stat-file inside content/geometrixx/en/products folder has modification time newer than any file from underlying tree including products folder then dispatcher considers such file is invalidated. Validation of all files which is not located in this file tree is determined by other stat-files.
  
Stat-files at the level with a value of statfileslevel property (level 4 in our case) only have the scope of responsibility with all underlying tree which begins from folder with this stat-file and expands down to lower levels of the file tree. 
That means that if stat-file inside content/geometrixx/en/products folder has modification time newer than any file from the underlying tree including products folder then dispatcher considers such file is invalidated. Validation of all files which is not located in this file tree is determined by other stat-files.

Hope you understand the invalidation mechanism and the use of .stat file,
We can understand the flush agent's configuration & manually agents for auto-invalidation when publishing pages and requests for manual invalidation will discuss in new article. 
Adobe Documentation References –






Comments

Post a Comment

Popular Posts

how to clear dispatcher cache in aem ?

-
Image
How to clear dispatcher cache in aem ? As you may know, the Dispatcher cache in Adobe Experience Manager (AEM) is used to improve the performance of your website by caching static resources and pages. However, sometimes you may need to clear the cache to ensure that the latest content and changes are displayed on your website. In this blog post, we'll show you how to clear the Dispatcher cache in AEM.   This method will clear the entire Dispatcher cache, including all cached pages and resources. Keep in mind that clearing the cache may affect the performance of your website, as it may take some time to rebuild the cache. Clear Cache using the Dispatcher Flush Agent You can use the Dispatcher Flush Agent. Follow these steps:   Log in to your AEM instance and navigate to http://localhost:4502/etc/replication/agents.author.html . Click on the "Dispatcher Flush" agent to open the agent's configuration page. Clic...
Read more

How to Increase Apache Request Per Second ?

-
Image
How to Increase Apache Request Per Second ? By default, Apache web server is configured to support 160 requests per second. As your website traffic increases, Apache will start dropping additional requests and this will spoil customer experience.  Here’s how to increase Apache requests per second. 1. Install MPM module We need to install MPM Apache module to be able to increase Apache requests per second. You can use mpm_worker or mpm_event module for this, instead of mpm_prefork module which consumes a lot of memory. You can easily install MPM module in Apache with following command For CentOS7/RHEL7 : Adjust /etc/httpd/conf.modules.d/00-mpm.conf Comment the line LoadModule mpm_prefork_module modules/mod_mpm_prefork.so by adding # in front of it. Uncomment the line LoadModule mpm_worker_module modules/mod_mpm_worker.so by removing # in front of it. For Ubuntu/Debian :  Use a2dismod / a2enmod to disable mpm_prefork and enable mpm_worker 2. Increase Max Connections in Apach...
Read more

Configure/Decoding AEM AuditLogs

-
Image
AEM Developers, Infrastructure Engineers / Dev-ops teams working in the financial domain regularly come across a challenge for event auditing in AEM. This helps in identifying most of the activities happening in AEM. Audit logs are a very effective way to debug the content issue & to know what all is happening in your environment and by whom. This article addresses in a simple way on how to enable the audit logs, its different ways, and how to understand the audit logs.  This article covers the following - How can we enable Audit logs in AEM. How can we read and understand the Audit logs/ tools to use it. Audit log on file system in crx-quickstart/logs folder.   Audit logs for User creation / Modification. How can you archive/purge the audit logs. How can we enable Audit logs in AEM?            By Default, the Audit logs are pre-configured in AEM, for a few basic operations of DAM and for all other operations of Pages ...
Read more

How to Configure CSP header in AEM , Dispatcher ?

-
Image
How to Configure CSP header in AEM ? Content Security Policy (CSP) is a security feature that helps prevent cross-site scripting (XSS) and other code injection attacks by restricting the sources from which a page can load resources. To implement a CSP header in an Apache web server, you can use the Header directive in your Apache configuration. Here are the steps to implement a CSP header in Apache: Determine your CSP policy: First, you need to determine your CSP policy. This policy defines the rules for what types of content can be loaded from which sources. You can use a CSP policy generator like the one available on the Mozilla Developer Network (MDN) website to generate a policy that meets your needs. Add the CSP header to your Apache configuration: Once you have your CSP policy, you can add the CSP header to your Apache configuration. To do this, open your Apache configuration file (usually located at /etc/httpd/conf/httpd.conf or a similar location depending on your setup) and ...
Read more

How to prevent DDoS in Apache ?

-
Image
Prevent DDoS in Apache & IP Block Automation DDoS (Distributed Denial of Service) attacks are a type of cyberattack that can cause serious damage to your web server. These attacks involve flooding your server with a huge volume of traffic, overwhelming its resources and causing it to crash. In this blog post, we'll discuss how to prevent DDoS attacks in Apache, without using any third part tool/application.     Available Options to Prevent DDoS : You can use various mentioned methods to achieve the same. But using WAF, CDN, etc will cost extra dollars. Which might not be necessary for a small scale application.   Use a Web Application Firewall (WAF): A WAF can help detect and block malicious traffic before it reaches your Apache server. It can also help block common attack vectors, such as SQL injection and cross-site scripting (XSS). Install mod_evasive: mod_evasive is an Apache module that helps detect and block DDoS attac...
Read more

How to protect AEM against CSRF Attack ?

-
Image
How to protect AEM against CSRF Attack ? Adobe Experience Manager (AEM) is a popular content management system that is widely used to develop and manage websites, mobile apps, and other digital experiences. However, like any other web application, AEM is vulnerable to cross-site request forgery (CSRF) attacks. CSRF attacks are malicious attacks where an attacker tricks a user into performing an action they did not intend to perform by exploiting the user's active session on a website. In this blog, we will discuss some measures that can be taken to protect AEM from CSRF attacks.   Implement CSRF protection in AEM:   The first and most important step to protect AEM from CSRF attacks is to implement CSRF protection in the application. AEM provides a built-in CSRF protection mechanism that can be enabled by setting the "sling.filter.methods" property in the OSGi configuration. This property specifies which HTTP methods are allowed to execute without requiring a CSRF ...
Read more

How Does S3 works with AEM ?

-
Image
How Does S3 works with AEM  ? Accommodating a huge amount of assets in any content management platform is challenging. Adobe Experience Manager offers an integration with the Amazon S3 storage solution, allowing binary data for images, documents and videos to be stored in an S3 bucket. Amazon S3 is highly performant and offers nearly infinite storage capacity.   When talking about terabyte storage, performance is everything. The choices made during the planning and architecting phase can literally make or break the performance of a CMS system and the websites running on it.  Adobe Experience Manager offers a number of storage methods, each offering a different way of storing data. Each of these options has its strengths and weaknesses. In AEM storage the mechanisms are called Micro Kernels, or MK for short.  In this article we will look at the AEM with S3 data store. For the detailed steps for S3 configuration you can refer -  https://www.aemrules.com/...
Read more

Difference between Adobe AEM Enterprise vs Adobe AEM as a Cloud Service

-
Image
Difference between Adobe AEM Enterprise VS Adobe AEM as a Cloud Service. ( AEMaaCS) Adobe Manages Services Enterprise (AMS) and AEM as a Cloud Service (AEMaaCS) are two different offering by Adobe for AEM.   While AMS is more of a hosting offers for AEM (similar to on-premise setups but run managed by Adobe), While AEMaaCS is a SaaS offering.    In AMS you can have access of your servers , can login and check the required files by your self & will have a dedicated Customer Success Engineer, while in AEMaaCS is completely SAS and you won’t have any access of servers , for everything you need Cloud manager.   Below are the Major difference between AMS Enterprise and AEMaaCS. Please note these difference are as of March 2023. Post this things may differ.  The following article can be referred for AEMaaCS details  AEM as a Cloud Service
Read more

Security best Practice in AEM

-
Image
 Adobe Experience Manager (AEM) is a popular content management system that is widely used by businesses to manage and publish digital content. With the increasing amount of sensitive data being stored and shared online, it's important for AEM users to be aware of the security features that the platform offers. In this blog, we'll discuss some of the key security features of AEM and provide tips for keeping your AEM instance secure. Authentication and Authorization AEM provides several options for authentication and authorization. Users can log in using their credentials, which can be verified using LDAP or other external identity providers. Once authenticated, users are assigned roles and permissions, which determine what actions they can perform within AEM. To keep your AEM instance secure, it's important to ensure that users only have the permissions they need to perform their jobs. For example, if a user doesn't need to publish content, they should not be given perm...
Read more